Stage 4b-1: the database learns who is allowed to report

Since stage 4a, Waystone has been able to record that someone asked for a server to start. What it can't do yet is hear back from the machine the server runs on. In stage 4b the agent on each machine connects to the controller and reports what's really running. 4b-1 is the half of that which lives in the database: what a "connection" means, what a report may change, and who is allowed to send one.

The connection itself, over gRPC on a Unix socket, comes in 4b-2. Doing the database half first means the rules are in place before anything can talk to them.

Every connection gets a number

When an agent connects, it says hello. The controller answers by raising that machine's session number, its epoch, by exactly one. From then on, every report has to carry that number.

That one rule covers a surprising number of failure cases. If an old connection hangs around after a newer one has taken over, its reports carry the old epoch and are refused. If two agent processes are accidentally running for the same machine, they keep replacing each other's session instead of quietly overwriting each other's data. And any operation that was still running when a new hello arrives is marked uncertain, because nobody knows whether the old connection finished it. Waystone says so rather than guessing.

A hello and a report for the same machine can't interleave halfway. They lock the same database row, so one always waits for the other. There's a test for each order.

Reports only move forward

Each report also has a sequence number that only goes up within a session. A server's stored state is only replaced by a report with a newer epoch and sequence, so a late or replayed report can't drag a running server back to "stopped".

Times come from the controller's own clock, when it receives the report. The agent's clock is kept for diagnostics only, so a machine whose clock is a year off can't fake fresh data or make old data look new.

A report must name a machine's own servers and nothing else. An entry for a server on another machine simply doesn't match. A report with duplicate entries, unknown states, malformed IDs or more than 256 servers is refused as a whole.

The database checks it too

The controller code does all of this, but it isn't the only thing standing guard. The web interface and the agents connect to PostgreSQL as the same database user, so a column permission alone can't tell "an admin" from "a machine". Two small triggers in the database itself enforce the rules no matter what code is calling:

• Only the machine itself can touch its own row, and its epoch can only go up by exactly one.
• Only a server's own machine can write its reported state, only with its current epoch, only moving forward, with the receipt time set by the database.
• No machine can ever change what a person asked a server to do.

The tests try every way around these rules I could think of, including partial updates that change one column and skip the rest. Each one is refused with a clear error.

Old news looks old

If a server hasn't been heard from in 45 seconds, the API now shows its state as unknown. The last thing it reported is kept beside it, so the interface can say "unknown, last seen running" instead of pretending it knows.

How it was tested

This stage adds 14 tests, again run against a real PostgreSQL database with no mocks. They cover the migration from the previous schema, both lock orders, eight agents saying hello at the same moment, and a 257th server rolling back without leaving anything half-done.

Then I broke the code on purpose, 34 different ways, one at a time: switching off each database rule, moving the server limit after the write, dropping the lock, shifting the 45-second boundary by a second. The tests caught 32 straight away. One of the two that slipped through was a real gap. A machine whose membership row said "admin" would have passed the store's node check. There's now a test for that, and it fails the way it should. The other can't actually happen, because a user ID can never equal a machine's ID.

Everything passes with Go's race detector, three times in a row, on two machines including the dev VM.

Commit: 6844212

What the screenshot shows

This is the real database on my development machine, followed by this stage's tests. The database calls a machine a node, so that's the word in the output. From top to bottom:

1. Migrations: versions 0, 1 and 2 are applied. Version 2 is this stage.
2. The two guard triggers: nodes_guard on the nodes table and servers_report_guard on the servers table. These are the checks from "The database checks it too".
3. Me, an admin, pretending to be the machine checking in: I try to update a machine's row by hand. The trigger refuses: only the node itself may update its row.
4. Me, an admin, writing a server's reported state: refused again: only the server's node may write report columns.
5. The tests: all 14 pass. Thirteen are in the store package, and TestServerStaleness, the 45-second rule, is in api.

Steps 3 and 4 are the point of the whole stage. I'm an admin of the organization, typing straight into the database with no controller code in the way, and it still says no.

What's next

In 4b-2 the agent itself arrives: a small program that connects to the controller over a Unix socket, says hello and reports what systemd says about each server. After that, stage 4c joins it all up. Asking for a start creates an operation, the agent carries it out, and its reports close the loop.

Waystone is open source under AGPL-3.0.