Stage 4c: start, stop and restart, for real

At the end of 4b-2 the controller could take requests and the agent could report what was running, but nothing connected the two. A start request was written down and then just sat there. Stage 4c joins them up. You ask the controller to start a server, the agent on that machine carries it out, and the result is recorded against the operation. Then all of it moves out of my terminal windows and into systemd, where it runs by itself and comes back after a reboot.

This came in two parts. 4c-1 is the code. 4c-2 is the real installation.

4c-1: from a request to a running server

When an operation is created, the controller nudges that machine's agent session. The session claims the oldest pending operation and sends it to the agent as a command, with up to four in flight at once. The agent asks Waystone's privileged helper to run it through systemd, then sends back a result. A restart is a stop, then a progress message, then a start, and it's one operation from beginning to end.

Every step checks the machine's session number, its epoch, inside the database. If a newer agent has taken over in the meantime, an old claim or an old result is refused and nothing is written. A result from a session that has already ended can never land on an operation it doesn't belong to.

Saying "I don't know" honestly

The hard part isn't the happy path. It's what Waystone says when something breaks halfway.

• The helper can't be reached and nothing was sent to it: the operation fails, and we know nothing ran.
• Anything was written to the helper without a reply coming back, even a partial message: the operation becomes uncertain. Waystone doesn't guess whether systemd acted. A partial message never makes the helper do anything, but the controller can't know how far it got.
• The controller restarts while an operation is running: on startup, before it accepts a single request, it marks those operations uncertain with controller_restarted.
• The agent dies mid-command: the next agent's hello marks the operation uncertain with session_replaced.

An uncertain operation blocks new operations on that server until a person looks at the server and records what really happened. The API answers previous_operation_uncertain until then. Stage 5 puts that decision in the web UI.

Never two actions on one server

The helper allows one start or stop per server at a time, across every connection. It also refuses to act while systemd itself is still busy with that unit, for example halfway through a stop. That comes back as server_busy. The one exception is a server waiting to restart after a crash, so a stop can still end a crash loop. Every start and stop also carries a deadline. If it's already late when the helper gets to it, the helper refuses rather than acting on a request the controller has given up on.

The database decides who may do what

Each kind of caller can only make the changes that belong to it. An agent can claim, report progress and finish. An admin can resolve an uncertain operation, under their own name. Nobody can rewrite history. This is enforced by a trigger in PostgreSQL, not by the Go code. 52 test cases run raw SQL as the application's real database role and check that the database itself refuses each one.

4c-2: a real installation

Until now I started the controller and agent by hand on the dev VM, plus a development harness from earlier stages. 4c-2 replaces all of that with four systemd units:

• Two sockets owned by systemd in a root-owned /run/waystone-controller/. api.sock is for the waystone-admin group, and agent.sock is for the agent's user only.
• The controller, running as its own unprivileged waystone-controller user. systemd starts it when either socket is first used and hands it both, by name.
• The agent, running as waystone-agent, starting at boot and restarting itself if it dies.

Both services get the same sandboxing as the helper. systemd-analyze security rates them 1.0, "OK". The development harness is deleted from the repository, and the release binaries contain no trace of it.

Two new tests read the unit files from the repository and check them against the code: socket paths, names, owners, modes, users, and every flag on each command line, run through the programs' real flag parsing. I planted 18 mistakes in the unit files, and all 18 were caught.

What the install found

The first real start turned up a genuine conflict between two earlier decisions. The agent checks, through the kernel, that the socket it connects to belongs to the controller's user. But with socket activation, systemd creates that socket, so the kernel reports root as the owner, and the agent refused to connect. Development mode never showed this, because there the controller made its own socket.

The fix is small. The agent now accepts the controller's user or root, and still refuses anyone else. That's safe because only root can create a socket in that directory, and root is trusted anyway. A new test covers it, and four planted bugs in that check were all caught.

Trying it for real

All of this ran on the dev VM against two real Minecraft servers.

• Every kind of operation. Start, restart and stop went through the API and the agent and finished succeeded. systemd showed the right state each time, with no job left behind.
• Queued while the agent is down. With the agent stopped, a start waited as pending. When the agent came back, it ran and succeeded.
• Busy servers. A start sent while systemd was halfway through a stop failed with server_busy, and a start after the stop finished succeeded.
• Killing the controller mid-stop. The operation became uncertain with controller_restarted. The log says startup recovery: 1 running operation(s) marked uncertain. A new start was refused until I resolved it.
• Killing the agent mid-stop. systemd restarted the agent, and the operation became uncertain with session_replaced.
• Rebooting the VM. Nobody touched anything afterwards. The sockets, controller and agent all came back by themselves, the agent reconnected at a new epoch and reported both servers, and a start and stop through the API both succeeded.

The two crash tests are timing-sensitive, since a stop can finish before the kill lands. The script only counts an attempt if it saw the operation running before the kill, and if the operation hadn't finished by then. Otherwise the attempt is inconclusive and it tries again. Both tests passed on their first counted attempt.

How it was tested

4c-1 adds 59 tests, most of them table-driven. The integration tests use real gRPC, real Unix sockets, real PostgreSQL and the real helper server code, with only systemctl faked. One test forces six steps of two sessions racing over the same server and checks that only one action ever runs.

Then I broke the code on purpose, 112 different ways, one at a time. 15 survived the first round. Most were real gaps, such as a machine being able to sneak a note in with its result, and each one now has a test. In the end 111 are caught, and the last one is impossible anyway, because an older database rule already forbids it.

Everything passes three times in a row under Go's race detector, locally and on the dev VM, and builds for Windows.

What the screenshot shows

This is the dev VM after the installation from 4c-2. My username is covered in the prompts and in each actor field, and the hostname is covered in the prompts. From top to bottom:

1. The four units. Both sockets, the controller and the agent are active (running). Each socket lists the controller as the service it triggers, and the controller lists both sockets under TriggeredBy.
2. Who owns the sockets. Both files belong to root. agent.sock has the group waystone-agent and api.sock has waystone-admin, and nobody else has any access.
3. The sandbox rating. systemd-analyze security gives the controller and the agent 1.0, "OK".
4. The log since boot. Startup recovery found 0 running operations to mark uncertain. The controller is listening on both sockets, and the agent connected at epoch 18, reporting 2 servers every 15 seconds.
5. Start, restart, stop. Each one comes back succeeded, and the line under it is what systemd says: running, running, then stopped, with an empty Job= each time. The restart's last phase was starting.
6. Killing the controller mid-stop. The script starts a server, asks for a stop, and kills the controller as soon as the stop is no longer pending. Afterwards the stop reads uncertain with controller_restarted, and the log line is startup recovery: 1 running operation(s) marked uncertain. A new start is refused with previous_operation_uncertain. Once the server really is stopped, I resolve the operation as succeeded. Its state stays uncertain, with that outcome recorded beside it.

What's next

Before the web panel, there's a security pass. Then stage 5, where Waystone gets a browser login and the start, stop and operation history above get buttons.

Waystone is open source under AGPL-3.0.