Stage 4a-1: Waystone now remembers everything it does

Through stage 3, Waystone could build a Minecraft server from nothing, install its jar and start or stop it, all without root. But it had no memory. Once a request was done, nothing recorded who asked for it, what happened, or whether it was still running. Stage 4 gives Waystone a controller that keeps that record, and stage 4a-1 is its foundation: the database.

Why this needed care

From here on, every action Waystone takes is written down as an operation before anything happens on a machine. That record is the source of truth, so it has to be hard to get wrong and impossible to quietly rewrite. Before I wrote a line of code, the stage 4 design went through several rounds of security review. The first two drafts were rejected, and the version I built from lists its rules one by one, each with a test that proves it.

What's in place

One database, two roles. Waystone uses PostgreSQL. One role owns the schema and runs migrations. The controller itself runs as a second role that can't change the schema, can't bypass the security rules, and can't delete anything. Both log in through the local socket with the operating system's own identity, so there are no database passwords to leak.

Every row belongs to an organization, and the database enforces it. All seven tables carry an organization ID, and row-level security is switched on and forced for every one of them. To see any rows, a request must name its organization and belong to it. Naming an organization you aren't a member of gets you nothing, not even an error that hints the rows exist. A request without that context sees zero rows and can write none.

One thing at a time per server. A server can have only one unfinished operation. The code checks for this, and a unique index in the database backs it up. In the tests, many simultaneous start requests for one server produce exactly one operation.

History you can't rewrite. Operations can never be deleted. The record of who asked, for which server and when can't be changed after it's written. A trigger in the database allows only real state changes, such as pending to running, or running to succeeded, so even a bug in my own code can't rewrite history.

Honest about uncertainty. Sometimes Waystone won't know whether an action actually happened, for example when a machine drops off mid-restart. That operation is marked uncertain, and the server stays locked until a person checks and records what really happened. The operation stays marked uncertain even after that, so history shows that the system itself never found out.

Safe retries. A client can attach an idempotency key to a request. If the connection drops and it retries with the same key, it gets the original operation back instead of starting the server twice. A key never expires while its operation is still unfinished.

Refusals are recorded too. When a request is denied, an audit entry records who tried what.

How it was tested

Every rule above has a test that runs against a real PostgreSQL database, with no mocks. That's 20 database tests, run with Go's race detector three times in a row. The real database on my development machine is migrated with the new controller migrate command. It holds one organization, one node and the two Minecraft servers I built in stage 3.

What the screenshot shows

This is the development machine's real database, checked by hand, followed by the test run. From top to bottom:

1. Migrations: controller migrate reports that the schema is applied.
2. Row-level security: PostgreSQL's own catalog lists the seven tables. rls says the rules are on. forced says they apply even to the table's owner. Both are true for every table. The database calls an organization a tenant, which is why that table is named tenants.
3. The controller's role with no identity: it asks how many servers it can see without saying who it's acting for. The answer is 0.
4. A stranger naming my organization: a user ID that isn't a member sets my organization as its context and asks again. Still 0, and no error.
5. Me, as the organization's admin: the same query now returns both stage 3 servers. Their process state reads unknown, and that's the honest answer. Nothing reports to this database yet.
6. The tests: 25 pass. The 20 that take a few hundredths of a second each are the database tests. The five at 0.00s check things that need no database, such as validating an idempotency key.

Steps 3 to 5 are the same question asked three times. Only the caller changes, and the database alone decides the answer.

Stage 4a-1 is committed as f020c2c.

What's next

Nothing talks to this database yet. Stage 4a-2 adds the controller's API on a local Unix socket, which knows who's calling from the operating system rather than from anything the caller sends. Then the agent connects (4b), and starting and stopping a server goes through recorded operations from end to end (4c). After that comes the part I've been waiting for: the web interface.

Waystone is open source under AGPL-3.0.