Adding servers

A server is any Linux host with Docker that the control plane can reach — or more precisely, any host that can reach the control plane. The direction matters, and it is the reason enrolment is one command rather than a firewall change.

Enrol

In the console: Servers → Add server. It prints a command to run on the machine, already carrying that server's one-time token:

curl -fsSL https://your-console/install/agent.sh | sudo bash -s -- \
  --token <one-time-token> --ca-fingerprint <fingerprint>

Two details in that line are worth noticing.

The script comes from your own control plane, not from us. It is served by the installation you just stood up, which is also the only thing that knows the token.

The CA fingerprint is on the command line. The agent checks the certificate authority it is handed against that fingerprint before it trusts anything — so a machine-in-the-middle on first contact cannot enrol the server into a control plane you did not install. This is the one moment trust is established, and it is pinned rather than assumed.

The agent then exchanges the token for a client certificate signed by that authority, and connects. The token is spent on first use.

Why nothing listens

The agent dials out. It opens one outbound mTLS connection to the control plane and keeps it; the control plane never initiates a connection to a server. So a server needs:

• no inbound port for QuorumKit,
• no public address,
• no VPN to the control plane,
• nothing but outbound reach to the control plane's gRPC address.

What a server does open

The only inbound ports are the ones the databases themselves publish, and you choose those per deployment. Everything else — replication, elections, health checks, the control plane's own traffic — either crosses the private mesh or travels back up the connection the agent already opened.

Between the nodes

Servers in one deployment talk to each other over a WireGuard mesh created for that deployment, with keys the control plane issues and rotates. Replication traffic, cluster elections and health checks all cross it, so none of that is exposed on the servers' own networks — and two deployments on the same physical hosts cannot see each other.

Checking on a server

Its page in the console carries the machine's real facts — cores, memory, disks, kernel, the Docker version, when it was last seen, and which agent version is on it. A server goes unreachable when the heartbeat stops.

On the machine itself:

systemctl status quorumkit-agent
journalctl -u quorumkit-agent -f      # what it is being asked to do
systemctl restart quorumkit-agent

Restarting is safe. The agent is a control-path process: the database containers keep running and keep serving across it, and the machine only stops taking instructions for the couple of seconds it takes to come back. When a server has gone unreachable, that log almost always says why — a network change, a full disk, a Docker daemon that is not running.

What the agent is allowed to do

A fixed vocabulary of tasks, and nothing outside it. It does not accept shell commands and will not run an arbitrary payload — the list of things it knows how to do is built into it, so the reach of a compromised control plane is bounded by that list rather than by what somebody can think of to send.

It also stores no database passwords of its own. Credentials are sealed by the control plane and delivered for one task at a time, rather than left on the host in plaintext.

Keeping agents current

Agents check their version against the control plane and update themselves. There is no command to run on each host after an upgrade; the control plane carries the matching agent binaries and hands one over when a version is behind. An agent that matches the control plane it talks to is the whole definition of up to date — which also means keeping your fleet current is keeping the control plane current, and that is one page over.

The care is all in the order it does things, because this is the one task that could otherwise leave a machine unreachable:

• the download is checksummed against what the control plane says it is serving, before anything is made executable,
• the new build is run once, to prove it runs on that host, before it replaces anything — which is what catches a wrong architecture or a truncated download,
• the previous build is kept beside the new one, so recovery is a rename rather than a reinstall.

Updates are spread across the fleet rather than all at once, so a hundred machines do not all download the same thing at the same instant.

The version each server is running is shown on that server's page, marked current or updating to …. The comparison shown is the same one the control plane makes when it decides whether to offer an update, so the page and the mechanism cannot disagree about what is behind.

Removing a server

Remove it from its page in the console. That is refused while any deployment still runs there — destroy or move those first, which is itself a plan you approve.

Removing it revokes that machine's certificate, so an agent you never uninstalled is not merely stale but rejected: its heartbeats stop being accepted and it cannot put itself back in your fleet.

To clean up the host afterwards:

systemctl disable --now quorumkit-agent
rm -f /usr/local/bin/quorumkit-agent /etc/systemd/system/quorumkit-agent.service
rm -rf /etc/quorumkit

That leaves the database containers alone. If you want those gone too, destroy the deployment through a plan before removing the server — that is the step that stops and removes them.