Deploying to a Single Server

In one line: with no orchestrator, the deploy script is the control plane — split the stack by change frequency, build the image on the box from a pinned SHA, and treat the health check as reachability, never as verification.

Two systems in this canon's evidence base deploy this way, and they are co-tenants on one Hetzner-class VPS: a hospital search pilot and a bookings platform (anonymized per the showcase convention; the artifact paths below make every claim checkable in the repositories themselves). Neither has Kubernetes, an image registry, or a platform team. What they have is a deploy script that refuses, and the refusals are where the design lives — the bookings platform's 47 committed deploy logs end in a success line 41 times, in an explicit refusal twice, and in an aborted build once.

Split the compose by change frequency — and treat the file set as part of the deploy's identity. Both projects separate stateful infrastructure (Postgres, Redis, object storage, an identity provider) from the application image. Infrastructure owns the named volumes and is brought up once; the application is the only artifact a normal release replaces. The pilot layers three files (infra / app / ssl); the bookings platform layers three (infra / app / dev) plus a fourth server-only overlay. A release is then up -d --no-deps <app services>, and down -v — which deletes the volumes — never appears in the release path at all.

The trap is that Compose does not persist -f across invocations: re-running with one file fewer reconfigures the service from only the files you named. Dropping the pilot's TLS overlay silently unbound port 443 and unmounted the certificates, while port 80 — declared in the base file — stayed up, so docker ps and the health check both looked fine and the site was dead over HTTPS until a human reported it. The fix is not a runbook note; it is one readonly COMPOSE_FILES="-f …app.yml -f …ssl.yml" constant that the deploy and rollback paths both read, so the two cannot drift apart.

Co-tenancy adds a second identity hazard. Compose adds the service name as a network alias on every network a service joins — including a shared external one — and a aliases: list is additive, so it cannot suppress that. Two stacks on one box each defining a service called keycloak therefore contend for the bare name on the shared network, and the loser's reverse proxy resolves login traffic into the wrong identity provider. That happened live, and it is not caught by either stack's health check because both stacks are healthy. The deploy script now strips the colliding alias after every bring-up, idempotently.

Build on the target host; never ship an image over the wire. The server already has the checkout and the toolchain. docker save | scp of the pilot's app image meant ~8 GB gzipped (~18.8 GB unpacked) and 30+ minutes saturating a home uplink, against a ~5-minute build on the box. Both projects' scripts therefore git fetch && git reset --hard origin/<branch> on the server and build there, tagging with the short SHA and pinning APP_IMAGE=<app>:<sha> — never :latest, so what is running is auditable. The side effect is worth as much as the bandwidth: a broken build fails before anything is recreated. One deploy of the bookings platform stopped on a frontend type error with the previous containers still serving.

Gate before you deploy, and log every bypass. Four gates, in the order they were learned:

# 0. serialize. mkdir is the portable atomic lock (flock is absent on macOS)
mkdir /tmp/<app>-deploy.lock.d || die "another deploy holds the lock"
trap 'rmdir /tmp/<app>-deploy.lock.d' EXIT

# 1. CI green for the SHA BEING DEPLOYED. Absent CI is not green; running is not green
gh run list --commit "$SHA" --json conclusion,status

The two projects pick different oracles for the same question, and both are right: the pilot gates on the SHA passed as the argument (the thing being deployed), the bookings platform on git ls-remote origin refs/heads/<branch> (the remote head the server will reset --hard to). Local HEAD is the wrong oracle either way. The bookings platform's gate is opt-in because its deployment target is a test environment; the pilot's is on by default because it serves patients. Same mechanism, different default, each with its reason stated in the script.

Gate 2 is the one the sources are most emphatic about: recreating a container is disruptive even when the diff is cosmetic. A dashboard-only redeploy dropped turn 3 of a live voice call, because a backend image rebuild is backend WebSocket churn regardless of what changed. Two more incidents killed running ingest jobs — the second one after the operator had written the pre-check, run it, watched it report a running job, and deployed anyway. A pre-check is a gate only if the operator halts.

Gate 3 is the override ledger. DEPLOY_FORCE=1 appends actor, SHA, gate and reason to a log the monthly consolidation census (§2.8) reads: habitual overrides mean the gate is misdesigned, and the census is where that gets noticed rather than normalized. The force flag is also scoped — in the pilot it waives CI and the pre-deploy smoke but explicitly cannot waive the post-deploy safety probe, because "CI is flaky today" is not a reason to stop checking whether the system gives medical advice.

Health is reachability. Verification is a separate step that can fail the deploy. This is the load-bearing rule, and the two projects reached it from opposite ends.

The bookings platform's health endpoint returns 200 as soon as Postgres and Redis answer — including against a schema several revisions behind. So its deploy runs migrations after health passes and treats "did not reach head" as a failed deploy. The log for ed2dde7 is the mechanism working: backend healthy. → Can't locate revision identified by '0032_loyalty' → REFUSE: migrations did not reach head — deploy NOT verified. The app was up. The deploy was not done.

The pilot migrates before the container starts and supplies the other half: a post-deploy probe of 30 real questions against the live system, whose failure triggers an automatic rollback. Its script states the rule outright — a probe that finds a bad image and then leaves it serving is a notification, not a gate.

Rollback targets what was actually running.

# capture BEFORE the new image goes live — afterwards the reference is gone
previous_image="$(docker inspect <container> --format '{{.Config.Image}}')"

Not a :previous tag (none exists) and not "the SHA before this one in git" (the server may be behind the default branch). Both projects use this same oracle, and both extract the decision — roll back iff a distinct previous image exists — into a pure predicate with a --self-test mode, so the recovery path is exercised without touching a container. That matters because rollback code is the code least likely to have run.

Two refinements. First, rollback is not finished when the old image is up: the pilot re-runs the probe against the restored image and escalates a second failure to a page-a-human state, because "we reverted" is a claim about the past and "it is safe now" is the claim that matters. Second, the database is deliberately not auto-downgraded — image-level rollback is safe only because migrations are additive, so the old image runs against the forward schema; the script prints the captured pre-upgrade revision and the exact manual downgrade command instead of running it, on the grounds that downgrade paths are far less exercised than forward ones.

Where no registry is kept, rollback is by SHA rather than by image: the bookings platform prints git reset --hard <prev> && docker compose up -d --build. Slower — a rebuild, not a cached image — but it needs nothing the box does not already have. Choose on whether you keep tagged images.

Backups: taken on a schedule, restored never. The backup design is the same in both projects and is well specified — a daily pg_dump --format=custom with 30-day retention, a daily mirror of the object store, a weekly copy of the Redis append-only file, and dashboards exported as JSON into git; the pilot adds a two-hourly forensic dump for incident evidence. Two properties are worth copying. The priority ordering is written down — which volume is CRITICAL, which is HIGH, which is merely reconstructable — so a restore under pressure has an order rather than an argument. And a backup is taken before a deploy, not only on a timer. What is not worth copying is the state of the restore path; see the declared gaps below. A backup you have never restored is a file, not a recovery.

First-time setup: the parts that are invisible until they bite. The routine half is uncontroversial and both projects do it identically — Docker Engine, a checkout under /opt/<app>, secrets generated with openssl rand into a chmod 600 env file that is never committed, UFW allowing only 22/80/443, fail2ban on SSH, key-only SSH with root login off, every infrastructure port bound to loopback and reached through an SSH tunnel. Four items are not routine:

Docker bypasses UFW. Publishing a port creates the DNAT rule and not the matching FORWARD ACCEPT, so ufw status can look correct while traffic is dropped — or, worse, while a port you never meant to expose is reachable. The pilot's hardening script writes explicit DOCKER-USER rules (RETURN for established, RFC1918 and loopback, then 80 and 443, then DROP). The gap cost 21+ inbound SIP INVITEs that reached the interface, matched PREROUTING DNAT, and died at the DROP; iptables -L DOCKER-USER -v -n --line-numbers is what surfaced it. Any newly published port needs its own ACCEPT, and a hand-added rule does not survive a reboot unless it is persisted.
The default log driver deletes the evidence. json-file discards a container's logs when the container is removed — so a deploy erases what the previous version was doing, which is how one demo's stdout was lost. Switching to journald only works paired with a persistent, bounded journald drop-in (Storage=persistent, a SystemMaxUse cap): without persistence it dies at reboot, without the cap it fills the disk.
Config formats that do not interpolate environment variables. Where a service reads YAML with no ${VAR} support, the substitution becomes a deploy-time step — and when that step lives in a YAML comment instead of a script, it gets skipped. Two services on one box came back from a reboot holding three unsubstituted placeholders between them — literal REDIS_PASSWORD_PLACEHOLDER and ${LIVEKIT_API_KEY} strings; they had been working only because a substituted copy sat in a running process's memory. Render the file in the deploy script from the env file, then assert no ${...} remains.
Anything installed by hand is not installed. A cron entry added once — a forensic backup, a certificate renewal hook — is not in the repository and does not survive a host rebuild. Both projects carry restart: policies on every container for the same reason; a container with no restart policy does not come back.

A reboot is the real test, and it is high blast radius. One accidental reset surfaced four latent failures at once — every one of them a piece of state that existed only in a running process. Schedule reboots into a window with a recovery runbook open; never reboot to clear something a command can clear. (The same incident is why diagnostic SSH gets bundled into one connection: a burst of short connections trips your own fail2ban jail mid-incident.)

A validator pinned to a superseded architecture is worse than none. The pilot ships a validate-deployment.sh; run today it exits 1 on two errors, because it validates a docker-compose.prod.yml that the compose split retired and a README-DEPLOYMENT.md that no longer exists — while checking neither of the compose files that actually deploy. Its sibling deploy-prod.sh is cited by the project's own release page as automating the release with rollback; it runs migrations against a service name absent from the compose file it passes, backs up a volume name neither compose file declares, and its rollback() function's restore step is a comment reading "In production, you would restore from the backup here." Meanwhile the 956-line design document that named the architecture opens with a DEPRECATED banner pointing at the Docusaurus guide that superseded it — the one artifact of the four that is honest about its own state. Deployment scripts rot exactly like documentation and are read under exactly the worst conditions, so they belong under §7.5's freshness discipline: one procedure is canonical, the superseded ones get a banner or get deleted.

Five contracts that only appear when you deploy onto a box someone else is already using. The rules above assume the server is yours. Joining a host that already serves live traffic adds failure modes that a single-tenant deploy never meets, and each of these was measured on a live box rather than reasoned about:

The firewall you think you have may exist for only one address family. A host with DOCKER-USER rules for IPv4 and an empty ip6tables DOCKER-USER publishes every 0.0.0.0-bound port to the public internet over IPv6 while the IPv4 equivalent is filtered. Verified live on a co-tenant's database: nc -6 -z <addr> 5434 → succeeded, where the IPv4 probe was refused. Loopback-binding every published port (§ above) is the fix; the lesson is that a firewall must be tested from outside on both families, because a rule set that covers one reads exactly like a rule set that covers both.
A migration gate belongs in the dependency graph, not in the runbook. Express it as a one-shot service the application services declare depends_on: { condition: service_completed_successfully }. Then nothing can start against an unmigrated schema — on this deploy or on any future restart. A runbook step saying "if migrations were not auto-applied, run them" is a reminder, and §7 has already ruled on reminders.
Build-time configuration needs an explicit rollback clause in the runbook. Values inlined at build (NEXT_PUBLIC_* in Next.js, VITE_* in Vite) are compiled into the assets. A restart cannot fix a wrong value and neither can an env edit — you rebuild. Write that sentence down: under demo pressure the instinct is to edit the env file and restart, which produces no change and burns the time you did not have.
Split liveness from readiness. /health answers "is this process alive" with no dependency calls; /ready aggregates the dependencies. Wire the container healthcheck to /health and the smoke test to /ready. Conflated, a transient database blip restart-loops an application that was fine.
Proxy defaults are tuned for request/response, and long-running work is neither. nginx's default 60-second proxy_read_timeout will 504 any genuinely long operation; proxy_buffering on breaks server-sent events. A stack with background investigations or streaming endpoints needs proxy_read_timeout 300s and proxy_buffering off on that vhost, and the symptom otherwise ("it works locally, it 504s through the proxy") wastes an afternoon.

Checklist. Provision once: Docker installed and version-checked; repository cloned; secrets generated and chmod 600; UFW to 22/80/443; DOCKER-USER rules written and verified from outside over both IPv4 and IPv6 and Docker restarted; fail2ban enabled; SSH key-only with root disabled; persistent bounded journald; DNS A record and TLS (certbot standalone plus a renewal hook, or an external proxy setting X-Forwarded-Proto/X-Forwarded-For — then update CORS origins to match); infrastructure up and every service reporting healthy; restart policies on every container; a migration gate in the dependency graph; backups scheduled with their priority order recorded; scheduled jobs installed and written down as reinstall-after-rebuild.

Every deploy: deploy lock acquired; CI green for the exact SHA being deployed; no in-flight work that a container recreate would kill; a database backup taken; previous image captured as the rollback target; the full compose file set named; image built on the host and tagged with the short SHA; migrations applied and confirmed at head.

After every deploy: health endpoint green and schema at head; the published ports are the ports you expect (docker ps --format '{{.Ports}}'); a real end-to-end request succeeds over the public hostname, not localhost; logs read for errors; caches that could mask the change flushed; and — where one exists — the falsification probe run and its artifact kept.

Evidence: the hospital pilot's scripts/deploy-prod.sh, scripts/deploy-app-on-pilot.sh, docker/deploy.sh, docker/harden-server.sh, scripts/validate-deployment.sh, docs/PILOT-RUNBOOK.md, docs/runbooks/2026-05-18-deploy-b1-b2-runbook.md, its Docusaurus docs/deployment/ guide, and the DEPRECATED docs/plans/2026-02-28-server-deployment-guide.md; the bookings platform's scripts/deploy-on-pilot.sh, scripts/pilot-preflight.sh, start-infra.sh, docker-compose.pilot.yml, its ADR-0012, and 47 committed .deploy-*.log runs. Every incident cited above is recorded in one of those artifacts or in the projects' operator memory; none of it is reconstructed from general knowledge. The five co-tenancy contracts come from a third artifact: a deployment plan for a shared Hetzner box, every claim of which was verified over SSH with non-mutating commands before it was written down — the IPv6 exposure, the empty ip6tables DOCKER-USER, the restart policies and the certificate-renewal state were all read off the running host.

Declared gaps — not covered because the sources do not establish them. Certificate renewal is documented as installed (a certbot renew cron with a --deploy-hook); nothing in either project records a renewal observed firing. Backups are taken daily and a weekly restore test is planned in the design document; no restore has been recorded, and the one scripted restore path is the comment quoted above — so these projects have backups, not a proven recovery. Both projects label their release "zero-downtime", but the shipped topology is a single app container brought down and back up behind a health wait, which is a short hard cut, not a rolling update. Alerting ships as rules with template contact points that route nowhere until real recipients are configured. And neither project deploys from CI: deployment is a human running a script over SSH, with CI as a gate, not a trigger. Do not read this section as covering blue/green, multi-node, autoscaling, or unattended continuous deployment — no evidence here speaks to any of them.