Skip to main content

Quality Gates

In one line: Three automated layers catch defects at progressively more expensive stages — and every gate must name its mechanism, because a gate that can't is a wish.

Quality gates catch defects early, where they are cheap. A lint error caught after editing costs seconds; caught at review, minutes; caught in production, hours or days.

Gate admission rule (meta). Every gate or ritual must declare: (a) per-occurrence cost, (b) enforcement mechanism — a hook, CI check, or script ("the agent will remember" is not a mechanism), (c) retirement condition, and (d) observation proof — evidence that the gate sees something. A gate that cannot name its mechanism is a wish, and wishes recur as incidents: prose-only rules get violated, sometimes within minutes of being written; mechanized rules hold. New gates without all four fields are rejected at canon review — the PR template carries the fields.

Field (d): observation proof. A named mechanism proves the gate exists. It does not prove the gate observed anything — and a gate that observed nothing is green for the same reason a gate that observed everything is green. Field (d) closes that gap at two different times:

  1. Demonstrated failure (admission time). The gate ships with a recorded instance in which it FIRES — a fixture, a test, or a mutation. Naming the mechanism is not enough; the mechanism must have been watched failing at least once, and that observation must live in the suite where it can be re-run, not in a memory of having tried it. Mechanism: scripts/check-test-coverage.sh — every shippable gate script declares # failing-case: <suite> "<test name>" (or # failing-case: none — <reason> where genuinely no firing case exists), and the meta-gate fails when the named test is not present in that suite, so the claim cannot drift away from the test it cites.

  2. Work witness (run time). A gate that enumerates a subject set prints the count of what it examined — files scanned, images built, tests collected, rows compared — and FAILS when that count is zero. Where an empty subject set is a legitimate state (a diff that touched no mapped file), the gate reports examined 0 explicitly instead of a success message. The job name is a claim; the count is the check. Mechanism: implemented in the kit's enumerating gates (check-adr-register.sh, check-doc-classification.sh, check-test-coverage.sh), each with a fixture pinning the empty-subject failure.

The corollary, stated plainly: a guard verified only where it passes is indistinguishable from a guard that cannot fail. Every instance below was measured on a real project, and every one of them was green: a Tier-1 "no exceptions" completeness guard that could not fail, because the loader synthesised every missing key as None — deleting a field from its data file was invisible; a CI job named Docker Build that was green for months while a profile filter built zero images; a test that captured four sections from a SPARQL fetcher while bypassing the query, so reverting the query to its broken version left the shipped test file green; a guard test that grepped a job's whole text for a function name that also appeared in a comment, so deleting the actual call still passed; a path-containment guard that passed unconditionally in its container because REPO_ROOT resolved to /. None of these lacked a mechanism. Each had one, and none of them had ever been seen to fail.

Each layer is independent; no single gate is the sole defense against any defect category.

Gate: live contract smoke. Any feature whose runtime path crosses an external SDK/API boundary ships with a marker-gated live smoke (3-5 calls, real credentials, ~cents) that runs before deploy. Mocked-only coverage of an external boundary is an unverified boundary — an outage can ship behind a fully green suite of mock-based tests precisely because the mock never exercised the real boundary.

Gate: migrated-schema oracle. Integration-test databases are built via the project's migration chain (alembic upgrade head), never via ORM metadata create_all. A create_all schema is a proxy oracle — it lacks migration-only DDL (partial indexes, constraints), so tests can pass against a database production will never run.

Gate: probe-gated flag flip. A feature flag moving to default-ON goes through templates/scripts/flip-flag.sh: no recorded falsification-probe artifact, no flip. A flip is the moment a rollout stops being reversible-by-default, so the evidence that the flag's failure mode was looked for — not just that the happy path worked — is recorded before the default changes, not reconstructed after an incident. Cost: one probe run per flip. Mechanism: templates/scripts/flip-flag.sh (refuses the flip without the artifact). Retirement: when the flag is deleted along with its dead branch (the §2.8 census work-list is where that happens).

Review line: name the oracle. Every review checklist includes: "for each verification claimed, name the production artifact it observes." A check that observes a proxy (a mock, an ORM-created schema, an intermediate frame, a spot-check subset) is recorded as a proxy, and the gap is either closed or accepted in writing. Most green-gate-but-still-broken incidents trace to a gate that measured a proxy.