Skip to main content

Quality Gates

In one line: Three automated layers catch defects at progressively more expensive stages — and every gate must name its mechanism, because a gate that can't is a wish.

Quality gates catch defects early, where they are cheap. A lint error caught after editing costs seconds; caught at review, minutes; caught in production, hours or days.

Gate admission rule (meta). Every gate or ritual must declare: (a) per-occurrence cost, (b) enforcement mechanism — a hook, CI check, or script ("the agent will remember" is not a mechanism), (c) retirement condition, and (d) observation proof — evidence that the gate sees something. A gate that cannot name its mechanism is a wish, and wishes recur as incidents: prose-only rules get violated, sometimes within minutes of being written; mechanized rules hold. New gates without all four fields are rejected at canon review — the PR template carries the fields.

Every new profile gate also declares demand tier, blocking status and discovery scope/limits, separately from mandatory validation. Declarations do not install enforcement. Release validation defines the advisory receipt checker and its limits; the release checklist owns the final-candidate full-run obligation for an adopted demand-only cadence (§7.9).

Field (d): observation proof. A named mechanism proves the gate exists. It does not prove the gate observed anything — and a gate that observed nothing is green for the same reason a gate that observed everything is green. Field (d) closes that gap at two different times:

  1. Demonstrated failure (admission time). The gate ships with a recorded instance in which it FIRES — a fixture, a test, or a mutation. Naming the mechanism is not enough; the mechanism must have been watched failing at least once, and that observation must live in the suite where it can be re-run, not in a memory of having tried it. Mechanism: scripts/check-test-coverage.sh — every shippable gate script declares # failing-case: <suite> "<test name>" (or # failing-case: none — <reason> where genuinely no firing case exists), and the meta-gate fails when the named test is not present in that suite, so the claim cannot drift away from the test it cites.

  2. Work witness (run time). Identify actual examined work through attributable output or an execution manifest: selected/executed files, built images, executed tests or compared rows. Expected nonempty work not observed is unassessed, not a pass. Legitimately unaffected scope is reported explicitly. Counts alone do not prove correct selection or outcomes. The kit’s enumerating checks print counts and test empty-subject behaviour; other adapters may use verified run artifacts.

The corollary, stated plainly: a guard verified only where it passes is indistinguishable from a guard that cannot fail. Every instance below was measured on a real project, and every one of them was green: a Tier-1 "no exceptions" completeness guard that could not fail, because the loader synthesised every missing key as None — deleting a field from its data file was invisible; a CI job named Docker Build that was green for months while a profile filter built zero images; a test that captured four sections from a SPARQL fetcher while bypassing the query, so reverting the query to its broken version left the shipped test file green; a guard test that grepped a job's whole text for a function name that also appeared in a comment, so deleting the actual call still passed; a path-containment guard that passed unconditionally in its container because REPO_ROOT resolved to /. None of these lacked a mechanism. Each had one, and none of them had ever been seen to fail.

Layers are complementary, not presumed statistically independent. Shared inputs, expectations or permissions can create correlated blind spots.

Gate: external contract evidence. The selected profile defines provider contract tests and any required sandbox/live check before release. Real calls require authorized credentials, data scope, environment and budget. Mocked tests support scoped local claims, not the real provider’s current contract. If required live evidence is unavailable, record a release gap; do not self-authorize calls or waive the requirement.

Gate: migrated-schema oracle. Integration-test databases are built via the project's migration chain (alembic upgrade head), never via ORM metadata create_all. A create_all schema is a proxy oracle — it lacks migration-only DDL (partial indexes, constraints), so tests can pass against a database production will never run.

Gate: probe-gated flag flip. A feature flag moving to default-ON goes through templates/scripts/flip-flag.sh: no recorded falsification-probe artifact, no flip. A flip is the moment a rollout stops being reversible-by-default, so the evidence that the flag's failure mode was looked for — not just that the happy path worked — is recorded before the default changes, not reconstructed after an incident. Cost: one probe run per flip. Mechanism: templates/scripts/flip-flag.sh (refuses the flip without the artifact). Retirement: when the flag is deleted along with its dead branch (the §2.8 census work-list is where that happens).

Review line: name the oracle. Name expected behaviour/provenance, observed artifact/revision/environment, actual work and limitations. A valid unit-level check is not defective merely because it uses a controlled boundary. A proxy cannot substantiate a broader real-system claim; close the gap or record an exception only where the accountable policy permits one.