Skip to main content

S4U Methodology — Showcase & Evidence

Case studies and evidence narratives moved out of the agent-facing canon (2026-06-12, canon v3): agents load rules (operating-card.md) and procedures (skills/); humans read this for the story and the numbers.

The delivery-flow dashboard, in demo mode — 16 September 2026​

What / why: the delivery-flow view that §7.9 and §7.10 describe was rendered as a single-page board and then rebuilt for progressive disclosure: one verdict sentence per screen, at most five dials, one chart, and every table behind a fold. The version published here runs on a fully synthetic, de-identified fixture so the layout and the provenance labelling can be reviewed without any live figure being read as approved.

Evidence: the hosted demo and Appendix P. What the demo shows honestly: a window in which automatic checks were switched off by a recorded decision, so stage attribution, CI minutes and rework share all read not assessed while landings and lead time stay measured. What it does not show: any adopter's real figures, any productivity claim, or a running collector. The fixture's numbers are illustration only.

Codex and Astra operating checkpoint — 10 September 2026​

What / why: TrustRelay moved primary implementation from Claude Code to Codex with the owner-chosen Astra configuration, retaining independent review and comprehensive CI. The owner selected targeted local regression/integration tests plus lint, types and review before pushing; a full local suite is not a universal prerequisite. See the operating page.

Evidence: PR #1421 was pushed at a01eaa24 after 125 targeted tests passed, with 96.31% coverage of the selected measurement; full frontend and advisory checks also passed. Comprehensive CI and independent review of that head were still pending at this checkpoint. A separate comprehensive local backend attempt recorded 14,837 passes and zero failures before reaching its time cap: no successful suite completion or final coverage was produced. It is incomplete, not green.

Measurement command and scope. The retained local receipt used Linux Python 3.13.15 in a disposable runtime, a native-filesystem checkout, migrated disposable services, and runner plugins for service ownership and per-test reporting. The selected pytest invocation was:

python -m pytest \
  tests/test_documented_script_paths_1093.py \
  tests/test_integration_gate_coverage_1085.py \
  tests/test_signal_capture_service.py::TestConfigFlags \
  -m 'not integration' \
  --cov=tests.test_documented_script_paths_1093 \
  --cov=tests.test_integration_gate_coverage_1085 \
  --cov=app.config --cov-fail-under=70 --cov-report=term-missing

Run from TrustRelay's backend directory with its installed dependencies and test infrastructure. This is the selected test/coverage command, not a standalone provisioning recipe. The original runner additionally loaded s4u_testcontainers_owner, s4u_diag_reporter and s4u_failure_context, wrote JSON/JUnit receipts and imposed a 300-second observer bound; these local helpers are not shipped by the methodology kit. Coverage counted 1,489 of 1,546 statements across app/config.py and the two named test modules. Including test modules is intentional for these detector tests; the figure must not be presented as product-code coverage. The frozen replay was based on ffeb61e5 plus the two reviewed test corrections subsequently committed in a01eaa24; the delivery owner confirmed the corrected source blobs match that receipt.

How to interpret it: These are project delivery receipts reported by the implementation coordinator, not a comparative experiment. They establish neither model superiority nor reduced cloud costs, token use or delivery time. Update the checkpoint after CI, review and deployment verification; do not infer a merge from a push. Appendix F remains a pointer to this evidence collection.

Case study: regulatory compliance as architecture​

Current interpretation (12 September 2026): this is a historical design account, not an application acceptance report or a legal determination. The correction below withdraws the former blanket logging, structural-impossibility and harm-comparison claims. It preserves the reported design motivations without borrowing current assurance from names of tables, types or regulations.

The transferable lesson is to identify applicable obligations early, assign accountable owners, and test the actual controls on the intended deployment. A methodology can organize that work; it cannot certify compliance.

Applicable Regulations​

AI Act classification depends on the intended use and applicable classification rules, not simply the presence of an LLM. The historical case used documentation, logging, transparency, oversight and robustness as design themes. This page does not establish that the entire Article 11–15 control set was independently required by other regimes, or that implementing similarly named features satisfied those provisions. Consult the applicable version and an accountable legal/compliance assessment. AI Act — official text and version history.

GDPR Article 22 concerns decisions based solely on automated processing that have legal or similarly significant effects, with conditions and exceptions; it is not a blanket rule covering every AI recommendation. Data minimisation, accuracy and storage limitation also matter. Determine the applicable lawful basis, safeguards, records, impact assessment and retention rules for the actual processing; do not infer a universal obligation to retain every conversation or operation indefinitely. GDPR — official text, especially Articles 5, 22, 25, 30 and 35.

Sector-specific record keeping requires its own applicability and retention decision. Record that decision and its evidence instead of importing a generic long-horizon minimum.

The Five Requirements for Every AI Output​

This historical heading is retained for incoming links. The five themes below are requirements to assess against the selected project profile, not proof that every output satisfies them.

1. Input provenance. References should identify the exact source/version supporting a claim and distinguish supported facts, inference and unavailable evidence. A reference-shaped object alone does not prove attribution, access permission or factual support; verify the producer and consumer paths.

2. Model identification. Retain the applicable model/deployment, prompt/template and context references for the authorized purpose. A version foreign key can support traceability but does not prove that the recorded version was the one actually executed.

3. Reviewable rationale, not hidden chain of thought. Keep concise conclusions, relevant source references, authorized tool outcomes and accountable decision reasons. Do not require private model reasoning or indiscriminate full-message retention. Logs need approved purpose, access, minimisation, retention and correction rules; they are not made necessary merely by calling them audit evidence.

4. Confidence and uncertainty. Report how uncertainty was assessed and the scope of any calibration evidence. A model-generated confidence number is not a probability of truth or a substitute for validation. Preserve explicit unknowns.

5. Audit integrity. Assess append-only restrictions using the actual writer and administrator privileges, event coverage and recovery behavior. A table that restricts updates does not alone prove tamper evidence, event completeness, or that omitted events can be detected. Verify the integrity and omission controls actually claimed.

The Non-Suppression Principle​

The historical name should be read as no silent loss of evidence, not indefinite promotion of every generated allegation to an active risk. Preserve traceable versions and dispositions under the applicable retention/privacy policy.

A later analysis may challenge an earlier finding. An accountable reviewer should be able to mark it confirmed, unresolved, corrected, superseded or rejected, with evidence and authority for that decision. Do not silently delete a valid concern, but do not keep a disproven claim active simply because it was recorded first.

A consequential reduction in scrutiny requires the appropriate authority and supporting evidence, recorded against the exact case and version. Assess both missed-risk harms and false-positive harms in context; this case does not establish an orders-of-magnitude comparison.

Evidence boundary: the earlier narrative reported provenance structures, version references and append-only records. Their existence is not proof of complete runtime enforcement. The methodology release does not certify that application's current roles, records, recovery or end-to-end acceptance. See the governed operating model for the portable distinction between policy, observation and authority.

Evidence metrics (former appendix-f)​

In one line: every claim about a codebase should ship with the command that produced it, so a skeptic can re-run it.

Do this: when you make an evidence-grade claim in your own docs, pin it to a commit and paste the exact collection command. The command examples below are historical reconnaissance for the stated repository layout, not a portable measurement harness. Confirm paths, branch, tools, error handling and the counted population before reusing them. File-name and text matches do not establish execution or semantic coverage.

Scope used by these commands: backend/app/ for production Python code, frontend/src/ for production TypeScript (excluding tests, dependencies, and generated files).

1. Codebase Scale​

Measure: total LOC, backend vs frontend split, API endpoint count, router-file count, ORM model count, service-module count. Each number below comes from one reproducible command — run them on your own repo.

Collection commands​

# Backend file count and LOC
find backend/app -name "*.py" -not -path "*__pycache__*" | wc -l
find backend/app -name "*.py" -not -path "*__pycache__*" | xargs wc -l | tail -1

# Frontend file count and LOC
find frontend/src \( -name "*.ts" -o -name "*.tsx" \) | wc -l
find frontend/src \( -name "*.ts" -o -name "*.tsx" \) | xargs wc -l | tail -1

# API endpoints
grep -r "@router\.\(get\|post\|put\|patch\|delete\)" backend/app/api/ --include="*.py" | wc -l

# API router files
find backend/app/api -name "*.py" -not -name "__init__.py" -not -path "*/deps/*" | wc -l

# ORM models
grep -c "^class.*Base):" backend/app/db/models.py

# Service modules
find backend/app/services -name "*.py" -not -name "__init__.py" | wc -l

# Pydantic model files
find backend/app/models -name "*.py" | wc -l

2. Development Velocity​

Measure: total commits, calendar vs active development days, commits per active day, the share of AI co-authored commits, and whether commits follow a conventional convention (feat/fix/docs/test with scope). The commands below derive each from git history.

Collection commands​

# Total commits
git rev-list --count master

# First and last commit dates
git log --reverse --format="%ai" | head -1
git log -1 --format="%ai"

# Active development days
git log --format="%ad" --date=short | sort -u | wc -l

# AI co-authored commits
git log --all --grep="Co-Authored-By" --oneline | wc -l

3. Testing & Quality​

Measure: backend/frontend test-file and test-function counts, the number of MOCK APPROVED comments and files mentioning testcontainers. These count source markers, not authenticated approvals, honest mocking or executed real-service tests. The commands below count each.

Collection commands​

# Backend test files
find backend/tests -name "test_*.py" | wc -l

# Backend test functions
grep -r "def test_" backend/tests/ | wc -l

# Frontend test files
find frontend/src \( -name "*.test.*" -o -name "*.spec.*" \) | wc -l

# Documented mock approvals (total comments)
grep -r "MOCK APPROVED" backend/tests/ | wc -l

# Files containing mock approvals
grep -rl "MOCK APPROVED" backend/tests/ | wc -l

# Files using testcontainers
grep -r "testcontainers\|TestContainer\|PostgresContainer" backend/ --include="*.py" -l | wc -l

4. Architectural Rigor​

Measure: number of Architecture Decision Records (with supersession tracking), Alembic migration count, and RLS-protected table count. These count artifacts associated with those practices, not their quality or effective tenant isolation. Assess actual policies, runtime roles and allowed/denied cases separately. The commands below count each.

Collection commands​

# ADR count
ls docs/adr/ | grep -c "^ADR-"

# Alembic migrations
ls backend/alembic/versions/*.py | wc -l

# RLS tables (grep + manual counting of TENANT_TABLES, DIAGNOSTICS_TABLES,
# and individual statements in migrations 023-030)

5. Living Documentation​

Measure: total doc pages, the breakdown by audience (architecture, ADR, API reference, strategy), and the code-to-documentation commit ratio. A ratio alone does not establish that the documentation matches the changed behavior or is useful to its audience. Track these in your own docs repo over time.

6. Methodology Infrastructure​

Measure: number of custom agent definitions (global vs project), persistent memory files, lifecycle skills, quality-gate hook layers, and MCP server integrations. These inventory possible components; they do not establish registration, execution or effective enforcement.

7. Updated Metrics (March 2026)​

Measure, over time: architecture pages carrying structured frontmatter, components mapped in the architecture index, backend documentation coverage (documented/total), and the chosen cross-tool instruction entry point. A symlink is only one possible adapter and does not itself prove host loading. Coverage is a ratchet — record it each cycle and watch the trend rather than chasing a one-time target.

Velocity Context​

In one line: co-author metadata records attribution conventions, not a measured division of labor or productivity advantage.

The historical collaboration combined human architecture/review with AI-assisted implementation and iteration. No controlled baseline here establishes equivalence to a larger team, saved effort or a causal delivery-speed improvement.

This methodology specification was itself designed using the brainstorm-to-spec-to-plan lifecycle it describes, authored collaboratively with an AI pair, and reviewed by fresh-context agents — a practical demonstration of the process.

Measuring whether the methodology is working​

In one line: use local measurements with explicit limits; do not borrow this page's dated project numbers as expected outcomes.

This page retains bounded historical project metrics and illustrative commands. Neither establishes effectiveness in another project. The kit also supplies a read-only reconnaissance report for an authorized repository; inspect its assumptions and scope before comparing runs:

bash scripts/methodology-health.sh <your-project-root>

It emits, from git and the file tree alone:

  • Feature flags currently defined versus retired in the last 30 days — the consolidation signal (do flags get removed once their work lands, or accumulate forever?).
  • ADRs added in the last 30 days — whether decisions are being recorded as they happen.
  • Design artifacts versus recent feature branches — a rough read on the one-design-artifact-per-feature rule.
  • Gate-config presence — whether the expected settings file is present; actual registration and execution require separate checks.
  • Oversized modules (over 3000 lines) — the structural-decay signal.

Mechanism: the report is generated by a checked-in script, so the numbers are reproducible and a skeptic can re-run them. Track the TREND across cycles, not any single absolute value — flags-retired-per-cycle going up, ADRs/month staying non-zero, design-artifact-per-feature holding near one, the gate config staying present.

What it does not measure: it counts artifacts, not their quality. A high ADR count says decisions are being written down, not that they are good decisions; a retired flag says the flag was removed, not that the consolidation was sound. The instrument is a tripwire for drift, not a verdict on correctness — that judgment still needs review. (Counting gate-saved incidents is intentionally out of scope until hooks persist block-events.)