Appendix A: Testing Standard

This is the full reference for the operational s4u-testing-standard skill. Use the adopted project's native runner, database, migration chain and identity system. Python/PostgreSQL/React examples are a reference profile, not mandatory replacements for enterprise tools. Stack choices implement the controls; they do not prove them.

The central rule: state the claim, exercise the relevant real behaviour, and report exactly what was observed. A test, a coverage percentage, a model review and a business acceptance decision are different kinds of evidence.

Table of Contents

1. Core Principles
2. PoC vs Production Mode
3. Coverage Targets
4. No Mocking by Default
5. Testcontainers Fixtures
6. Temporal Workflow Testing
7. Async Python Testing Patterns
8. HTTP Boundary Mocking
9. Security Tests
10. API Contract Alignment
11. Evidence Requirements
12. Frontend Testing
13. Quick Reference
14. Mutation Discipline

1. Core Principles

Principle 1: Tests Must Exist Before Merge

Changed behaviour needs appropriate tests and fresh evidence before merge. For bugs, first reproduce the failure; for clear requirements, use red–green–refactor. Approved synthetic exploration may use code-first ordering, but does not remove the verification requirement. A prose-only change does not need an artificial code test; instructions for agents need scenario validation.

Principle 2: Tests Must Catch Real Bugs

Derive expected values independently of the implementation. Verify outputs, state, rejection paths and relevant interactions. A mocked call assertion can establish what was sent to a double; it does not establish that a user was persisted or that an external provider accepted it.

Exercise ordinary successful/empty and adverse/error expectations, plus applicable missing, unknown and not-assessed states. Zero is valid for a fully assessed empty collection, not a substitute for an unavailable or partial assessment. Expected results are pinned and independently reviewed before candidate evaluation. Record the author/reviewer and actual independence limits; implementation authors can propose oracle corrections with reason and independent approval, not silently weaken tests to match code. Synthetic boundaries and approved sanitized goldens complement one another. Captured production output is an observation until its meaning is independently approved; an uncaptured or unverified golden remains a gap.

For example, a create-record integration case verifies the returned identity and the intended stored record through the real migrated database. An adverse case verifies an unauthorized attempt did not persist a record. Merely asserting that a mocked create() was called establishes neither outcome.

Principle 3: Real Services Always (No Mocking by Default)

The historical heading means real-service claims require real-service evidence, not that every unit test needs a database. Controlled dependency injection and error simulation are legitimate unit techniques. Keep the subject under test real and disclose each substituted boundary. See Section 4.

Principle 4: Deterministic Tests Only

Control time, random inputs and state wherever possible. Use reproducible seeds or recorded generated examples, verified isolation, and condition-based waits with deadlines. Real-time tests are appropriate when timeout, scheduling or process cancellation is the actual property being measured; arbitrary sleeping is not a readiness check.

An engine-managed timer requires the engine's test API, not a patched application clock. Bound both each remote query and the overall polling loop. Retries that eventually hide an unexplained failure do not establish reliability.

Principle 5: Fast by Default

Separate fast developer feedback from complete candidate/release evidence. Slow tests remain required when they cover required behaviour. Registering a pytest marker supplies metadata; selection requires an actual selector/configuration. An environment variable only works if something reads it. pytest marker semantics.

In prose: define the expected cases before execution, use quick checks while developing, then verify that all required cases actually ran on the candidate. A green subset cannot substitute for the required set.

2. PoC vs Production Mode

Also declare synthetic prototype, real-data pilot or production use. PoC is a delivery-breadth label, not a privacy or security exemption. Before real personal/confidential data, tenant boundaries, authentication or external effects are used, verify the applicable safeguards and critical journeys. If unavailable, keep the demonstration synthetic and record the unassessed boundary.

PoC Mode

Exploration may defer unrelated edge-case breadth and use test-after ordering under the adopted policy. Touched error paths, material risks and real-data boundaries remain covered. Tests and the applicable layer coverage are required before merge. No deadline authorizes an unreviewed change in data use.

Production Mode

Use the adopted comprehensive checks for changed behaviour, critical user journeys, security, integration and operations. The reference coverage targets below are minimums, not acceptance guarantees. Bug fixes reproduce first; release requires the authorized decision on the exact candidate, not a developer's “all green” statement.

3. Coverage Targets

By Mode

The following are reference-profile policies, not scientifically proven universal thresholds or automatically installed controls. A different enterprise profile must explicitly define its evidence and approval.

Measure	Synthetic PoC	Production profile
Business logic/state-machine line coverage	90%	90%
Each other applicable layer's line coverage	70%	90%
Overall line coverage	70%, plus the layer floors	90%, plus the layer floors
Inventoried failure branches with dedicated tests	Touched/risk-relevant cases required	85% minimum; critical cases still required
Integration cases exercising intended real services	Report actual scope	80% minimum; required boundaries still required

Declare denominators and classification: lines in each layer, inventoried failure branches, and which integration cases actually use their intended service. Zero subjects cannot become 100%; report unassessed or justified non-applicability. An aggregate can hide a critical untested layer, and a high integration ratio can hide one missing critical provider.

By Layer (PoC Mode Overrides)

Define layer membership in project configuration. A simple-looking API or UI can enforce critical authorization or business decisions; do not classify it as low risk merely because it is “CRUD” or “glue.” Profile changes need the appropriate approved decision, not an agent's silent edit to a local instruction.

Verification Commands

Use the real runner's coverage configuration and test selection. For the Python reference, an overall --cov-fail-under=70 command does not establish the separate 90% business-logic floor. Report the actual scopes and exclusions. Coverage configuration files do not prove that CI runs them.

4. No Mocking by Default

The Rule

A unit double can isolate the caller; it cannot certify the replaced service. Pure functions with sample inputs are not mocks. Keep real integration checks for applicable runtime boundaries and document the limitations of simulated responses.

What Is Forbidden (Without Approval)

Do not present SQLite as verification of PostgreSQL-specific SQL/RLS/migrations, an API emulator as the real provider, or a mocked function as evidence that function works. Required real-boundary checks cannot be deferred merely by adding a comment or choosing a convenient test label.

When Mocking Is Allowed

Technique	What it can establish	What it leaves open
Injected unit dependency or fault	Caller behaviour under controlled inputs/failures	Actual dependency behaviour
Scoped HTTP transport fixture	Real client's request assembly and response handling	Provider contract and environment
Component API-module double	Rendering and component interaction	Hidden client serialization/auth and real backend
Service emulator	Behaviour exposed by that emulator/version	Provider-specific semantics and deployment permissions
Authorized provider test account	Observed test-environment contract	Unchecked production differences

Approval Process

Use the adopted test policy for ordinary unit doubles. If a required real-boundary check is deferred or replaced, retain a linked MOCK APPROVED exception with affected scope, rationale, accountable approver/date, remaining real check, expiry/review condition and release consequence. A template placeholder is not approval. An exception cannot authorize effects outside its owner's remit.

5. Testcontainers Fixtures

Containers or isolated managed instances can provide disposable services. Neither a container name nor a fresh client proves isolation or production parity.

Prerequisites

Verify permitted execution, supported images, capacity, intended service version and owned cleanup scope. Test setup must not contact production or acquire live credentials by default.

PostgreSQL Fixture

Build the schema through the actual migration chain, not ORM create_all. Choose isolation according to connection ownership:

Application behaviour	Suitable approach	Proof required
All work uses one test-controlled connection	Migrated schema, outer transaction and verified savepoint joining	Application commit cannot escape outer teardown
Independent connections, server or workers	Isolated migrated database/namespace or verified reset protocol	Committed state cannot leak to the next test/worker

SQLAlchemy's external-transaction recipe is relevant to the first case, not a fence around unrelated connections.

The positive witness matters: an empty or wrong database also returns no rows. Run the cases in both orders and in the project's supported parallel configuration. Stop or effectively fence writers before reclaiming/resetting their resources.

MinIO (S3-Compatible) Fixture

Allocate a unique owned bucket/prefix per test or worker, verify the upload/read outcome, then clean only that namespace after writers stop. A new S3 client and a shared constant test-bucket do not prevent collisions or leftover objects. An S3-compatible test server still leaves provider-specific permissions and behaviour unverified.

Session vs Function Scope

Container lifetime and data lifetime are separate. Sharing startup may improve speed; function-scoped sessions do not undo committed rows. Measure startup cost and prove cleanup rather than assuming a fixed number of seconds or unconditional session-scope safety.

6. Temporal Workflow Testing

The Correct Pattern: WorkflowEnvironment.start_time_skipping()

When using Temporal's Python SDK, its time-skipping test environment can drive engine timers. Test workflow orchestration, activities and real integrations at their appropriate boundaries; mocked activities do not establish those activities' external effects. Use supported SDK versions and the actual project configuration. Temporal testing guidance.

What NOT to Use

Do not assume a signal call means all subsequent business processing completed, or that a query immediately afterwards must return the eventual state. Do not patch the application's wall clock to control engine timers. Wait for a defined observable state with a total deadline and bounded individual queries.

Testing Signals and Queries

Use unique workflow IDs/task queues where isolation requires them. Exercise the accepted workflow's success, rejection, duplicate/late signals, retry, cancellation and timeout cases. Verify actual authority checks at ingress; a test calling an internal signal method does not prove an end user is authorized to send it.

7. Async Python Testing Patterns

pytest-asyncio Configuration

Choose the plugin's supported mode for the actual test environment; asyncio_mode=auto is a reference option, not universal across all async frameworks. Verify collected/executed tests and fixture lifecycle. Do not infer execution because the function starts with async def.

asyncpg Compatibility: CAST Syntax

SQLAlchemy textual SQL with named binds can use CAST(:data AS jsonb) to avoid adjacent colon ambiguity. Raw asyncpg uses positional parameters such as $1; valid PostgreSQL :: casts are not universally forbidden. Verify the actual adapter and emitted query rather than replacing every cast mechanically. See the stack reference.

Async Fixture Patterns

Bound I/O and await the intended operations. Confirm completion and stop independent writers before teardown. A fixture containing a comment to delete objects does not implement cleanup; a yield and new client do not isolate committed data.

8. HTTP Boundary Mocking

The Pattern: Mock the Wire, Not the Function

For a client-contract test, exercise the real client with a controlled transport. Assert method, destination, relevant headers, serialization, response handling, rejected/malformed responses and timeouts. Use synthetic credentials. The function under test must not itself be replaced.

Why This Pattern

It preserves client logic while isolating an external dependency. A higher-level caller test may legitimately replace that client, but then establishes only caller behaviour. A configured fixture must reject unexpected requests rather than accepting arbitrary arguments and producing an always-valid response.

respx vs httpx.MockTransport

Both can provide controlled HTTP responses in the Python reference stack. Pick the one that exposes the assertions and lifecycle needed by the test. Neither is a live provider check. Use equivalents in other stacks; do not install a new mocking library merely to match this example.

9. Security Tests

Required whenever a protected boundary is in use, including real-data pilots. Do not defer applicable checks until a product is labelled production.

Authentication Enforcement

For each critical protected action, verify missing, invalid, expired and insufficient credentials as applicable, plus an intended authorized success. Check effects, not only status. Follow the API's approved error contract rather than treating a particular status code as universal.

Tenant Isolation

Prove tenant A can access its permitted record and cannot read or modify tenant B's existing record. Verify absence/malformed context, cross-tenant writes, exports, background work and pooled context reset where in scope. A 404 might be a nonexistent route or absent fixture; it is not proof of RLS.

PII Leak Prevention

Use clearly synthetic sensitive sentinels, a positive event witness and a capture point after the actual redaction processors/sink relevant to the claim. Assert forbidden values are absent and the expected sanitized event exists. An empty log stream is not a passing redaction test.

structlog.testing.capture_logs() normally bypasses configured processors; explicitly include the relevant path or capture its final output. caplog can be valid when stdlib routing is configured. Verify all claimed sinks and keep raw test diagnostics private. structlog testing behaviour.

Row-Level Security Verification

Test the effective runtime connection role on the real migrated schema. PostgreSQL superusers and roles with BYPASSRLS bypass policies; FORCE ROW LEVEL SECURITY can apply policies to table owners, not those bypass roles. Verify positive access, cross-tenant denial and state reset. PostgreSQL RLS.

An empty SELECT does not cover INSERT/UPDATE/DELETE, another connection role, a background job or an administrative path. Keep those scopes explicit.

10. API Contract Alignment

The Problem

Producer types, serialized payloads, published schemas and consumer behaviour can diverge. Matching source declarations alone does not establish runtime compatibility.

The Practice

Choose one approved contract authority: code-first or schema-first according to the project. Pin its revision, verify generated artifacts are current, compare compatibility, and test actual producer/consumer serialization. A generated OpenAPI description is not business approval or proof that a deployed endpoint matches it.

Change the producer and affected consumers together when required, or document the supported version overlap and migration plan. A backwards-compatible additive field need not force every consumer to mirror it mechanically. Legacy and new systems may coexist; record which contract/version each uses and test both sides of the boundary.

11. Evidence Requirements

Minimum Evidence (Both Modes)

A receipt records exact subject/revision, command, runner/environment, expected and executed cases, result/exit status, coverage scope, adverse outcomes, and accessible sanitized artifacts. Report skipped, deselected, unavailable and excepted checks separately.

For release outputs use the universal protocol and the v1 receipt contract. Evidence binds candidate/profile/corpus versions, declared surface, expected/observed results and execution context. Record corpus source, redaction transformations, author/reviewer and known limits in adopter-owned evidence; do not add undeclared top-level schema fields. Compare authoritative record, transported payload and actual UI/API/CLI/ artifact/event output separately. UI claims require browser/visual observation; headless products use their appropriate actual output, not a compulsory browser. Account for every inventoried surface, including optional/inapplicable ones.

Do not invent a passing output block for illustration and later treat it as execution evidence. Examples in this appendix specify test obligations; they are not completed application tests.

Production Mode Additional Evidence

Include required security, integration, recovery and critical-journey evidence for the candidate, plus remaining exceptions and accountable acceptance. Provider sandbox checks do not establish production configuration. A repository SHA does not by itself identify database state, model version or deployed infrastructure.

What Constitutes Valid Evidence

Distinguish pass, fail, unassessed and approved exception. Positive counts and exit 0 are necessary for some claims, but assertions and selection still matter. Link detailed access-controlled artifacts instead of copying private data, source or credentials into public reports. Verification evidence is not release authority.

12. Frontend Testing

Stack

Use the adopted runner and design system. React Testing Library with the selected Jest or Vitest configuration is a reference choice; do not replace a working runner without justification.

Patterns

Separate component rendering/interaction, real API-client transport behaviour, accessibility evaluation and critical end-to-end journeys. Test loading, pending, accepted, rejected, withdrawn, cancelled, unknown and recoverable errors where relevant. An optimistic state or successful network request is not an accepted business decision.

Frontend Mock Policy

An API-module double can isolate a component but hides the client's serialization/auth. Pair it with real-client transport tests and required real integration. Browser journeys with simulated backends may be useful UI evidence; label them simulated, not end-to-end business acceptance.

13. Quick Reference

Commands

Use actual project selectors. For pytest, pytest tests/ -m "not slow" selects tests not marked slow, including those carrying other markers; it does not prove all required fast cases were collected or executed. pytest tests/ follows configured defaults, which may themselves deselect/skip. Inspect configuration and report the expected/executed set. A RUN_SLOW_TESTS switch has no effect unless implemented. pytest markers.

Coverage Targets Summary

Use Section 3, including separate layer floors and declared denominators. Do not copy a flat 70% PoC figure without its business-logic floor.

Forbidden List (Without MOCK APPROVED Comment)

The historical heading points to the scoped exception policy in Section 4. A comment cannot turn a simulated boundary into real evidence. Ordinary unit doubles follow the approved test policy; deferred required checks require the proper decision.

Mock Approval Template

Record: MOCK APPROVED reference; boundary and revision; rationale; accountable owner/date; remaining real check; expiry/review condition; release consequence. Link the actual decision. A placeholder such as “[architect]” is not an approver identity.

14. Mutation Discipline

Mutation challenges whether a check notices changed behaviour. It complements coverage and adverse cases; no single mutant proves every relevant defect is detected.

14.1 The three outcomes (INCONCLUSIVE is not a pass)

Outcome	Meaning	Next action
KILLED	Intended tested behaviour detects the mutation	Record assertion and scope; not universal correctness
SURVIVED	Relevant test executes and still passes	Inspect binding, assertions and semantic equivalence
INCONCLUSIVE	Selection, application, environment or failure attribution is incomplete	Repair evidence/harness before scoring

Establish a green baseline with expected cases actually executed, one intended mutation applied, changed bytes, and the same required cases executed under mutation. Then attribute the failure. A collection/import error, missing environment, crash or arbitrary timeout is not automatically a kill. A demonstrated bounded deadline violation can count when that is the intended liveness assertion. Test runners distinguish failure classes; inspect their receipts, not only nonzero status. pytest exit-code meanings.

After safe restoration, run the required final selection and retain its passing receipt. A baseline, an intended failing mutation and a restored passing run are separate witnesses; synthetic guard tests do not verify captured product outputs.

14.2 Mutate DATA as well as code

Mutate what the guard actually reads: code, reference JSON/YAML, schema, thresholds or policy configuration. A mutation to an unused copy proves nothing. Use independently chosen missing/invalid values and assert the intended rejection rather than counting red jobs.

Reader-enumerating guard tests. Declare the discovery scope: searched roots/languages, access forms and known alias, reflection, generated-code or external-reader gaps. Discover consumers independently of the expected register, then assert registration and, where actually checked, routing through the policy owner. Merely iterating the register cannot discover an omitted reader. In isolated fixtures, add an unregistered in-scope reader and verify the intended failure; also verify an allowed registered reader passes. Report gaps as unassessed. This proves only the structural properties asserted within that discovery scope, not every runtime reader or how a caller interprets the answer. Retain independent behavioral tests for downstream use. (Canon §2.11.)

14.3 A surviving mutation means suspect the test first

Check subject binding and meaningful assertions first, then whether the mutant is semantically equivalent or the requirement is wrong. A survivor is not automatically a defective test; document the investigated outcome. Never assume one project's observed mutation results generalize to every test suite.

14.4 The shipped probe

templates/scripts/mutation-probe.sh is a local, adapter-reported probe, not independent proof of correctness. Use only disposable, isolated work and trusted harness commands. It executes those commands with the caller's permissions; it is not a sandbox.

The 4.0 candidate requires adapter v1. This deliberately replaces the old line-count/raw-exit convention: raw test-runner output now yields INCONCLUSIVE. The collection command must emit only a JSON array of unique test IDs; the execution command must emit only the following shape, derived from actual execution:

{
  "schema_version": 1,
  "cases": [
    {
      "id": "test_business_rule",
      "outcome": "passed"
    }
  ]
}

Outcomes are passed, assertion_failed, skipped or error. Report setup, import, teardown and infrastructure failures as errors, not assertion failures. Execution exit 0 must agree with all-passed results; exit 1 must agree with explicit assertion failures. Other exit codes, duplicate/missing/extra cases, skips, errors and any selection change are inconclusive. The baseline must pass and both runs must execute the same nonempty selection. The kit's synthetic shell fixtures demonstrate this protocol; a general pytest/Jest adapter is not shipped. Build and adversely test an adapter for the adopted runner before using its results. A hard-coded or dishonest receipt can still lie.

The probe's exit 0/1/2 means adapter-reported KILLED/SURVIVED/INCONCLUSIVE. Inspect the failing assertion and its relationship to the intended change; the label is not acceptance or release authority. Commands have a per-phase time limit (default 60 seconds, configurable from 1 to 3600), bounded stdout and their own POSIX process group. The group is killed before restoration, including on timeout or INT/TERM/HUP. This does not contain escaped sessions, external writers or remote side effects. Python subprocess process/session controls.

One regular, single-link file of at most 4 MiB is supported. A cooperating lock prevents two probes from sharing that subject. Restoration checks file identity, mode and expected bytes; a detected edit/replacement or uncertain cleanup preserves the intervening state and the original backup, returns INCONCLUSIVE and leaves the lock for reviewed recovery. The original is in the subject's sibling .<filename>.s4u-mutation.lock/original. Stop all writers and inspect both versions before restoring or removing recovery files. No SIGKILL, power-loss, hostile-filesystem or general stale-writer fencing guarantee is claimed.

14.5 When a mutation is required

The reference profile calls for mutation or equivalent adverse evidence for silent guards/gates, tests written after code, and data-driven validators. Record what changed, which expected case detected it, the attributable result and recovery. Whole-suite mutation engines are optional project choices, not a shipped universal gate.

The governed factory consumes scoped evidence. It does not turn a successful test run into business approval or permission to release.