Testing Standard (Summary)
In one line: verify the claimed behaviour, distinguish unit simulations from real integration, and report scoped evidence—not confidence or a green subset.
The operational procedure is skill:s4u-testing-standard; Appendix A is the full reference. Use the adopted project's equivalent tools rather than forcing Python, PostgreSQL or a specific container package.
Key principles:
- Keep the subject real. Controlled unit doubles and error injection are legitimate. They do not replace evidence for a claimed real dependency. A deferred required real-boundary check needs an actually authorized, scoped exception; a
MOCK APPROVEDcomment alone is not approval. - Control timing and state. Use controlled clocks, isolated resources and bounded condition-based waits. Real elapsed-time tests are appropriate when timeout/cancellation is the property tested, not as a substitute for readiness.
- Prove selection and execution. Identify the expected non-empty case set, actual execution, assertions, environment and results. A marker or environment variable does not configure a runner automatically. Fast local checks do not remove required slower candidate/release evidence.
- Test protected boundaries before use. Synthetic prototypes may defer unrelated breadth; real-data pilots must verify applicable privacy, authorization, tenant isolation and critical journeys before using those boundaries. A 404 without a positive witness does not prove RLS.
- Challenge the oracle. Use mutations or equivalent adverse inputs for silent guards and data-driven checks. Attribute failure to the intended behaviour; import errors, arbitrary timeouts or any nonzero exit do not automatically prove a kill. Inspect the shipped mutation probe's documented limitations before use.
For output validation, exercise independently approved ordinary/empty and adverse/error expectations; distinguish verified zero from missing, unknown or not-assessed data. Compare authoritative record → payload → actual observable output on each applicable UI, API, CLI, batch-artifact or event surface. Preserve provenance and independent approval of pinned expectations, including reviewed corrections; production captures alone are observations. Verify alternate-path control parity and record permissible exceptions. A root-cause explanation alone does not clear a defect: verified correction or authorized permissible acceptance is required. Detailed procedure lives in release validation, with testing specifics in Appendix A.
Reference-profile coverage targets:
| Measure | Synthetic PoC | Production profile |
|---|---|---|
| Business logic/state-machine line coverage | 90% | 90% |
| Each other applicable layer | 70% | 90% |
| Overall line coverage | 70%, plus layer floors | 90%, plus layer floors |
| Inventoried failure branches with dedicated tests | Touched/risk-relevant cases required | 85% minimum; critical cases still required |
| Integration cases exercising intended real services | Report actual scope | 80% minimum; required boundaries still required |
Define layer membership and denominators; zero is not 100%. These are reference policies, not scientific guarantees or automatically installed metrics. An aggregate cannot hide an untested business layer; a high ratio cannot excuse a missing critical provider. Different approved enterprise profiles retain explicit evidence and authority.
Mechanism and limits: the supplied adapters can invoke configured commands, and reviewer checklists can identify gaps. They do not prove every expected test ran, establish production parity or implement the entire testing policy. A human-reviewed test receipt and required candidate checks remain necessary. Report pass, fail, unassessed and approved exception distinctly; none alone grants business acceptance or release permission.