What the runner reports
Each case ends in one of:
| Result | Meaning |
|---|---|
| pass | Every MUST check in the case held. |
| FAIL | A MUST check did not hold. |
| known | Every check that did not hold is listed in the known failures file. |
| not run | The adapter has no function for this case. |
| advisory | A SHOULD check did not hold. Printed beside the case's result. |
The runner exits 0 only when nothing failed, nothing was left unrun, and no
known failure has started passing. It exits 2 if a vector file does not match
SHA256SUMS or the runner's own QR check fails.
Known failures
A known failures file lists { file, case, check, requirement, reason }
entries. It exists so a signer that fails a case today can still use the runner
to catch regressions, and it changes no expected value. A listed failure is
still a failure of the profile. If a listed case starts passing, the run fails
until the entry is removed, so the list cannot go on describing a bug that has
been fixed.
Rendered from conformance/README.md, the same bytes that ship with the code. SHA-256 d0c681a5e7b7f55bcc96ecd33bb0ab07dd944b4fe2bb50f640ee79a85db3917e