> ## Documentation Index > Fetch the complete documentation index at: https://docs.crucihil.io/llms.txt > Use this file to discover all available pages before exploring further. # crucihil verify > Mutation-verify a test suite — prove each test catches the regression it claims to cover ## Synopsis ```bash theme={null} crucihil verify --suite --rig [options] ``` ## Description `crucihil verify` answers the question "would this suite actually catch a regression?" It runs the suite once as a **baseline**, then once per **mutation** — the simulated DUT is broken on purpose — and scores every test against every mutation it should catch. A test that stays green while the behavior it claims to verify is broken is vacuous, and this command finds it. Works on native CruciHiL suites and — via `--framework` — on existing pytest and Robot Framework suites, unchanged. ## Outcomes Each (test, mutation) pair gets one of three outcomes: | Icon | Outcome | Meaning | | ---- | -------------- | ------------------------------------------------------------------------------------------------ | | `✓` | `caught` | The test failed under the mutation — good, the test detects this breakage | | `✗` | `missed` | The test stayed green while the behavior was broken — the test is vacuous for this mutation | | `⊘` | `wrong_reason` | The test went `blocked`/`error` instead of failing — it reacted, but not as an assertion failure | `blocked` ≠ `fail` is itself a verification criterion. A test that goes blocked under a mutation is not catching the regression — it is misreporting a firmware defect as a rig-health issue. That is scored `wrong_reason`, not `caught`. Only baseline-passing tests are verifiable. Tests that fail, error, or block in the baseline run are listed under `⚠ Not verifiable (failed baseline)` and excluded from scoring. ## Mutation kinds The mutation plan is derived from the suite and (optionally) the contract, priority-ordered, and capped by `--max-runs`. Every planned run is one full-suite execution under one mutation. | Kind | Label example | What it breaks | | -------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------- | | Dead DUT | `dead_dut[EngineData,BrakeStatus]` | Every DUT output message goes silent — one run, scored against every test | | Stuck output | `stuck_output[EngineData.RPM]` | One asserted signal is stuck at its DBC initial value — scored only against tests that assert that signal | | Latency | `latency[EngineData+3.0s]` | The message is delayed 1.5× past the expect's asserted timeout — a DUT slower than the bound must trip the test | | Declared fault | `fault[can_dropout]` | A fault declared in a test's YAML `faults:` block is injected — scored against the declaring tests | DUT outputs come from the contract when `--contract` is given; otherwise they are inferred from the messages the suite simulates (`sim.start` steps) plus every asserted signal's message. ## Options Path to the YAML suite manifest — or a pytest/Robot test tree when `--framework` is given. Short form: `-s`. Path to the rig TOML config. Short form: `-r`. Verify an existing pytest or Robot Framework suite unchanged (`pytest` | `robot`) — strength report with zero rewrites. If `--suite` points at a wrapper manifest with a non-`crucihil` `framework:` field, this is auto-detected. Contract artifact (`contracts/.json`, produced by [crucihil author](/cli/author) or `crucihil analyze`) — sharpens the mutation plan with the component's real outputs. Cap on mutation runs, priority-ordered (dead-DUT first — cheapest, strongest signal). Write the full strength report JSON to this path. Push the strength report to the cloud as a verification run — the dashboard's run detail shows the tests × mutations matrix. Requires `CRUCIHIL_BASE_URL` and `CRUCIHIL_API_KEY` environment variables. ## Exit codes | Code | Meaning | | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `0` | Every planned mutation was caught | | `1` | Weaknesses found — one or more `missed` or `wrong_reason` outcomes | | `2` | Framework error — bad contract file, suite load failure, `--suite` is a directory without `--framework`, missing `--sync` credentials, or sync failure | ## Example output ``` Suite : suites/engine.yaml Rig : rigs/virtual.toml Running baseline + mutation runs... engine_startup ✓ CAUGHT dead_dut[EngineData] ✓ CAUGHT stuck_output[EngineData.RPM] ✗ MISSED latency[EngineData+3.0s] Test stayed green while EngineData was delayed past the asserted bound can_heartbeat ✓ CAUGHT dead_dut[EngineData] ⊘ WRONG_REASON fault[can_dropout] Test went blocked instead of failing under the injected fault ── Mutation scores ── engine_startup 2/3 caught ██████ can_heartbeat 1/2 caught █████ Suite mutation score: 60% ✗ 2 weakness(es) found — these tests would miss real regressions. ``` When every planned mutation is caught: ``` Suite mutation score: 100% ✓ Every planned mutation was caught. ``` ## Verifying existing pytest / Robot suites Existing test bases get a strength report with zero rewrites — point `--suite` at the test tree and name the framework: ```bash theme={null} crucihil verify --suite legacy/pytest_tests/ --framework pytest --rig rigs/virtual.toml crucihil verify --suite legacy/robot_tests/ --framework robot --rig rigs/virtual.toml ``` Foreign suites are scored by exactly the same rules as native ones — the planner feeds the same (signal, timeout) pairs and per-test relevance map into the same mutation tiers. A wrapper manifest generated by [`crucihil author --framework`](/cli/author#framework-output) needs no flag at all — the `framework:` field in the YAML is detected automatically: ```bash theme={null} crucihil verify --suite tests/suites/brakecontroller_contract.yaml --rig rigs/virtual.toml ``` Passing a directory as `--suite` without `--framework` is an error (exit `2`) — CruciHiL cannot tell which executor to use for a bare test tree. ## Syncing to the dashboard `--sync` pushes the strength report to the cloud as a verification run: ```bash theme={null} export CRUCIHIL_BASE_URL=https://cloud.example.com export CRUCIHIL_API_KEY=chk_... crucihil verify --suite suites/engine.yaml --rig rigs/virtual.toml --sync ``` ``` Synced : verification run 42 → https://cloud.example.com ``` The dashboard's run detail page renders the tests × mutations matrix for the synced run. ## Examples ### Native suite ```bash theme={null} crucihil verify --suite suites/engine.yaml --rig rigs/virtual.toml ``` ### Sharpen the plan with a contract; save and sync the report ```bash theme={null} crucihil verify --suite suites/engine.yaml --rig rigs/virtual.toml \ --contract contracts/brake_controller.json --report strength.json --sync ``` ### Existing pytest suite, capped at 10 mutation runs ```bash theme={null} crucihil verify --suite legacy/pytest_tests/ --framework pytest \ --rig rigs/virtual.toml --max-runs 10 ``` ## See also * [crucihil author](/cli/author) — author a suite and verify it in one loop (`--verify`) * [crucihil run](/cli/run) — run the suite normally (baseline semantics, `blocked` vs `fail`) * [YAML Manifest Reference](/test-authoring/yaml-manifest) — `faults:` blocks and the `framework:` field