crucihil verify

Synopsis

crucihil verify --suite <path> --rig <toml> [options]

Description

crucihil verify answers the question “would this suite actually catch a regression?” It runs the suite once as a baseline, then once per mutation — the simulated DUT is broken on purpose — and scores every test against every mutation it should catch. A test that stays green while the behavior it claims to verify is broken is vacuous, and this command finds it. Works on native CruciHiL suites and — via --framework — on existing pytest and Robot Framework suites, unchanged.

Outcomes

Each (test, mutation) pair gets one of three outcomes:

Icon	Outcome	Meaning
`✓`	`caught`	The test failed under the mutation — good, the test detects this breakage
`✗`	`missed`	The test stayed green while the behavior was broken — the test is vacuous for this mutation
`⊘`	`wrong_reason`	The test went `blocked`/`error` instead of failing — it reacted, but not as an assertion failure

blocked ≠ fail is itself a verification criterion. A test that goes blocked under a mutation is not catching the regression — it is misreporting a firmware defect as a rig-health issue. That is scored wrong_reason, not caught.

Only baseline-passing tests are verifiable. Tests that fail, error, or block in the baseline run are listed under ⚠ Not verifiable (failed baseline) and excluded from scoring.

Mutation kinds

The mutation plan is derived from the suite and (optionally) the contract, priority-ordered, and capped by --max-runs. Every planned run is one full-suite execution under one mutation.

Kind	Label example	What it breaks
Dead DUT	`dead_dut[EngineData,BrakeStatus]`	Every DUT output message goes silent — one run, scored against every test
Stuck output	`stuck_output[EngineData.RPM]`	One asserted signal is stuck at its DBC initial value — scored only against tests that assert that signal
Latency	`latency[EngineData+3.0s]`	The message is delayed 1.5× past the expect’s asserted timeout — a DUT slower than the bound must trip the test
Declared fault	`fault[can_dropout]`	A fault declared in a test’s YAML `faults:` block is injected — scored against the declaring tests

DUT outputs come from the contract when --contract is given; otherwise they are inferred from the messages the suite simulates (sim.start steps) plus every asserted signal’s message.

Options

path

required

Path to the YAML suite manifest — or a pytest/Robot test tree when --framework is given. Short form: -s.

path

required

Path to the rig TOML config. Short form: -r.

string

Verify an existing pytest or Robot Framework suite unchanged (pytest | robot) — strength report with zero rewrites. If --suite points at a wrapper manifest with a non-crucihil framework: field, this is auto-detected.

path

Contract artifact (contracts/<component>.json, produced by crucihil author or crucihil analyze) — sharpens the mutation plan with the component’s real outputs.

integer

default:"20"

Cap on mutation runs, priority-ordered (dead-DUT first — cheapest, strongest signal).

path

Write the full strength report JSON to this path.

boolean

default:"false"

Push the strength report to the cloud as a verification run — the dashboard’s run detail shows the tests × mutations matrix. Requires CRUCIHIL_BASE_URL and CRUCIHIL_API_KEY environment variables.

Exit codes

Code	Meaning
`0`	Every planned mutation was caught
`1`	Weaknesses found — one or more `missed` or `wrong_reason` outcomes
`2`	Framework error — bad contract file, suite load failure, `--suite` is a directory without `--framework`, missing `--sync` credentials, or sync failure

Example output

Suite : suites/engine.yaml
Rig   : rigs/virtual.toml
Running baseline + mutation runs...

  engine_startup
    ✓ CAUGHT       dead_dut[EngineData]
    ✓ CAUGHT       stuck_output[EngineData.RPM]
    ✗ MISSED       latency[EngineData+3.0s]
        Test stayed green while EngineData was delayed past the asserted bound
  can_heartbeat
    ✓ CAUGHT       dead_dut[EngineData]
    ⊘ WRONG_REASON fault[can_dropout]
        Test went blocked instead of failing under the injected fault

  ── Mutation scores ──
  engine_startup                           2/3 caught  ██████
  can_heartbeat                            1/2 caught  █████

  Suite mutation score: 60%

  ✗ 2 weakness(es) found — these tests would miss real regressions.

When every planned mutation is caught:

  Suite mutation score: 100%

  ✓ Every planned mutation was caught.

Verifying existing pytest / Robot suites

Existing test bases get a strength report with zero rewrites — point --suite at the test tree and name the framework:

crucihil verify --suite legacy/pytest_tests/ --framework pytest --rig rigs/virtual.toml
crucihil verify --suite legacy/robot_tests/  --framework robot  --rig rigs/virtual.toml

Foreign suites are scored by exactly the same rules as native ones — the planner feeds the same (signal, timeout) pairs and per-test relevance map into the same mutation tiers. A wrapper manifest generated by crucihil author --framework needs no flag at all — the framework: field in the YAML is detected automatically:

crucihil verify --suite tests/suites/brakecontroller_contract.yaml --rig rigs/virtual.toml

Passing a directory as --suite without --framework is an error (exit 2) — CruciHiL cannot tell which executor to use for a bare test tree.

Syncing to the dashboard

--sync pushes the strength report to the cloud as a verification run:

export CRUCIHIL_BASE_URL=https://cloud.example.com
export CRUCIHIL_API_KEY=chk_...

crucihil verify --suite suites/engine.yaml --rig rigs/virtual.toml --sync

  Synced : verification run 42 → https://cloud.example.com

The dashboard’s run detail page renders the tests × mutations matrix for the synced run.

Examples

Native suite

crucihil verify --suite suites/engine.yaml --rig rigs/virtual.toml

Sharpen the plan with a contract; save and sync the report

crucihil verify --suite suites/engine.yaml --rig rigs/virtual.toml \
  --contract contracts/brake_controller.json --report strength.json --sync

Existing pytest suite, capped at 10 mutation runs

crucihil verify --suite legacy/pytest_tests/ --framework pytest \
  --rig rigs/virtual.toml --max-runs 10

CLI Reference

Synopsis

Description

Outcomes

Mutation kinds

Options

Exit codes

Example output

Verifying existing pytest / Robot suites

Syncing to the dashboard

Examples

Native suite

Sharpen the plan with a contract; save and sync the report

Existing pytest suite, capped at 10 mutation runs

See also

​Synopsis

​Description

​Outcomes

​Mutation kinds

​Options

​Exit codes

​Example output

​Verifying existing pytest / Robot suites

​Syncing to the dashboard

​Examples

​Native suite

​Sharpen the plan with a contract; save and sync the report

​Existing pytest suite, capped at 10 mutation runs

​See also

Synopsis

Description

Outcomes

Mutation kinds

Options

Exit codes

Example output

Verifying existing pytest / Robot suites

Syncing to the dashboard

Examples

Native suite

Sharpen the plan with a contract; save and sync the report

Existing pytest suite, capped at 10 mutation runs

See also