Mutation-test a plan before an agent edits code

A plan is useful when a reviewer can tell what must remain true, which surfaces may change, what evidence will be required, and when work must stop. Those are testable declarations. Whether the prose is persuasive is not.

This lab turns a narrow implementation plan into a JSON contract, applies four synthetic mutations, and runs a zero-dependency offline validator. It does not interpret requirements, judge architecture, run the commands named by the plan, or prove that an implementation will be correct.

Put policy outside the plan

The plan describes proposed work. A separate policy describes the reviewer’s expectations:

• changes are limited to src/ and test/;
• infra/ and .github/ are explicitly denied;
• no new dependency is approved;
• unit and manifest checks must be declared; and
• rollback must include both a strategy and a verification step.

Keeping these documents separate matters. A plan should not grant itself a wider scope by editing the rule it is evaluated against.

Keep requirements separate from results

testRequirements names commands that a future implementation must run. executionEvidence records a later observation with a status and artifact reference. The intact fixture is in the proposed phase, so it declares two tests and contains no execution evidence.

A proposed plan that says a test passed is rejected. A verified record must cite one non-conflicting, passed evidence record for every declared test. The validator checks only that the evidence reference is present and lexically shaped; it does not open that artifact, authenticate its producer, or decide whether the test was adequate.

Four mutations, four explicit failures

The scope-widening result is not natural-language contradiction detection. The policy exposes path prefixes as data, and the validator compares each declared change path against them. Similarly, the dependency result is a set-membership check, not a vulnerability assessment.

What the validator rejects before policy evaluation

The schema fails closed on unknown fields, wrong types, blank required values, duplicate test or evidence IDs, unknown evidence references, and paths outside a deliberately small portable lexical grammar. That grammar allows slash-separated ASCII letters, numbers, dots, underscores, and hyphens; it rejects absolute paths, traversal segments, backslashes, colons, controls, percent escapes, and glob characters. These are string checks only: the validator does not resolve a filesystem path, follow a symlink, account for every operating-system normalization rule, or inspect any named file.

The validator never executes testRequirements.command. Command strings are data in this lab. Running a plan requires a separate, authorized executor with its own filesystem, process, network, credential, and approval boundaries.

Reproduce the mutation run

Prerequisite: Node.js 24 or newer. The recorded run used Node.js 24.19.0 and built-in modules only. It used no network, credentials, packages, or user repository content.

• Plan-contract validator source, served as plain text
• Synthetic policy
• Intact proposed plan
• Missing-rollback mutation
• Scope-widening mutation
• Dependency mutation
• Unsupported-evidence mutation

Download the files into one directory, rename the validator from .mjs.txt to .mjs, then run:

node plan-contract-validator.mjs plan-contract-policy.synthetic.json plan-contract-valid.synthetic.json plan-contract-missing-rollback.synthetic.json plan-contract-scope-widen.synthetic.json plan-contract-dependency.synthetic.json plan-contract-evidence-claim.synthetic.json

The intact plan returns allow; all four mutations return reject. The combined process exits 2 because at least one inspected plan was rejected. That nonzero exit is the expected successful lab outcome.

Use the result as a review aid

Mutation testing asks a useful question: if a risky change entered this document, would the declared review contract notice? A passing intact fixture shows that the contract can represent one known-good example. Rejected mutations show only that these exact structural deviations were detected.

The lab does not establish that the allowed paths are correct, the rollback will work, the commands are safe, the tests cover the behavior, or a future agent will obey the plan. Those require human review and executed evidence in the real environment.

Before adopting a similar contract, choose policy terms that are visible and mechanically comparable. Avoid claims such as “detect architectural contradictions” unless the contradiction is represented as explicit data with a deterministic rule.