Is automated pull-request feedback actionable?

An automated review is useful when it helps a person make a focused decision about a change. Comment count is not that measure. A long review can repeat itself, point outside the diff, or suggest broad cleanup without identifying a defect. A short review can direct attention to one human-adjudicated issue and propose a testable next step.

This guide includes an original, deterministic calibration lab. It compares invented feedback with explicitly stipulated synthetic reference issues for three tiny synthetic diff texts. No qualified person adjudicated the shipped examples. No model, review service, repository account, parser, or vendor product was run. The result evaluates only the supplied fixtures and mechanical matching policy—not correctness, actionability, or the quality of any automated reviewer.

Start with the review contract

GitHub pull-request reviews support line comments, suggestions, and the decisions comment, approve, or request changes. GitHub also recommends focused changes with enough context for a reviewer to understand the purpose. PR-Agent documents separate review and improvement tools and exposes controls such as a maximum finding count and whether tests or security should be reviewed. These interfaces differ, but they share a practical handoff: a finding must help a human decide what to inspect or change.

For this lab, a useful finding has five parts:

1. a location inside the changed files;
2. a category that matches the answer key;
3. an action—fix, add a test, or ask a bounded question;
4. a concrete next step; and
5. no duplicate of an earlier finding.

That is a calibration policy, not a universal definition of code quality. A real review may correctly raise an issue outside this fixture taxonomy. Human review remains responsible for intent, architecture, and merge decisions.

Three synthetic diff records, three review shapes

The fixture includes short, inert unified-diff text plus changed-file metadata and explicitly stipulated synthetic reference locations; it contains no private code and nothing is executed. One case is labeled with an off-by-one boundary issue. One is labeled with an omitted failure-path test. One has no reference issue. The candidate findings were deliberately constructed to include a localized report, a duplicate, an unmatched design question, and a suggestion outside the changed files.

The scorer mechanically matches only exact file, line, and category tuples. The shipped reference issues and each finding's actionable, not-actionable, or needs-discussion disposition are stipulated example labels, not human review evidence; the script merely counts them. It separately reports:

• expected issues found and missed;
• unmatched findings (not automatically false positives);
• duplicate findings;
• stipulated example dispositions, without deriving them from prose; and
• findings outside the changed-file boundary.

It does not infer correctness or actionability from code or free text, rank prose, or decide whether a pull request should merge. A localized reference match means only that a finding names the same tuple as the stipulated key.

Run the offline calibration kit

Prerequisite: Node.js 24 or newer. Download the files together, rename the script from .txt to .mjs, and run:

• Calibration scorer
• Synthetic review fixture
• Blank review worksheet
• Expected result

node pr-feedback-calibration.mjs pr-feedback-calibration.synthetic.json result.json

The script uses Node.js built-ins, performs no network request, and refuses to overwrite evidence. Repository tests execute it twice in isolated directories and require byte-identical results. Adversarial tests reject unknown fields, unsafe or duplicate identifiers, invalid lines, duplicate answer-key issues, undeclared files, and unsupported actions or categories.

The worksheet is a valid, deliberately obvious starter record. Replace every replace-with-... value, the invented diff text, and the example labels before treating a run as project evidence.

Use the result to tune the workflow

Start with a small set of representative, rights-cleared diffs and an answer key adjudicated by qualified people. Keep the answer key separate from candidate feedback. Compare each configuration, inspect every miss and unmatched finding, then change one review instruction or threshold at a time. Preserve the exact fixtures, configuration, adjudicators' criteria, and result so a later run is comparable.

Do not turn this tiny synthetic exercise into a product ranking or an accuracy claim. A useful next step is to build a project-specific calibration set from reviewed, non-sensitive changes and have more than one qualified reviewer resolve disagreements before treating the labels as reference evidence.

Primary sources:

• GitHub pull-request reviews
• GitHub guidance for reviewable changes
• PR-Agent tool documentation
• PR-Agent review configuration
• Ponytail repository

The Ponytail capture motivated the minimal-change question; its benchmark claims were not reused. All fixtures, labels, scoring rules, worksheet fields, conclusions, and limitations in this guide are original.