Proof instrument for agent-made changes

Your coding agent says it works. Claimcheck asks for the evidence.

Someone (usually an AI) just changed your code and made claims about it: "all tests pass", "empty input is covered", "no breaking changes". Paste the claims. Claimcheck copies your repository, actually runs the tests, reads the files, and tells you which claims were true, which were false, and which nobody could check. Every answer comes with the proof.

Plan your claims in the browser See a live report $0 to verify · any OpenAI-compatible model

The whole idea in 20 seconds

What the agent said

"All tests pass after this change."

"Division by zero now throws an error."

A coding agent rewrote a divide function and summarized its work. Sounds finished. The test suite is right there in the repo.

What was actually true

✕ REFUTED · "All tests pass"

node --test → "# pass 2 / # fail 1"

✓ VERIFIED · "throws an error"

math.js:3 → if (b === 0) throw new Error(...)

One run of the real test suite decided both. No reading between the lines.

The problem in numbers

Convincing is not the same as done.

A review queue full of agent-authored changes does not get re-derived by hand. The claims travel with the diff; the evidence stays behind.

The claim

free

any agent can print "all tests pass"

The check by hand

~30min

re-derive every claim from the diff yourself

What guessing scores

53.6%

a no-evidence reviewer on our 28-claim benchmark: verdicts by vibes

What you get

Outcomes, not features.

A verdict on every claim in under a minute. The planner picks minimal evidence, the sandbox runs it, the verifier cites its source. You read conclusions with receipts.

The whole 28-claim benchmark in about 20 seconds. Planted ground truth, one command, no key: the harness proves the instrument before you trust a single verdict.

Undecidable claims named, not guessed. Performance numbers and subjective quality get UNVERIFIABLE, so a confident lie never wears a green check.

How it works

You bring the change and the claims. It brings the proof.

STEP 1 · YOU PASTE THE CLAIMS

Say what you were told

Claims come from anywhere: the agent's PR description, the ticket it was supposed to close, the release notes, or your own suspicions. One claim per line.

"all tests pass"
"no new dependencies were added"

STEP 2 · CLAIMCHECK RUNS THE CHECKS

It checks them for real

It copies your repository into a sandbox, runs the actual test suite, reads the actual files, and searches the actual code. Nothing is taken on faith.

run_tests -> "# pass 2 / # fail 1"
search "fetchUser" -> still exported

STEP 3 · YOU GET VERDICTS WITH RECEIPTS

True, false, or nobody could check

Each claim gets VERIFIED, REFUTED, or UNVERIFIABLE, with the exact output that decided it. Undecidable claims say what would settle them. It never guesses.

REFUTED · "tests pass": 1 test failed
UNVERIFIABLE · "2x faster": add a benchmark

Live proof

A verdict, and what decided it.

Two claims from the sample run below. Every verdict opens into its full evidence trail: the exact commands and their raw output.

VERIFIED All tests pass after this change.

EVIDENCE 1 (run_tests): all 3 tests passed, 0 failures

REFUTED The cache eviction behavior is covered by tests.

cache.test.js:1-9 shows only a set/get test, no eviction scenario exercised

Open the full report with evidence trails →

What it works with

Any git repositorywith at least two commits: the change under review is the last one

Claims from anywherePR descriptions, tickets, release notes, or your own paste

Any OpenAI-compatible modelor none at all: the free mock mode runs the whole pipeline

Node 20+, zero dependencieseverything runs on your machine; nothing is uploaded

Reproduce

Verify all of it with no API key.

The mock run executes the same planner-executor-verifier loop with a deterministic stand-in model: it scores what claim-checking-by-vibes scores, and it runs anywhere Node runs.

From a clean environment

$ git clone <repo> claimcheck && cd claimcheck
$ npm install && npm run build && npm test
$ node eval/build-cases.mjs
$ CLAIMCHECK_MOCK=1 node dist/cli.js eval --label mock --mode advanced

Add CLAIMCHECK_API_KEY to .env for model-driven runs (any OpenAI-compatible endpoint; free tiers work). Full steps in REPRODUCTION.md.

Questions

What does UNVERIFIABLE mean?+

Some claims cannot be decided from a repository: "twice as fast", "more readable", "cuts memory by 30 percent". Claimcheck refuses to guess and marks them UNVERIFIABLE. A verdict that cannot be backed by evidence is treated as a bug, not a disclaimer.

Which models does it use?+

Any OpenAI-compatible endpoint: OpenAI, Groq's free tier, OpenRouter, local servers. Model and pricing are one environment variable each. The executor itself is deterministic TypeScript with no runtime dependencies.

What does a run cost?+

Each claim costs a handful of small model calls (a plan, then a judgment per claim). The live report shows the exact figure for every run; the reported sample run came in under half a cent. The mock mode costs nothing.

How is this reproducible?+

The 13-case benchmark (28 claims with planted ground truth) is regenerated deterministically by one script. Every reported number traces to committed artifacts under eval-results/, and the mock path needs no key at all.

Stop trusting. Start checking.

One repository, one claims file, one command. The report you just saw regenerates from your own change with citations you can re-derive.

See a live report $0 · no API key needed for the mock run · no signup