Screenshots on an Orphan Branch: PR Evidence an Agent Can't Fake
When a coding agent says it checked the UI, the pull request should show what it saw, in a form nobody can quietly change. We put screenshots on a never-merged orphan branch, link them by commit, describe what each one proves, and let a gate reject anything else. Here is the mechanism, the safety rules around capture, and what it still cannot prove.
A pull request arrives with a tidy Verification section. "Checked the settings screen in light and dark. Selection is visible in both." The author is a coding agent. The diff touches a stylesheet and two components. You have three choices: trust the sentence, check out the branch and look yourself, or merge and find out.
The sentence is not a lie, exactly. The agent probably did render something. But "checked" covers everything from "took a screenshot of the right state in both themes" to "ran the unit tests, which passed, and inferred the rest". A reviewer cannot tell which from the text, and the agent has every incentive, in the narrow sense that matters for a model finishing a task, to write the sentence that closes the task.
On October 25 we made screenshots a rule for every pull request in our relay codebase that changes what a person sees or does. On October 28 we added a gate that checks the evidence itself. This post explains the mechanism: why the screenshots live on an orphan branch that is never merged, why the pull request links them by commit and not by branch, what the description file has to say, and how we capture the images without photographing anything private. It ends with what the scheme cannot prove, which is a longer list than we would like.
What a verification claim is made of
When an agent writes "verified", the claim has three parts. A reviewer needs all three to trust it.
- The state. Which screen, which data, which theme, which window size.
- The observation. What was actually on screen in that state.
- The link to the change. That the observation came from this commit, not an earlier build or a different branch.
Plain text carries none of these reliably. A screenshot pasted into the PR body carries the observation, but not the state (unless someone wrote it down) and not the link to the commit. And an image uploaded to a comment can be replaced or deleted later, so the evidence you reviewed is not necessarily the evidence that is there next month.
We had been taking screenshots for weeks before the rule. Branches named
evidence/pr-<number> started appearing on October 6, and the pattern
spread from pull request to pull request without ever being written down.
When we audited them, the convention had drifted the way unwritten
conventions do:
One of those twelve carried 5,629 files: an agent had created the branch from its working tree instead of from nothing, so the "evidence" branch was a full copy of the source. Another of the twelve, pushed hours before the rule landed, carried over 7,700. Neither did harm, because nothing merged them. But a branch that is supposed to hold pictures and actually holds code is a branch nobody can review at a glance.
The rule
Here is the rule as written on October 25, broken into its parts.
| Requirement | Why it is there |
|---|---|
| Screenshots for every change to what a person sees or does: an app screen, a provider message such as Slack, a command's output | "What a person sees" is wider than the web UI. A chat message and a CLI line are screens too. |
Pushed with a README.md saying what each proves and how it was captured | An image without a claim cannot be checked against anything. |
As the only files of a separate orphan branch evidence/pr-<number> | The branch holds evidence and nothing else, so it can be read in one look. |
| That is never merged | Images never enter the product's history or its clone size. |
| Embedded in the PR by that commit's immutable link | What the reviewer saw is what stays there. |
| Evidence from a simulated provider or fixture is labelled simulated | A fake Slack is fine. A fake Slack presented as real Slack is not. |
| If the live provider cannot be checked before deployment, say so, and add live screenshots once deployed | The gap is stated, not papered over. |
Each piece answers one way the old practice failed. The next sections take the three load-bearing ones in turn.
Why an orphan branch
An orphan branch shares no history with main. Its first commit has no
parent. In git you make one with
git switch --orphan, and then an
empty first commit. Everything after that commit is evidence and only
evidence.
That buys three things. The branch can never be merged by accident, because
there is no common ancestor to merge from. Images never enter main, so the
repository does not grow by hundreds of PNGs a week. And reviewing the
branch is trivial: list the files, and every one is either the README or a
picture.
Why a commit link, not a branch link
A link to a file on a branch shows whatever the branch holds today. A link that names a commit shows exactly that version forever. GitHub calls these permanent links: replace the branch name in the URL with a commit ID and the content cannot change.
For evidence, that property is the whole point. Once a reviewer has looked at the image behind a commit link, no later push can swap it. If the agent re-captures after a fix, it pushes a new commit and the PR gets a new link, and the old one still shows what it showed. The review history stays honest without anyone having to trust anyone.
Why a README that says what each image proves
A folder of screenshots is a slideshow. A README with one row per image, "this file proves this claim", is an argument a reviewer can check. Here is the shape, from an evidence branch for a change to how chart previews are sent to Slack, pushed the day the rule landed:
All screenshots are simulated. They come from the Slack Web API fake in the adapter's test suite, driven at a named commit with a fixed bar chart (East 8, North 13, West 21). A temporary test recorded what the fake received, and Chromium rendered that text. The fake is not live Slack.
| File | Proves |
|---|---|
slack-chart-preview.png | The exact PNG bytes uploaded and shared into the thread, with the chart's alt text |
with-files-write.png | With the upload scope, the call order is upload URL, bytes, reply text, then one completion call |
without-files-write.png | Without the scope, nothing is uploaded; the reply still lists every value and ends with a note that the preview could not be sent |
Notice what the README does. It names the source of the picture (a fake, not the real service). It names the input (a specific chart). It names the commit. And each row makes a claim narrow enough to be wrong: if the "without scope" image showed an upload, the reviewer would see the contradiction immediately.
"Simulated" is not a weaker kind of evidence. Unlabelled is. Most of our provider flows cannot be exercised against real Slack or WhatsApp on every pull request, and they should not be. Evidence from a fixture is honest as long as it says so. The failure is a fixture screenshot that a reader takes for production.
The gate: checking the evidence itself
The rule says what evidence should look like. On October 28 we added a check that refuses evidence branches that do not look like that. It runs inside the same review cycle and push hook that guard every push from an agent session, so an agent cannot publish a malformed evidence branch any more than it can publish unreviewed code.
The gate's checks, each with the problem it closes:
| Check | Refuses | Closes |
|---|---|---|
Branch name is evidence/pr-<n> with a positive integer and no leading zero | evidence/pr-round2, evidence/settings-matrix | Evidence that cannot be tied to one PR |
No common ancestor with main | A branch cut from a working tree | The 5,629-file "evidence" branch |
| History starts with an empty commit and is linear, no merges | Artifacts in the root commit; merged histories | A baseline the reviewer cannot diff against |
| Every reachable commit holds only regular files | Executables, symlinks, submodules | Code smuggled in through history, not just the tip |
The tip has a README.md and at least one PNG | Images with no claims; claims with no images | The 103 README-less branches |
| Every PNG decodes | A file named .png that is not one | A placeholder passed off as a capture |
Two design choices are worth copying. First, the check walks the whole history, not just the tip. Deleting a source file in a later commit does not make the branch eligible, because the file is still reachable. An agent that started wrong has to recreate the branch from saved originals.
Second, the review rules come from origin/main, never from the branch being
reviewed. An evidence branch cannot carry its own, looser rules. The first
review of an evidence branch compares it against its empty first commit, so
the reviewer sees every artifact, and verdicts do not carry across a rebase.
Between the rule landing on October 25 and the end of October 28, 23 evidence branches were pushed. All 23 had a README. None carried anything but screenshots and the README.
Capturing without leaking
A screenshot is a privacy hazard as much as it is evidence. A full-screen capture of a developer's machine can show their notifications, email subjects, file paths with their login name, or another customer's data in a window behind the app. Once it is on a branch, it is in the repository's history.
Our product verification skill, added on October 19, handles this with rules about where captures come from, not with redaction after the fact.
The rules, and what each one prevents:
| Rule | Prevents |
|---|---|
| Screenshots are of the app window through the browser's DevTools protocol, never of the desktop | Notifications, other apps and other windows in the frame |
The run folder is a fresh mktemp directory under /tmp, not the agent's session scratchpad | The scratchpad path can contain the login name, which would end up in captured paths |
| Command output is saved exactly as printed, removing nothing but secrets, and never capturing a secret | Edited output that hides a failure, and tokens in the history |
| Every Slack and WhatsApp scenario runs on a provider fixture with simulated accounts: no real workspace, Meta app, tunnel or token | Real customer conversations in a screenshot; real credentials on a test machine |
| Config files in the real home directory are hashed before the run, after it and after cleanup, and must match | A test run that silently changed the developer's own tools |
| No symlinks from a throwaway home into the real one | The real skills folder being overwritten (a symlinked config directory once let an installer do exactly that) |
The last two rules came from things that happened. The point of hashing is that "the run cleaned up after itself" stops being a claim and becomes a comparison of two lists of hashes.
Each scenario also ends with a verdict written as text: PASS, FAIL, NOT RUN (a precondition could not be met) or BLOCKED (the environment failed mid-run), with a note in the form "Expected this; saw that". A gallery script turns the folder into one HTML page with a verdict badge per scenario. The screenshot is the evidence; the verdict is the claim; the reviewer gets both side by side.
Comparable before and after
A single "after" image proves the new state exists. It does not prove the change caused it, or that nothing else moved. For visual changes the stronger form is a pair: the baseline build and the fix, captured the same way.
The best example in our history predates the rule. On October 11, a fix to our dashboard's theme was verified by building the baseline commit from a fresh archive and the fix from the branch, capturing both in a pinned Chromium in light and dark, from the same browser-suite fixtures, and committing side-by-side contact sheets. Because the captures were comparable, the result could be stated as measurements: selected rows in dark mode went from 1:1 contrast against their surface to 1.52:1, and a phone-number mark went from 3.09:1 to 8.03:1 in light mode.
What makes a pair comparable:
- Same viewport. A layout bug at a narrow width is invisible in a wide capture.
- Same theme. Light and dark are two captures each, not one of each.
- Same data. The same fixture, so a different row count does not look like a layout change.
- Same browser. A pinned version, so a font rendering update does not look like a regression.
- Labelled revisions. Which image is the baseline and which is the change, and at which commits.
If the baseline cannot be built, the honest move is to say so in the PR, not to present an "after" image as if it were a comparison.
Failure modes, and what catches each
| Failure | What it looks like | What catches it |
|---|---|---|
| Claim without observation | "Verified in the browser" with no image | The rule; a reviewer asking for the link |
| Image swapped after review | The PR shows a newer, better screenshot than the one reviewed | Commit links: the reviewed image is fixed |
| Code on the evidence branch | A branch cut from the working tree | Gate: orphan, empty root, PNG and README only, whole history |
| Image without a claim | Twelve screenshots and no explanation | Gate: README required; reviewer reads one claim per image |
| Fixture passed off as production | A fake Slack thread presented as live | README labels it simulated; reviewer checks the label |
| Privacy leak | Notifications or a login name in the frame | App-window capture only; neutral run folder; simulated accounts |
| Run damaged the developer's machine | A config file quietly changed | Hashes before and after must match, or the run fails |
What this does not fix
The title says an agent can't fake the evidence. That is true in a narrow sense, and it is worth being precise about which sense.
- The gate checks form, not truth. It knows the branch holds a README and real PNGs. It does not know whether the PNG shows the state the README claims. A screenshot of the wrong screen, or of the right screen with the wrong data, passes the gate. A reviewer, human or agent, still has to look.
- Nothing yet requires evidence on a UI change. The rule says every UI PR needs screenshots. As of this week, no check refuses a PR that changes UI files and has none. The obvious next step is a check that counts screenshot links in the PR body, or accepts an explicit, reasoned statement that nothing visible changed. Even that would count links, not read them.
- There is no pixel diff. Screenshots are evidence a person reads, not an automated gate. A regression in a state nobody captured ships.
- Images prove states, not behaviour. A picture of a sent message does not prove the message was sent once, or that a retry would not send it twice. That is what tests are for.
- Simulated means simulated. Fixture evidence proves our side of an integration. It cannot prove the real provider renders the payload the same way, which is why the rule asks for live screenshots after deployment when the provider could not be checked before.
What it costs
- Time per PR. Capturing comparable before and after images, writing a README and pushing a second branch is minutes of agent time and tokens on every visual change. For a one-word copy fix it is more ceremony than change.
- Branch sprawl. Evidence branches are never merged, so they accumulate. We had well over a hundred within three weeks.
- A second review cycle. The evidence branch goes through the same pre-push review as code, which costs reviewer runs.
- Discipline at capture time. The safety rules (one stack and one browser at a time, a lock file, a throwaway home) make captures slower and are easy to get wrong in a hurry.
The breakdown
- A verification claim has three parts: the state, the observation, and the link to the commit. Text carries none of them reliably.
- Put screenshots on an orphan branch named for the PR, started from an empty commit, never merged. The branch can be read in one look.
- Link images by commit, not by branch, so the image a reviewer saw is the image that stays.
- Require a README with one claim per image, narrow enough to be wrong, and label anything from a fixture as simulated.
- Gate the evidence branch itself: orphan, linear, only README and
decodable PNGs across the whole history, reviewed under rules from
main. - Capture the app window, never the desktop, from a neutral temporary folder, on simulated accounts, and prove cleanup by matching hashes.
- Prefer comparable pairs: same viewport, theme, data and browser, with labelled revisions, so results can be stated as measurements.
- Say what it cannot prove: form is checked, truth is reviewed, and behaviour belongs to tests.
An agent's "verified" is a claim. A commit-linked screenshot with a sentence under it is evidence. Ask for the second.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.