Agent Verification
How to know if your AI agent's context is reliable. Trust ordering, memory hygiene, diff reading, browser smoke testing, and verification questions — the practices that prevent acting on stale or extrapolated claims.
When working with persistent-memory AI agents, not all context inputs are equally trustworthy. The trust order is: files read in this session beat live state observation, which beats tool results, which beats recent memory entries, which beats older memory and inferences from filenames. Memory is hypothesis; current state is truth. Apply the framework whenever an agent's claim could drive non-trivial work.
- Tier 1 — Files read in this session: direct, current, authoritative
- Tier 2 — Live state observed in the browser: empirical ground truth (screenshots, page renders, real flows)
- Tier 3 — Tool results from this session: API responses, git output, network fetches
- Tier 4 — Memory entries written or verified this week and not contradicted by reads
- Treat as suspect — Memory older than a few days; inferences from filenames or commit messages without opening the file; recall from many turns earlier in the same conversation
- Phrasing convention: "memory suggests X" vs "I read this file and confirmed X" — let the audience know your evidence source
Limitations: Requires the agent to have file or browser access — not all environments expose this. Doesn't help when the question is about external systems with no API access. Discipline-dependent: agents and operators can drift back to memory-first reasoning under time pressure.
When an AI agent reports a PR is ready and the build passes, that confirms intent compiled — not that behavior actually changed. Always read the diff for non-trivial PRs, especially UI changes where dead code can pass build but ship zero behavior change. The polymorphic-component pitfall is the most common case: a prop is added correctly but in a code branch that's only reached when the prop wouldn't have any effect anyway.
- Open the PR diff and verify the changes match the original ask
- For polymorphic components, confirm variant routing reaches the modified branch
- For new files or imports, verify they are actually wired into the render path
- Check that the modified code is reachable for the user state the change targets
- Compare the agent's summary against the actual file deltas — agents drift mid-PR
Limitations: Doesn't catch runtime issues (hydration errors, missing env vars, database failures) — pair with browser smoke test for those. Requires you or your agent to be able to read the PR; not useful if you're just trusting a CI green light without inspecting the diff.
When an agent references memory entries about repo state, code behavior, or shipped features, cross-check against current files before acting. Memory entries are most dangerous when they confidently assert 'X is missing' or 'Y hasn't shipped' — those assertions drive engineering work that's wasted if the feature has actually shipped. Default to skeptical for any 'missing' or 'deferred' claim more than a few days old.
- Open the file or run a quick grep to verify the memory claim
- When memory says "deferred" or "not shipped", default to skeptical
- When memory and code disagree, code wins — update or kill the memory the same turn
- Watch for memory entries older than 1–2 weeks on fast-moving projects
- Treat memory as hypothesis; current code as truth — phrase claims accordingly
Limitations: Discipline-dependent — agents will sometimes paper over memory-vs-code conflicts rather than surface them. Doesn't help if memory was wrong from the start (i.e., never reflected reality). Requires an audit pass on memory periodically, which itself takes time.
HTTP smoke-test every preview deploy before relaying an agent's 'ready' claim. Surface 500s, runtime errors, and DOM-level issues that build pass cannot catch. The smoke test is the empirical complement to the diff read — together they catch nearly all merge-time regressions for typical web app changes.
- HTTP fetch the preview URL and confirm 200 status
- Pull runtime logs to check for errors during page render
- Take a screenshot to verify the visual state matches expectations
- Walk the actual user flow (sign-in, key actions) end-to-end
- For complex changes, compare screenshots before and after the change
- Worked example: Better Auth's signUp.email and signIn.email accept a callbackURL parameter but do NOT auto-redirect — only signIn.social (OAuth) does. An AI agent assumed all three flows auto-redirected, the build passed, the diff looked clean, but users were stranded on the sign-up page after submit. Caught only by walking the actual cold-start signup flow on the preview deploy. Library behavior inherited from training data is a common hidden bug source.
Limitations: Requires browser access tools or manual testing time. Can miss subtle bugs that don't surface in the obvious flows you walk. Not a replacement for proper end-to-end tests on critical paths — it's a sanity check, not a guarantee.
Direct questions about evidence-source surface where an agent is reasoning from current data versus extrapolating from filenames, memory, or prior sessions. The agent doesn't need to be deceptive for these to matter — most context drift is unconscious. Bake these into your default prompts when accuracy matters: strategic decisions, dispatch specs, architectural recommendations.
- "What have you actually read in this session?"
- "Where's that claim coming from — current code or memory?"
- "Are you reasoning from a file or extrapolating from a name?"
- "What would you check first before specing a dispatch on this?"
- "When was that memory written, and have you verified it against current code?"
Limitations: Effectiveness depends on the agent's truthfulness and self-awareness — most modern agents will answer honestly when asked, but the questions can't fully replace empirical verification for high-stakes decisions. Best paired with at least one of the other practices.
The most dangerous memory says 'X is deferred'
| Practice | Catches | Cost | Speed | When to apply |
|---|---|---|---|---|
| Trust Ordering | Acting on stale or extrapolated context | Free (process) | Instant | Always — baseline discipline |
| Read the Diff | Dead code, missed integration, drift from spec | 5–10 min per PR | Slow | Every non-trivial PR before merge |
| Cross-Reference Memory | Stale claims about repo state | 1–2 greps + a read | Fast | Before specing dispatches or architecture |
| Browser Smoke Test | Runtime errors, visual regressions, integration bugs | 1–2 min per deploy | Fast | Every preview deploy before merge |
| Verification Questions | Agent extrapolation, soft claims | Free (prompt overhead) | Instant | Strategic decisions where accuracy matters |
Memory is hypothesis; code is truth
When you're about to act on a non-trivial claim about repo state — whether to spec a dispatch, change architecture, or merge a PR — open the file. Most expensive bugs come from acting on stale claims that have since been resolved by the codebase. The verification effort is small (a few greps, a read, a smoke test); the cost of acting on stale data accretes. The five practices above compound: trust ordering sets the discipline, read the diff + browser smoke test catch merge-time regressions, cross-reference memory prevents wasted dispatch work, and verification questions surface where the agent is soft before you act.