Engineering leadership
How do I know whether an AI agent built the right thing?
Ask three different questions: Was this the right problem to solve? Did the change satisfy the accepted behavior? Is the implementation safe and maintainable? Passing tests answers only part of the second and third questions.
Use three independent verdicts
Product fit compares the work with the user, problem, outcome, and strategy. Contract fit compares the implementation with accepted requirements. Engineering quality evaluates the code itself.
Keep the verdicts separate. A well-written implementation can solve the wrong problem, while a strategically correct change can still violate an interface contract.
Right problem
Trace the work to an active problem and persona, then confirm that the chain remains coherent.
Right behavior
Map each accepted clause to complete test, static, or reviewed evidence.
Right implementation
Review security, reliability, maintainability, and repository conventions.
Require evidence at the pull-request head
An agent can start from valid context and still finish after that context changes. Re-evaluate the final head against current intent before merge.
Helen’s Foundry connects the pull request to live product artifacts and code symbols. Its findings show whether the problem is missing context, contradictory intent, a stale premise, or an exact clause violation.
Avoid the single-score trap
A composite score can hide one disqualifying failure. Keep blocking contract evidence distinct from advisory semantic judgments and unavailable evaluations.