V1 — Two agents
Post-hoc Review
Agent A writes the answer first. Agent B then creates review criteria and checks the answer. If changes are needed, it sends feedback to Agent A.
More detailsAI Engineering Case Study
Evaluating Agentic Consensus Workflows
I built and compared different AI review workflows to see when extra agents improve quality and when the extra cost is worth it.
Why I built it
I built this project after seeing the same pattern many times: one AI agent creates an answer, but a second agent finds missing details, edge cases, or problems. The first agent then improves its answer. The second agent checks it again.
This happened when reviewing pull requests, creating AI skills, and building evaluation frameworks. I sometimes had to move an answer back and forth between different models until the result was good enough.
How I built it
I use LangGraph to build the workflows and manage the steps between agents. I use LangSmith to trace runs and inspect what each agent does. The app also evaluates final answers against saved criteria.
LangGraphLangSmith
LLM EvaluationMulti-Agent Workflows
Cost AnalysisTracing & Observability
Three ways to reach consensus
The small workflow became an experiment. I created three versions to compare their answers, cost, tokens, and speed.
V2 — With a moderator
Moderated Review
A moderator first makes the user's problem clearer and creates fixed criteria. Agent A writes the answer, and Agent B reviews it against those criteria.
More details
V3 — with an adversarial reviewer
Adversarial Review
This version also starts with a moderator. But Agent B tries to prove that the answer is not ready. It approves only when it cannot find a real blocker.
More detailsWhat I compare
I run the same tasks through different workflows and model combinations, then compare the results.
QualityHow well the final answer meets the criteria
IterationsHow many revisions were needed
TokensTotal input and output usage
CostProvider-reported model cost
LatencyTotal time for the workflow
Questions I want to answer
Does an intake moderator improve the result enough to justify its extra cost?
Does an adversarial reviewer find more useful problems than a normal reviewer?
Can a cheaper model review almost as well as an expensive model?
When does another review round stop being useful?
Which workflow gives the best balance between quality, cost, and speed?