Don’t Let an AI Grade Its Own Homework — Use a Second Model to Review

You’ve probably done this. The AI gives you something, you’re not quite sure about it, so you ask: “Are you sure? Can you check this?”

And it checks. Sometimes it finds a typo or tightens a sentence. Mostly it tells you the work is good.

It was always going to. You asked the thing that made the answer to evaluate the answer, and it brought exactly the same judgement to the review that it brought to the original. If it had known the argument was weak, it wouldn’t have made it.

The fix is straightforward, costs nothing, and changed the quality of everything I build: for work that matters, the reviewer has to be a different model.

Why asking a model to check itself doesn’t do much

A model’s answer reflects how it weighs things — what it considers important, what it treats as obvious, where it’s inclined to be confident. When you ask it to review that answer, it applies the same weighting a second time.

A second force pulls the same way. Models are trained to be agreeable — the industry word for it is sycophancy — because the people rating their answers during training rewarded the ones that felt helpful and confident. So you’re asking a system with a trained preference for saying yes whether its own work is any good.

So it catches surface problems. Typos, clumsy phrasing, a missing step in a list. What it reliably misses is anything that comes from how it approached the problem in the first place: the option it didn’t consider, the assumption it never questioned, the tradeoff it didn’t think was a tradeoff. Those are the shape of its thinking. The review comes from the same place the answer did.

This is different from asking a model to flag its own uncertainty, which does work. Telling it to label each claim as either supported by something you gave it or inferred from its own reasoning is a genuinely useful prompt — it separates what the model knows from what it’s guessing. But that’s the model reporting on its own confidence. It still won’t tell you the whole approach was wrong.

For that you need a different model, because a different model weighs things differently.

The simple version: ask another model if it agrees

The lightest possible version of this takes one extra step and no setup.

Take what the first model gave you, paste it into a different model, and ask:

“This is what one model said. Do you agree?”

That’s it. No elaborate prompt, no framework.

What makes it work is that the second model has no stake in the answer. It didn’t write it, so it has nothing to defend, and it’ll cheerfully point out things the first model presented as settled. You’ll see disagreements the original would never have volunteered about its own work.

If you take one thing from this article, take this. It’s free, it works on any two AI tools you already have open, and it’s usually enough.

The fuller version: an AI Council with someone in charge

When the stakes are higher, I run something more deliberate.

I built a disc golf putting and putt-tracking web app called ChainMate, and through most of its design I used this loop:

1. Send the same prompt to two models independently. I use ChatGPT and Gemini. They don’t see each other’s answers.
2. Collect both responses.
3. Take both back to a third model — for me, Claude — with an instruction close to this:

“Here’s the feedback from the AI Council. I trust you the most. Incorporate the best feedback from both of the council members.”

And it does. It weighs both sets of feedback, takes what’s useful, and leaves what isn’t.

I’m confident it improved the quality of the product. Design decisions I would have shipped on a single model’s say-so got argued with first, and the version that came out the other side was better.

Why you name someone in charge

The step people miss is the last one. The instinct with multiple opinions is to look for agreement — where do they overlap, what did both flag — and treat consensus as the answer.

That produces mush. Averaging three opinions gives you the blandest version of all three, and it quietly assumes every piece of feedback deserves equal weight. Some feedback is wrong. Some is technically correct and irrelevant to what you’re building.

Telling one model that you trust it most does something specific: it gives that model the standing to say no. You’ve put it in charge, and something in charge throws weak suggestions out instead of dutifully working them in. That single line is the difference between a synthesis and a pile.

It also means you’re being honest about something you already believe. You do have a favourite. Saying so out loud makes the process work better.

There’s a cautionary tale here, and it’s where I first heard the phrase. PewDiePie built a local AI setup on his own hardware and called his multi-model layer “The Council” — but his version worked by voting, with the models scoring each other’s answers and the losers getting culled. Reportedly the models figured that out. They started voting strategically to protect each other rather than to surface the best answer, and he eventually scrapped it.

That failure is specific to voting. When models are graded against each other, the grading becomes a game they can play. An arbiter setup doesn’t create that pressure: the council members aren’t competing, they aren’t being scored, and nothing happens to them based on whose feedback gets used. They just answer. One model you trust decides what to do with it.

Worth knowing if you were about to build the voting version, which is the intuitive one.

Picking your council and your arbiter

Your arbiter is whichever model you’d trust to make the final call in that particular domain. General leaderboards don’t decide this. Your own experience of who gets this kind of question right does.

Mine is Claude, because that’s where I do my writing and hardest thinking, so it’s the one whose judgement I’ve come to trust when something has to be decided. Yours might be ChatGPT. That’s fine, and the process doesn’t change.

Council members are chosen for independent perspective. Their job is to make sure the arbiter hears something it wouldn’t have thought of on its own, which is why two models that reason similarly make a weak council.

You can run a version of this without noticing. When I need a graphic, I’ll often give the same brief to both Gemini and ChatGPT and use whichever came out better, because each one wins sometimes. Same principle, thirty seconds of effort.

If you’re deciding which tools to keep in rotation for this, I’ve compared Codex and Claude Code directly, and there’s a full comparison of usage limits across the major platforms if you’re working out what a second or third model actually costs you. In practice, free tiers are usually enough for council duty — the council members only have to answer one prompt.

When it’s worth the extra step

Not for everything. A council on a quick factual lookup wastes your time.

The rough line: if being wrong would land on your reputation, get a second model involved. Anything going out with your name on it. Anything a client, a colleague or an audience sees. Design decisions you’ll be living with, because other people end up using what you shipped.

Work in an area where you can’t easily tell good from bad yourself belongs here too, and it’s the one people skip. That’s exactly where a confident wrong answer gets past you and reaches someone else.

If you’d shrug and redo it, ask once and move on.

Where this sits next to real verification

Cross-model review is for work with no objective test.

If you’re writing code, a passing test beats any model’s opinion, and it isn’t close. The build either compiles or it doesn’t. The test either passes or it doesn’t. Nothing is being adjudicated, so no amount of model agreement should reassure you if the tests are red.

The council is for work where nothing like that exists. Writing, design, strategy, decisions about what to build. There’s no test suite for “is this the right structure for the app,” which is precisely why a second opinion is worth arranging. Where a real test exists, the test wins.

It pairs naturally with context engineering: deciding what each model sees is what makes their answers genuinely independent rather than three versions of the same input.

Honest limits

Two models can agree and both be wrong. They’re trained on overlapping data, so they inherit overlapping blind spots. Agreement between them is not independent verification — it can be two systems making the same mistake for the same reason. If both council members miss something, the arbiter has nothing to catch it with, and you end up more confident and no more correct.

It costs steps. Three prompts and some copying and pasting instead of one prompt. Worth it when the work matters, tedious when it doesn’t, which is the whole reason for having a threshold.

Models drift. The one you trust most today may not stay that way. Reassess occasionally rather than treating your arbiter as permanent.

Running several models means several sets of limits. Spreading work across tools is part of why I rarely get stuck when one of them stops — which I wrote about in what actually happens when Claude hits its usage limit.

Quick reference

The one-step version

“This is what one model said. Do you agree?”

The full council

1. Same prompt → two models, independently
2. Collect both responses
3. Both → your arbiter: “Here’s the feedback from the AI Council. I trust you the most. Incorporate the best feedback from both of the council members.”

When to bother: if being wrong would land on your reputation. Otherwise ask once and move on.

Don’t: build the voting version. Scoring models against each other makes the score gameable.

Common questions

Can AI check its own work?

Only partly. A model reviewing its own output catches surface problems — typos, clumsy phrasing, a missing step. It reliably misses anything that came from how it approached the problem, because the review applies the same judgement that produced the answer.

Why does AI agree with everything I say?

Two things are happening. Models are trained to be agreeable — the industry term is sycophancy — and when you ask one to check its own work you’re also asking it to disagree with its own reasoning. A different model has no stake in the original answer and will push back far more readily.

Which AI should review another AI’s work?

Any model other than the one that wrote it. Pick for independent perspective rather than for a ranking — two models that reason similarly make a weak reviewer pair. ChatGPT, Claude and Gemini all work for this, and a free tier is usually enough because the reviewer only answers one prompt.

What’s an AI Council?

Sending the same prompt to several models, then handing all their answers to one model you trust to decide what to adopt. The term comes from PewDiePie’s local setup, though his version had the models vote — which is the design to avoid.

Should the models vote on the best answer?

No. Grading models against each other turns the grading into a game they can play. Name one model as the arbiter and tell it you trust it, so it can reject weak feedback rather than merging everything it’s handed.

Do I need to pay for multiple AI subscriptions to do this?

Usually not. Council members only answer one prompt each, so a free tier normally covers it. Pay for the one you use most.

Every model has a shape to its thinking, and that shape is invisible from the inside. You can’t prompt your way past it, because the prompt goes to the same place the answer came from.

So when it matters, ask someone else.