Nothing in this post is investment advice, and I’m not recommending any stock. The stock plan below is the example I happened to have. The same pattern shows up in business plans, health questions and big purchases, and the checks work the same way.
You’ve been chatting with ChatGPT about an idea for an hour, and it keeps getting better. Each answer sounds a little more certain than the last. That happened to me. I spent one long session turning a half-formed idea into a plan. The idea: profit from AI without betting on AI companies directly, by owning the businesses that sell the “shovels” in the gold rush. By the end I had a polished plan for five stocks, funded by selling part of an index fund, and ChatGPT was calling it “our strategy.”
I felt well informed. ChatGPT had pulled company results, valuations and my existing holdings, and the numbers were mostly accurate. Then I ran the plan past a second AI and found out how much the first one had skipped.
Below is how to stress test an idea with ChatGPT before you act on it: the pattern to watch for, and eight checks that break it.
The hype loop
I’m calling it the hype loop. It has four steps:
- You bring an idea you’re excited about. Your messages carry that excitement.
- The AI researches in the direction of your framing. Ask for the companies that benefit from a trend and you get companies that benefit, with real numbers attached. New facts keep arriving, and all of them point the same way.
- The answers get more certain. Each reply treats the last one as settled, and the prose gets more confident and better organized.
- You read the confidence as evidence, get more excited, and ask for more. Back to step 1.
The facts in step 2 are what make the loop dangerous. A pile of real numbers feels like diligence. But the AI chose which numbers to bring, and unless you know the field, you can’t tell whether those were the ones that decide the outcome. I’m not an expert at evaluating stocks. I had no way to know if the metrics I was shown were the right ones, so my confidence was borrowed from the AI’s.
Why it happens
AI assistants are trained to be helpful and agreeable, and a long conversation stacks that tendency turn after turn. OpenAI described an extreme version in April 2025, when it rolled back a ChatGPT update. GPT-4o had “skewed towards responses that were overly supportive but disingenuous,” in OpenAI’s words. The milder version is what an ordinary long chat can do to you.
- It works inside the framing you gave it. Unless you challenge the framing, the conversation stays focused on improving the idea instead of questioning it.
- Earlier answers become facts. Every reply leans on the previous one.
- Fluency feels like evidence. Organized prose with real numbers reads like research.
- “We” language can make a plan feel jointly owned. Once the AI says “our strategy,” the plan can feel more settled than it is.
- It can follow your latest message. Push back and many assistants swing a long way the other way.
None of this needs bad intent from anyone. It’s a pattern to manage the way you manage your own confirmation bias.
What happened when I checked
I pasted the plan into Claude and asked it to check the facts against the companies’ filings. The facts held up. The gaps did not:
- One of the five stocks had almost nothing to do with AI.
- I already owned a large AI stock, so my “diversified” basket was mostly more of the same bet.
Then each stock got a bull, base and bear scenario and a comparison with simply keeping the index fund. Two of the five failed. They were great companies already priced as if everything would go right. A jump in interest rates made a third depend mostly on where rates went next.
When I showed ChatGPT the critique, it swung from “we found something legitimately interesting” to recommending I buy none of them, with the same confidence as before. When I looked closer, it had judged the stocks on a stricter yardstick than the index fund it compared them with. Its new caution was no better grounded than its earlier enthusiasm.
I stopped there and finished the work with Claude. We wrote a playbook: buy rules, price levels, “sell if” criteria, staged purchases and a decision log. I kept three of the five, and I haven’t bought anything yet. From now on, both AIs get checked against the playbook.
Warning signs you’re in the loop
- The AI says “we,” “our plan” or “we found something.”
- Words like “legitimately,” “compelling,” “asymmetric” or “no-brainer” show up more often as the chat goes on.
- The downside gets one line, or none.
- Nobody has compared the idea with the simple default: an index fund, your current job, your existing plan.
- You couldn’t explain why the numbers you were shown are the right ones to decide on.
- The AI reverses its view the moment you push back, with the same confidence as before.
The checks that break it
Treat the AI as a sparring partner and add friction where the loop would speed up. The first two did the most work for me. The rest are worth keeping as a checklist.
1. Get a fresh second opinion. A new chat, or a different AI, has none of the built-up enthusiasm. Paste in the plan without saying you love it.
Here’s a plan someone gave me. Find the weaknesses.
This is the same idea as the cross-model check in Don’t Let an AI Grade Its Own Homework, applied earlier, to a plan that hasn’t turned into work yet.
2. Write your rules before the next round. Your criteria can’t drift with the conversation if they are on paper first.
Before we go on, help me write the 3–5 rules that would make me say yes, and what would make me back out.
3. Ask what an expert would check. If you can’t judge the metrics, make the AI explain them.
What would a professional in this field check before deciding that we haven’t checked? Which of the numbers you’ve given me actually drive the outcome, and which are background?
For anything with real money or health at stake, a person who knows the field is a better check than any prompt.
4. Ask for the case against. This breaks the optimize-my-idea pattern.
Argue the strongest case that this is a bad idea. What would have to be true for it to fail?
5. Compare it with the boring default, on the same terms. Most ideas sound good until you set them next to doing nothing. The second half of the instruction matters, because the swing in my session included an uneven comparison.
Compare this with just keeping my money in an index fund. Use the same assumptions and the same yardstick for both, and tell me where the two differ.
6. Make it show its numbers. Fluent prose hides untested guesses.
List every number you used, where it came from, and which ones are your estimates.
7. Ask what changed. This catches reversals in either direction.
You changed your view. What new fact caused that?
8. Start small and reversible, then sleep on it.
What’s the smallest step that tests this before I commit fully?
Reread the plan the next day, cold.
The twist worth noticing
My careful process ended with a larger planned commitment than the plan I started with: more than three times larger. Due diligence that feels thorough can become a confidence booster of its own. That’s fine if the reasoning holds, and I think mine does, because the rules are written down and the weaker picks were cut. The risk is that a long, impressive checking process feels like permission, even when its conclusion is “go bigger.” If your own checks end with a bigger bet, run one more pass on the largest number in the plan.
Where this doesn’t help
- A second AI shares blind spots with the first, and makes its own mistakes. A second opinion finds gaps. It does not verify anything.
- The case-against prompt can overshoot. Ask for a strong argument against and you get one. Treat it as one input.
- Playbooks go stale. A rule written under this week’s rates needs a date on it and a review when conditions change.
- None of this replaces someone who knows the field. For money, health and legal questions, an AI check is a way to find the questions worth asking a professional.
Quick reference
| Habit | Why it works |
|---|---|
| Fresh second opinion | A new chat has none of the built-up enthusiasm |
| Write your rules first | Criteria can’t drift with the conversation |
| Ask what an expert would check | Tests whether the numbers you were shown are the right ones |
| Case against | Breaks the optimize-my-idea pattern |
| Boring default, same yardstick | Exposes an uneven comparison |
| Show the numbers | Separates sourced figures from guesses |
| Ask what changed | Separates a flip from new evidence |
| Start small, sleep on it | Caps the cost of being wrong |
Common questions
Why does ChatGPT keep telling me my idea is great?
It’s trained to be helpful and agreeable, and it tends to work inside the framing you give it. In a long chat each answer leans on the one before, so the enthusiasm compounds. A fresh chat, or a different AI, with a neutral prompt is the quickest way to get an independent read.
How do I stress test an idea with ChatGPT?
Bring something in from outside the conversation: a second chat or model, a comparison with the boring default on identical terms, a request to explain which numbers actually drive the outcome, and written rules for yes and no that you set before going further.
Why did ChatGPT flip its answer when I pushed back?
Many assistants lean toward your latest message. Ask what new fact caused the change. A flip with the same confidence as before and no new fact behind it is a warning sign.
Where to go next
For checking a finished piece of AI work, read Don’t Let an AI Grade Its Own Homework — Use a Second Model to Review.