Chain of Thought Prompting: When It Helps, When Not

Chain of thought prompting is the technique of asking a model to reason through the steps before it answers. It came out of research in 2022 showing that models given room to work through intermediate steps solved multi-step problems far more reliably than models asked for the answer directly.

Four years on, the landscape has changed: frontier models now reason internally by default. That has not made the technique useless. It has changed what it is for. Today the value is not triggering reasoning — it is directing it, and being able to audit it.

What actually happens#

When a model answers immediately, it commits to a direction in its first few words and then justifies it. When it works through steps, each step conditions the next, and errors surface where you can see them.

For a business user the difference shows up on tasks like these:

• Comparing three options against five criteria.
• Working out a budget allocation with constraints.
• Diagnosing why a campaign underperformed.
• Deciding whether a claim is supported by the data you provided.

On all four, a direct answer sounds equally confident and is more often wrong.

The 2026 version of the technique#

"Let's think step by step" was the phrase that made the technique famous. On current models it is close to a no-op. What still produces a measurable improvement is specifying your steps.

Weak:

Think step by step, then tell me which channel to cut.

Strong:

Work through this in order, showing your work at each stage:
1. List each channel with spend, leads and cost per lead.
2. Flag any channel where the sample is under 30 leads as
   statistically unreliable.
3. Rank the reliable channels by cost per qualified lead.
4. Note any channel with a strategic reason to keep despite
   poor efficiency.
5. Only then recommend what to cut, with the number that
   justifies it.

If data for a step is missing, say so instead of estimating.

The second version is not a trick. It is your analysis process, written down. That is the whole skill, and it is why prompt engineering is really specification.

When chain of thought earns its cost#

Use it when all of these are true:

• The task has multiple dependent steps.
• A wrong answer is expensive — a recommendation, a calculation, a claim you will publish.
• You want to audit the logic, not just the conclusion.
• Latency is not critical.

Typical wins: financial modelling, root-cause analysis, evaluating tenders, legal-adjacent reading, any comparison with more than three criteria.

When to skip it#

• Lookups and retrieval. "What is our refund policy?" needs no deliberation.
• Short copy. Reasoning about a headline usually produces a worse headline and a paragraph of explanation.
• Classification with good examples. Few-shot prompting is faster and more accurate here.
• High-volume automation. Reasoning multiplies tokens and latency across thousands of runs; use it only on the step that needs it.
• Anything real-time. A customer waiting on a chat widget will not wait for deliberation.

A practical rule I give clients: reason on decisions, not on drafts.

Structuring reasoning so it is useful#

Three patterns that work in real deliverables.

Separate the thinking from the answer. Ask for the reasoning in one section and the conclusion in another, so you can paste the conclusion without editing out the working.

Ask for the assumptions explicitly. "List every assumption you made before concluding." This catches the invented input that would have wrecked the recommendation.

Force a counter-argument. "Then argue the strongest case against your own recommendation." This single line has saved more client decisions than any other prompt I use.

Chain of thought in agents#

The technique matters more, not less, once you move to agents. An agent that plans before acting is the same idea applied across tool calls: state the plan, execute step by step, check the result of each step before continuing.

That is exactly why a written plan is the first thing a good terminal agent produces, and why catching a wrong plan at step one is the cheapest intervention available. See how to build an AI agent and free agentic AI courses for how this scales.

Cost, honestly#

Reasoning costs tokens and time. On a single important decision that is irrelevant. On an automated pipeline running 5,000 times a day it is your entire budget.

The pattern that works: a cheap fast model for routing, extraction and classification, and a reasoning model for the one step where judgement is required. Most teams over-apply reasoning to everything, then complain about the bill. See Claude Code pricing for how that plays out in agent work.

A worked example from a real client question#

The question was whether to keep a trade show that cost 18,000 EUR a year. The direct answer was "yes, trade shows build relationships": true, generic, useless.

The structured version asked for five steps. List every lead attributed to the show over two years. Mark which ones closed. Calculate cost per closed deal against the same figure for paid search. Note what the show provides that a channel comparison cannot capture. Only then recommend, stating assumptions.

The working revealed the actual problem at step one: nobody had attributed the leads, so 60% of them were untracked. The recommendation that came out was neither keep nor cut. It was "fix attribution this year, decide next year." That answer only exists because the reasoning was visible before the conclusion.

That is the business case for chain of thought in one sentence: it surfaces the missing input before you make a decision on top of it.

Mistakes to avoid#

• Treating a magic phrase as the technique. Specify your steps instead.
• Reading only the conclusion. If you are not going to check the reasoning, you did not need the reasoning.
• Trusting confident arithmetic. Verify numbers independently; a plausible calculation is not a correct one.
• Letting reasoning leak into client deliverables. Separate the sections, then send the answer.
• Using it as a substitute for data. Reasoning over missing information produces well-structured guesses. Supply the data.

The short version#

Chain of thought is no longer a trick you add to prompts. It is the habit of writing your own analysis process into the instruction, asking for assumptions and counter-arguments, and reading the working before you accept the answer.

That habit is lesson four of my free prompt engineering course, and Anthropic's prompt engineering documentation covers the model-side detail. If you want your team applying it to real decisions with someone checking the work, that's what my AI training packages are for.