Prompting Techniques: Baseline Beat Every Chain-of-X Technique

Machine learning comparisons often focus on which model wins. This experiment asked a different question: do named prompting techniques actually improve output quality - or do they just sound like they should?

The Setup

The benchmark ran a planned 150-call grid: 3 models × 10 techniques (9 Chain-of-X variants + baseline) × 5 tasks. The models were llama-3.3-70b-versatile, llama-3.1-8b-instant, and gemma2-9b-it via Groq's hosted API. A third model, gemma2-9b-it, was supposed to run - it didn't produce a single usable result. Groq had decommissioned it between when the script was written and when it ran; all 50 of its calls returned the same model_decommissioned error. Combine that with 6 more calls lost to rate-limit errors on the judge step, and 94 of the planned 150 runs (62.7%) actually produced a scored result.

Every surviving response was scored by an LLM judge on correctness, clarity, and completeness (1–10 each, averaged into a quality_score), alongside the usual latency and token metrics. An honest benchmark reports its failure rate, not just its winners.

The Headline Result: Techniques Cost More and Deliver Less

Every technique scored below baseline. Averaged across all nine, quality came in at 9.14 versus baseline's 9.58, while latency more than doubled - 3.64s versus 1.61s, a 2.27× tax for a net quality loss. You can pay more and get less; nine techniques out of nine did exactly that.

This isn't just an artifact of one lucky task. Baseline was the top scorer on logic_puzzle (10.0) and math_compound_interest (10.0), and stayed above 9.0 on both explanation tasks and the summarization task. There was no task category where a Chain-of-X technique reliably beat plain-old asking-the-model-directly.

The Failure Mode: When the Scaffold Doesn't Fit the Task

Averages hide the worst part of the story. Two techniques didn't just underperform on the categorical logic puzzle - they collapsed.

llama-3.3-70b-versatile under Chain of Density scored a 2.0 on the logic puzzle (correctness 2.0, clarity 1.0), even though the same model scored 8.5–9.8 on every other task under that same technique. The judge's note: *"overly convoluted and ultimately incorrect."* The model spirals into a run-on sentence that contradicts itself in real time - concluding "Bob likes red" is correct, then incorrect, then correct again, without ever committing to an answer.

llama-3.1-8b-instant under Chain of Numerical Reasoning did almost exactly the same thing on the same task: a 2.0 quality score, with the judge flagging that the response *"concludes that both Bob and Carol like red,"* directly violating the puzzle's own constraint. Meanwhile that same model scored a perfect 10.0 on the math task under the same technique.

Both failures happened on the same task, under two different techniques, on two different models. That's not noise - it's a pattern: a scaffold built for one kind of reasoning actively hurts a task that needs a different kind. The reasoning task category's quality scores had a standard deviation of 2.47 - by far the highest of any category - almost entirely driven by these two collapses.

The Speed Trap: Chain of Density Looks Efficient Until You Read the Output

Chain of Density is the one technique that actually beat baseline on latency (1.58s vs. 1.61s) and produced far fewer output tokens (162 vs. baseline's 405 on average). On paper, that's a compression technique doing exactly what it promises.

But its average quality was the worst of all ten conditions tested (8.17). Outside the logic-puzzle catastrophe, its failures were quieter: llama-3.1-8b-instant's Chain-of-Density summary of WWI causes scored a 7 on completeness - it compressed away information the judge considered load-bearing, not just filler.

Why This Happens: The Scaffold Is a Prior, and Priors Can Be Wrong

A "Chain of X" system prompt is, functionally, a strong prior about how to solve the problem - think step-by-step, compress to essentials, translate to symbols, reason numerically. When that prior matches the task, it's neutral at worst. When it doesn't, the model doesn't ignore the instruction and reason normally - it tries to force the task into the scaffold's shape anyway.

A categorical logic puzzle has no numbers to compress and no density to extract. Instead of falling back to plain deduction, both models produced long, self-contradicting chains - text that looks like reasoning in progress but never resolves. A model given the wrong lens doesn't refuse to look through it - it just reports back what it thinks it's supposed to see.

Baseline has no such prior. It's free to reach for whatever reasoning shape actually fits - plain step-by-step logic for the puzzle, plain arithmetic for the interest calculation - which is exactly why it topped both of those task types.

Key Findings

• Baseline beat every technique, on average and on latency. 9.58 avg quality at 1.61s, versus 9.14 average and 3.64s across the nine Chain-of-X techniques - a 2.27× latency tax for a net quality loss.
• Technique-task mismatch produces catastrophic, not gradual, failure. Chain of Density (70B model) and Chain of Numerical Reasoning (8B model) both scored 2.0/10 on the same categorical logic puzzle.
• The same technique doesn't fail the same way across models. Chain of Numerical Reasoning scored 9.48 on llama-3.3-70b-versatile overall but only 7.70 on llama-3.1-8b-instant.
• Compression trades completeness for speed, quietly. Chain of Density hit the lowest average quality (8.17) despite the best latency.
• More output tokens didn't buy more quality. Correlation between output_tokens and quality_score across all 94 scored runs was -0.07 - essentially zero.
• A third of the benchmark died before it started. gemma2-9b-it was decommissioned by the provider mid-experiment - a reminder that hosted-API benchmarks are benchmarking a moving target.

The Real Takeaway

If you're building a production system and reaching for a named prompting technique because it sounds rigorous, ask what it's actually costing you. In this benchmark it cost 2.27× the latency for a quality score that was lower, not higher, than just asking the question directly - and on one task, it produced a confidently wrong, self-contradicting answer at more than double the latency.

The technique isn't free insurance against a bad answer - it's a bet that your task matches the technique's assumptions, and that bet doesn't always pay off. Before you add scaffolding to a system prompt, benchmark it against doing nothing. Sometimes doing nothing wins.

Which "Chain of X" technique have you tried in production, and did you ever benchmark it against a plain baseline - or did it just sound like it should work?