Why Doesn't Model Size Predict When an AI's Writing Falls Apart?

It's a comfortable assumption: the bigger the model, the longer it holds together. Throw more parameters at a language model and it should be able to write further before it starts repeating itself, drifting off-topic, or forgetting who its characters are. That assumption is why "just use a bigger model" is the default fix whenever a small open-weight LLM starts producing garbage a few paragraphs in.

We tested that assumption directly. We took four open-weight models spanning a 5x parameter range — 82M to 410M — gave them identical story prompts and an identical generation budget, and measured exactly how their coherence behaved as the text got longer. The model that collapsed hardest wasn't the smallest one. The model with the single sharpest coherence cliff in the entire dataset was the largest one we tested. And the two models closest in size to each other told almost opposite stories.

Parameter count, in our results, was not the variable that mattered. Distillation was.

1. The Experiment

Four models, three of them base (non-instruction-tuned) causal language models, continuing the same five story openings — one each in fantasy, mystery, sci-fi, memoir, and literary fiction:

Each model continued each of the five prompts for an identical budget of 800 new tokens — no model was allowed to write less than another, so nothing below can be explained by generation length alone. Decoding was deliberately vanilla: nucleus sampling (top-p 0.9, temperature 0.8), with repetition_penalty and no_repeat_ngram_size both left off on purpose. Those settings exist specifically to suppress the kind of repetition we were trying to measure — turning them on would have hidden the exact failure mode this experiment is about.

The generated text was split into sequential ~150-word windows, and five automated metrics were computed at each window. Plus a TF-IDF similarity-to-opening score (topic drift), a named-entity recurrence rate (does the model keep its established cast, or invent a new one every window?), and average sentence length. All five were validated against synthetic text with known coherent/repetitive/drifting properties before being trusted on real model output — worth mentioning because one early version of the entity-recurrence metric had a real bug (it double-counted old characters as "recurring" even after the story had moved on), and catching that before running the real experiment changed what the metric could actually detect.

One honest caveat up front: the original design called for six models across three size tiers, including two larger instruction-tuned models (TinyLlama-1.1B, Qwen2.5-1.5B-Instruct). Both failed to generate on every single prompt due to a chat-template compatibility issue that's still being run down. Everything below is the four-model, two-tier comparison that actually completed — which, as it turns out, already complicates the story enough on its own.

2. The Numbers, Model by Model

Here's the topline: a composite coherence index (blending all five metrics, 0 to 1, higher = better) tracked across the full length of each model's generations.

That chart alone should make you distrust "bigger = more stable." Four very different shapes are hiding in there.

The One That Held Steady: GPT-2 (124M)

GPT-2 opened at a coherence index of 0.668 at 150 words and closed at 0.614 at 650 words — a loss of about 8%, and the flattest decay rate of any model tested at -0.081 points per 100 words. It even recovered mid-generation, dipping to 0.552 at 450 words before climbing back to 0.614 by the end. For the smallest non-distilled model in the set, this is not what the naive story predicts.

The One That Imploded: DistilGPT-2 (82M)

This is the outlier that makes the whole post worth writing. DistilGPT-2 started at 0.620 — respectable, in line with the others — and by its final window at 750 words had collapsed to 0.062. That's a 90% loss, roughly an order of magnitude worse than any other model's endpoint. The repetition chart makes the mechanism visible:

By word 543, DistilGPT-2's 4-gram repetition rate hit 0.978 — essentially every four-word sequence in that window was a repeat of one already seen. Its average sentence length, meanwhile, ballooned to 150 words in its final window — a single run-on stretch with almost no terminal punctuation. That's not "the writing got worse." That's the model stuck in a loop it never escaped.

The One With the Sharpest Cliff: Pythia-410M (410M)

Here's where "bigger is more stable" really falls apart. Pythia-410M produced the single sharpest coherence collapse of the entire experiment: between word 532 and word 541 — a nine-word window — its coherence index dropped 0.163 points, a rate of -1.814 per 100 words, more than twice as steep as DistilGPT-2's worst moment (-0.774/100 words) and over twenty times steeper than GPT-2's (-0.081/100 words).

And yet Pythia-410M finished the experiment with a coherence index of 0.689 — the highest final score of any model, and a net loss from its starting value of just 1.8%, the best retention in the dataset. The largest model we tested had the worst instantaneous crash and the best overall outcome. Both things are true at once.

The Best Start, the Steepest Ordinary Decline: GPT-2 Medium (355M)

GPT-2 Medium opened highest of all four (0.729) and actually peaked at 300 words (0.806) before settling into a more ordinary decline, closing at 0.633 — a 13.1% net loss, the second-worst retention in the set, worse than the 124M-parameter GPT-2 it's three times the size of. Bigger, in this one comparison, meant a larger absolute drop.

3. Why Does This Happen?

Two mechanisms are doing the work here, and neither of them is "parameter count."

First: distillation trades away exactly the slack you need for long-form writing. DistilGPT-2 was built by compressing GPT-2's knowledge into roughly two-thirds the layers. That's a great trade for short completions, where DistilGPT-2 performs nearly as well as its parent at a fraction of the cost. But long-form generation is a repeated-passes problem — every new token depends on the model correctly weighing everything that came before, over and over, for hundreds of steps. Distilling a model is a lot like re-saving a JPEG at a lower quality setting: at a glance, indistinguishable from the original. It's only when you stretch it — zoom in, or in this case, keep generating for another five hundred words — that the compression artifacts nobody noticed at first come to dominate the whole picture.

Second: our entity-recurrence metric only means what you think it means if you check it against repetition first. Look at DistilGPT-2 and GPT-2 at the 300-word mark: both show a perfect entity recurrence rate of 1.0, which sounds like healthy narrative continuity — the model is sticking with its established characters. But DistilGPT-2's 4-gram repetition rate at that same checkpoint was already 0.491, and GPT-2's was 0.402. Compare that to Pythia-410M and GPT-2 Medium at the same checkpoint, both under 0.05 repetition. A model that's stuck looping the same handful of phrases will trivially score perfect "recurrence" on its own character names — not because it remembered them, but because it never stopped saying them.

A model can recur the same three character names for six hundred words and still be falling apart — recurrence only means continuity if the words around it aren't also repeating.

The lexical-diversity chart tells the same story from a different angle:

DistilGPT-2's distinct-2 score is the only one that spends real time near zero (0.022–0.047 across several windows) rather than merely dipping.

4. Key Findings

• Distillation, not raw scale, was the dominant risk factor. DistilGPT-2 (82M) lost 90% of its starting coherence; GPT-2 (124M) — barely larger, and not distilled — lost only 8%. Size explains almost none of that 82-point gap.
• The largest model had the sharpest single collapse, and the best overall outcome. Pythia-410M's -1.814/100-word crash between words 532–541 was the steepest in the dataset, over 20x GPT-2's rate — yet it ended with the highest final coherence score (0.689) and the smallest net loss (1.9%) of any model. Instantaneous volatility and long-run stability are not the same measurement, and a model can score badly on one while scoring best on the other.
• Bigger sometimes meant a bigger fall, not a smaller one. GPT-2 Medium (355M) started highest (0.729) but retained less of its coherence (86.9%) than the plain, three-times-smaller GPT-2 (91.9%).
• Entity recurrence needs a repetition check before you trust it. At the 300-word checkpoint, DistilGPT-2 and GPT-2 both showed "perfect" 1.0 entity recurrence rates alongside repetition rates of 0.49 and 0.40 respectively — a signature of parroting, not narrative continuity, that foreshadowed DistilGPT-2's later collapse.
• Data density thins out fast past the first few checkpoints. The heatmap on the right shows exactly where each model's generation actually reached — DistilGPT-2's data extends furthest (to 750 words) precisely because its repetition loops are cheap, low-entropy text that pads out word count without saying anything new.

5. The Real Takeaway

If you're picking an open-weight model for a long-form writing feature — a story generator, a long chat assistant, an autonomous report-writer — "count the parameters" is not a reliable proxy for "will this hold together." In our tests, the model most likely to fail catastrophically wasn't the smallest model in absolute terms; it was the one built by compression. Long-form coherence isn't a parameter-count story — it's a compression story.

That has a concrete, practical consequence: if cost pressure is pushing you toward a distilled model to save on inference, that trade-off may cost you disproportionately more at the exact use case — long, uninterrupted generation — where you can least afford a silent collapse into a repetition loop. A support chatbot that degenerates after 700 words of conversation, or a story-generation feature that quietly starts looping the same sentence, is a much worse failure than a slightly slower response — and standard short-completion benchmarks, which is most of what distilled models get evaluated on, will never catch it.

The large-tier comparison — whether a 1B+ instruction-tuned model changes any of this — is still an open question here; that part of the experiment didn't run. Given how much distillation mattered more than scale in these four, I wouldn't bet on size alone answering it either.

Which of your own model-selection decisions were actually made on parameter count when they should have been made on how the model was trained?