The Same Model, Before and After: What Post-Training Actually Changes

Post 3 of 7. Post 2 showed a base model looping, rambling, and drifting dark, and claimed none of it was malfunction. This post tests the claim that assistant behavior is installed, not emergent, using the cleanest comparison I could construct for free.

---

Comparing GPT-2 to a modern chat model proves nothing. They differ in architecture, training data, scale, and training procedure, four confounded variables. Any behavioral difference could be attributed to any of them, which means it can't be attributed to anything.

What you want is a controlled pair, and the open-model ecosystem happens to provide one: Qwen2.5-0.5B and Qwen2.5-0.5B-Instruct. Same architecture. Same parameter count. Same pre-training lineage. The instruct version is the base version plus post-training, supervised fine-tuning and preference optimization. Any behavioral difference between them isolates the one variable that differs. This is as close to a controlled experiment as you can run without training anything yourself, and both models are small enough to run on a laptop CPU.

Before the results, the two mechanisms being tested.

What SFT and RLHF actually are

Supervised fine-tuning (SFT) is continued training on a curated dataset of (instruction, ideal response) pairs. The technical detail that surprised me: the objective doesn't change. It's still next-token prediction, the same loss function as pre-training. What changes is the data distribution, instead of predicting the next token of arbitrary web text, the model now predicts the next token of consistently-formatted exchanges where an instruction is always followed by a helpful answer that then stops. Do that enough and "answer, then stop" becomes the high-probability continuation of any question. The behavior is installed through the same mechanism that installed everything else.

RLHF (reinforcement learning from human feedback) refines what SFT roughs in. Humans rank multiple model outputs for the same prompt; a separate reward model is trained to predict those rankings; the language model is then optimized, typically with PPO, to produce outputs the reward model scores highly. Where SFT teaches by imitation ("write like these examples"), RLHF teaches by preference ("of the things you might write, this kind is better"), which can push toward qualities that are hard to demonstrate but easy to rank: appropriate hedging, better formatting, refusing bad requests.

The detail nobody tells beginners: chat templates

Before any comparison can be fair, there's a trap. Instruction-tuned models were trained on conversations wrapped in special structural tokens, Qwen's look like this:

<|im_start|>user
How do I boil an egg?<|im_end|>
<|im_start|>assistant

Feed an instruct model your prompt raw, without this wrapper, and you're handing it input unlike anything in its fine-tuning distribution. It often degrades toward base-model rambling, and you'd wrongly conclude post-training did nothing. The playground handles this with the tokenizer's built-in template method:

messages = [{"role": "user", "content": prompt}]
formatted = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

Base models get raw text; chat models get the template. Every comparison below respects this, and the is_chat flag in the playground's model registry exists for exactly this reason. If you've ever seen someone conclude a chat model is bad after prompting it raw, this is usually why.

Experiment 1: The question

Prompt: "How do I boil an egg?", the same prompt from Post 2's opening.

GPT-2 (base) produced fluent, confident, food-adjacent chemistry fiction:

We use a cooking oil that has an enzyme called lactose in it. Lactose is a fatty acid that is extracted from plants. It is the building block of proteins and carbohydrates that make up the human body. Lactose is converted into a sugar called sucrose...

Every clause is false. Lactose is a milk sugar, not an enzyme, not a fatty acid, not extracted from plants. But notice the form of the failure: grammatical, declarative, textbook-cadenced. Pre-training optimizes for plausible continuation, and false statements delivered in encyclopedia register are extremely plausible continuations. Truth was never in the objective.

Qwen2.5-0.5B-Instruct produced this:

Boiling an egg is a simple and effective way to cook it. Here's how you can do it:

1. Prepare the Egg: Use fresh eggs for best results.
2. Place the Egg in a Bowl: Place the boiled egg in a bowl of cold water or ice water. This will help the yolk escape from the egg and make it easier to handle.

Structured, on-task, formatted, direct. The gap in form between these two outputs is the entire visible effect of post-training, and it's enormous.

Now read step 2 again. It instructs you to place "the boiled egg" in cold water, before any boiling has occurred. The answer refers to the result of a step it never performed, and the "yolk escape" claim is nonsense besides. The formatting is immaculate; the procedure is broken.

I want to state the finding carefully, because it became the seed of the entire next post: post-training improved form dramatically and correctness only somewhat. The instruct model is genuinely better, it addressed the question, structured an answer, adopted a helpful register. It is not therefore right. Fluency and correctness are separate axes, post-training targets the first directly and the second only obliquely, and a well-formatted answer creates a halo of competence that the content doesn't always earn. Humans fall for that halo. So, it turns out, do smaller AI judges.

Experiment 2: The prompt that wasn't a question

The comparisons above use an instruction, which is the instruct model's home turf. What happens on a prompt that isn't one? I fed both models a creative fragment: "When the sun shone and the birds were flying".

The base model did what base models do, continued the scene. It spun narrative, invented dialogue, stayed in the fiction. Unpredictable, ungrounded, but formally appropriate: a story fragment got story continuation.

The instruct model replied:

As an AI language model, I don't have feelings or emotions like humans do, but I can tell you about the sounds of nature when the sun shines and the birds fly...

Nobody asked it anything. There was no question in the prompt, no request, no second person. But post-training has burned "I am an assistant responding to a user" so deeply into the model that it reframed a story fragment as a query requiring a disclaimer about its own nature before answering.

This is sometimes called over-alignment, and it's the honest counterweight to Experiment 1: post-training is not free. The base model was a flexible text-continuer with no goals; the instruct model has goals, and they intrude even where they don't belong. You trade raw flexibility for reliability-of-persona, and at 0.5B parameters the trade is visibly crude. (Larger models manage the same trade more gracefully, but the trade is still there.)

Experiment 3: The dark prompt, revisited

Post 2 ended with GPT-2 continuing "Once upon a time" into looping violence. The same prompt through the instruct models produces harmless storytelling or gentle deflection, same prompt, opposite character of output. Combined with the persona intrusion above, the pattern is consistent: post-training changes what the model is trying to do, not what it knows. The base model isn't more ignorant than its instruct sibling; it's un-aimed.

What "conversation" actually is

One more mechanism belongs in this post, because only post-trained models can hold a conversation and I'd never questioned what a conversation is.

Language models are stateless. Between API calls, nothing persists, no memory, no session, no accumulated understanding. When a chatbot "remembers" what you said three turns ago, the application re-sent the entire conversation history with your newest message, every single turn:

full_messages = [{"role": "system", "content": system_prompt}]
full_messages += conversation_so_far          # every prior turn, resent
formatted = tokenizer.apply_chat_template(full_messages, ...)

The memory is an illusion maintained by the client. Three consequences fall straight out of the mechanism: each turn is more expensive than the last (the input grows); conversations have a hard ceiling (the model's context window); and someone has to decide what to drop when you hit it. The playground's chat tab surfaces the input token count on every turn so you can watch the cost of the illusion accumulate.

Testing this taught me a small lesson in experimental hygiene. My first memory test, tell the model my favorite color, then ask what it is, passed, but for the wrong reason: the color was still sitting in the first user message of the resent history, so the model could answer by re-reading my words rather than by using its own prior turn. A test that can pass without the mechanism under test isn't a test. The fixed version had the model compute something (4 + 9 + 15), then asked it to recall its own answer, information that exists only in the assistant turn. It answered 28. That's context threading actually demonstrated, and the flawed first version is exactly the kind of thing I'd have never noticed if I weren't writing it up.

The scoreboard

One line, if you keep nothing else: pre-training gives a model everything it knows; post-training gives it something to want. The wanting is what makes it useful, and the wanting is also what makes its failures, over-refusal, persona intrusion, confident formatting around wrong content, distinctly its own.

Try it yourself: In the live playground, the Model Zoo tab runs any prompt across the base/instruct pair side by side, try an instruction, then a story fragment, then something ambiguous. The Clubhouse tab is the chat interface; run the arithmetic-recall test yourself and watch the token counter grow.

Caveats: A 0.5B instruct model is a weak specimen of post-training, frontier RLHF produces subtler behavior than anything shown here, and some findings (the persona intrusion especially) rest on a handful of prompts. One model family. The mechanisms are general; the specific outputs are anecdotes, chosen because they're reproducible and legible, not because they're representative in a statistical sense.