Building an Instrument, Not a Model: Why My First ML Project Was a Playground

Post 1 of 7 in a series on learning LLMs by building an observation instrument for them. Code is public, and there's a live version you can try, links at the end.

---

The standard advice for learning machine learning hands-on is to train something. A tiny GPT on Shakespeare, a digit classifier, a sentiment model. I started down that road and stopped, because I noticed the advice optimizes for the wrong thing. Training a toy model teaches you the plumbing, tensors, loss curves, optimizer flags. It teaches you surprisingly little about why a language model behaves the way it does, because a toy model barely behaves at all.

So I inverted the project. Instead of training a model, I built the instrument that observes trained ones: a playground where I could feed real models real inputs and watch, measurably, repeatably, what they do. Tokenize text and see the pieces. Generate with different sampling settings and see the output change character. Put a base model next to its instruction-tuned sibling and see what post-training actually bought. Score outputs with metrics and with other models acting as judges, and then check whether the judges themselves can be trusted.

This series is a write-up of what that instrument taught me. The claim I'll defend across five posts: you learn the concepts of the LLM pipeline faster by probing real models than by training fake ones, because real models fail in ways that make the concepts legible.

What an LLM is, in three sentences

A large language model is a function that takes a sequence of tokens and outputs a probability distribution over what token comes next. Text generation is just calling that function repeatedly, predict, pick, append, repeat. Everything else you associate with these systems, answering questions, holding conversations, refusing requests, is either emergent from that loop or deliberately installed afterward by a second phase of training.

That third sentence is the load-bearing one, and most of this series is about testing it. The lifecycle splits in two:

A base model given "How do I boil an egg?" has no reason to answer you. It has only ever been trained to continue text, so it might continue with more questions, a forum-style digression, or something worse. The fact that ChatGPT-style systems answer is not a property of language modeling, it's a property of post-training. Post 3 demonstrates this with a controlled comparison; for now, hold it as the hypothesis.

The zero-cost constraint was a methodology, not a budget

I set one rule at the start: everything runs locally, on CPU, for free. Hugging Face transformers, small open models, no API bills during development.

I chose this to remove excuses, but it turned out to be the best pedagogical decision in the project. The constraint forced me onto small models, GPT-2 at 124M parameters, Qwen2.5-0.5B, TinyLlama-1.1B, and small models fail visibly. GPT-2 will loop the same sentence at you. It will invent chemistry. A 0.5B judge model will grade nonsense as a 4/5 and fabricate a justification. Frontier models make the same category of errors, but rarely and subtly, hidden behind fluency. Small models put the seams of the technology where you can see them, and the seams are where the concepts live.

There's a corollary I didn't anticipate: when a small model fails at something and a slightly larger or better-trained one succeeds, the difference isolates what the extra capability or training is actually for. Cheap models make good controls.

The instrument itself

The architecture is deliberately boring: a Python backend that owns the models, a React frontend that owns the interface, HTTP between them.

React frontend  <--HTTP/JSON-->  FastAPI backend  <-->  transformers models (CPU)

The backend is the lab bench. It holds a model registry, a dictionary mapping short names to Hugging Face model IDs, with a flag for whether each model is chat-tuned, and lazy-loads each model into memory the first time it's requested:

MODEL_REGISTRY = {
    "gpt2":                  {"hf_id": "gpt2", "is_chat": False},
    "qwen2.5-0.5b-base":     {"hf_id": "Qwen/Qwen2.5-0.5B", "is_chat": False},
    "qwen2.5-0.5b-instruct": {"hf_id": "Qwen/Qwen2.5-0.5B-Instruct", "is_chat": True},
    "tinyllama-1.1b-chat":   {"hf_id": "TinyLlama/TinyLlama-1.1B-Chat-v1.0", "is_chat": True},
}

_loaded_models = {}

def get_model(model_key):
    if model_key not in _loaded_models:
        info = MODEL_REGISTRY[model_key]
        tok = AutoTokenizer.from_pretrained(info["hf_id"])
        mdl = AutoModelForCausalLM.from_pretrained(info["hf_id"])
        mdl.eval()
        _loaded_models[model_key] = (tok, mdl)
    return _loaded_models[model_key]

Two details matter here. Models load once and stay resident, loading a model per-request would make every call take minutes. And the is_chat flag isn't cosmetic: chat-tuned models expect their input wrapped in a specific conversation format, and feeding them raw text degrades them. That detail gets a full section in Post 3, because nobody tells beginners about it and it silently ruins comparisons.

Each endpoint on the backend maps to a concept:

The frontend wraps these in a deliberately playful skin, the tabs are named Sandbox, Swing Set, Merry-Go-Round, Model Zoo, Report Card, Judge's Bench, Clubhouse, because I was going to spend weeks in this tool and dashboards you enjoy get used more than dashboards you don't. Every experiment in this series is reproducible through those tabs; you don't need to touch the code to replay any finding.

Where the series goes

Post 2 starts at the bottom of the stack: tokenization and decoding. It opens with the first output I ever got from GPT-2, a sentence repeating itself into oblivion, and explains, mechanically, why greedy decoding produces loops, what temperature actually does to the probability distribution, and why the same frozen weights can produce text with completely different character. It also shows what happens when you tokenize Hindi with a tokenizer trained on English, which is worse than you think.

Post 3 is the controlled experiment at the heart of the series: Qwen2.5-0.5B base versus Qwen2.5-0.5B-Instruct. Same architecture, same lineage, same size, the only variable is post-training. The differences are dramatic, and so are the things that don't change. It also covers what "conversation" actually is, since only post-trained models can hold one.

Post 4 is the one I'd keep if I could only keep one. I built traditional metrics, found their blind spots, then ran a four-case study using three LLMs of different sizes as judges, and found that my smallest judge gave nearly identical scores to a good answer, a fabricated answer, and a completely off-topic answer, inventing justifications for each. Does a bigger judge grade more accurately? The answer turned out to be: not monotonically, and the rubric matters as much as the judge.

Post 5 covers the distance between "runs on my laptop" and "deployed for you to use", containerization surprises, a model whose name understates its size by 10x, and what the whole thing actually costs to serve. It exists because the live demo linked below exists, and getting there was its own education.

Caveats, stated once and early

This is a learning instrument, not a benchmark suite. The models are small and the experiments are qualitative, designed to make mechanisms visible, not to estimate effect sizes. Where I make claims, I show the outputs they rest on, and where the evidence is thin (it's often n = one prompt), I say so. The value here is that every observation is reproducible in about thirty seconds, by you, in the live playground.

Try it yourself: live playground · repo. Start with the Sandbox tab, type your own name and look at what the tokenizer does to it. That's the subject of the next post.