One orchestrator, three brains: building a provider-agnostic movie agent

I built a conversational agent for movie discovery. You ask it something in plain English — "what's Dune 2 about, who's in it, and where can I stream it?" — and it figures out which TMDB API calls to make, makes them, and streams the answer back. No buttons, no dropdowns, no "search then click the result." Just chat.

That part is table stakes for an LLM agent in 2026. The part I actually wanted to solve was different: can the same orchestrator run on Claude, Gemini, or a local open model, swapped with one environment variable, and stay reliable when the cheaper model makes mistakes?

It can. Here's how, and here's where the clean abstraction leaks.

Try the live demo · Source on GitHub

(The demo is rate-limited to 5 questions per session — see the cost section below for why.)

What it does

CineLens is a FastAPI backend that streams over SSE. The tool layer is six TMDB endpoints wrapped in an MCP (Model Context Protocol) server — search, movie details, cast, recommendations, watch providers, and person lookups. Data models are Pydantic v2. The whole thing is async end to end.

A typical exchange looks like this:

You: what's Dune 2 about and who's in it? Agent: (calls search_movie → get_movie_details → get_cast) Dune: Part Two follows Paul Atreides as he unites with the Fremen... Starring Timothée Chalamet, Zendaya, Rebecca Ferguson... You: where can I stream it? Agent: (calls get_watch_providers) It's on Max in the US...

The user never picks a tool. The model does.

The agentic loop

The core is a standard tool-use loop in orchestrator.py, capped at ten rounds:

1. Append the user message to session history.
2. Loop (up to 10 rounds):
   a. Stream one model turn: system prompt + full history + tool schemas.
   b. The model emits interleaved text events (streamed to the client) and
      tool_use events (buffered, not shown).
   c. Append the assistant turn to history.
   d. No tool calls this turn? Done — we have the final answer.
   e. Otherwise: execute every requested tool, append results to history, loop.
3. If ten rounds pass with no plain-text answer, return a graceful fallback
   instead of looping forever.

The important thing here is what the orchestrator doesn't contain. There's no if user_intent == "streaming": call get_watch_providers. Zero routing logic. Tool selection is entirely the model's job, and it makes that decision from three inputs:

• the full conversation history,
• a system prompt describing the domain and the rules,
• the live list of tool names, descriptions, and JSON schemas, introspected at runtime from the MCP server — so the model always sees the real tools, never a hardcoded list that can drift out of sync with the code.

There's one more input that did more work than I expected: two few-shot examples baked into the message history from turn one. One walks through a normal detail lookup; the other handles an ambiguous multi-result search. They anchor the ordering of tool calls far more reliably than prose instructions did. Telling the model "always search first" in the system prompt is a suggestion it sometimes ignores. Showing it a transcript where searching happens first is an example it imitates. For anchoring behavior, one good example beat three paragraphs of rules.

The reliability lesson: enforce rules in code, not prompts

Here's the rule that mattered most: search_movie must run before any other tool, and a movie_id is only valid if search_movie set it earlier in the same session. Every other tool depends on a real movie ID. Let the model call get_cast with a hallucinated ID and you get garbage or an API error surfaced to the user.

I could have written that rule into the system prompt and hoped. I didn't, because LLMs are not reliable rule-followers — and the whole point of this project was to run models that follow rules less reliably than Claude. So the constraint lives in deterministic middleware:

def enforce_prerequisites(tool_name: str, state: MovieSessionState) -> None:
    if tool_name in _GATED_TOOLS:
        if not state.search_performed:
            raise PrerequisiteError("search_movie must run first")
        if state.movie_id is None:
            raise PrerequisiteError("no movie_id set for this session")

This is a synchronous, pure-function check that runs against session state before a gated tool ever touches TMDB. If the prerequisites aren't met, it raises instead of making the call.

The nice part is what happens next. The orchestrator catches PrerequisiteError and converts it into a friendly, natural-language tool result — "I need to search for a movie first. What would you like to look up?" — and feeds that back into the message history as if it were the tool's output. The model sees its own mistake reflected as tool output and self-corrects on the next turn. It's a feedback loop, not a crash.

The general principle I'd pull out of this: prompting is a hint, middleware is a guarantee. Anything genuinely non-negotiable — data integrity, ordering constraints, security boundaries — belongs in code the model can't route around. Save the prompt for shaping behavior you'd merely prefer. This distinction becomes the load-bearing decision later, when the model getting things wrong stops being hypothetical.

Session state

Each conversation gets an AgentSession holding:

• a MovieSessionState dataclass (movie_id, movie_title, search_performed, region),
• a fresh MCP server instance bound to that state,
• the running message history, seeded with the few-shot prefix.

History is trimmed to 20 messages to bound token growth, always preserving the few-shot examples at the front. Sessions live in memory keyed by session_id, with a DELETE /chat/{session_id} endpoint to reset. Nothing exotic — but binding the MCP server to the session's state is what makes the prerequisite check above able to reason about "did this conversation search yet."

The provider abstraction

This is the piece I'm most happy with, because it turned "switch the agent's entire brain" into a config change.

Everything routes through one abstract interface. The orchestrator talks to this and nothing else — it has no idea whether Claude or Gemini is on the other end:

class LLMProvider(ABC):
    async def stream_turn(self, system, messages, tools, max_tokens):
        """Yield normalized events: {"type": "text" | "tool_use", ...}"""

    def format_assistant_message(self, collected) -> dict:
        """Collected blocks -> this provider's history format."""

    def format_tool_results(self, results) -> dict:
        """Tool results -> this provider's history format."""

Each concrete provider owns three translation jobs:

1. Message format. Anthropic's tool_use / tool_result content blocks and Gemini's FunctionCall / FunctionResponse Part objects are genuinely different shapes. The provider converts the internal format to the vendor's wire format and back.
2. Streaming. Each vendor streams its own chunk protocol; the provider re-emits it as the two normalized event types the orchestrator understands (text, tool_use).
3. Partial tool-call buffering. Both vendors stream function-call arguments as fragments — Anthropic as input_json_delta, Gemini accumulated and parsed — so the provider buffers the partial JSON and only commits the tool call once the block closes.

The factory is the whole switch:

_PROVIDER_DEFAULTS = {
    "anthropic": "claude-haiku-4-5-20251001",
    "gemini": "gemini-3.1-flash-lite-preview",
}

def get_provider() -> LLMProvider:
    name = os.environ.get("PROVIDER", "anthropic").lower()
    model = os.environ.get("MODEL", _PROVIDER_DEFAULTS.get(name, ""))
    if name == "anthropic":
        return AnthropicProvider(model=model)
    if name == "gemini":
        return GeminiProvider(model=model)
    raise ValueError(f"unknown provider: {name}")

Changing the agent's brain is two env vars: PROVIDER=gemini, MODEL=gemini-3.1-flash-lite-preview. No code change, no logic redeploy.

The reason this was even possible is MCP. Tool definitions — name, description, JSON schema — live independent of whichever LLM consumes them. Swapping providers never means rewriting tools; only the adapter translates MCP's schema into that vendor's function-calling format. I followed the general shape of Anthropic's Agent SDK patterns (system-prompt-driven autonomous tool selection, structured tool_use/tool_result turns, streaming) but implemented directly against the async clients rather than the SDK, specifically so the same loop could re-target Gemini's very different native SDK without being boxed in.

Where the abstraction leaks

Clean abstractions promise you can convert every vendor to one common JSON shape and be done. Reality has protocol-level requirements that leak through, and this project has a perfect example.

Gemini's "thinking" traces come back with an opaque thought_signature that the API validates when you replay the history on the next turn. You have to carry that signature through the message history untouched — not the thought text, the signature blob — or follow-up turns get rejected. There's no equivalent concept on the Anthropic side. So "just normalize everything to a common format" quietly breaks: the common format has to be lossy enough to be shared but faithful enough to preserve a vendor's non-obvious replay requirements. The abstraction survives, but only because the Gemini provider knows to smuggle that signature through. Worth remembering the next time an interface looks suspiciously clean.

Gemini vs. a local model: "free" isn't "cheap"

This is a portfolio project. If strangers try it, my cost exposure needs to stay near zero — but it's a demo of engineering quality, not a cost experiment, so reliability still matters. That framed the model decision.

Why not self-hosted open models (Ollama + Qwen)? Zero API cost isn't zero cost; you trade API spend for hosting spend. The free tiers I'd actually deploy a portfolio project to don't have the RAM or VRAM to run even a 7B model. A real local deployment needs a VPS or GPU box — recurring cost plus ops burden. And tool-calling reliability drops meaningfully with smaller open models. In a system with a hard tool-ordering constraint, that means more middleware rejection/retry cycles, which the user sees as extra "let me try that again" turns. Local models earned a place as an optional mode for people who clone the repo (PROVIDER=ollama) — not as the backbone of a public-facing demo.

Why Gemini won for the public deploy. gemini-3.1-flash-lite-preview is priced so a few dozen visitors costs cents, not dollars. No hosting or ops burden — it's an API call with the same operational shape as the Anthropic path, so the provider abstraction covers it with zero new infrastructure. Reliability is close enough to Claude for a tool-calling agent that the middleware safety net rarely has to work hard. And it pairs with the rate limiting I'd already shipped (5 questions/day per session) as a second, independent cost lever: model choice caps per-request cost, rate limiting caps volume.

This is also where the middleware from earlier pays off. A cheaper model that violates the tool-ordering constraint more often is survivable specifically because the prerequisite check catches every violation deterministically and turns it into a self-correcting nudge. The reliability layer is what makes the cost decision safe.

The framing I'd leave you with: for a small public demo, the real constraint is total cost of ownership — hosting plus ops — not API price per token. A cheap hosted model behind a hard rate limit beats "free" local inference once you account for where a portfolio project actually gets deployed.

What's next

A few honest open items from my own security review, kept visible on purpose: CORS is still permissive, sessions have no auth, and in-memory session storage doesn't survive a restart or scale past one process. None of those matter for a single-instance portfolio demo; all of them would matter the moment this became real. Naming them is part of the engineering, not an afterthought to it.

The through-line, if there is one: let the model decide what it's good at deciding — which tool serves the user's intent — and put everything that must not go wrong into code that doesn't care how clever the model is. Then the choice of model becomes a dial you can turn for cost, not a rewrite.

---

CineLens is live at cinelens.gayathri.dev, and the full source — orchestrator, provider adapters, MCP tool layer, and the prerequisite middleware — is on GitHub. If you clone it, set PROVIDER=ollama to run it against a local model and watch the middleware earn its keep.