Switching LLM providers mid-conversation in LangChain: what breaks, and why

Why a running thread is different

A LangGraph thread is saved as a list of messages, and each message carries content in the shape the provider that wrote it produced. When the next turn runs, the whole list is sent to the current model. Switch provider mid-thread and the new model receives the old provider's messages on every turn from then on - not once, every time, because the checkpoint replays them. If the new provider cannot read even one of them, the thread is not failing intermittently. It is bricked.

Problem 1: content blocks the new provider rejects

Modern chat messages are not plain strings; their content is a list of typed blocks. Anthropic's models write thinking blocks for extended reasoning and tool_use blocks for tool calls, and accept images as {"type": "image", "source": {...}}. OpenAI's reasoning models produce reasoning items. Other providers use other shapes.

Each provider's integration converts LangChain messages into its own wire format, and some are strict. ChatOllama's converter accepts a handful of block types - text, tool use, image URLs and LangChain's standard data blocks - and raises ValueError: Unsupported message content type on anything else. So a thread that started on Claude with extended thinking, then moved to a local Ollama model, fails on every message from the switch onwards. The same happens with an image a user sent earlier, if your application stored it in Anthropic's format.

The fix is to sanitise at the conversion point rather than rewriting history. Subclass the chat model and override the one method that converts messages for the wire, so all of its entry points - invoke, stream and their async forms - pass through it. In it: drop reasoning blocks (no provider accepts another's reasoning back), translate images into the shape this provider understands, degrade anything else unknown to text, and pass supported blocks through untouched so a normal turn costs nothing extra. The checkpoint keeps the original messages; only what goes on the wire changes, so switching back loses nothing. Olano builds every Ollama model through such a subclass for exactly this reason.

Problem 2: a model with no known context window

Each LangChain chat model can carry a profile, a dictionary of capabilities that includes max_input_tokens. The profiles come from a table shipped inside each provider's package, so a model released after the package was published has none. Usually that only means a capability check is skipped. In Deep Agents it changes how conversations are summarised.

Deep Agents' summarisation sizes itself from the profile. With a known window it summarises when the conversation reaches 85% of the window and keeps the most recent 10%. Without one it falls back to fixed values: summarise at 170,000 tokens and keep the last 6 messages. Six messages is a few tool calls. The visible symptom is an agent on a brand-new model that forgets the start of a short session - nothing errors, it simply loses the thread.

Fix it at model construction: if the model's profile has no max_input_tokens, set it from a source you trust - your own model catalogue, or the provider's documentation. Better still, configure summarisation explicitly in tokens or fractions of the window so a missing profile cannot change behaviour silently. Olano attaches the window from its synced model catalogue whenever the package lacks one, and resolves summarisation from a per-deployment and per-agent policy, never from the message-count fallback.

The dangerous case is a setting that is accepted and ignored: an output cap set as max_tokens on a model that reads num_predict appears to work in configuration and does nothing at runtime. Map settings to each provider's spelling in one place, applied to every model you build, and test that the provider-specific field is set rather than that the generic one was passed. For reasoning effort, derive the offered levels from the model itself - its package profile first, your catalogue second - so no part of your interface can offer a level another part drops.

Problem 4: provider packages are not free to import

Supporting many providers means installing many integration packages, and some are heavy. Importing one can load hundreds of modules and take half a second or more, and in the past at least one registered a vendored protobuf definition that collided with another provider's package and crashed with a TypeError - not the ImportError that code guarding an optional import usually catches. Libraries above you can also import provider packages eagerly to support optional features.

Three habits keep this under control. Test whether a provider is installed with importlib.util.find_spec(), which does not execute the package. Import a provider's package only when an agent actually uses that provider. And measure: the memory and start-up time of a process that never touches a provider should not include it. In Olano, deferring provider imports that the agent framework made eagerly cut the first agent's start-up memory by more than half.

Retries are provider-specific too

One last difference that only shows under load. The retry behaviour you get depends on the SDK under each integration: the official OpenAI SDK retries 408, 409, 429 and 5xx responses and connection errors, other SDKs differ, and a gateway in front of several providers can relay an upstream failure as a 4xx that no SDK retries. Decide which failures are transient for your setup, retry those around the single model call with jittered back-off, and never retry a credit or billing refusal (it only delays an honest message) or an authentication error (it needs a person, not a loop). The streaming guide covers the one retry that must never happen: after the user has seen output.

A switching checklist

Before you let users change an agent's model mid-thread: sanitise content blocks at the conversion point for every strict provider; make sure every model has a context window, from its profile or your catalogue; set summarisation explicitly; map output and reasoning settings to each provider's spelling; and run a thread that has used every provider you support through every other one, with images and reasoning enabled, as an automated test.