Handling Thinking Blocks So a Model Upgrade Isn't a Rewrite

What breaks, and why it keeps breaking

The failure has a recognisable shape. An agent that has run in production for months starts returning 400s the week a new model version is switched on. Or, far worse, it keeps working — and quietly gets less coherent on the long, tool-heavy tasks that were the reason you built it. Nothing in the agent's own logic changed. What changed is that the model now enforces a rule about how its own reasoning is carried between turns, and the integration has been breaking that rule for months without ever being told.

The rule exists because reasoning became state. When a model thinks before it answers, that thinking is not decoration; it is working memory for the rest of the task. Providers could have returned it as readable text and let you decide what to do with it. Instead all three of the major ones converged on the same design: the API returns an encrypted block, unreadable to your application, which you replay on the next request so the model picks up where it left off. It is closer to a session token than to content.

The concrete failure most teams meet first is Google's. Multi-turn function calling against a Gemini thinking model fails with INVALID_ARGUMENT and the message Function call is missing a thought_signature when the signature attached to the function-call part is not sent back on the following request. It bites hardest when Gemini is routed through an OpenAI-compatible tool-call path, because such a path is defined by the fields it knows about and silently discards the ones it does not. Your gateway is not malfunctioning. It is doing exactly what it was written to do, to a field that did not exist when it was written.

This keeps happening because the bug is in the plumbing rather than the prompt. Most hand-built integrations treat the messages array as a scratch pad to be reassembled from parts on every request. That was harmless when a message was just text. It stopped being harmless the moment part of that array became a cryptographically bound artefact whose validity depends on everything in front of it.

• All three major providers now return an opaque reasoning artefact you cannot inspect and must replay.
• The replay rules are provider-specific and tighten between versions — a working integration is not a permanently working integration.
• The costly failure is the silent drop, not the error. A 400 pages someone; a dropped block shows up in no metric you currently have.
• Reasoning travels forward across model versions and never backward, which makes fallback and cost routers the riskiest pattern in this whole area.
• The fix is architectural: append-only history, opaque artefacts tagged with the model that produced them, and explicit boundaries where models change.
• Get that right once and the next model version is a config change rather than a rewrite of the agent loop.

This guide uses Anthropic's Claude Fable 5.1, released on 1 September 2026, as the worked example, because it is the clearest published statement of these rules anywhere. That is a deliberate choice and not the subject. The model version is the illustration; the invariants are the content. If you are reading this a year or two from now with a different model id in your config, the section headings will still tell you what to build.

What a reasoning artefact actually is

Strip the branding away and every provider is doing the same thing. The model produces internal reasoning. The provider encrypts it, attaches it to the response, and expects to see it again on your next request. Your application is a courier. It carries a sealed envelope from one turn to the next and is not the intended recipient.

The names differ. OpenAI calls the encrypted reasoning field encrypted_content. Anthropic attaches an encrypted signature to its thinking blocks. Google calls them thought signatures — encrypted representations of the model's internal thought process that preserve reasoning state across multi-turn and multi-step conversations, attached to function-call parts in the API response. Their only purpose is to be passed back on the next turn so the model can resume. That is worth reading twice, because it tells you exactly how to treat them: they are not data you own, and there is nothing in them for you.

The practical consequence is that reasoning has joined the small set of things you are obliged to store and return verbatim, alongside tool-call identifiers. It differs from a tool-call id in one uncomfortable way: you cannot look at it and tell whether you have mangled it. A truncated, re-encoded or key-reordered artefact looks exactly like a healthy one until the API rejects it, or worse, until it does not.

Why providers made it opaque

There are two reasons usually given, and both are plausible. Raw reasoning traces are commercially sensitive — they are close to the thing a competitor would most like to distil. And they are a safety surface: a visible chain of thought is a visible target for manipulation, and providers would rather the model reason in private and answer in public. Whatever the mix, the effect on your architecture is identical, so the motive matters less than the constraint.

One thing worth knowing, filed under awareness rather than action: researchers reported during 2026 that encrypted reasoning envelopes returned by provider APIs can, in some circumstances, be replayed into weaker sibling models in a way that exposes the underlying reasoning trace. This is not a reason to avoid extended thinking, and there is nothing here to configure. Treat the envelopes as sensitive material at rest — the same handling you would give a session token in your logs and your database — and carry on.

The three providers, side by side

Read that last column as a spectrum of politeness rather than a spectrum of severity. Google's behaviour is the friendliest of the three, because a hard error is a bug report you receive on the day you introduce the bug. Anthropic's silent drop, in its default configuration for older accounts, is the one that costs you a quarter before anybody notices. This is the same asymmetry that makes an exit plan for a forced vendor migration worth writing before you need it: the migrations that hurt are the ones that succeed loudly and fail quietly.

The four invariants a provider-agnostic design must satisfy

Before any code, the four properties your integration needs. Every provider-specific rule discussed in this guide is a consequence of one of them, which is why designing to the invariants rather than to a particular API is what makes the next version a config change.

1. Opacity. Reasoning artefacts are bytes. Your code never parses them, never re-serialises them, never normalises them, never validates them against a schema, and never tidies them up. If an artefact passes through a JSON library that reorders keys or re-encodes strings, assume it is damaged.
2. Provenance. Every artefact is stored with the model id that produced it and the turn it belongs to. Not the model you are about to call — the one that generated it. Without this you cannot make a safe decision at a model boundary, because you will not know what you are carrying.
3. Prefix immutability. Once a turn is written, nothing in front of it ever changes again. Not the message text, not the ordering, not the system prompt, not the tools array, not the bytes behind a referenced image. Append-only is not a stylistic preference here; it is the precondition for replay.
4. Explicit boundaries. When the model or the provider changes mid-conversation, your code decides what happens to the artefacts. Not the API. A boundary your code owns produces a log line; a boundary the API handles for you produces a mystery.

A design that survives the next model

What follows is a shape rather than a library. It fits in a few hundred lines in any language, and the point of each piece is to make one of the invariants structurally true instead of merely intended.

An append-only conversation store

Make history append-only at the storage layer, not by convention. Convention survives exactly as long as the person who wrote the convention. If your code has a function that mutates a past turn, that is the bug — you do not need to find the caller that is misusing it, because the existence of the function guarantees that one exists or soon will.

"""Append-only conversation store.

Two rules the rest of the codebase is structurally unable to break:
  1. append() is the only writer. There is no update(), no delete(),
     and turns() hands back an immutable view.
  2. Reasoning artefacts are stored as opaque bytes, tagged with the
     model id that produced them. Nothing here parses them.
"""

from dataclasses import dataclass
from typing import Any, Callable, Literal

Provider = Literal["anthropic", "openai", "google"]


@dataclass(frozen=True)
class ReasoningArtifact:
    """Whatever the provider handed back. We never look inside."""
    provider: Provider
    model_id: str        # the model that PRODUCED it, not the next one
    field_name: str      # "thinking" | "encrypted_content" | "thought_signature"
    payload: bytes       # verbatim, byte for byte, as received


@dataclass(frozen=True)
class Turn:
    role: str
    model_id: str | None                    # None for user turns
    blocks: tuple[dict[str, Any], ...]      # provider-shaped content blocks
    artifacts: tuple[ReasoningArtifact, ...] = ()


class ConversationLog:
    def __init__(self) -> None:
        self._turns: list[Turn] = []

    def append(self, turn: Turn) -> None:
        self._turns.append(turn)

    def turns(self) -> tuple[Turn, ...]:
        return tuple(self._turns)           # read-only view, no mutation path

    def trim_oldest(self, n: int) -> None:
        """The only legal removal: a leading run, oldest first."""
        self._turns = self._turns[n:]


def strip_artifacts_at_boundary(
    turns: tuple[Turn, ...],
    target_model_id: str,
    can_read: Callable[[str, str | None], bool],
) -> tuple[list[Turn], list[str]]:
    """Drop reasoning artefacts the target model cannot read.

    Do this deliberately, at the boundary, and log what went. If you
    let the API drop them for you, the loss is invisible and surfaces
    only as a quality regression with no cause attached to it.
    """
    kept: list[Turn] = []
    dropped: list[str] = []
    for turn in turns:
        if turn.artifacts and not can_read(target_model_id, turn.model_id):
            dropped.append(turn.model_id or "unknown")
            kept.append(Turn(turn.role, turn.model_id, turn.blocks, ()))
        else:
            kept.append(turn)
    return kept, dropped

can_read is deliberately injected rather than hard-coded. It is the one piece of this design that genuinely is provider knowledge, it will change with every model family, and keeping it as a small table you can edit is the difference between a config change and a code change when the next version lands.

The opaque-blob envelope

The ReasoningArtifact record above carries four fields and no cleverness. Store the payload as bytes rather than as a parsed structure, and resist every instinct to improve it. The three most common ways teams destroy an artefact without noticing are all well-intentioned: round-tripping the whole message through a JSON normaliser that sorts keys; passing it through a Pydantic or dataclass model that drops unrecognised fields; and a logging layer that truncates long strings for readability and then writes the truncated version back into the store.

A gateway is the fourth, and it is the most common of all in practice. Any proxy that presents an OpenAI-compatible surface is, by construction, a field filter. If your requests pass through one, verify that unknown fields survive the round trip in both directions before you trust it with a thinking model — this is the specific hazard that turns Gemini's missing-signature error into a puzzling intermittent failure. Our guide to choosing between LiteLLM, OpenRouter and Portkey covers the wider trade-offs; the test that matters for this article is narrow and worth writing down: send a request with an artefact, capture what the provider actually received, and diff the bytes.

A per-turn overlay instead of history injection

Here is the anti-pattern that catches the most careful teams, because it looks like good engineering. You want the model to know something that is true only for this request — the current queue depth, a compliance reminder, the user's plan tier, the fact that a retry is in progress. So you splice it into an earlier turn, and remove it again next time so it does not accumulate.

# ANTI-PATTERN: mutating history to deliver a per-turn reminder.
# The first turn is rewritten on every request, so every thinking
# block after it is bound to a prefix that no longer matches.

import copy

def build_request(history, reminder, user_text):
    messages = copy.deepcopy(history)
    # Splice a status line into the FIRST user turn...
    messages[0]["content"][0]["text"] += f"\n\n[status] {reminder}"
    messages.append({"role": "user", "content": [{"type": "text", "text": user_text}]})
    return {
        "model": MODEL,
        "system": build_system_prompt(),   # rebuilt from a dict every call
        "tools": build_tools(),            # ...so is this one
        "messages": messages,
    }

# Three separate edits in one request: the spliced status line, a
# re-serialised system prompt, and a re-serialised tools array. Any
# one of them is enough. The deepcopy makes it feel safe, because
# the stored history is untouched -- but the request is what the
# binding check reads, and the request is different every time.

The fix is to stop editing and start appending. Anthropic's documented mechanism, as of September 2026, is a mid-conversation system message, which can be scoped to a single turn with clear_at: "next_user_message" behind the mid-conversation-system-clear-at-2026-08-21 beta header. For providers without an equivalent, the general shape is the same and just as effective: append a fresh turn carrying the per-request context, rather than reaching backwards into one that already exists.

# FIXED: the reminder arrives as a turn-scoped system message.
# Nothing already in the history is touched, so every earlier
# thinking block stays bound to a prefix that still matches.
#
# Envelope simplified -- check the current API reference for exact
# field placement. The architectural point is append, never splice.

request = {
    "model": MODEL,
    "system": STATIC_SYSTEM_PROMPT,   # built once at startup, never rebuilt
    "tools": TOOLS,                   # the same object, request after request
    "messages": log.turns_as_blocks() + [
        {
            "role": "system",
            "content": [{"type": "text", "text": f"[status] {reminder}"}],
            "clear_at": "next_user_message",     # applies to this turn only
        },
        {"role": "user", "content": [{"type": "text", "text": user_text}]},
    ],
}

headers = {
    # Required for clear_at, as of September 2026:
    "anthropic-beta": "mid-conversation-system-clear-at-2026-08-21",
}

Note what else changed in that second version. STATIC_SYSTEM_PROMPT and TOOLS are module-level constants built once, not functions called per request. That is not tidiness. Rebuilding either between requests in the same conversation invalidates every thinking block that follows, and a dictionary that serialises in a different order on a different process is a rebuild whether you intended one or not.

A tools-change channel that is not "rebuild the array"

Agents that change their available tools mid-task — unlocking write operations after an approval step, loading a tenant-specific integration, narrowing the surface once the task phase is known — hit this immediately, because the obvious implementation is to construct a new tools array and send it. On Fable 5.1 that is an edit to the prefix, and every thinking block after it is invalidated. Anthropic's documented answer is mid-conversation tool changes, which express the delta rather than replacing the array.

There is a pleasant side effect worth knowing about: the same discipline keeps the prompt cache warm. A stable prefix is a cacheable prefix, so the work you do to protect reasoning blocks pays for itself twice. If you are placing cache breakpoints around a tool layer, the mechanics of programmatic tool calling and cache breakpoints covers where the boundaries belong.

One more version-specific change belongs in the same mental bucket, because it is the same category of breakage. As of September 2026, Fable 5.1 does not support forced tool use: tool_choice of {"type":"any"} or {"type":"tool","name":"..."} returns a 400 invalid_request_error, while {"type":"auto"} and {"type":"none"} are unchanged. The documented replacements are strict tool use — strict: true with tool_choice: {"type":"auto"} — or structured outputs, in both cases alongside a prompt that states plainly when the tool applies. The migration lesson is not about tool_choice. It is that a request field held in one config constant is a one-line change, and the same field hard-coded across six call sites is a project.

A model-transition policy

This is where the Fable 5.1 rules become genuinely instructive, because Anthropic has written down something the other providers have so far left implicit. Every thinking block records which model produced it, and preservation is one-directional: Fable 5.1 reads earlier models' thinking blocks, and no earlier model reads Fable 5.1's. Moving a conversation forward onto the newer model keeps its reasoning. Moving it backward loses it, for whichever turns run there.

When a request carries a block the target model cannot read, the API drops it before the model sees it. Dropped blocks do not count toward input_tokens and are not billed, which sounds like a courtesy and is in fact the problem: there is no cost signal, no error, and nothing on the invoice to notice. With the thinking-binding-controls-2026-08-01 beta header the drop is reported in a top-level input_transformations array. Without it, the drop is silent.

The second Fable 5.1 rule concerns edits rather than models. Editing anything before a thinking block invalidates it — the system prompt, the tools array, or an earlier message. A replayed invalidated block is rejected with a 400 whose message reads The block is bound to a different conversation. If you would rather continue than fail, send the thinking-binding-controls-2026-08-01 beta header with thinking.block_binding.prefix_mismatch_behavior: "drop_block"; the drop is then reported in input_transformations with reason: "prefix_binding_mismatch".

And a detail that will cost somebody a confusing afternoon: the check is enforced for accounts created on or after 31 August 2026. For accounts older than that, the API records the mismatch but acts on it only when the request explicitly sets thinking.block_binding.prefix_mismatch_behavior. Two teams running byte-identical code can therefore observe different behaviour. A Bengaluru team whose staging organisation was opened in September will see hard failures that a London fintech's four-year-old production account does not — or, more painfully, the reverse: staging is quiet, production is quiet, and only the new contractor's fresh sandbox reveals that the integration has been editing history since launch.

Two structural observations about that table. First, everything in the invalidating half is something your code chose to do; none of it is the model being fragile. Second, if you are using a managed harness — Claude Code, claude.ai, Claude Managed Agents, or the Claude Agent SDK — the prefix is kept intact for you and most of this is already handled. Hand-built messages arrays are where it breaks, which is a fair description of every production agent that started life as a prototype.

Trimming context without invalidating anything

Append-only does not mean unbounded, and this is the objection people raise first. Long-running agents accumulate history that has to go somewhere. The constraint is not that you may never remove anything; it is that removal has a legal direction.

Trim from the oldest end only. Removing a leading run of thinking blocks, oldest first, keeps everything after it valid. Removing a block from anywhere else invalidates every block that follows it — which means the intuitive optimisation, "drop the reasoning from turns four and seven because they were unusually long", is precisely the wrong move. It is also the one a well-meaning cost-reduction ticket will produce, because from the outside it looks like removing the biggest items first.

Where the provider offers server-side trimming, prefer it outright. Anthropic's context editing and compaction do not count as edits for binding purposes, which is a strong reason to use the platform's mechanism rather than your own even if yours would otherwise be smarter. The general principle transfers: any trimming the provider performs on its own side of the boundary cannot break a binding the provider itself computed. Our guides to compaction for long-running agents and to context engineering for long-horizon agents cover what to keep and what to discard; the rule specific to reasoning artefacts is simply that the decision of where to cut is not free.

Routing and fallback: the direction problem

Now the part that turns a correct integration into an incorrect one without changing a line of the agent loop. Reasoning preservation is one-directional, and a router is a device for changing models. Put those two facts together and the risk profile of common routing patterns falls out immediately.

The escalation pattern deserves the emphasis because it is so widely recommended on cost grounds and so rarely examined on this axis. Consider a London fintech whose routing layer sends the first three turns of every support conversation to a small model and escalates to a frontier model when a confidence heuristic trips. The escalation is fine. What is not fine is the retry path underneath it, which sends the conversation back down to the small model when the frontier model is rate-limited — and does so mid-conversation, carrying blocks the small model cannot read. Nobody wrote that as a policy. It emerged from a sensible retry decorator meeting a sensible router.

The fix is unglamorous: make the model a property of the conversation rather than of the request. Choose at the start, based on whatever signal you have, and hold it. If you must escalate, escalate once and never come back down within the same conversation. If your cost model genuinely requires per-turn routing, then reasoning artefacts must be stripped at every hop by design, and you should account for the quality cost rather than discovering it. And whichever you choose, prove it before it reaches everyone: shadow and canary deploys for model upgrades exist precisely for the class of change whose failure mode is a slow quality drift rather than an exception.

A migration checklist for an existing codebase

Run this against the agent you already have. Most of it is grep, and the first four items find the majority of real defects.

1. Find every write to the message list. Anything that is not an append is a candidate bug. Index assignment, insert(), slice assignment, and any helper with a name like rewrite, patch or normalise.
2. Find every deep copy of the history followed by an assignment into an existing element. The copy makes it look safe; the request is still different.
3. Check whether the system prompt and tools array are built once per process or once per request. If per request, either build them once or prove they serialise byte-identically. Prefer building them once.
4. Search for per-request text spliced into an earlier turn: timestamps, "the current date is", user plan tier, queue depth, retrieved-document reminders, retry counters.
5. Confirm reasoning artefacts are stored rather than discarded — and stored as received, not through a JSON normaliser, a schema-validating model, or a truncating log formatter.
6. Confirm every stored artefact carries the model id that produced it. If it does not, you cannot implement a safe boundary, only a hopeful one.
7. Enumerate every place the model can change mid-conversation: fallbacks, cost routers, A/B splits, retry paths, manual overrides, feature flags. Each one is a boundary that needs a decision.
8. Check any image or document referenced by URL. Does the same URL serve identical bytes on a later request? A regenerated thumbnail or a re-encoded PDF is an edit even though the link is unchanged.
9. Test the gateway. If requests pass through a proxy or an OpenAI-compatible shim, verify that unknown fields survive in both directions. Capture what the provider actually received and diff it.
10. Run one session with the provider's diagnostic enabled and read what it says it removed. This is the item that turns the previous nine from a code review into a measurement.

Common pitfalls

Cleaning up the history. Somebody notices that stored conversations contain large unreadable blobs and writes a tidy-up job. Reasoning artefacts are exactly the thing that looks like junk and is not. Give them a name in your schema that makes their purpose obvious — reasoning_artifact_opaque beats extra_data — and put a comment above the column.

The nightly refactor that moves the prefix. A prompt-builder refactor that changes nothing semantically — reordering two paragraphs of the system prompt, alphabetising the tool definitions, upgrading a serialisation library — invalidates every thinking block in every live conversation. Long-lived conversations are the ones that suffer, and they are also the ones you care most about.

Environments that disagree. The account-age split described above means a passing staging run is not evidence. Confirm which behaviour each of your organisations is subject to before you conclude that a fix worked, and prefer to set prefix_mismatch_behavior explicitly everywhere so that all your environments behave alike regardless of when the account was created.

The gateway that normalises. Covered above and worth restating, because it is the failure that survives a correct application. Your code can satisfy all four invariants and still lose the artefact three hops downstream.

Assuming the 400 is the worst case. It is the best case. An error is a fact you receive on the day of the change. The drop is the same information delivered as a rumour, six weeks later, in the form of a support-ticket backlog nobody can explain.

What to log so the next model version is a config change

The diagnostic Anthropic recommends is direct, and it answers a question you cannot answer by reading your own code: run a session with prefix_mismatch_behavior: "drop_block" and log input_transformations. Every entry is a block the API removed before the model saw it, with a reason attached. A healthy integration produces an empty array.

# Diagnostic run: ask the API to report what it removed, then log it.
# As of September 2026 the field names below are Anthropic-specific;
# the practice is not. Run this against REPLAYED PRODUCTION
# conversations, not a synthetic three-turn script -- the edits that
# bite are almost always in the long tail.

response = client.messages.create(
    model=MODEL,                      # the Fable 5.1 id from your config
    max_tokens=2048,
    system=STATIC_SYSTEM_PROMPT,
    tools=TOOLS,
    messages=replayed_history,
    thinking={
        "type": "enabled",
        "budget_tokens": 4096,
        "block_binding": {"prefix_mismatch_behavior": "drop_block"},
    },
    extra_headers={"anthropic-beta": "thinking-binding-controls-2026-08-01"},
)

transformations = getattr(response, "input_transformations", None) or []
for t in transformations:
    log.warning(
        "thinking_block_dropped conversation=%s reason=%s",
        conversation_id,
        getattr(t, "reason", "unknown"),      # e.g. "prefix_binding_mismatch"
    )

# Contract test: a healthy integration reports nothing at all.
assert not transformations, (
    f"{len(transformations)} thinking block(s) dropped -- "
    "the request prefix changed between turns"
)

Turn that assertion into a contract test that replays a small set of stored conversations against the current model on every build, and you have converted an invisible architectural property into a red pipeline. This belongs in the same suite as your prompt and agent regressions — running evals in CI for prompt and agent regression testing covers how to keep that suite fast enough that people do not disable it.

Beyond the fingerprint, five things are worth logging permanently. The model id per turn, not per conversation, so a mid-conversation hop is visible in the record. The provider, artefact field name and byte length for every artefact stored, which catches truncation. Whether your own boundary code stripped an artefact, and why. The provider's transformation or diagnostic field verbatim, wherever one exists. And the beta headers in force on the request, because a behaviour that depends on a header is a behaviour you will misdiagnose the day someone removes it.

None of this is exotic engineering, and that is rather the point. The teams for whom a model upgrade is a two-line change are not the ones with the cleverest agent loops. They are the ones whose conversation store refuses to be edited, whose artefacts carry their provenance, and whose model boundaries are places in the code rather than accidents in a router. Everything in this guide follows from those three habits, and none of them depends on which model id is current. The primary sources are Anthropic's release notes for what is new in Claude Fable 5.1 and Google's documentation on thought signatures in the Gemini API. Both will be superseded. The invariants will not.