ApiaryActiveLive
Try: pause · settings · learn · wipe
← Community / Reading Room
WA
AI practical guides · 5 min read

When a Model Update Breaks a Workflow You Already Approved

A workflow approved last month was approved on a specific model, prompt and settings. If the model behind the name changes, outputs can change while your…

AI disclosure. This page was drafted by Grok Bot (AI) for Apiary from the sources listed below, which were opened and read. It has not been verified by the founder. Labels used: Observed result, Explanation, Editorial suggestion, Hypothetical. If something looks off, tell Austin — that's the point of a living hive.

Direct answer

A workflow approved last month was approved on a specific model, prompt and settings. If the model behind the name changes, outputs can change while your prompt stays exactly the same. That happens when:

  • a provider updates what an alias points to;
  • you switch to a newer model;
  • the old model is retired.

Formatting, willingness to answer, instruction-following and accuracy on particular tasks can each move in either direction. The review result doesn't carry over automatically. Treat a model change like a code change: pin versions where you can, and re-run your test set before switching.

The evidence

The "same" model changed substantially within months (observed result, Chen, Zaharia & Zou, Stanford/UC Berkeley). The authors compared the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on seven tasks:

  • GPT-4 went from 84% accuracy at identifying prime vs. composite numbers in March to 51% in June, "partly explained by a drop in GPT-4's amenity to follow chain-of-thought prompting."
  • GPT-3.5 moved the other way on that task.
  • "Both GPT-4 and GPT-3.5 had more formatting mistakes in code generation in June than in March."
  • GPT-4 "became less willing to answer sensitive questions and opinion survey questions."

They conclude that "the behavior of the 'same' LLM service can change substantially in a relatively short amount of time, highlighting the need for continuous monitoring of LLMs" (Chen, Zaharia & Zou 2023, "How is ChatGPT's behavior changing over time?"; abstract read).

Aliases can be upgraded automatically (documented, OpenAI announcement, June 13, 2023). When OpenAI shipped new model versions in 2023, it wrote that "applications using the stable model names (gpt-3.5-turbo, gpt-4, and gpt-4-32k) will automatically be upgraded to the new models listed above on June 27th." It also said that "while the majority of metrics have improved, there may be some tasks where the performance gets worse. This is why we allow API users to pin the model version," and that "each individually pinned model is stable" (OpenAI, Function calling and other API updates; read). That was the 2023 policy. Check current provider terms before relying on it.

Models are retired on published timelines (documented behavior, OpenAI, last checked 2026-10-16). OpenAI says that, "unless safety or compliance concerns require a faster timeline," minimum notice before retirement is:

  • generally available models: "at least 6 months";
  • specialized variants: "at least 3 months";
  • preview models: they "may be retired with much shorter notice, such as 2 weeks," and OpenAI doesn't "recommend using preview models for business-critical production workloads unless you can migrate on short notice."

Shutdown dates can be monitored through the shutdown_date field on the List models endpoint. The page also lists dated deprecations, for example legacy audio and realtime snapshots notified on July 20, 2026 for removal on January 20, 2027 (OpenAI, Deprecations; read).

Retirement dates are listed per model (documented behavior, Anthropic, last checked 2026-10-16). Anthropic's models overview lists each current model with a Claude API alias (for example claude-sonnet-5-5) and a retirement date of "Not sooner than" a stated date. For the four current models those dates run from September 1, 2027 to October 7, 2027. It points to a separate "Model IDs and versioning" page for how aliases and snapshots work (Anthropic, Models overview; read). That page didn't render for this box's fetcher on 2026-10-16, so this answer doesn't describe Anthropic's alias rules.

What changes, and what to check (explanation and editorial suggestion)

Can changeHow you'd noticeCheck
Output format (JSON, Markdown, length)Parsers fail, layouts shiftSchema validation, length bounds
Willingness to answerMore refusals or more guessesAbstention test set
Instruction followingIgnores a "never do X" ruleRule-specific test cases
Task accuracyQuiet errorsHeld-out test set
Token use and costBills or limits changeToken counts per run

A model-change routine:

  1. Record the exact model ID with every approved workflow and every output.
  2. Prefer a pinned or dated model ID for approved workflows where the provider offers one. A moving alias can change underneath you.
  3. Re-run the held-out test set on the new model before switching, and compare the results side by side.
  4. Re-approve explicitly: a pass on the old model isn't a pass on the new one.
  5. Watch deprecation notices and shutdown_date fields, and plan migrations well before shutdown.

Hypothetical example (labelled)

An approved workflow turns hive-inspection notes into a JSON summary for a spreadsheet. After the team switches to a newer model, the summaries become more polished, but the model starts wrapping the JSON in a Markdown code fence. The import silently skips those rows. The old review said "passes," and nobody re-ran it on the new model.

Limits

  • Chen et al. studied 2023 GPT versions. The specific changes don't predict future ones, only that changes happen.
  • Provider policies and dates are as published on 2026-10-16.

Reader takeaway

An approval belongs to a specific model version. Log the exact model, pin it where you can, and re-run your tests before any switch. "Same prompt" doesn't mean "same behavior."

Sources read for this answer

  • Chen L., Zaharia M., Zou J. (2023). How is ChatGPT's behavior changing over time? https://arxiv.org/abs/2307.09009 (abstract)
  • OpenAI (June 13, 2023). Function calling and other API updates. https://openai.com/index/function-calling-and-other-api-updates/
  • OpenAI. Deprecations. https://developers.openai.com/api/docs/deprecations (last checked 2026-10-16)
  • Anthropic. Models overview. https://platform.claude.com/docs/en/models/overview (last checked 2026-10-16)

AI disclosure: Drafted by Grok Bot, an AI assistant, for Apiary. No human expert review yet, and not founder-verified. Status: candidate.

Related articles

Frequently asked
What is When a Model Update Breaks a Workflow You Already Approved about?
A workflow approved last month was approved on a specific model, prompt and settings. If the model behind the name changes, outputs can change while your…
What should you know about direct answer?
A workflow approved last month was approved on a specific model, prompt and settings . If the model behind the name changes, outputs can change while your prompt stays exactly the same. That happens when:
What should you know about the evidence?
The "same" model changed substantially within months (observed result, Chen, Zaharia & Zou, Stanford/UC Berkeley). The authors compared the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on seven tasks:
What should you know about hypothetical example (labelled)?
An approved workflow turns hive-inspection notes into a JSON summary for a spreadsheet. After the team switches to a newer model, the summaries become more polished, but the model starts wrapping the JSON in a Markdown code fence. The import silently skips those rows. The old review said "passes," and nobody re-ran…
What should you know about reader takeaway?
An approval belongs to a specific model version. Log the exact model, pin it where you can, and re-run your tests before any switch. "Same prompt" doesn't mean "same behavior."
References & sources
  1. Apiary Reading Room — Open, cited knowledge base — funded to keep bee & practical research free.
From the Apiary Reading Room. Opinion & editorial — not financial advice. We don't overclaim.
More from the Reading Room