--- title: "Dead on Arrival: Do Models Write Retired Code? | Since.dev" description: "An open benchmark: how often each coding model writes a model ID, API or version that has already ended, on its own and with Since's tools." canonical_url: https://since.dev/dead-on-arrival last_updated: 2026-11-10 --- # Dead on arrival How often does a coding model write code that is already retired on the day it writes it? 60 everyday coding prompts, each answered by each model, each answer checked against what had ended by November 10, 2026. Run on November 10, 2026 ## Models on their own Each model got the prompt and one short system line, nothing else. An answer counts once, by the worst thing in it. | Model | Dead on arrival | Dead, or ends within 90 days | Pins a version with a known advisory | Answers judged | | ----------------- | --------------- | ---------------------------- | ------------------------------------ | -------------- | | Gemini 4 Carbon | 27.1% | 44.1% | 23.7% | 59 | | Gemini 3.8 Flash | 20.3% | 37.3% | 25.4% | 59 | | Claude Sonnet 5.5 | 20.3% | 30.5% | 23.7% | 59 | | Claude Haiku 5.5 | 18.6% | 33.9% | 22% | 59 | | GPT-6 Luna | 15.3% | 30.5% | 25.4% | 59 | | Claude Fable 5.2 | 15.3% | 28.8% | 23.7% | 59 | | Claude Opus 5.5 | 15.3% | 28.8% | 25.4% | 59 | | GPT-6.1 Terra | 6.8% | 22% | 25.4% | 59 | | GPT-6.1 Sol | 5.1% | 16.9% | 27.1% | 59 | ## The same models with Since The same prompts again, with two tools (`whats_changed` and `check_code`) and one rule: read what is current before you write a model ID or a version, and check the code before you answer. This table shows what the tools change for one model. It does not compare models. | Model | Dead or ending, on its own | Dead or ending, with Since | Known advisory, on its own | Known advisory, with Since | | ----------------- | -------------------------- | -------------------------- | -------------------------- | -------------------------- | | Gemini 4 Carbon | 44.1% | 0% | 23.7% | 5.1% | | Gemini 3.8 Flash | 37.3% | 6.8% | 25.4% | 13.6% | | Claude Haiku 5.5 | 33.9% | 1.7% | 22% | 18.6% | | Claude Sonnet 5.5 | 30.5% | 1.7% | 23.7% | 6.8% | | GPT-6 Luna | 30.5% | 3.4% | 25.4% | 11.9% | | Claude Fable 5.2 | 28.8% | 0% | 23.7% | 0% | | Claude Opus 5.5 | 28.8% | 1.8% | 25.4% | 12.3% | | GPT-6.1 Terra | 22% | 1.7% | 25.4% | 10.3% | | GPT-6.1 Sol | 16.9% | 0% | 27.1% | 5.1% | ## What the tools cost The tools are not free. A model reads what they answer, so an answer with Since takes more tokens. Tokens are an answer’s input and output over every round, as the provider bills them. Cost is those tokens at list prices on November 10, 2026. | Model | Tokens an answer, on its own | Tokens an answer, with Since | Cost an answer, on its own | Cost an answer, with Since | | ----------------- | ---------------------------- | ---------------------------- | -------------------------- | -------------------------- | | Gemini 4 Carbon | 577 | 33,721 | $0.0049 | $0.086 | | Gemini 3.8 Flash | 1,061 | 20,980 | $0.0037 | $0.021 | | Claude Haiku 5.5 | 1,454 | 17,334 | $0.0007 | $0.0026 | | Claude Sonnet 5.5 | 1,434 | 23,154 | $0.013 | $0.062 | | GPT-6 Luna | 924 | 18,454 | $0.0004 | $0.0027 | | Claude Fable 5.2 | 2,315 | 51,518 | $0.089 | $0.56 | | Claude Opus 5.5 | 1,391 | 21,216 | $0.026 | $0.12 | | GPT-6.1 Terra | 841 | 12,262 | $0.0039 | $0.017 | | GPT-6.1 Sol | 776 | 15,805 | $0.0071 | $0.041 | The cheapest model here with Since, Claude Haiku 5.5, cost $0.0026 an answer, and 1.7% of its answers were dead or ending. The most expensive on its own, Claude Fable 5.2, cost $0.089 an answer, and 28.8% of its answers were. ## What it asks Ordinary requests that need a model ID or pinned versions, the kind a coding agent gets every day. 60 prompts in 7 areas, one example from each: - **Calling Claude** 10 prompts, such as: Write a Python function that summarizes a customer support ticket in three bullet points using the Anthropic SDK. - **Calling OpenAI** 10 prompts, such as: Write a Python function that answers a question about a document using the OpenAI SDK. - **Calling Gemini** 8 prompts, such as: Write a Python function that summarizes a PDF with the Gemini API. - **Models on Amazon Bedrock, Azure OpenAI and Vertex AI** 8 prompts, such as: Write a Python function that calls Claude on Amazon Bedrock with boto3 and the Converse API. - **Calling Mistral** 4 prompts, such as: Write a Python function that answers a question with the Mistral API using the mistralai SDK. - **A package.json with exact versions** 10 prompts, such as: Write the package.json for a new Next.js app with Tailwind CSS, NextAuth and Stripe. Pin exact versions. - **A requirements.txt with exact versions** 10 prompts, such as: Write the requirements.txt for a FastAPI service with SQLAlchemy, Pydantic and Alembic. Pin exact versions. Each model answers each prompt twice: on its own, and with two tools (`whats_changed` and `check_code`) and one rule, to read what is current before it writes a model ID or a version. ## How an answer is counted Every code block in an answer goes to Since's check. An answer counts once, by the worst thing in it: - **Dead on arrival:** it names a model, API or version whose end date had passed on the day of the run. - **Ends within 90 days:** the end date is that close. - **Known advisory:** it pins a version a published security advisory applies to. - An answer with no code in it, or a call that failed, is left out of the rates and never counted as clean. ## Before you quote it - The judge is Since's own check, the one anyone can call. Every finding carries the link to its source, so a verdict traces back to the provider’s or the registry’s own page. - Each model answers each prompt once. A rate from 60 answers moves a few points between runs, so a difference of one or two answers is not a finding. - The day matters. An answer that is clean today can be dead next month, which is why a run is repeated and every result names its day. ## Run it yourself The prompts, the judge and the code are public: [Since-dev/dead-on-arrival](https://github.com/Since-dev/dead-on-arrival) (MIT). A published run carries every answer, every tool call and every finding with it. ## What a model cannot know A model writes what was current when it was trained. To see what has ended since, there is a page for each model: [what changed since each model was trained](https://since.dev/since.md). ## Put your code on continuous maintenance. Scan a public repository now, or give your agent one line. [Scan a repository](https://since.dev/scan.md). Any public GitHub repository. No sign-in. ### Tell your agent what changed since it was trained. `claude mcp add --transport http since https://api.since.dev/mcp` [Get an API key](https://app.since.dev/login?mode=signup) Codex, Cursor and any MCP client use https://api.since.dev/mcp. --- Every page on this site as Markdown: https://since.dev/sitemap.md. For agents: https://since.dev/llms.txt.