--- title: "tokenizers Breaking Changes on crates.io | Since.dev" description: "What each tokenizers release removed, renamed or changed: 19 breaking changes in 2 versions on crates.io, each with its source." canonical_url: https://since.dev/packages/crates/tokenizers last_updated: 2026-04-27 --- # tokenizers breaking changes Every breaking API change we have recorded in tokenizers on crates.io: 19 breaking changes in 2 versions. Below are its newest 2 versions, removals and renames first, each with its source. Newest change April 27, 2026 ## 0.23.1 14 breaking changes, first published April 27, 2026. - **Removed** Python 3.9 is no longer supported, so users on 3.9 must stay on tokenizers 0.22.x or upgrade Python. [github.com](https://github.com/huggingface/tokenizers/releases/tag/v0.23.1) - **Behavior changed** On the free-threaded build, setters and getters now return PyResult, so a poisoned lock raises PyException instead of panicking. [github.com](https://github.com/huggingface/tokenizers/releases/tag/v0.23.1) - **Behavior changed** `add_tokens` Tokens added with normalized=True have their content normalized when added, so re-saved tokenizer.json files may differ in the added_tokens block. [github.com](https://github.com/huggingface/tokenizers/releases/tag/v0.23.1) - **Signature changed** `RobertaProcessign.__init__` Methods that returned Any now return real types, so mypy --strict may report new errors; stub files moved from tokenizers//\_\_init\_\_.pyi to tokenizers/.pyi, and the RobertaProcessign \_\_init\_\_ surface is affected. [github.com](https://github.com/huggingface/tokenizers/releases/tag/v0.23.1) - **Signature changed** `tokenizers::Normalizer` Was trait Normalizer; now trait Normalizer: Sync. [crates.io](https://crates.io/crates/tokenizers/0.23.1) - **Signature changed** `tokenizers::tokenizer::AddedVocabulary::add_special_tokens` Was fn add_special_tokens(&mut self, tokens: &\[AddedToken\], model: &impl Model, normalizer: Option<&N>) -> usize; now fn add_special_tokens(&mut self, tokens: impl IntoIterator, model: &impl... [crates.io](https://crates.io/crates/tokenizers/0.23.1) - **Signature changed** `tokenizers::tokenizer::AddedVocabulary::add_tokens` Was fn add_tokens(&mut self, tokens: &\[AddedToken\], model: &impl Model, normalizer: Option<&N>) -> usize; now fn add_tokens(&mut self, tokens: impl IntoIterator, model: &impl Model... [crates.io](https://crates.io/crates/tokenizers/0.23.1) - **Signature changed** `tokenizers::tokenizer::Normalizer` Was trait Normalizer; now trait Normalizer: Sync. [crates.io](https://crates.io/crates/tokenizers/0.23.1) - **Signature changed** `tokenizers::tokenizer::TokenizerImpl::add_special_tokens` Was fn add_special_tokens(&mut self, tokens: &\[AddedToken\]) -> usize; now fn add_special_tokens(&mut self, tokens: impl IntoIterator) -> Result. [crates.io](https://crates.io/crates/tokenizers/0.23.1) - **Signature changed** `tokenizers::tokenizer::TokenizerImpl::add_tokens` Was fn add_tokens(&mut self, tokens: &\[AddedToken\]) -> usize; now fn add_tokens(&mut self, tokens: impl IntoIterator) -> Result. [crates.io](https://crates.io/crates/tokenizers/0.23.1) And 4 more in this version. ## 0.22.0 5 breaking changes, first published August 29, 2025. - **Signature changed** `tokenizers::DecodeStreamError::InvalidPrefix` Was InvalidPrefix; now InvalidPrefix {token_id: u32, expected_prefix: String, actual_string: String}. [crates.io](https://crates.io/crates/tokenizers/0.22.0) - **Signature changed** `tokenizers::models::unigram::Unigram::iter` Was fn iter(&self) -> UnigramIterator; now fn iter(&self) -> UnigramIterator<'\_>. [crates.io](https://crates.io/crates/tokenizers/0.22.0) - **Signature changed** `tokenizers::step_decode_stream` Was fn step_decode_stream(tokenizer: &TokenizerImpl, id: u32, skip_special_tokens: bool, ids: &mut Vec, prefix: &mut String, prefix_index: &mut usize) -> Result> where M: Model, N... [crates.io](https://crates.io/crates/tokenizers/0.22.0) - **Signature changed** `tokenizers::tokenizer::DecodeStreamError::InvalidPrefix` Was InvalidPrefix; now InvalidPrefix {token_id: u32, expected_prefix: String, actual_string: String}. [crates.io](https://crates.io/crates/tokenizers/0.22.0) - **Signature changed** `tokenizers::tokenizer::step_decode_stream` Was fn step_decode_stream(tokenizer: &TokenizerImpl, id: u32, skip_special_tokens: bool, ids: &mut Vec, prefix: &mut String, prefix_index: &mut usize) -> Result> where M: Model, N... [crates.io](https://crates.io/crates/tokenizers/0.22.0) ## Check your code These are the changes. To see which of them reach your code, [scan a public repository](https://since.dev/scan.md), or have your coding agent call `check_code` with your manifest or lockfile over [MCP](https://since.dev/docs.md#mcp). [The API](https://since.dev/docs.md#api) reads every change in every version, not only the newest. On Team and up, a fix pull request opens for the changes that reach a connected repository. Nothing merges or deploys. ## Put your code on continuous management. Scan a public repository now, or give your agent one line. ### See how out of date your code is. [Scan a repository](https://since.dev/scan.md). Any public GitHub repository. No sign-in. ### Tell your agent what changed since it was trained. `claude mcp add --transport http since https://api.since.dev/mcp` --- Every page on this site as Markdown: https://since.dev/sitemap.md. For agents: https://since.dev/llms.txt.