tokenizers breaking changes

0.23.1

14 breaking changes, first published April 27, 2026.

• Removed
  Python 3.9 is no longer supported, so users on 3.9 must stay on tokenizers 0.22.x or upgrade Python. github.com
• Behavior changed
  On the free-threaded build, setters and getters now return PyResult, so a poisoned lock raises PyException instead of panicking. github.com
• Behavior changed add_tokens
  Tokens added with normalized=True have their content normalized when added, so re-saved tokenizer.json files may differ in the added_tokens block. github.com
• Signature changed RobertaProcessign.__init__
  Methods that returned Any now return real types, so mypy --strict may report new errors; stub files moved from tokenizers/<sub>/__init__.pyi to tokenizers/<sub>.pyi, and the RobertaProcessign __init__ surface is affected. github.com
• Signature changed tokenizers::Normalizer
  Was trait Normalizer; now trait Normalizer: Sync. crates.io
• Signature changed tokenizers::tokenizer::AddedVocabulary::add_special_tokens
  Was fn add_special_tokens<N: Normalizer>(&mut self, tokens: &[AddedToken], model: &impl Model, normalizer: Option<&N>) -> usize; now fn add_special_tokens<N: Normalizer>(&mut self, tokens: impl IntoIterator<Item = AddedToken>, model: &impl... crates.io
• Signature changed tokenizers::tokenizer::AddedVocabulary::add_tokens
  Was fn add_tokens<N: Normalizer>(&mut self, tokens: &[AddedToken], model: &impl Model, normalizer: Option<&N>) -> usize; now fn add_tokens<N: Normalizer>(&mut self, tokens: impl IntoIterator<Item = AddedToken>, model: &impl Model... crates.io
• Signature changed tokenizers::tokenizer::Normalizer
  Was trait Normalizer; now trait Normalizer: Sync. crates.io
• Signature changed tokenizers::tokenizer::TokenizerImpl::add_special_tokens
  Was fn add_special_tokens(&mut self, tokens: &[AddedToken]) -> usize; now fn add_special_tokens(&mut self, tokens: impl IntoIterator<Item = AddedToken>) -> Result<usize>. crates.io
• Signature changed tokenizers::tokenizer::TokenizerImpl::add_tokens
  Was fn add_tokens(&mut self, tokens: &[AddedToken]) -> usize; now fn add_tokens(&mut self, tokens: impl IntoIterator<Item = AddedToken>) -> Result<usize>. crates.io

And 4 more in this version.

0.22.0

5 breaking changes, first published August 29, 2025.

• Signature changed tokenizers::DecodeStreamError::InvalidPrefix
  Was InvalidPrefix; now InvalidPrefix {token_id: u32, expected_prefix: String, actual_string: String}. crates.io
• Signature changed tokenizers::models::unigram::Unigram::iter
  Was fn iter(&self) -> UnigramIterator; now fn iter(&self) -> UnigramIterator<'_>. crates.io
• Signature changed tokenizers::step_decode_stream
  Was fn step_decode_stream<M, N, PT, PP, D>(tokenizer: &TokenizerImpl<M, N, PT, PP, D>, id: u32, skip_special_tokens: bool, ids: &mut Vec<u32>, prefix: &mut String, prefix_index: &mut usize) -> Result<Option<String>> where M: Model, N... crates.io
• Signature changed tokenizers::tokenizer::DecodeStreamError::InvalidPrefix
  Was InvalidPrefix; now InvalidPrefix {token_id: u32, expected_prefix: String, actual_string: String}. crates.io
• Signature changed tokenizers::tokenizer::step_decode_stream
  Was fn step_decode_stream<M, N, PT, PP, D>(tokenizer: &TokenizerImpl<M, N, PT, PP, D>, id: u32, skip_special_tokens: bool, ids: &mut Vec<u32>, prefix: &mut String, prefix_index: &mut usize) -> Result<Option<String>> where M: Model, N... crates.io

Check your code

These are the changes. To see which of them reach your code, scan a public repository, or have your coding agent call check_code with your manifest or lockfile over MCP. The API reads every change in every version, not only the newest. On Team and up, a fix pull request opens for the changes that reach a connected repository. Nothing merges or deploys.