tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuing_subword_prefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1.
| Métrique | Valeur | Interprétation |
|---|---|---|
|
AV (Vecteur d'Attaque)
Attack Vector
|
Network (N)
|
Exploit possible à distance via réseau (HTTP, API, etc.) |
|
AC (Complexité d'Attaque)
Attack Complexity
|
Low (L)
|
Peu de conditions requises pour exploiter |
|
PR (Privilèges Requis)
Privileges Required
|
None (N)
|
Aucune authentification requise |
|
UI (Interaction Utilisateur)
User Interaction
|
Unknown (P)
|
Interaction inconnue |
|
S (Périmètre)
Scope
|
Unchanged (U)
|
Impact reste dans le périmètre du composant compromis |
|
C (Impact Confidentialité)
Confidentiality Impact
|
None (N)
|
Aucune divulgation d'informations |
|
I (Impact Intégrité)
Integrity Impact
|
None (N)
|
Aucune modification de données possible |
|
A (Impact Disponibilité)
Availability Impact
|
High (H)
|
Interruption totale du service possible |
tokenizers BpeBuilder Buffer Overflow via merge token