<|im_end|>

Representative of the seven “ghost” special tokens in Kimi K3's tokenizer: ChatML-era names from the K2 generation, hard-coded into the tokenizer yet registered entirely outside the declared vocabulary.

"<|im_end|>"

Observed behaviors

• 01A hard-coded K2-leftover list in tokenization_kimi.py registers <|im_end|>, <|im_user|>, <|im_assistant|>, <|start_header_id|>, <|end_header_id|>, <|im_system|> and <|im_middle|> at ids 163840–163846, while config.json declares vocab_size 163840 (last valid id 163839) — all seven are out of range.huggingface.co
• 02The two encode paths disagree: tok.encode goes through tiktoken and yields ordinary BPE ids [27, 91, 348, 11129, 91, 29], while the tokenizer(...) call returns the out-of-range [163840] — an IndexError at the embedding table (device-side assert on GPU). Any deployment that tokenises untrusted input via tokenizer(...) can be crashed by a plain prompt: a crash/DoS shape, not prompt injection (K3's real protocol tokens are a different set and cannot be forged this way).huggingface.co

References

• Kimi-K3 Discussion #65: out-of-range K2-leftover special tokens — Hugging Face