Skip to main content
Back to catalog

␣日本毛片免费视频观看

A Chinese porn-spam title fragment (with a leading space), emblematic of GPT-4o's vocabulary pollution; “fixed” by GPT-5.

Raw token

" 日本毛片免费视频观看"

marks a real space; the symbol itself is not part of the token, and copying preserves it.

Observed behaviors

  • 01In xcloche's tests, GPT-4o could not read the token at all.note.com
  • 02Verified by Haruhiko Okumura: GPT-5 reads it normally — OpenAI patched the gap in later training, a rare “fixed” case.okumuralab.org
  • 03MIT Technology Review covered the pollution problem of such Chinese porn-spam tokens in GPT-4o's vocabulary.technologyreview.com
Observed or documented on
GPT (OpenAI)
Behavior types
Unspeakable
Tokenizer
o200k_base
Discovered by
xcloche