Back to catalog
␣ForCanBeConvertedToF
The second member of the ForCanBeConverted triplet (cl100k id 80370) — a single C#-compiler-flavored fragment that BPE split into three adjacent tokens (80369–80371).
Raw token
" ForCanBeConvertedToF"
␣marks a real space; the symbol itself is not part of the token, and copying preserves it.
Observed behaviors
- 01A “polysemantic” glitch token: together with ForCanBeConverted, the two most variable tokens in the source — perceived as wildly different words every time; nondeterministic at temperature 0 on gpt-3.5-turbo (bold-flagged).LessWrong
- Observed or documented on
- GPT-3.5 / GPT-4
- Behavior types
- Polysemantic
- Tokenizer
cl100k_base- Discovered by
- Matthew Watkins