A practical lab · about 16 minutes
Interactive learning lab
How AI works · guide 2 of 8
The token and cost laboratory
0/4 core signals
Saved only on this device
Token lab · BriefingScene 1 / 5
Models consume chunks, not words
Common strings can stay whole while rare strings, code and punctuation split.
Models consume chunks, not words ready.
Thefastestwaytoimprove→?
A tokenizer maps text chunks to numeric IDs before the model sees them. Exact splits vary across tokenizer and model families, so rough word counts are estimates rather than billing records.
Tokens are lookup-table units.
Input and output both consume budget.
Use the target tokenizer for exact counts.