ttok 1.0

Simon Willison shipped ttok 1.0, a release of his command-line token-counting tool. The version bump came after he upgraded from ttok 0.4 with uv tool upgrade ttok, piped a file into the new build, and noticed it was still defaulting to the GPT-4 tokenizer when it should default to the GPT-5/GPT-6 tokenizer. He describes that switch as a reasonable excuse to finally cut a 1.0 release.

The underlying question was whether the GPT-5 family and GPT-6 actually share a tokenizer, since OpenAI has not confirmed it. Willison points to a commit by William Liu reporting an experiment suggesting they are likely the same: across all seven GPT models tested (5.5, 5.6 Sol/Terra/Luna, and 6 Astra/Sol/Luna), every model reported 44,794 tokens and matched the others on all 31 fixtures, with GPT-6 introducing no input-count change on that corpus.

For developers, token counts drive prompt sizing, context-window budgeting and per-request cost estimates. A tokenizer change would break those numbers, so a 1.0 default that tracks the current model family removes a silent source of drift. The caveat is that the evidence is a third-party experiment rather than vendor documentation, so anyone depending on exact counts may want to verify against their own inputs.

Why it matters: teams using ttok for quick token counts will now get figures matching the newer OpenAI models by default. The GPT-6 tokenizer equivalence is inference from a community experiment, not an OpenAI confirmation, so treat counts near a context limit with some caution.