Introduction
In the world of advanced language models, the cost discussion often revolves around pricing per million tokens. However, this approach can be misleading. Why? Because 'tokens' are not standardized across different models. What does the price of a token really mean, and why is it crucial to understand this nuance?
Understanding Tokens and Tokenization
A 'token' is a unit of data processed by language models. Each model provider has its own tokenization mechanism, splitting the same text into a different number of tokens. For instance, Anthropic's Claude model can produce about 30% more tokens from the same text compared to its previous versions. This means that even if the price per million tokens remains unchanged, the actual cost to process the same text can vary significantly.
Comparison Between GPT and Claude
Let's take a concrete example: a TypeScript file. According to the measurements, Claude generates up to 1.73 times more tokens than GPT for the same file. In effective terms, this means the listed rates of $5/$25 for Opus 4.8 behave like $7.50/$37.50. For Sonnet 5, after its introductory window, this translates to behaving like $4.50/$22.50.
Impact on Billing
A model's billing is calculated by multiplying the number of tokens generated by the price per token. But due to the difference in tokenization, two models listing a price of $5.00 per million tokens can produce very different bills for the same text. This underscores the importance of not relying solely on listed prices to estimate costs.
Measurement Methodology
To obtain these figures, a series of tests was conducted. 16 types of files were used, ranging from English prose to TypeScript code, JSON schemas, and Chinese text. Each file was analyzed with the production tokenizers of the concerned models, and results were verified by real API calls under production conditions.
Conclusion and Implications
Understanding token dynamics is essential for optimizing the costs of using frontier models. Tech decision-makers must incorporate this dimension into their AI solution deployment strategy.
Let's discuss your project in 15 minutes.