Introduction
In the realm of large language models (LLMs), performance is often measured in terms of tokens per second (tok/s). This metric is crucial for evaluating how quickly a model can generate content, yet it is seldom understood outside technical circles. So, what does it really mean when a model operates at 10, 100, or 800 tok/s? Let's break it down.
Understanding the Concept of a "Token"
Before diving into token speed, it is essential to understand what a "token" is. Tokens are units of data that language models use to process textual input. They can be whole words, chunks of words, or even individual characters, depending on the tokenization used. For example, the word "processUserInput" might be broken into three tokens: "process", "User", and "Input". On average, in English, a sentence consists of about 1.3 tokens per word.
Speed in Context
Imagine working on a Raspberry Pi generating 5 tok/s. This model might seem slow, especially compared to a typically hosted model capable of 60 tok/s, or even Groq models reaching 200 tok/s. But what do these numbers represent? At 5 tok/s, you'll get about 3.8 words per second, which is sufficient for simple applications or basic text generation tasks.
At the other end, Cerebras-class systems with 800 tok/s can produce responses at a speed where the main limitation becomes the user's ability to keep up with what is being generated. For developers and decision-makers, understanding these performances allows for evaluating the suitability of LLMs for specific applications.
Differentiation by Content Type
The type of content being generated heavily influences the perception of token speed. For instance, code is generally more token-dense than prose. This means that a given tok/s rate can feel faster or slower depending on whether it is text or code. In a coding environment, 30 tok/s might feel swift, whereas for narrative text, it could be perceived as slower.
Benchmarks vs. Reality
Token speed benchmarks, such as "47 tok/s on an M3" or "180 tok/s on a 4090," are often cited in technical discussions. However, these figures might not fully reflect the real user experience. For example, a demonstration at 30 tok/s might feel fast in a text context but slow in a coding context.
Conclusion
Ultimately, understanding tokens per second speed requires putting these numbers in perspective with the content type and user application. Decision-makers and developers can thus better align the capabilities of LLMs with their specific needs.
Let's discuss your project in 15 minutes.