Token speed visualizer

What you are watching

Each burst of text is about one token. A token is a piece of a word, a short word, or a symbol. The slider sets how many of those pieces appear per second. The default code test writes one local-inference test function after another, and the stream continues indefinitely. The output follows the newest tokens automatically.

Code mode is syntax-colored source, and it is tokenized more densely than prose because symbols and short words stand alone. Text mode is a normal reply. Think mode alternates dim, italic reasoning with the answer. The think-length control sets how many reasoning tokens appear before the answer continues. Agent mode writes, calls a tool, pauses while that tool “runs,” then continues. The pause is wall-clock time, so a fast decode rate does not erase it.

Presets are feel-targets, not a measurement of a named server. A typical local 7B often sits near 10 to 30 tokens per second. Groq-class and extreme are here so a hosted rate is obvious next to hardware you might actually put in a building.

Reference rates

Tokens per second, and the kind of setup that usually feels like that.

tok/s	What it feels like
5	Constrained edge CPU. A short note is a wait.
10	Typical local 7B. Usable if one person is reading along.
20	Local 7B on a mid GPU. A paragraph finishes while you watch.
30	Comfortable local chat for a single user.
60	Fast local, or a small hosted model. The cursor stops being the focus.
100	Fast hosted decode.
200	High-end local GPU, or a quick hosted path.
400	Groq-class decode. A paragraph arrives almost at once.
800	Extreme. The limit of this demo’s presets. The reply is simply there.

Prefill and time to first token

Decode is the writing. Prefill is the reading that happens first. Time to first token is that read, plus any queue. A short question is cheap. A chart, a filing, or a repository is not.

The context control holds the stream until context tokens ÷ prefill speed have elapsed. Sixteen thousand tokens read at 4,000 tokens per second take four seconds before the first word. The default context is off, so the page starts immediately. Turn a context on when you want the silence.

On premises, both numbers belong to the machine in the building. A fast decode spec does not describe a 64k packet. Measure the packet you actually send.

Why this matters on premises

SAI runs on computers you own. Notes, filings, and chats stay on your network. They are not sent to a public cloud, and they are not sent back to us. The owner can turn outbound internet off. That privacy has a speed: the GPU or CPU in the room, shared by the people in the room.

A private 7B that writes at 20 tokens per second can be the right default for one clinician or one analyst. The same model feels wrong if ten people submit long contexts at once, or if an agent must call tools between every paragraph. A larger model is not automatically better if it decodes slower than the smaller one, or if prefill turns the first token into a long pause.

This visualizer does not connect to a model, and it does not bill anyone. Token boundaries are an approximation meant for feel. Prose breaks near four characters per token. Code is denser. It is not a specific model’s vocabulary, and it is not a purchasing spec. For how SAI is installed and kept on site, see private AI and how it stays on site. To put hardware in a building, request an on-site visit. All tools are listed on the tools index.

Developed by shahir.ai. The visualizer runs indefinitely until paused or cleared. A copied link stores the rate, mode, context, prefill speed, and think length. It does not store pasted text.

Token speed FAQ

What is tokens per second?

Tokens per second is how fast a model writes. A token is a piece of a word, a short word, or a symbol. At 10 tokens per second a paragraph arrives slowly. At a few hundred, the same paragraph is on screen almost at once.

What is prefill and time to first token?

Prefill is the work of reading the prompt before any output. Time to first token is that wait. A 16,000-token context read at 4,000 tokens per second takes about four seconds before the first word. The context control on this page simulates that from a context size and a prefill rate.

Why does token speed matter for local and private AI?

On-premises models run on hardware you own, so the pace depends on that machine. A private 7B model on a modest GPU often feels near 10 to 30 tokens per second. A long chart, filing, or codebase delays the first token even when decode is quick. Seeing the pace helps a team judge whether a local model will feel usable for notes, code, or tool-using agents.

Is my text sent to shahir.ai?

No. Tokenization, streaming, and anything you paste stay in the page. The shareable link records rate, mode, context, prefill speed, and think length only.

How accurate is the tokenization?

It is an approximation meant for feel, not a claim about one model’s vocabulary. Prose breaks near four characters per token. Code is denser, because operators and short words become their own tokens. Use it to compare speeds, not to bill a provider.