1.4 Tokens: How Are They Counted, and Why Do Prompts in English Help Save a Little Money?

One token is not one letter. Although sometimes it is. A token is a unit of meaning in an LLM. It can be a whole word, part of a word, or a punctuation mark. Understanding how tokens work makes it possible to use a neural network more effectively. That is what we will talk about today.

What does a single token look like in English and Russian?

Neural networks do not read text like humans do. They break the entire text into parts using special algorithms called tokenizers.

What the text is broken down into:

• Word parts: the most frequently used words may consist of a single token, while rare words are split into syllables

• Punctuation and spaces: punctuation marks, spaces, and emojis are also counted as separate tokens

• Case sensitivity: The word "Hello" and the word "hello" can contain different numbers of tokens

Example of visual separation

• English phrase: AI is future - is recognized as 3 tokens ( AI, is, future).

• Russian phrase: ИИ - это будущее - may turn into 6–8 tokens ( И, И, -, э, то, буд, ущ, ее).

Using English can save a considerable number of tokens

If you write a prompt in English, you usually use 2–3 times fewer tokens to convey the same meaning. You can also load much more English-language material into the context than material in another language.

What language is best for writing prompts when coding?

Large language models are trained primarily on English-language texts—at least, LLMs have been fed more of those than anything else. And English is much simpler than other languages, such as Russian or Chinese. Modern reasoning models very often translate a prompt from your native language into English before providing an answer.

Ideally, it’s better to write prompts in English—then the LLM will understand your intentions more accurately, and there will be fewer translation difficulties.

However, there’s one very important nuance. If you don’t have a fluent, rich, technical command of English, it will be much easier and more effective for you to express your thoughts in your native language—you’ll be able to explain your request to the neural network much more clearly and precisely. The neural network will translate it into English, and you’ll get a more expected and predictable result. The LLM will spend more reasoning tokens, and the answer will take a little longer, but IMO, that’s a small price to pay for solving your task.