Tokens: AI Language Building Blocks

Reading time: approx. 5 min

In this moment, we build on your knowledge from "Fundamentals of AI Thinking" by digging deeper into tokens. Tokens are the smallest units that AI models use to process, understand, and generate text. Understanding tokens is the key to being able to communicate effectively with AI and get the answers you want.

1. What is a token?

A token is the basic building block in AI language processing. It can represent different parts of text:

• A whole word: For example "school" or "student".
• A part of a word: Like "un-" in "understanding", "pre-" in "predict", or "co-" in "cooperation".
• Punctuation: Period (.), comma (,), exclamation mark (!).
• Spaces: Even spaces can be counted as their own token by some models.

When you write a prompt, the AI model breaks down your text into tokens and assigns each token a unique numeric ID. It is these IDs that the model then processes internally. This system makes it possible for the AI to handle all languages and even unusual words by breaking them down into known, smaller parts.

2. Tokenization in practice

Let us look at an example to visualize how text is converted into tokens:

Example: The sentence "English teachers can use AI in their teaching."

can be divided into tokens roughly like this:

• English
• teachers
• can
• use
• AI
• in
• their
• teaching
• .

In this example, it becomes a total of 9 tokens (excluding possible handling of spaces). Important to note is that different AI models use different "tokenizers", which means that exactly how a sentence is divided and how many tokens it results in can vary somewhat between different AI services (for example, between ChatGPT, Gemini, or Claude).

3. How AI models process tokens

Before we talk about practical consequences, let us understand what happens when the AI processes tokens. When you send a prompt to an AI model:

1. Tokenization: Your text is broken down into tokens (as we saw above)
2. Probability calculation: For each position in the answer, the model calculates the probability of which token should come next
3. Token selection: The model chooses the next token based on these probabilities (this is affected by settings like temperature, which we will learn more about in upcoming moments)
4. Repetition: The process repeats for each new token until the answer is complete

This means that each token that is generated affects the choice of the next token, which explains why AI can build coherent and logical answers.

4. Why token counting is important

Keeping track of tokens is not just a technical detail, it has direct practical consequences for how you use AI:

1. Cost and performance: More tokens require more computing power and energy. This means that prompts with many tokens can take longer to process, consume more energy, and, if you use commercial APIs, they can also cost significantly more. For schools that use AI services, it is therefore important to teach students to formulate efficient, concise prompts.
2. Context window: Each AI model has a limited "memory capacity", called a context window. This is the maximum number of tokens that the model can "remember" and process simultaneously in a conversation. If your prompt, plus any previous chat history, exceeds this window, the model will "forget" the oldest parts of the text. This leads to it not being able to base its answer on all the information you gave it.
3. Precision in the prompt: Since the model's "memory" is limited, it is crucial that you place the most important information in your prompt where it has the greatest chance of being noticed. Avoid unnecessary introductions or "fluff" that take up valuable tokens at the beginning of the prompt.

5. Interactive token tools

To get a better feel for tokens, you can try interactive tools online. Many AI developers and platforms offer so-called "tokenizer playgrounds" where you can:

• Paste your text.
• See exactly how it is broken down into tokens.
• Get an exact number of how many tokens your text consists of.
• Experiment with shortening or restructuring your prompts to see how it affects the number of tokens. An example of such a tool is OpenAI's Tokenizer.

6. Practical tips

Understanding tokens helps you formulate more effective prompts:

• Use token counters: Teach yourself and your students to use online tools to count tokens. This helps you understand how "long" your prompts actually are in the AI's eyes and to optimize them.
• Cut out "fluff": Encourage students to be direct and concise. Long, vague sentences with unnecessary information only take up tokens without adding value. Short, clear instructions often give more precise AI answers.
• Iterate and test: It is part of the process to experiment! Encourage students to test several different formulations of a prompt. Sometimes a small rephrasing can drastically reduce the number of tokens and lead to a better result.

7. Reflection exercise

To apply this knowledge in practice, consider the following:

1. Choose a prompt you have used recently with an AI model. Use an online tool to count the number of tokens. How many were there?
2. Can you rewrite the same prompt so that it uses 20% fewer tokens, without it losing its original meaning or instruction? How did you shorten it?
3. Discuss with a colleague: How would you best introduce the concept of "tokens" to your students so they understand what it means for AI communication?

Next moment: Context window: Why the model forgets - now that we have understood what tokens are, we will explore how many tokens the AI model can handle simultaneously (its "memory") and how you can effectively work within these limits to get consistent and relevant answers.