What is Ollama? Your Toolkit for Local AI
Reading time: approx. 7 min
In the previous lesson, we saw the advantages of running AI locally. Now we take the practical step and get to know Ollama, a fantastic tool that greatly simplifies the process of downloading, installing, and running large language models on your own computer. With Ollama, a complicated technical process is transformed into a single simple command.
What You Will Learn
- What Ollama is and how it works.
- How to install Ollama.
- The basic commands for downloading and running an AI model.
The Basics: What Does Ollama Do?
Think of Ollama as an "app store" and "startup assistant" for local AI models. Without Ollama, you would need to download model files manually, configure complex settings, and manage Python environments.
Ollama packages all of this for you. When you ask Ollama to run a model, it automatically handles:
- Download: Retrieves the correct model file from a secure library.
- Configuration: Adjusts the settings for your specific computer hardware.
- Execution: Starts the model and gives you a simple chat prompt to interact with.
Hardware Context: The tests in this course were performed on a computer with Ubuntu (a Linux operating system) and an NVIDIA RTX 4070 graphics card with 8GB VRAM. This is a solid mid-range solution. Performance on your computer may vary, but many smaller models work excellently even on computers without such a powerful graphics card, including Mac computers with Apple Silicon (M1/M2/M3).
The Installation Process: Step by Step
Ollama is available for Linux, macOS, and Windows. Installation is surprisingly simple and takes only a few minutes.
For Windows and macOS: Go to ollama.ai and download the installation file for your operating system. Installation occurs via a standard installer, just like any other program.
For Linux (Ubuntu/Debian): Open a terminal and run the command:
curl -fsSL https://ollama.ai/install.sh | sh
After installation, you can verify that everything works by typing ollama in the terminal. You should then see a list of available commands.
Practical Application: Your First Steps with Ollama
Once the program is installed, you manage everything via a terminal or command window. For Windows users, use "Command Prompt" or "PowerShell".
Step 1: Download a Model You choose a model from the Ollama library. We will test several in this course. To download a model, type:
ollama pull <modelname>
For example, to download the Llama 3.1 model (8 billion parameters):
ollama pull llama3.1:8b
Step 2: Run the Model Once the model is downloaded, you start a conversation with the command:
ollama run <modelname>
For example:
ollama run llama3.1:8b
After a short moment, a prompt >>> will appear in the terminal. Now you can type your questions directly to the AI model. Everything you write and all responses stay on your computer.
Step 3: Exit When you are done, type /bye to end the conversation.
Practical Tips for Beginners
Model Sizes and Performance: Models come in different sizes, usually labeled as 7b, 8b, 12b (billion parameters). Larger models provide better quality but require more resources. As a rule of thumb:
- 7-8b models: Work on most modern computers
- 12b+ models: Require at least 16GB RAM or a powerful graphics card (however, we run Gemma3:12b on 8GB VRAM without problems in this course)
The First Time You Run a Model: The download can take time depending on your internet connection. An 8b model is approximately 4-5GB in size. Therefore, plan your first installation when you have plenty of time and stable internet.
Useful Commands:
ollama list- Show installed modelsollama rm <modelname>- Remove a model to save disk spaceollama show <modelname>- Display information about a model
Why the Terminal?
Although there are graphical interfaces built on Ollama, the terminal gives you the most direct and powerful control. It is also the fastest way to switch between different models to see which fits best for a specific task. Think of it as a direct line to the AI engine.
If you have never used the terminal before, it may seem cumbersome at first, but after a few uses it becomes an invaluable tool. Once you get used to the terminal, you will prefer it because it is both simple and fast and gives you full control over what happens.
Next Steps
Now that you know how to use Ollama, it is time for the fun part: seeing what the different models can do! In the next lesson, we begin our benchmark review by looking at Gemma3, a model that has proven to be exceptionally good for pedagogical and linguistic tasks.