How to Run AI Models Locally with Ollama: A Beginner's Guide

To run AI models locally, install Ollama, download (“pull”) an open-weight model, and start chatting with it from your terminal. The whole process takes a few minutes, and after the model is downloaded it runs on your own computer without sending your prompts to a cloud service.

This guide walks through each step, explains how to pick a model your machine can actually handle, and shows how other apps can use the model once it is running.

What Ollama is (and why people use it)

Ollama is a free tool that runs open-weight language models on your own computer. It handles the hard parts for you: downloading model files, loading them into memory, and using your graphics hardware when it can.

People use it for a few common reasons:

• Privacy. Your prompts and files stay on your machine.
• No per-request bill. Once a model is downloaded, using it costs only electricity.
• Offline work. It keeps working on a plane or a bad connection.
• Experimentation. You can try many different open models side by side.

If you are still deciding whether local is right for you, our comparison of local vs cloud AI models covers the trade-offs in detail.

What you need before you start

You do not need a special AI workstation, but memory matters more than anything else.

• A Mac, Windows PC or Linux machine. Ollama supports all three.
• Enough memory (RAM). The whole model has to fit in memory while it runs. More memory means you can run larger, smarter models.
• Free disk space. Model files range from a couple of gigabytes for small models to many tens of gigabytes for large ones.
• A terminal. You will type a few short commands. No programming is required.

On Apple silicon Macs, memory is shared between the processor and graphics, which makes them surprisingly good at running local models for their size.

Step 1: Install Ollama

1. Go to the official Ollama website and download the installer for your operating system.
2. Run the installer like any other app. On a Mac, drag it into your Applications folder and open it.
3. Open a terminal and type ollama to confirm it is installed. You should see a list of available commands.

Ollama runs a small background service once it is open. That service is what loads models and answers requests.

Step 2: Pick a model

Ollama has a public model library with many open-weight model families, such as Llama, Qwen, Gemma and Mistral. Most families come in several sizes, usually described by their number of parameters (for example, “small”, “medium” and “large” variants).

The single most important rule: pick a small model first if your Mac has less memory. A small model that runs smoothly is far more useful than a large one that crawls or fails to load.

Here is a rough way to think about sizes:

Model size	Good for	What to expect
Small	Laptops with modest memory, quick questions, simple edits	Fast, but makes more mistakes on hard tasks
Medium	Machines with more memory, everyday writing and coding help	A good balance of quality and speed
Large	High-memory workstations, harder reasoning tasks	Best quality locally, but slow and memory-hungry

Each model page in the library lists its available sizes and the approximate download size. As a rule of thumb, the download size gives you a hint of how much memory the model needs, plus some extra headroom for your other apps.

Step 3: Pull the model

“Pulling” downloads the model to your computer. In your terminal, run:

ollama pull <model-name>

Replace <model-name> with the name shown in the library, including the size tag if you want a specific variant. The download can take a while for bigger models, and you only need to do it once.

To see which models you have downloaded, run:

ollama list

Step 4: Run it and start chatting

Now start an interactive chat:

ollama run <model-name>

You will get a prompt where you can type questions and read the answers as they stream in. Type /bye to exit.

The first response may take a moment while the model loads into memory. Later responses are usually faster because the model stays loaded for a short time.

A few tips for good results:

• Keep prompts clear and specific, the same way you would with a cloud assistant.
• Small models do better with shorter, focused tasks than with long, multi-part requests.
• If answers are slow, close memory-heavy apps or switch to a smaller variant.

Step 5: Use the model from other apps

Ollama is not just a chat window. While it is running, it exposes a local API on your computer, so other tools can send it prompts. Many editors, note-taking apps and AI agent tools have an “Ollama” or “local model” option that connects to it.

This is where local models become really useful. You can point a coding agent or a writing tool at your local model and keep the whole workflow on your machine.

For example, BotQuarters lets you put teammates on local models through Ollama alongside cloud-based agents in the same team, so you can keep simpler tasks local and send harder ones to a stronger model.

Managing models and disk space

Models take up real space, so it is worth tidying up now and then.

• Remove a model you no longer use with ollama rm <model-name>.
• Update a model by pulling it again; Ollama downloads only what changed.
• Check what is loaded with ollama ps, which shows models currently in memory.

If you experiment a lot, keep one small “daily driver” model and delete the rest after testing.

Common problems and quick fixes

The model is very slow

This almost always means the model is too large for your memory, so your computer is swapping to disk. Try a smaller size of the same model family.

The model fails to load

Check free memory and disk space. Quit other heavy apps, then try again with a smaller variant.

Answers are poor or off-topic

Small models have limits. Give more context, break the task into smaller steps, or try a different model family that is better suited to your task, such as one tuned for code.

Long documents get cut off

Every model has a context window, which is how much text it can consider at once. Very long inputs may exceed it. Split the document into sections, or look for a model and settings with a larger context.

When local models are (and are not) a good fit

Local models are excellent for private notes, quick drafts, summarizing your own files, and small coding tasks. They are also great for learning how language models behave.

They are less ideal for long, complex agent work, like refactoring a big codebase or planning a multi-step project. There, larger cloud models still tend to be more reliable and much faster on typical hardware. Many people use both: local for private and routine tasks, cloud for heavy lifting. If you are planning that kind of mixed setup, our guide to AI agent costs explains how to decide where each task should run.

FAQ

Is Ollama free to use?

Yes, Ollama itself is free and open source. The models in its library are open-weight models that you can download at no cost, though each model has its own license that you should check for commercial use.

Can I run AI models locally on a Mac with limited memory?

Yes, as long as you pick a small model first. Small models run well on modest machines, and you can move up to a medium size later if you have memory to spare.

Do local AI models need an internet connection?

You need a connection to download Ollama and pull models. After that, the model runs entirely on your computer and works offline.

Can coding agents use Ollama models?

Many agent tools can connect to Ollama’s local API. Results depend heavily on model size: small local models can handle simple, well-scoped tasks, but larger models are usually needed for complex coding work. Our AI coding agents guide covers how to match tools to tasks.