Local LLMs on a Raspberry Pi 5: Honest Numbers & Setup

The Exact Symptom

Related: Midjourney vs DALL-E vs Flux: A Practical Image-Tool Comparison Framework →

Related: [Mac Mini M4 Local LLM Benchmarks: Real Tokens/Sec →](/article/local-llm-mac-mini-m4-real-benchmarks)

You bought a Raspberry Pi 5, flashed a fresh OS, installed Ollama, and told it to run Llama 3. The console paused for about thirty seconds, the board got incredibly hot, and then you were instantly kicked back to the command line with this exact error string: llama runner process has terminated: signal: killed.

Or maybe it did load, but you are watching the text crawl across your terminal at half a word per second, wondering if you did something wrong or if the hardware is just this slow.

This guide is for developers and hobbyists who want a dedicated, offline text generation node on their network. I will give you the actual limitations of running an LLM on a single-board computer, the exact configurations required to prevent memory crashes, and the honest tokens-per-second numbers you can expect. There is no magic here. We are bound by the physics of ARM CPUs and unified memory architecture. Let us get into the real numbers.

My Hardware and OS Setup

Related: 9 Free AI Coding Tools Every Developer Should Try in 2026 →

Related: Local Embedding Models Compared: Nomic, BGE, E5, mxbai →

You cannot run large language models on the base model Raspberry Pi without significant pain. My specific test bench is a Raspberry Pi 5 with 8GB of RAM. The 4GB model will technically boot tiny models, but you will hit swap memory immediately, reducing generation speeds to a crawl. If you need a refresher on memory constraints, check out this guide on running a local LLM on 8GB of RAM to understand how tight this ceiling actually is.

Cooling is strictly mandatory. I am using the official Raspberry Pi Active Cooler. If you try to run an LLM bare-board, the CPU will hit 85 Celsius in under twenty seconds and heavily throttle your clock speeds. I am also using the official 27W USB-C Power Supply. Standard phone chargers will trigger a low-voltage warning under full multi-core load.

For the operating system, I am running Raspberry Pi OS Lite 64-bit. I chose the Lite version because the desktop environment wastes precious RAM that we need for the model weights. The OS is installed on a fast SanDisk Extreme Pro MicroSD card. An NVMe hat would speed up model loading times significantly, but it has exactly zero impact on tokens-per-second once the model is loaded into RAM.

Installing Ollama on ARM64

Related: Sora 2 Review: OpenAI's Video Model Is Finally Useful for Real Work →

Related: Q4 vs Q5 vs Q8 Quantization: Which GGUF to Actually Download →

There are several ways to run inference on edge devices. If you are comparing runners on a desktop PC, you might look at Ollama vs LM Studio vs Jan. On the Raspberry Pi, your choice is basically made for you: stick with Ollama. It runs as a lightweight headless daemon, has native ARM64 support out of the box, and compiles the llama.cpp ARM NEON optimizations perfectly for the Pi's Cortex-A76 cores.

Installation is exactly one command. SSH into your Pi and run this:

curl -fsSL https://ollama.com/install.sh | sh

This script detects your architecture, downloads the correct binaries, and creates a systemd service automatically. By default, Ollama binds to localhost. If you are running the Pi headless and want to send API calls to it from your main PC, you need to expose the port.

Open the systemd service override file:

sudo systemctl edit ollama

Add these exact lines in the empty file that opens:

[Service]
Environment="OLLAMA_HOST=0.0.0.0"

Save the file, reload the daemon, and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Benchmarks and Real Tokens Per Second

Related: DeepSeek-R1 Repeats Itself or Outputs Gibberish: The 4 Settings That Fixed It →

To get accurate benchmarks, we cannot rely on the eyeball test. I measured these numbers by querying the Ollama API with a standard 500-word essay prompt and forcing it to return a single JSON payload instead of streaming the text.

Here is the exact curl command I used from my desktop to benchmark the Pi:

curl http://192.168.1.100:11434/api/generate -d '{
  "model": "phi3:mini",
  "prompt": "Write a 500 word essay about the history of the CPU.",
  "stream": false
}'

When it finishes, the JSON response includes eval_count (the number of tokens generated) and eval_duration (the time taken in nanoseconds). I calculated the speeds below by dividing the token count by the duration in seconds.

Are these numbers good? It depends entirely on your use case.

Llama 3 at 2.8 tokens per second is slightly slower than a fast reader reads. You will not want to use it as a real-time chatbot, but for background batch processing like summarizing cron job logs, it is highly reliable. Phi-3 Mini at 6.4 tokens per second is actually quite pleasant to watch stream into a terminal, and Qwen 0.5B feels instantaneous.

The bottleneck here is not just the CPU cores. It is memory bandwidth. The LPDDR4X RAM on the Pi 5 peaks at around 34 GB/s. To generate a single token, the system has to read the entire model from RAM into the CPU registers. An 8B model takes about 4.7 GB of space. Even with perfect overhead, 34 divided by 4.7 gives a theoretical maximum of around 7 tokens per second. We get 2.8 because of system overhead and OS background tasks.

Fixing the Out of Memory Error

Related: DeepSeek-R1 Shows Its <think> Tags in the Output — Here Is How I Strip Them →

If you attempt to run anything larger than 3B parameters, you will likely encounter the Linux OOM (Out of Memory) killer terminating your process. Unlike PC builds where you hunt for CUDA out of memory fixes to manage dedicated VRAM, the Pi 5 uses unified memory. The CPU and the GPU share the same 8GB pool.

To maximize what is available to the LLM, we need to completely starve the Pi's GPU. We are running headless anyway.

Open your boot configuration file:

sudo nano /boot/firmware/config.txt

Scroll to the bottom and add or modify this line:

gpu_mem=16

This restricts the GPU to just 16MB of RAM, leaving the rest for the operating system and Ollama. Save and exit.

Next, we must increase the system swap space. When Ollama loads a model, Linux will occasionally try to page out idle background processes to swap to make room for the model weights. The default 100MB swap file on Raspberry Pi OS is too small.

Open the swap configuration file:

sudo nano /etc/dphys-swapfile

Change the swap size from 100 to 4096 (which creates a 4GB swap file):

CONF_SWAPSIZE=4096

Restart the swap service to apply the change immediately:

sudo systemctl restart dphys-swapfile

Reboot the Pi. Your llama runner process has terminated: signal: killed errors will stop for any 7B or 8B model quantized at Q4 or below.

Managing Thermals So You Do Not Melt It

Related: How to Run DeepSeek-R1 Offline on Mac mini M4 (Step-by-Step 2026 Guide) →

Running an LLM pegs all four Cortex-A76 cores at 100% utilization. If you are running an inference job that takes five minutes, the heat generation is severe.

To monitor your temperatures while the model runs, open a second SSH session and use this command:

watch -n 1 vcgencmd measure_temp

With the official Active Cooler installed, idle temperatures sit around 42C. The moment I send a prompt to Llama 3, the fan audibly ramps up to maximum RPM. Under sustained load, the temperature stabilizes at exactly 64C. This is perfectly safe. The Pi 5 does not begin soft thermal throttling until 80C, and it hits hard throttling at 85C.

If you are using a passive heatsink case, be warned. I tested a standard aluminum block case. The Pi hit 80C within forty seconds of generation, dropped its clock speeds from 2.4GHz down to 1.5GHz, and my tokens per second fell by 40%. You must have a fan.

What Did NOT Work

I spent a lot of time trying to optimize this setup, and several heavily recommended internet solutions failed completely.

First, running a web frontend natively on the Pi is a bad idea. I installed Docker and pulled Open WebUI so I could have a nice ChatGPT-style interface on my local network. The background Node processes and Docker daemon consumed nearly 1.5GB of RAM. When I tried to load Llama 3, the Pi immediately locked up and crashed.

If you want a GUI, host the web UI on a different machine and point it at the Pi. If you attempt this split setup and the UI fails to connect, review the steps for when Open WebUI in Docker cannot reach Ollama. Usually, it is because you forgot to set the OLLAMA_HOST variable mentioned earlier.

Second, multi-modal models (vision models) are practically unusable. I loaded LLaVA 1.5 to try and analyze a simple 800x600 JPG image. The model loaded, but evaluating the image prompt took the CPU over four minutes before it even began generating text. Vision encoders are heavily optimized for matrix multiplication on GPUs, and the Pi simply lacks the architecture to handle image embeddings in any reasonable timeframe.

Finally, attempting to run anything larger than an 8B model is a waste of time. I tried downloading the severely compressed Llama 3 70B IQ1_S (a 1-bit quantization). It is roughly 18GB on disk. Even with a massive swap file configured on an NVMe drive, the generation speed was measured in seconds-per-token, not tokens-per-second. Stick to 8B parameters and below.

FAQ

Can I use a Raspberry Pi 4 instead?

Yes, but do not expect usability. The Pi 4 memory bandwidth and older CPU cores bottleneck generation speeds to less than one token per second for 7B models. It is a novelty, not a practical tool.

Does the Raspberry Pi AI Kit (Hailo-8L) help with LLMs?

No. The Hailo-8L NPU is designed for vision tasks like object detection in video feeds. It does not have the VRAM or the architecture required to accelerate large language models. LLMs run strictly on the CPU.

Why does the first prompt take so long to start?

Ollama unloads models from RAM after 5 minutes of inactivity by default. Your first prompt forces the Pi to read a 4GB file from the SD card into memory. Subsequent prompts start much faster.

Is an NVMe SSD required for this to work?

No. An NVMe SSD will drastically reduce the initial model load time compared to a MicroSD card, but once the model is sitting in system RAM, generation speeds are identical on both storage mediums.