Ollama Not Using My NVIDIA GPU in WSL2: The Fix That Finally Worked

Ollama Not Using My NVIDIA GPU in WSL2: The Fix That Finally Worked

Symptom: generation runs at 3 tokens/sec, the fans stay quiet, and ollama serve logs "no compatible GPUs were discovered" — while nvidia-smi inside WSL2 happily prints your card. I lost most of a Saturday to this. The cause was never the GPU.

Hardware: RTX 3060 12 GB, Windows 11 23H2, WSL2 Ubuntu 22.04, Ollama 0.6.x.

Step 1 — Confirm what Ollama actually sees

Related: Midjourney vs DALL-E vs Flux: A Practical Image-Tool Comparison Framework →

Related: Ollama "connection refused on 127.0.0.1:11434": The 5 Causes I Have Actually Hit →

ollama ps

If the PROCESSOR column says 100% CPU, the model is not on the GPU. Then read the server log, which names the real reason:

journalctl -u ollama -n 100 --no-pager | grep -i -E "gpu|cuda|library"

Step 2 — Install the driver on Windows, never inside WSL2

Related: 9 Free AI Coding Tools Every Developer Should Try in 2026 →

Related: Ollama Pull Fails with "max retries exceeded" or EOF: How I Get Downloads to Finish →

The single biggest mistake. Installing a Linux NVIDIA driver inside WSL2 breaks the passthrough. Only the Windows driver is needed; WSL2 receives /usr/lib/wsl/lib/libcuda.so.

ls -l /usr/lib/wsl/lib/libcuda.so*
nvidia-smi

If libcuda.so is missing, update the Windows driver and run wsl --shutdown.

Step 3 — Install the CUDA toolkit for WSL

Related: Sora 2 Review: OpenAI's Video Model Is Finally Useful for Real Work →

Related: Ollama Filled My Disk: How I Moved the Models Directory Safely →

wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update && sudo apt install -y cuda-toolkit-12-4

Step 4 — Reinstall Ollama after CUDA exists

Related: How to Clone Your Voice with AI in 2026 (Free and Paid Options) →

Related: Ollama Error "model requires more system memory": How I Fixed It in 10 Minutes →

Ollama picks its backend at install time. Mine had installed the CPU-only build before CUDA was present, and nothing short of reinstalling changed that:

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl restart ollama

That was the actual fix for me. ollama ps flipped to 100% GPU and 8B q4_K_M went from 3.1 to 41 tokens/sec.

Step 5 — Force full offload if it is still split

Related: Ollama Answers Get Cut Off Mid-Sentence: num_ctx vs num_predict Explained →

/set parameter num_gpu 99      # inside the REPL

A split like 35% CPU / 65% GPU means the model does not fit in VRAM. Drop the quantization or num_ctx rather than fighting it — details in CUDA out of memory fixes.

My working numbers

FAQ

Do I need Docker or the container toolkit? Only if you run Ollama in a container. Bare-metal WSL2 does not need nvidia-container-toolkit.

AMD GPU in WSL2? ROCm passthrough is still unreliable there; I run AMD cards on native Linux instead.

Still slow on GPU? Context size and keep-alive dominate after the offload is fixed — see Ollama running slow and the local LLM guide, plus the broader AI tools guide.