Running Two Local Models on One GPU Without Crashing

My local stack needs two models running at the same time: a chat model for writing and reviewing code, and a small embedding model that indexes my notes for retrieval. For a week I ran them on one 16 GB card and got a crash roughly every second query — usually the embedding call dying with an out-of-memory error mid-index.

The fix was not a bigger card. It was doing the arithmetic once and then telling Ollama the limits instead of letting it improvise.

First, know what each model actually costs

Related: Why Ollama's First Response Is Slow (Cold Start Fix) →

VRAM usage is not just the file size on disk. Three things are competing for the card:

1. Model weights. Roughly the size of the GGUF file. An 8B Q4_K_M is about 4.7 GB; a 14B Q4_K_M is about 8.5 GB.
2. KV cache. Scales with context length, layers, and heads. On an 8B model, expect very roughly 0.5 GB per 4K of context at fp16 cache.
3. Compute buffers and overhead. Several hundred megabytes, plus whatever your desktop environment already holds.

On Linux, check what is already gone before you start:

nvidia-smi --query-gpu=memory.total,memory.used --format=csv

My desktop session alone was holding 900 MB. That matters when you are trying to fit two models into a fixed budget.

The budget that worked on 16 GB

Related: Ollama Not Using My NVIDIA GPU in WSL2: The Fix That Finally Worked →

Here is the allocation I settled on, measured with nvidia-smi while both models were resident:

That headroom is not waste. When a long prompt arrives, prefill temporarily needs more than steady state, and if there is nothing spare the allocation fails. I aim to leave at least 20 percent of the card free. Every crash I logged happened when steady-state usage was above 85 percent.

Tell Ollama how many models it may keep

Related: Ollama Running Slow? 7 Fixes That Actually Worked on My Machine →

By default Ollama will load models on demand and evict them under pressure, which is exactly the thrashing you do not want when one of them is being called on every keystroke.

Two variables control this:

OLLAMA_MAX_LOADED_MODELS=2
OLLAMA_NUM_PARALLEL=1

OLLAMA_MAX_LOADED_MODELS caps how many models stay resident together. Set it to the number you genuinely need — mine is 2. OLLAMA_NUM_PARALLEL controls concurrent requests per model, and each parallel slot allocates its own KV cache. Leaving it at 4 on a tight card silently multiplies your cache cost by four. For a single-user desktop, 1 is correct.

Set them in the systemd override so they persist:

sudo systemctl edit ollama

[Service]
Environment="OLLAMA_MAX_LOADED_MODELS=2"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_KEEP_ALIVE=-1"

keep_alive of -1 matters here: without it the embedding model unloads between indexing runs and the reload competes with the chat model for memory at the worst possible moment.

Pin the context window per model

Related: Ollama vs LM Studio vs Jan: I Used All Three for a Month →

This is where most of my savings came from. The chat model does not need a 32K window for reviewing a function, and the embedding model needs almost none.

Create a Modelfile per role:

FROM qwen2.5-coder:7b
PARAMETER num_ctx 8192
PARAMETER num_gpu 99

ollama create coder-8k -f Modelfile

Dropping the chat model from 32768 to 8192 tokens freed 3.4 GB on my card — more than the entire embedding model costs. If you routinely paste whole files, try 16384 before you reach for a bigger GPU. There is more on how these parameters behave in building custom Ollama models with a Modelfile.

Choose a small embedding model deliberately

Related: How to Run a Local LLM on 8GB of RAM (What Actually Works in 2026) →

People often pair a 7B chat model with an embedding model that is far larger than the job requires. For document retrieval over personal notes, a compact embedding model is both faster and dramatically cheaper in memory:

I use nomic-embed-text. The quality difference against the larger option on my own corpus was small enough that it did not justify the memory. We compared these properly in the best local embedding models.

What to do when it still will not fit

Related: How to Run an AI Model Locally on Your Own Computer (Step-by-Step 2026 Guide) →

If the arithmetic says no, you have four honest options, roughly in order of how much I like them:

Drop a quantisation level. Moving the chat model from Q5_K_M to Q4_K_M saved 1.1 GB with a difference I could not reliably detect in code review.

Run the embedding model on CPU. Embeddings are short, batched, and latency-tolerant. Forcing them off the GPU frees the card entirely for chat:

FROM nomic-embed-text
PARAMETER num_gpu 0

On my Ryzen, indexing 400 notes on CPU took 90 seconds instead of 35. Perfectly acceptable for a background job.

Partially offload the chat model. Set num_gpu to a layer count below the model total and the remainder runs on CPU. This is a real slowdown — around 40 percent fewer tokens per second on my test — so treat it as a last resort rather than a tuning knob.

Use a smaller chat model. A well-chosen 3B for autocomplete plus a 7B invoked only for harder questions often beats one mid-size model doing both jobs badly.

Verifying it actually holds

Once configured, load both and watch the card under real load:

ollama run coder-8k "summarise this repo layout" &
curl -s http://localhost:11434/api/embeddings -d '{"model":"nomic-embed-text","prompt":"test"}' > /dev/null
ollama ps
nvidia-smi --query-gpu=memory.used --format=csv -l 1

You want two rows in ollama ps, both showing 100% GPU, and peak memory that stays under about 85 percent of the card during a long prompt. If a model reports a CPU percentage, it did not fit and Ollama silently split it — that is your warning sign, not the crash that comes later.

I left this running for three weeks with no out-of-memory failures. The card was never the problem; the default settings were. If you are hitting hard OOM errors even with one model, the causes are slightly different and covered in fixing CUDA out of memory on local LLMs, and if you are picking hardware in the first place, the budget GPU guide has the current numbers.