CUDA Out of Memory: 9 Fixes That Actually Worked on My GPU

The error looks like this:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.50 GiB
(GPU 0; 8.00 GiB total capacity; 6.21 GiB already allocated;
 512.00 MiB free; 6.85 GiB reserved in total by PyTorch)

Read it properly before changing anything. Already allocated is your model weights plus activations. Reserved minus allocated is fragmentation PyTorch is holding but not using. If reserved is much bigger than allocated, fix 6 is your problem, not the model size.

Fix 1: cut the context window (biggest single win)

Related: GPT-5 Is Here: Everything You Need to Know About OpenAI's Most Powerful Model Yet →

Related: How to Run a Local LLM on 8GB of RAM (What Actually Works in 2026) →

KV cache grows linearly with context length and it is usually larger than people expect. Halving context from 8192 to 4096 freed 1.8GB on a 7B model for me.

# llama.cpp / Ollama
ollama run mymodel
/set parameter num_ctx 4096

# transformers
model.generate(..., max_length=2048)

Fix 2: load in 4-bit

Related: Will AI Coding Agents Replace Developers? We Asked 100 Engineers →

Related: Ollama vs LM Studio vs Jan: I Used All Three for a Month →

A 7B model is ~14GB in fp16 and ~4GB in 4-bit. This is the difference between "does not fit" and "plenty of room".

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
cfg = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype="float16")
model = AutoModelForCausalLM.from_pretrained(name, quantization_config=cfg, device_map="auto")

Quality cost is real but small for chat and writing; it is more noticeable for maths.

Fix 3: find out what is really using your VRAM

Related: The 27 Best AI Tools in 2026 (Tested for 90 Days) →

Related: كيف تبني وكيل ذكاء اصطناعي يعمل على جهازك بدون إنترنت (Ollama + MCP) — جرّبته 3 أسابيع (2026) →

Half the time the culprit is a zombie process from a crashed run, or your desktop compositor.

nvidia-smi
# or continuously:
watch -n1 nvidia-smi

Kill the leftovers:

nvidia-smi --query-compute-apps=pid,used_memory --format=csv
kill -9 <pid>

On a headless server, that alone reclaimed 1.2GB for me once.

Fix 4: reduce batch size / image resolution

Related: ChatGPT vs Claude 4: Which AI Should You Actually Pay For in 2026? →

Related: NVIDIA Blackwell Ultra: What It Means for AI Startups in 2026 →

For diffusion models this is the equivalent of fix 1. Batch of 4 at 1024×1024 is roughly four times the memory of batch 1. Generate sequentially and stop fighting it.

Fix 5: offload layers to CPU

Related: Google Gemini 3 Ultra Review: Has Google Finally Caught Up? →

Related: Nvidia's Next Move: What the Rubin Roadmap Means for AI Buyers Right Now →

Partial offload is much better than nothing. In llama.cpp-based tools:

# put 20 of 32 layers on the GPU, rest on CPU
./llama-cli -m model.gguf -ngl 20

Tune -ngl down until it loads. Each layer you move to CPU costs speed but buys VRAM.

Fix 6: fix fragmentation

Related: Midjourney vs DALL-E 4 vs Flux 1.1: The Definitive AI Image Generator Comparison →

Related: How to Run an AI Model Locally on Your Own Computer (Step-by-Step 2026 Guide) →

If reserved is far above allocated:

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

And inside long-running loops:

import torch, gc
del outputs
gc.collect()
torch.cuda.empty_cache()

empty_cache() alone does not help if you still hold a reference to the tensor — the del matters.

Fix 7: gradient checkpointing (training only)

Related: 9 Free AI Coding Tools Every Developer Should Try in 2026 →

model.gradient_checkpointing_enable()

Roughly 30–40% memory saved for about 20% slower training. For fine-tuning on consumer GPUs this is not optional.

Fix 8: turn off other GPU consumers

Related: Sora 2 Review: OpenAI's Video Model Is Finally Useful for Real Work →

Browser hardware acceleration, video calls, and the desktop environment can hold 500MB–1.5GB. On Linux, dropping to a TTY freed 900MB on my 8GB card. On Windows, disable hardware acceleration in Chrome before a big run.

Fix 9: accept that the model does not fit

A 70B model does not run on 12GB of VRAM in any usable way, regardless of tricks. Use a smaller model or an API. Recognising this saves days.

Quick reference: what fits in what

FAQ

Does adding system RAM fix CUDA OOM? No. It only helps if your tool supports CPU offload, and then it costs speed.

Why does it fail after ten minutes of working fine? Growing KV cache or a memory leak in a loop. See fixes 1 and 6.

Is PYTORCH_NO_CUDA_MEMORY_CACHING=1 a good idea? It removes fragmentation and destroys performance. Use it for diagnosis, not production.

Starting from zero? Read the full walkthrough first: How to run an AI model locally on your own computer, then come back here.