Q4 vs Q5 vs Q8 Quantization: Which GGUF to Actually Download

The GGUF Download Dilemma

Related: Midjourney vs DALL-E vs Flux: A Practical Image-Tool Comparison Framework →

Related: Local LLMs on a Raspberry Pi 5: Honest Numbers & Setup →

You finally found the Hugging Face repository for the local LLM you want to run. You click on the "Files and versions" tab, expecting a simple download button. Instead, you are staring at a wall of 15 different .gguf files with names like llama-3-8b-instruct.Q4_K_M.gguf, Q5_K_S.gguf, and Q8_0.gguf.

If you download the wrong one, you either run out of VRAM and crash with a ggml_alloc_graph_build: not enough space in the buffer error, or you download a heavily compressed version and the model loses its ability to write basic logic.

I run local models daily on everything from a 16GB M2 Mac to a desktop rig with a 24GB RTX 3090. Over the last year, I have tested almost every quantization level llama.cpp supports.

Quantization is just rounding. The original AI models are trained at 16-bit (fp16) precision. Quantizing them down to 8-bit (Q8) or 4-bit (Q4) reduces the file size and RAM requirements drastically. But it is not a linear trade-off. There are hard drop-offs where the model suddenly becomes stupid. Here is exactly which file you should download, depending on your hardware and your use case.

Q4_K_M: The Daily Driver

Related: 9 Free AI Coding Tools Every Developer Should Try in 2026 →

Related: Mac Mini M4 Local LLM Benchmarks: Real Tokens/Sec →

If you just want an answer and don't want to think about it, download the Q4_K_M file.

Q4 means the model's weights have been compressed to roughly 4 bits per parameter. The K_M stands for "K-quant Medium." Older quantization methods (like Q4_0) crushed every single weight down to 4 bits equally. K-quants are smarter. They analyze the neural network layers and keep the most critical attention tensors at 6 bits, while crushing the less important feed-forward layers down to 4 bits.

The result is a model that requires half the VRAM of a Q8, but retains about 98% of the original unquantized model's intelligence.

For general chat, summarizing PDFs, or extracting data from text, you will not notice a difference between Q4_K_M and the uncompressed model. An 8B parameter model at Q4_K_M takes up about 4.8GB of VRAM. Add another 1GB for a decent context window, and you are sitting comfortably at 5.8GB of VRAM. This makes Q4_K_M the absolute perfect fit if you are running an 8GB graphics card like an RTX 3060, 4060, or a base M1/M2 Mac.

Q5_K_M: The Sweet Spot for Coding and Logic

Related: Sora 2 Review: OpenAI's Video Model Is Finally Useful for Real Work →

Related: Local Embedding Models Compared: Nomic, BGE, E5, mxbai →

While Q4 is great for natural language, code generation is significantly more fragile.

If an LLM drops a word in an essay, you skip over it. If an LLM drops a closing parenthesis or hallucinates a variable name in a Python script, the entire execution fails. Because of this, when I am running a model specifically for coding tasks inside Cursor or Continue.dev, I step up to Q5_K_M.

At 5 bits, the model retains nearly perfect precision in its logical reasoning layers. I also use Q5 specifically when dealing with complex tool calling. For example, if I am trying to pipe commands through custom tools and need to get MCP servers explained and configured by an autonomous agent, the strict JSON output required by the agent breaks far less frequently on Q5 than it does on Q4.

For an 8B model, the Q5_K_M file is about 5.7GB. Once loaded with context, you are looking at around 6.8GB of VRAM. It still barely fits on an 8GB GPU, but you won't be able to have heavy browser tabs or hardware-accelerated apps running in the background without risking system RAM spillover.

Q8_0: Almost fp16, but usually a waste of RAM

Related: DeepSeek-R1 Repeats Itself or Outputs Gibberish: The 4 Settings That Fixed It →

The Q8_0 file represents 8-bit quantization. It is massive.

For an 8B model, a Q8 file weighs in at about 8.5GB. With a standard 8k token context window, you will need nearly 10GB of VRAM to run it entirely on your GPU. If you have a 12GB or 16GB card, you might think, "I have the space, I should just run Q8 to get the best quality."

This is a trap.

The perplexity gain (the measure of how accurate the model is) from Q5 to Q8 is mathematically negligible. In blind tests, you will not be able to tell the difference. But you will notice the lost RAM.

More importantly, if you have 16GB of VRAM, running an 8B model at Q8 is a massive waste of your hardware's potential. Instead of running a small 8B model at Q8, you should use that 16GB VRAM to run a larger, vastly smarter 14B or 12B model at Q4. A Q4 version of Qwen 2.5 14B will completely destroy a Q8 version of Llama 3 8B in every single benchmark. Always prioritize running a larger parameter model at Q4 over a smaller parameter model at Q8.

Furthermore, if you miscalculate your VRAM and load a Q8 model that overflows your GPU's VRAM by even 500MB, llama.cpp will offload those remaining layers to your system CPU RAM. Your generation speed will immediately drop from 60 tokens per second down to 3 tokens per second. If you ever find your Ollama running slow, blindly downloading Q8 files and spilling over VRAM limits is the most common culprit.

The VRAM Math (Comparison Table)

Related: DeepSeek-R1 Shows Its <think> Tags in the Output — Here Is How I Strip Them →

To make this concrete, here is exactly how the math breaks down for a standard 8B parameter model (like Llama 3) across the three main quantization levels.

Note the "Context VRAM." The GGUF file size is not your total RAM footprint. As you chat with the model, it builds a KV cache (Key-Value cache) to remember the conversation. This cache is usually unquantized (16-bit), meaning it eats RAM quickly.

Note: Base VRAM needed is slightly higher than file size because of CUDA/Metal overhead.

How I test them locally (The exact setup)

Related: How to Run DeepSeek-R1 Offline on Mac mini M4 (Step-by-Step 2026 Guide) →

When a new model drops, I don't guess. I download the Q4_K_M and run it directly through the terminal to watch the exact memory mapping.

If you are using Ollama, you can bypass downloading the file manually and just specify the quantization tag directly from their registry. Open your terminal and run:

ollama run llama3.1:8b-instruct-q4_K_M

Note: If you run this command and immediately get a networking error instead of a download bar, you likely have a local port binding issue. Check my guide to fix Ollama connection refused on 127.0.0.1:11434 before proceeding.

While Ollama is great, it abstracts away the exact VRAM numbers. To see the raw truth of what the GGUF file is doing to my hardware, I prefer downloading the raw .gguf file from Hugging Face and running it through the raw llama-cli binary.

Here is the exact command I use to test a model and offload all layers to the GPU:

./llama-cli -m ~/models/Meta-Llama-3-8B-Instruct.Q4_K_M.gguf \
  -p "Write a python script that implements a basic HTTP server." \
  -n 512 \
  -ngl 99 \
  -c 8192

The -ngl 99 flag tells the system to attempt to put 99 layers (essentially all of them) on the GPU. The -c 8192 sets the context window to 8k.

When you run this, watch the startup logs before the text generation begins. You are looking for a line that looks exactly like this:

llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors: VRAM used: 4892.44 MiB

If the logs say offloaded 20/33 layers to GPU, you have hit your VRAM limit. The remaining 13 layers are sitting on your slow system RAM. At this point, you either need to drop your context window (-c 4096), or you need to delete the GGUF file and download a lower quantization level.

What did NOT work

Over the last year of building local AI rigs, I have chased a lot of dead ends trying to optimize VRAM. Here is what I tried that completely failed:

FAQ

Question: What does the K_M stand for in Q4_K_M?

Answer: It stands for "K-quant Medium." Instead of compressing every layer equally to 4 bits, it keeps highly important attention layers at 6 bits and compresses less important layers to 4 bits, preserving much more intelligence than older Q4_0 methods.

Question: Is Q4 faster at generating text than Q5?

Answer: Marginally, but usually not enough to notice. Text generation speed is primarily bottlenecked by memory bandwidth. Because Q4 is a smaller file, less data needs to be moved through your GPU memory bus, yielding slightly higher tokens-per-second than Q5 or Q8.

Question: Should I download the GGUF or the Safetensors version?

Answer: If you are running the model locally on your own hardware using Ollama, LM Studio, or llama.cpp, you must download the GGUF file. Safetensors are unquantized base files used mostly by researchers, Python scripts (Transformers library), or for cloud deployment on vLLM.

Question: Can I run Q8 if I have exactly 8GB of VRAM?

Answer: Practically, no. An 8B model at Q8 takes about 8.5GB of space, plus another 1GB for the context cache. It will overflow your 8GB VRAM limit, spill into your system RAM, and run incredibly slowly. Use Q4_K_M or Q5_K_M instead.