Fix Stable Diffusion Out Of Memory on a 6GB VRAM GPU

The Error: CUDA out of memory

Related: Fixing Painfully Slow Whisper Transcription →

You click generate, wait five seconds, and the terminal spits out a wall of red text.

RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 6.00 GiB total capacity; 5.12 GiB already allocated; 18.00 MiB free; 5.23 GiB reserved in total by PyTorch)

I run an RTX 2060 with 6GB of VRAM. It is plenty for most 1080p gaming, but Stable Diffusion eats it alive if you use the default settings. When you load a model, PyTorch reserves a chunk of your video memory. As the image generates, noise calculation and the final VAE decoding process require even more memory. If that number crosses 6144 MB, the process terminates immediately.

This guide is for people using Automatic1111 or Forge on Windows or Linux, trying to get decent generation speeds without crashing. We are going to change your startup arguments, tweak the attention mechanism, and implement a strict workflow for upscaling.

If you are running Linux via WSL2, make sure your drivers are mapped correctly first. I have had similar memory and hardware detection issues where Ollama not using my NVIDIA GPU in WSL2 drove me crazy until I fixed the Windows CUDA toolkit paths. Assuming your GPU is detected correctly, let's fix the memory limits.

The Magic Startup Arguments

Related: Fix Open WebUI Showing No Models in the Dropdown →

The single biggest impact comes from your startup script. On Windows, this is webui-user.bat. On Linux, it is webui-user.sh. Open this file in a basic text editor like Notepad.

Look for the line that says set COMMANDLINE_ARGS=. If it is blank, we are going to add three specific flags.

set COMMANDLINE_ARGS=--medvram --xformers --opt-split-attention

Here is exactly what these flags do:

The --medvram flag prevents Stable Diffusion from loading the entire model into VRAM at once. Instead, it breaks the model into three parts (cond, unclip, and the main model) and swaps them in and out of VRAM as needed. This adds a slight delay to your generation time, but it drops idle VRAM usage from 4.5GB down to about 2.5GB. Do not use --lowvram unless you absolutely have to. It swaps data out to your system RAM aggressively and will turn a 10-second generation into a 3-minute generation.

The --xformers flag installs and enables the xFormers library. This is a highly optimized cross-attention implementation. It mathematically reduces the memory overhead during the actual image generation steps. When you launch the webui with this flag the first time, it will download a few hundred megabytes of dependencies.

The --opt-split-attention flag forces PyTorch to use cross-attention optimization if xformers fails to load for any reason. It acts as a safety net.

Cross-Attention Optimization Settings

Related: Secure Ollama with Nginx, HTTPS, and a Password →

VRAM limits are not just about the file size of your model. The way your graphics card calculates the diffusion steps matters heavily.

Start your WebUI and go to the "Settings" tab. Look on the left sidebar for "Optimizations". Scroll down until you find the dropdown menu labeled "Cross attention optimization".

By default, this is usually set to "Automatic" or "Doggettx". Change this directly to "xformers". Apply settings and reload the UI.

If you get an error in your terminal saying xformers is not installed, or if you are running an older AMD card where xformers is not supported, select "sdp attention" instead. Scaled Dot Product (SDP) attention is built natively into PyTorch 2.0 and newer. It uses almost the exact same amount of VRAM as xformers and runs at nearly identical speeds. I tested both on my 2060 and saw less than a 50MB difference in peak VRAM allocation.

Resolution and Batch Size Limits

Related: How to Stop Ollama From Unloading Models (keep_alive) →

You cannot generate 1024x1024 native images on SD 1.5 with 6GB of VRAM without trickery. The math simply does not fit.

Keep your base resolution at 512x512 or 512x768 for portrait aspect ratios.

Your "Batch size" must stay at 1. Batch size tells the GPU to process multiple images in parallel. Changing batch size to 4 will instantly quadruple your active VRAM usage and trigger an out of memory error. If you want to generate 4 images, change the "Batch count" slider instead. Batch count runs the generations sequentially. It uses the exact same amount of VRAM as a single image, it just loops the process four times.

To get high-resolution images, you have to use the Hires.fix toggle. Set your base resolution to 512x512. Check Hires.fix. Set the upscale multiplier to 1.5x or 2x. Choose the "Latent" or "R-ESRGAN 4x+" upscaler. This renders the image small, then scales it up carefully.

However, upscaling introduces a new crash point: the VAE.

The VAE (Variational Autoencoder) turns the latent noise into the final visible pixels. This happens at the very end. If your progress bar hits 100% and then throws a CUDA out of memory error, the VAE is to blame. To fix this, go to the Extensions tab and install "Multidiffusion upscaler / Tiled VAE".

Similar to how chunking text matters when building a local RAG chatbot so you do not blow up context limits, we need to chunk our upscaling. Tiled VAE cuts the final image decoding into 256x256 squares. It processes one square at a time, keeping VRAM completely flat at the end of the generation. Once installed, just check the "Enable Tiled VAE" box on your main txt2img tab.

Model Types: SD 1.5 vs SDXL on 6GB

Related: Fix Continue in VS Code Not Connecting to Ollama →

Standard SD 1.5 checkpoints are about 2GB. They fit comfortably in 6GB of VRAM. SDXL base models are 6.5GB just sitting on the hard drive.

Can you run SDXL on a 6GB card? Yes, but it requires discipline.

First, you must use --medvram-sdxl in your startup arguments instead of just --medvram.

Second, you need to be aware of the fp16 VAE issue. Many 16-series and 20-series Nvidia cards produce completely black squares when using the default SDXL VAE in half-precision. The common advice is to add --no-half-vae to your arguments. Do not do this on a 6GB card. Forcing the VAE into full precision 32-bit mode doubles its memory footprint and guarantees a crash on high resolutions. Instead, download the specific "fixed fp16 VAE" created by madebyollin from HuggingFace, place it in your models/VAE folder, and select it in your settings.

Honestly, if you want to run SDXL, I strongly recommend abandoning Automatic1111 and downloading the Forge webui. Forge is a fork of Automatic1111 designed specifically to manage memory better. It looks identical to Automatic1111, but the backend memory swapping is vastly superior. Heavy local models are universally painful if the backend is not optimized. I saw this exact same resource hogging when I had Ollama running slow before I properly configured my GPU layer offloads.

The VRAM Usage Breakdown

Related: A One-File Docker Compose Stack for Ollama and Open WebUI →

To show you exactly what to expect, I tracked peak VRAM usage in my terminal using the nvidia-smi command. All tests were run using the CyberRealistic V4.1 checkpoint (SD 1.5) with 20 steps of Euler a.

Using xformers and Tiled VAE allows you to generate images twice as large while using less VRAM than the default 512x512 setup.

What Did NOT Work

I spent three nights trying random Reddit suggestions that ended up being complete dead ends. Do not waste your time on these.

Turning on Nvidia's "Sysmem Fallback" in the driver control panel. Nvidia added a feature where if your GPU runs out of VRAM, it borrows standard DDR4 system RAM to keep the program from crashing. It sounds great, but GPU memory is incredibly fast and system RAM is incredibly slow. When Stable Diffusion spilled over into my system RAM, my generation speed dropped from 4 iterations per second to 6 seconds per iteration. A 10-second image took 2 minutes. It is better to leave this setting on "Prefer No Sysmem Fallback" so the app crashes immediately and you know you need to lower your resolution.

Disabling browser hardware acceleration. A common tip is to close Chrome because it uses 500MB of VRAM. This is true, but closing my browser means I cannot read prompt tutorials or use the WebUI effectively. Turning off hardware acceleration entirely made my browser lag when scrolling through image grids. Just keep your tabs to a minimum.

Dockerizing the application. I thought isolating the environment in Docker might restrict memory leaks. The container overhead actually consumed an extra 300MB of RAM. I also ran into internal networking headaches trying to expose the ports, similar to the nightmare I had when Open WebUI in Docker cannot reach Ollama. Run it natively on your host OS.

FAQ

Question: Do I need to reinstall PyTorch to fix CUDA errors?

No. PyTorch comes bundled correctly if you used the standard installation script. Updating it manually often breaks dependencies unless you know exactly which CUDA version matches your Nvidia driver.

Question: Why does Hires.fix cause OOM even with medvram?

Hires.fix multiplies your pixel count before passing it to the VAE. To fix this, use the Tiled VAE extension. It breaks the image into smaller chunks during the final decoding step so VRAM stays low.

Question: Is 6GB VRAM enough for Stable Diffusion 3?

No, SD3 is too large. Even the medium version struggles heavily on 8GB cards. Stick to SD 1.5 customized models or highly optimized SDXL checkpoints using the Forge web interface.

Question: Will adding system RAM help my VRAM issues?

Not for rendering speed. Once your GPU runs out of VRAM, Nvidia drivers can use system RAM as a fallback, but it slows generation from seconds to minutes. You want to stay entirely inside VRAM.