A One-File Docker Compose Stack for Ollama and Open WebUI

The Problem With Manual Local AI Setups

Related: Fix Stable Diffusion Out Of Memory on a 6GB VRAM GPU →

You boot up an Open WebUI Docker container, point it at your local Ollama installation, and get hit with an immediate WebUI could not connect to Ollama error message. You try changing the host to host.docker.internal. That fails. You try exposing the ports manually. Then you realize Docker is not even seeing your graphics card, so your local LLM is crawling at 2 tokens per second on your CPU.

I wasted hours trying to run Ollama as a native host service and Open WebUI as a Docker container. The networking bridge between a host machine and a Docker container is notoriously fragile. If your firewall rules change or Docker rebuilds its internal network, the UI simply stops talking to the backend.

If you are just looking for a way to run open-source models with a clean interface tonight, you need them isolated in the same Docker network. If you are debating between Ollama vs LM Studio vs Jan, keeping your setup containerized makes it incredibly easy to test options without polluting your host operating system.

Here is the exact setup I use on my Ubuntu workstation with an RTX 4090 to deploy both services simultaneously, pass the GPU through, and link them securely.

Prerequisites for GPU Passthrough

Related: Fixing Painfully Slow Whisper Transcription →

If you want hardware acceleration, Docker needs permission to talk to your Nvidia card. If you skip this, Ollama will run in CPU-only mode.

First, ensure your Nvidia drivers are installed and functioning. Run this command to verify:

nvidia-smi

If you see a table listing your GPU and VRAM, you are good. Next, you need the Nvidia Container Toolkit. This is the bridge between Docker and your physical hardware. On Debian or Ubuntu systems, the installation looks like this:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update sudo apt-get install -y nvidia-container-toolkit sudo systemctl restart docker

Once Docker restarts, your machine is ready for the compose file.

The One-File Docker Compose Solution

Related: Fix Open WebUI Showing No Models in the Dropdown →

Create a new directory for your AI stack. I put mine in ~/ai-stack. Inside that directory, create a file named docker-compose.yml.

Copy and paste this exact configuration:

version: '3.8'

services: ollama: image: ollama/ollama:latest container_name: ollama restart: unless-stopped ports: - "11434:11434" volumes: - ./ollama_data:/root/.ollama deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu]

open-webui: image: ghcr.io/open-webui/open-webui:main container_name: open-webui restart: unless-stopped ports: - "3000:8080" environment: - OLLAMA_BASE_URL=http://ollama:11434 volumes: - ./webui_data:/app/backend/data depends_on: - ollama

Breaking Down the Configuration

Related: Secure Ollama with Nginx, HTTPS, and a Password →

This file does a few very specific things that prevent the most common errors people hit.

First, look at the deploy block under the ollama service. This is the modern Docker implementation for hardware acceleration. Older guides tell you to use the --gpus all flag in a run command or a legacy runtime setting. The deploy > resources > reservations format is the official standard for Docker Compose v3.8 and guarantees that the container claims exactly one Nvidia GPU.

Second, the volumes mapping is critical. I map ./ollama_data to /root/.ollama because if you do not map this directory, every single model you download will disappear the moment the container restarts. A standard 8B model is about 4.7GB. You do not want to re-download that every time you update Docker.

Third, the environment variable in the open-webui block is the actual fix for the communication error. Setting OLLAMA_BASE_URL=http://ollama:11434 tells Open WebUI to use Docker's internal DNS to resolve the name ollama directly to the adjacent container. It never attempts to route out to your host machine's localhost. This entirely prevents the frustrating Ollama connection refused on 127.0.0.1:11434 scenario.

Finally, depends_on ensures Docker starts Ollama before it attempts to boot the UI, preventing crash loops on startup.

Running the Stack and Initial Setup

Related: How to Stop Ollama From Unloading Models (keep_alive) →

Open your terminal, navigate to the directory where you saved the file, and bring the stack up in detached mode:

docker-compose up -d

Docker will pull the latest images for both Ollama and Open WebUI. This takes a few minutes depending on your internet connection. Once it finishes, run docker ps to verify both containers are running.

You now have a blank slate. Ollama is running, but it has no models loaded. To get your first model, you can either download it through the UI or directly via the Ollama container CLI. I prefer the CLI for the initial pull because you get a clear progress bar.

Execute this command to pull Llama 3:

docker exec -it ollama ollama run llama3.1

Once it finishes, open your web browser and navigate to http://localhost:3000.

You will be greeted by the Open WebUI login screen. The first account you create on this screen automatically becomes the administrator. Enter an email and password (this is all stored locally on your machine in the ./webui_data folder) and log in.

Select Llama 3.1 from the drop-down menu at the top, and start chatting.

Hardware Impact: CPU vs GPU Performance

Related: Fix Continue in VS Code Not Connecting to Ollama →

It is worth understanding exactly why we went through the trouble of installing the Nvidia Container Toolkit. Running an LLM on your CPU is possible but painful. The math required to generate tokens relies heavily on memory bandwidth, which standard DDR4 or DDR5 RAM lacks compared to VRAM.

Here is a look at the performance difference when I run an 8-billion parameter model (Llama 3, 4-bit quantized) on my setup.

If you skip the deploy block in the compose file, you get the CPU performance numbers. If you mapped the hardware correctly, you get the GPU numbers. It transforms the experience from a novelty to a practical daily tool.

If you start running larger 32B or 70B models, you will inevitably push past your VRAM limits. When that happens, the model generation will either slow to a crawl (spilling over to system RAM) or crash the container entirely. If you hit those walls, you will need to implement specific CUDA out of memory fixes, mostly involving limiting your context window in Open WebUI's advanced settings.

What Did NOT Work

When building this stack, I tried several approaches that failed miserably. I am listing them here so you do not waste your time trying to outsmart the basic compose setup.

Extending the Setup Later

Once you have this core setup running reliably, you can start pushing the boundaries of what local AI can do. Because Open WebUI is a very active project, it supports external tool integrations.

You can expose custom scripts or access local filesystems by integrating Model Context Protocol servers later on. I use this to let my local models read my code repositories. If you want to dive into that eventually, keeping everything containerized makes it much easier to add new services to this compose file. You can read how MCP servers explained fit into a local architecture, but wait until your base stack is stable before adding more containers.

Updating and Maintenance

Because everything is stored in the mapped ./ollama_data and ./webui_data directories, updating this stack is completely safe. You will not lose your chat history or downloaded models.

To update both Ollama and Open WebUI to their latest versions, navigate to your ~/ai-stack folder and run:

docker-compose pull
docker-compose up -d

Docker will download the new image layers, stop the old containers, and recreate them with the exact same volumes and network settings attached. I do this roughly once a week, as the Open WebUI team ships bug fixes and new features constantly.

You can also monitor the internal logs if something ever acts up. If Open WebUI gets stuck loading a chat, check the backend logs with:

docker logs open-webui --tail 50

This will output the last 50 lines of activity. Usually, it will tell you if you ran out of memory or if a model file became corrupted.

This single file replaces dozens of terminal commands, background services, and path variables. Save it, run it, and spend your time actually using the models instead of doing systems administration.

FAQ

Question: Can I run this stack on an Apple Silicon Mac?

Yes. Mac Docker Desktop handles translation automatically. Remove the deploy block completely from the ollama service. Apple's Metal API will automatically use the unified memory of your M1/M2/M3 chip without specific hardware passthrough lines in the compose file.

Question: Where are my downloaded models actually stored on my hard drive?

Because of the volume mounts in the compose file, your models are stored inside the ollama_data folder located in the exact same directory where you saved your docker-compose.yml file.

Question: How do I access Open WebUI from my phone on the same Wi-Fi?

Find your host computer's local IP address (e.g., 192.168.1.50). On your phone's browser, navigate to http://192.168.1.50:3000. You do not need to change the Docker file; the port mapping - "3000:8080" exposes it to your entire local network.

Question: Why does Open WebUI say a model is unavailable when I select it?

This usually means Ollama crashed in the background while trying to load the model into VRAM. Check your available memory using nvidia-smi. If the model is too large, restart the stack with docker-compose restart and choose a smaller quantized model.