Build Custom Ollama Models with a Modelfile

The Problem: Repeating Yourself in Every Chat

Related: Fix Stable Diffusion Out Of Memory on a 6GB VRAM GPU →

You just installed Ollama and pulled a model. You open your terminal, type ollama run llama3.1, and paste in a paragraph telling the model to act like a cynical senior systems programmer who only gives terse answers. It works perfectly for that session.

But then you close the terminal. The next time you boot up the model, it is back to being a cheerful, overly verbose AI assistant.

You find yourself keeping a text file on your desktop just to copy and paste your system prompts every time you start a new chat. If you are integrating Ollama into a script or a local app, you are passing that same massive string of text in every single API call. This wastes tokens, clutters your code, and is incredibly annoying.

The fix is building a custom model using an Ollama Modelfile.

A Modelfile is essentially a Dockerfile for LLMs. It allows you to define a base model, inject a permanent system prompt, tweak model parameters like temperature, and save the whole thing as a new, ready-to-run model. I use this constantly on my home server to keep distinct "personalities" available on port 11434 without having to manage client-side prompts.

What Actually is an Ollama Modelfile?

Related: Fix ComfyUI "Torch Not Compiled With CUDA Enabled" Error →

When you pull a model like mistral or llama3, Ollama downloads a multi-gigabyte weight file (usually in GGUF format) and an associated JSON manifest.

When you create a Modelfile, you are not copying those gigabytes of weights. You are just creating a tiny metadata file that points to the original weights but applies a new layer of instructions on top. You can have fifty different custom models based on llama3.1, and they will only take up a few kilobytes of extra disk space.

This is incredibly efficient. If you are already pushing the limits of your hardware and running a local LLM on 8GB of RAM, you do not want to duplicate heavy weight files just to change a system prompt. The Modelfile handles this gracefully through layered architecture.

Creating Your First Modelfile

Related: A One-File Docker Compose Stack for Ollama and Open WebUI →

Let us build that cynical senior developer model.

First, create a new directory for your project and create a plain text file named Modelfile. You do not need any file extension.

mkdir ~/ollama-custom-models
cd ~/ollama-custom-models
touch Modelfile

Open Modelfile in your text editor of choice (nano, vim, or VS Code). We only need two commands to start: FROM and SYSTEM.

FROM llama3.1:8b

SYSTEM """ You are a grumpy, veteran C++ and Rust programmer. You are annoyed by basic questions. You give extremely short, accurate answers. You never use the words "delve", "explore", or "crucial". You only provide code if explicitly asked. """

The FROM instruction tells Ollama which base model to use. If you do not have it downloaded yet, Ollama will fetch it during the build process. The SYSTEM instruction sets the permanent system prompt. Notice the triple quotes ("""). You need these for multi-line prompts.

Building and Testing Your Custom Model

Related: Fixing Painfully Slow Whisper Transcription →

With the file saved, you need to compile it into a model Ollama can recognize. We do this with the ollama build command.

You need to give your new model a name. I will call this one grumpy-dev.

ollama build grumpy-dev -f ./Modelfile

When you hit enter, you will see output that looks like this:

transferring model data 
using existing layer sha256:87048bcd552163b84177ebaf14db5a2305a46875b253b2164a2f8b5a153202e8 
using existing layer sha256:8c17c2ebb0ea011ddb6da6984e0cdeafbb7145749f7ce25b59631a02120e29b1 
using existing layer sha256:7c23fb3618770b5d9bc37f5979f427f7dbde314b1ce33a1e94443916d7a4cb2d 
using existing layer sha256:2e0493f67d0cf0c4ce33b00de1ed2a297be7d4722883395c8ba0fb6c7c427c3d 
using existing layer sha256:a6a8bc825fcb98faaa14eecf1dc0b0be812d4538e12c1b4edc3a30c5e7b2ff63 
creating new layer sha256:c29ab9f7d2f9390234a98a0024fbd9b3b847849e8a8de4887342fb216bd4d320
writing manifest 
success 

Notice how it says "using existing layer" for most of the steps. It is recycling the weights from the base llama3.1:8b model. The "creating new layer" step is just your system prompt being baked in.

Now, run your custom model:

ollama run grumpy-dev

Ask it a question: > How do I reverse a string in Python?

It will likely respond with a one-liner and zero polite filler, exactly as instructed. You can now use grumpy-dev in any API call or UI connected to your Ollama instance.

Tweaking Parameters: Temperature and Context Window

Related: Fix Open WebUI Showing No Models in the Dropdown →

System prompts are just the beginning. The real power of a Modelfile is locking in specific parameters so the model always behaves predictably.

Open your Modelfile again. We are going to add the PARAMETER directive.

FROM llama3.1:8b

PARAMETER temperature 0.1 PARAMETER num_ctx 8192 PARAMETER num_predict 500

SYSTEM """ You are a grumpy, veteran C++ and Rust programmer. You are annoyed by basic questions. You give extremely short, accurate answers. You never use the words "delve", "explore", or "crucial". You only provide code if explicitly asked. """

Here is what these parameters do:

• temperature 0.1: This makes the model highly deterministic and less creative. A lower temperature is perfect for coding tasks where you want exact syntax, not creative variations. The default is usually 0.8.
• num_ctx 8192: This sets the context window to 8,192 tokens. By default, Ollama often sets this to 2048 to save memory. Bumping this up lets you paste in larger chunks of code. Just be careful. If you set this too high, you will crash your GPU. I have written extensively about CUDA out of memory fixes if you push this number too far and your terminal starts throwing allocation errors.
• num_predict 500: This caps the maximum number of tokens the model will generate in a single response. Since we want terse answers, this is a good safety net to prevent runaway generation.

Rebuild the model to apply the changes:

ollama build grumpy-dev -f ./Modelfile

Using a Modelfile to Load Raw GGUF Files

Related: Secure Ollama with Nginx, HTTPS, and a Password →

One of my favorite uses for Modelfiles is loading raw models downloaded straight from HuggingFace. Sometimes the exact quantization I want is not available on the official Ollama registry, but someone like TheBloke or Bartowski has uploaded the .gguf file.

You can download that .gguf file to your local machine and point a Modelfile directly at it.

First, download the model file to your drive. Let us say you saved it to /home/user/models/mistral-7b-v0.1.Q4_K_M.gguf.

Create a new Modelfile:

FROM /home/user/models/mistral-7b-v0.1.Q4_K_M.gguf

TEMPLATE """{{ if .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }}{{ if .Prompt }}<|im_start|>user {{ .Prompt }}<|im_end|> {{ end }}<|im_start|>assistant """

PARAMETER stop "<|im_start|>" PARAMETER stop "<|im_end|>"

Build it the exact same way:

ollama build local-mistral -f ./Modelfile

When importing raw GGUFs, you often have to specify the TEMPLATE and stop tokens manually. The template tells Ollama how to format the chat history so the model understands who is speaking. If you do not include the right stop tokens, the model will finish answering your question and then start hallucinating the user's next question.

Modelfile vs. API System Prompts

You might be wondering when you should use a Modelfile versus just sending a system prompt via your chat application. Often, people try to configure system prompts in a web interface, run into Docker networking issues, and get stuck. If you have ever fought with a setup where Open WebUI in Docker cannot reach Ollama, you know that minimizing complex client-side configuration saves a lot of headaches.

Here is how I decide where to put my instructions:

If a behavior needs to be foundational to the model regardless of who or what is calling it, use a Modelfile.

What Did NOT Work

I have messed up plenty of custom models trying to get clever with Modelfiles. Here are a few dead ends I hit so you can avoid them.

Using single quotes for multi-line prompts When I first started, I tried writing SYSTEM "My long prompt \n on multiple lines". It failed horribly. The build command threw a syntax error. You must use the triple quotes (""") for anything that spans more than one line.

Assuming ollama build will fix a corrupted base model Once, I pulled a model, but my internet dropped at 99%. The base model was corrupted. I tried running ollama build to create a custom version, assuming the build process would verify and re-download the broken base layer. It did not. The build succeeded, but running the model threw a segmentation fault. If your custom model crashes immediately, delete the base model with ollama rm <modelname> and pull it again.

Messing with TEMPLATE on registry models When you use FROM llama3.1, Ollama already knows the exact Jinja template and stop tokens that Llama 3 requires. I once tried to override the TEMPLATE directive just to see if I could make it use a different format. The model immediately degraded into outputting garbage characters and infinite loops. Unless you are importing a raw .gguf file yourself, leave the TEMPLATE directive alone.

Ignoring memory constraints with context windows I tried setting PARAMETER num_ctx 32768 on a 7B model while running on an old laptop with 16GB of shared RAM. The model built fine, but the moment I sent a prompt, the system froze and eventually killed the Ollama process. Setting the parameter in the Modelfile does not magically optimize the model. You still have to respect the physical limits of your hardware. If you are struggling with this, figuring out how much RAM you actually need for local AI is mandatory before tweaking num_ctx.

FAQ

Question: Where does Ollama save the models I build?

Ollama stores all models, including custom ones, in a hidden directory. On Linux, this is /usr/share/ollama/.ollama/models. On macOS, it is ~/.ollama/models. You will find them inside the blobs and manifests folders.

Question: Can I share my custom Modelfile with someone else?

Yes. The Modelfile is just a plain text file. You can email it or push it to GitHub. As long as the other person has Ollama installed, they can run ollama build on your file to get the exact same model.

Question: How do I view the Modelfile of an existing model?

You can inspect any installed model by running ollama show <modelname> --modelfile. This will print the exact base model, parameters, template, and system prompt it currently uses.

Question: Does building a custom model take up double the hard drive space?

No. Ollama uses a layered filesystem. Your custom model just creates a small metadata layer containing your system prompt and parameters. It shares the heavy, gigabyte-sized weight files with the base model, taking almost no extra space.