Local Embedding Models Compared: Nomic, BGE, E5, mxbai

The Symptom: Trash Retrieval and Dimension Errors

Related: Midjourney vs DALL-E vs Flux: A Practical Image-Tool Comparison Framework →

Related: [Mac Mini M4 Local LLM Benchmarks: Real Tokens/Sec →](/article/local-llm-mac-mini-m4-real-benchmarks)

You ask your local AI a question based on your documents. The vector database retrieves three completely irrelevant paragraphs. The LLM hallucinates an answer. You look at your terminal, tweak a configuration, restart the script, and get hit with this:

chromadb.errors.InvalidDimensionException: Embedding dimension 384 does not match collection dimension 768

This article is for developers who are tired of sending their private data to OpenAI's text-embedding-ada-002 API. If you are in the middle of building a local RAG chatbot and your search results are terrible, your LLM is not the problem. Your embedding model is.

I recently spent a week tearing down my local RAG pipeline to benchmark the four most popular open-weight embedding models: Nomic, BGE, E5, and mxbai. I ran all of these locally on a machine with a single Nvidia RTX 3090 (24GB VRAM) and 64GB of system RAM, using Ollama to serve the models.

Here is exactly how they compare, which one you should use, and the undocumented quirks that will break your setup if you do not handle them in your code.

The Contenders at a Glance

Related: 9 Free AI Coding Tools Every Developer Should Try in 2026 →

Related: Local LLMs on a Raspberry Pi 5: Honest Numbers & Setup →

Before we look at the code, you need to understand the physical constraints of these models. Embeddings are just lists of floating-point numbers. The "Dimension" is how long that list is. A higher dimension theoretically captures more semantic nuance, but it takes up more RAM and disk space in your vector database.

The "Max Tokens" is the context window. If you feed a document longer than this limit into the model, the model will just ignore the rest of the text.

To pull any of these to your local machine, open your terminal and run the pull command.

ollama pull nomic-embed-text
ollama pull mxbai-embed-large

If your terminal throws an error here, your daemon probably dropped. Read my guide on fixing Ollama connection refused on 127.0.0.1:11434 before continuing.

Test Environment and Basic Implementation

Related: Sora 2 Review: OpenAI's Video Model Is Finally Useful for Real Work →

Related: Q4 vs Q5 vs Q8 Quantization: Which GGUF to Actually Download →

To test these, I used a dataset of 5,000 internal engineering wiki pages. I chunked the markdown files using LangChain's RecursiveCharacterTextSplitter with a chunk size of 500 and an overlap of 50.

Instead of relying on heavy frameworks to do the embedding, I prefer hitting the Ollama API directly. It gives you exact control over what is happening and exposes silent errors that wrappers often hide. Here is the exact Python function I used to generate embeddings across all tests.

import requests

def get_embedding(text, model_name): url = "http://localhost:11434/api/embed" payload = { "model": model_name, "input": text } response = requests.post(url, json=payload)

if response.status_code == 200:
    # The API returns a list of embeddings under the 'embeddings' key
    return response.json()["embeddings"][0]
else:
    raise Exception(f"Failed to embed: {response.text}")

Example usage

vector = get_embedding("Deploy the application using docker-compose up -d", "nomic-embed-text") print(f"Generated vector with {len(vector)} dimensions.")

Nomic-Embed-Text: The Long Context Default

Related: DeepSeek-R1 Repeats Itself or Outputs Gibberish: The 4 Settings That Fixed It →

If you want a safe default that "just works" for almost any document size, use Nomic.

The massive advantage of nomic-embed-text is its 8192 token limit. Most local embedding models tap out at 512 tokens. If you pass a 2,000-token text block to BGE, it silently drops the last 1,488 tokens. Nomic will actually read and embed the whole thing.

Nomic also uses a technique called Matryoshka Representation Learning. This means the embedding is structured so that the most important information is front-loaded in the vector. While Ollama outputs Nomic as 768 dimensions by default, you can actually slice the Python array down to 256 dimensions before inserting it into your database, and it still retains about 90% of its accuracy.

# Truncating a Nomic embedding to save database space
full_vector = get_embedding("System architecture overview", "nomic-embed-text")
small_vector = full_vector[:256] 

The only catch with Nomic is that you technically should use prefixes. For documents you are indexing, you prepend search_document: . For the questions the user asks, you prepend search_query: . I found that omitting these only dropped my retrieval accuracy by a few percentage points, but it is best practice to include them.

mxbai-embed-large: The Precision Pick

Related: DeepSeek-R1 Shows Its <think> Tags in the Output — Here Is How I Strip Them →

Created by Mixedbread AI, mxbai-embed-large became my favorite model for pure semantic precision on short chunks.

When I queried my database for "How do I reset the staging database password?", Nomic returned a mix of database setup guides and password policies. Mxbai returned the exact bash script I needed as the number one result.

Mxbai outputs 1024 dimensions. It does not require any special prefixes for queries versus documents, which makes your application code much cleaner. You just pass the raw text in, and you get a highly accurate vector out.

The main limitation is the 512-token context window. You must ensure your chunking strategy strictly enforces text blocks smaller than ~400 words. If your system crawls during the embedding process, it usually means your batches are too large and spilling into system RAM. If you hit this, check out my troubleshooting steps for Ollama running slow.

BGE and E5: The Asymmetrical Trap

Related: How to Run DeepSeek-R1 Offline on Mac mini M4 (Step-by-Step 2026 Guide) →

BAAI's bge-large-en-v1.5 and Microsoft's multilingual-e5-large are both incredible models that dominate the MTEB (Massive Text Embedding Benchmark) leaderboards. But they are a trap for beginners.

Both of these are asymmetrical models. This means the model maps "queries" and "passages" into slightly different vector spaces so they align better during a cosine similarity search. If you do not format your strings exactly how the model expects, your RAG pipeline will fail silently. Your vectors will insert, but your similarity scores will be garbage.

For E5, you must prepend your database chunks with passage: and your user searches with query: .

# How to properly use E5 in your RAG code
document_text = "The server is located in the US-East region."
user_question = "Where is the server?"

Indexing step

doc_vector = get_embedding(f"passage: {document_text}", "multilingual-e5-large") db.insert(id="doc1", vector=doc_vector)

Retrieval step

query_vector = get_embedding(f"query: {user_question}", "multilingual-e5-large") results = db.search(query_vector)

BGE has a similar requirement, but it only demands a prefix for the query, not the passage. You must prepend user questions with: Represent this sentence for searching relevant passages: .

I eventually dropped BGE from my daily driver stack. Having to hardcode that specific string into my retrieval functions felt brittle, and mxbai offered identical performance without the formatting hassle. I only use E5 now if I am specifically dealing with a mix of Spanish and English documents, as its multilingual capabilities are significantly better than the others.

What Did NOT Work

I hit several dead ends while building out this test suite. Here is what you should avoid.

Mixing dimensions in the same ChromaDB collection. If you embed your first 100 documents with Nomic (768 dimensions), then switch your code to use mxbai (1024 dimensions) and try to insert a new document into the same collection, ChromaDB will throw the InvalidDimensionException from the beginning of this article. Vector databases require uniform dimensions. If you change models, you must wipe your database and re-embed everything.

# Wiping a local ChromaDB instance to start fresh
rm -rf ./chroma_db_storage/

Ignoring silent truncation. I tried feeding a massive 4,000-word block of JSON logs into mxbai to see if it would figure it out. The Ollama API returned a 200 HTTP status and a standard 1024-dimension vector. It did not throw an error. But when I searched for data located at the bottom of that JSON file, it never matched. The model silently discarded everything after the 512th token. You must enforce chunk sizes in your application logic.

Running Docker containers without host networking. When I tried to use a web UI to test my RAG setup, it could not connect to my local embedding models. My Python scripts worked, but the UI failed. If you are running your frontend in a container, you might hit network isolation. See my fix for why Open WebUI in Docker cannot reach Ollama.

Using a tiny embedding model with a massive LLM. I initially tried using all-minilm-l6-v2 (a tiny 384-dimension model) to save VRAM, paired with llama3:70b for generation. It was a complete waste of hardware. The 70B LLM is a genius, but if the tiny embedding model retrieves the wrong documents, the genius LLM will just give you a highly articulate, completely wrong answer. Spend the extra 1GB of VRAM to run a modern embedding model like Nomic or mxbai.

The Final Verdict

If you are just starting and want a robust system that handles various document sizes without strict formatting rules: use Nomic.

If you have strict chunking (under 500 tokens), want the highest possible accuracy for English technical documents, and want clean code: use mxbai.

I highly recommend writing a small Python script to embed the exact same 10 documents with both models, running a cosine similarity search for a known fact, and looking at the raw score. Your specific dataset will always be the final judge.

FAQ

Question: Can I use different embedding models for different collections?

Yes. Your vector database separates data by collections (or indexes). You can use Nomic for a wiki_docs collection and mxbai for a code_snippets collection, as long as the dimensions within a single collection remain identical.

Question: How much VRAM do local embedding models actually need?

Very little. Models like Nomic or mxbai only require about 1GB to 1.5GB of VRAM to run comfortably. You can easily run them alongside an 8B parameter LLM on a standard 8GB graphics card like an RTX 3060 or 4060.

Question: Why are my similarity scores all exactly the same?

You likely forgot to normalize your vectors, or you are running an asymmetrical model like BGE or E5 without using the required query: or passage: prefixes. Without prefixes, the model places queries and text in different vector spaces.

Question: Should I use cosine similarity or dot product?

If your vectors are normalized (which Ollama does by default for models like Nomic and mxbai), cosine similarity and dot product will mathematically return the exact same ranking. Dot product is slightly faster to compute at scale.