Connect Claude Code CLI to Local Ollama Models

Why Connect Claude Code CLI to Local Models?

Related: Set Up Aider CLI with Ollama and Qwen 2.5 Coder →

Anthropic's Claude Code CLI is one of the most effective terminal-based agentic coding tools available. It reads your codebase, runs git commands, edits multi-file projects, and executes bash scripts autonomously. However, relying entirely on the Anthropic API gets expensive fast. A single complex refactoring session across a large repository can burn through $5 to $15 in API credits as the tool continuously re-injects code contexts and system prompts.

Beyond the cost, sending entire proprietary codebases to an external cloud API creates compliance and privacy issues for many teams.

Running Claude Code CLI offline with local models powered by Ollama solves both problems. You get the terminal workflow of Claude Code while keeping your code 100% local and running at zero incremental cost.

Achieving this requires a middle layer. Claude Code is hardcoded to speak the Anthropic Messages API (/v1/messages) schema, whereas Ollama exposes an OpenAI-compatible or native Ollama API. By placing LiteLLM in the middle as an translating proxy server, we can fool the Claude Code CLI into thinking it is talking to Anthropic's cloud when it is actually routing requests directly to a local GPU running Ollama.

+------------------+      Anthropic API      +---------------+      Ollama API      +-----------------+
| Claude Code CLI  |  -------------------->  | LiteLLM Proxy |  ----------------->  |  Ollama Model   |
| (Terminal Agent) |  http://localhost:4000  | (Translator)  |  http://localhost:11434 | (Qwen2.5-Coder) |
+------------------+                         +---------------+                      +-----------------+

Here is how to set up this pipeline on Linux or macOS, configure the local proxy, and deal with the inevitable schema edge cases.

Hardware and System Requirements

Related: Run Bolt.diy Locally with Ollama: Free v0 Alternative →

Local code agents do not behave like simple chat interfaces. Because Claude Code CLI passes system prompts, directory trees, file payloads, and tool definitions with every turn, your local model needs both high context capacity and strong function-calling (tool-use) capabilities.

If your local model cannot reliably produce structured JSON for tool calls, Claude Code will fail instantly with parsing errors.

Here are the tested baseline hardware specs for this setup:

• Minimum Specs: Apple Silicon M-series (32GB Unified Memory) or an Nvidia GPU with 16GB VRAM (e.g., RTX 4080 / RTX 3090). This handles 14B parameter models like qwen2.5-coder:14b.
• Recommended Specs: Apple Silicon M2/M3/M4 Max (64GB+ Unified Memory) or 24GB+ VRAM (e.g., RTX 4090 or dual RTX 3090s). This lets you run qwen2.5-coder:32b at 4-bit quantization with a full 32k context window.
• Software Requirements: Python 3.10+, Node.js 18+, Ollama v0.3.14+, and pip.

Step 1: Install and Test Ollama with a Tool-Capable Coder Model

Related: Kokoro TTS in Open WebUI: Zero-Latency Local Voice Setup →

First, you need Ollama installed and serving a model capable of tool calls. Standard instruction-tuned models like Llama 3.1 often struggle with Claude Code's complex system prompts. The qwen2.5-coder series is currently the most reliable open-weight model family for this specific task.

Open your terminal and pull the 14B or 32B Qwen2.5-Coder model:

# For GPUs with 16GB-24GB VRAM / 32GB Mac
ollama pull qwen2.5-coder:14b

For GPUs with 24GB+ VRAM / 64GB+ Mac (Highly Recommended)

ollama pull qwen2.5-coder:32b

Next, verify that your local Ollama instance is running by checking its API baseline:

curl http://localhost:11434/api/tags

You should receive a JSON response listing your installed models.

To handle large code repositories without running out of memory, set Ollama's context window environment variables. By default, Ollama limits context lengths to 2048 or 4096 tokens, which will truncate Claude Code's system prompts.

Add these to your shell profile (~/.zshrc or ~/.bashrc):

export OLLAMA_NUM_PARALLEL=1
export OLLAMA_MAX_LOADED_MODELS=1

Restart Ollama so these variables take effect.

Step 2: Install and Configure LiteLLM Proxy

Related: Connect Windsurf IDE to Local Ollama: Step-by-Step Setup →

LiteLLM acts as an API gateway. It accepts incoming requests matching the Anthropic Messages API format from Claude Code, rewrites them into Ollama's format, sends them to Ollama, and converts the streamed output back into Anthropic-compliant JSON chunks.

Install LiteLLM with its proxy dependencies:

pip install 'litellm[proxy]'

Create a dedicated configuration directory and a config.yaml file to map Anthropic's model names to your local Ollama model:

mkdir -p ~/.config/claude-ollama
nano ~/.config/claude-ollama/config.yaml

Paste the following YAML configuration inside ~/.config/claude-ollama/config.yaml:

model_list:
  # Map Claude 3.5 Sonnet requests to local Qwen 32B
  - model_name: claude-3-5-sonnet-20241022
    litellm_params:
      model: ollama_chat/qwen2.5-coder:32b
      api_base: http://localhost:11434
      num_ctx: 32768
      max_tokens: 8192
      temperature: 0.0

Map Claude 3 Haiku requests (used by Claude Code for fast summarization)

• model_name: claude-3-haiku-20240307 litellm_params: model: ollama_chat/qwen2.5-coder:14b api_base: http://localhost:11434 num_ctx: 16384 max_tokens: 4096 temperature: 0.0

Fallback wildcard mapping for any generic sonnet request

• model_name: claude-3-5-sonnet litellm_params: model: ollama_chat/qwen2.5-coder:32b api_base: http://localhost:11434 num_ctx: 32768

general_settings: master_key: sk-dummy-key-for-local-use drop_params: true

litellm_settings: set_verbose: false json_logs: false

Key points in this config:

1. model_name: Matches the exact model string Claude Code CLI requests under the hood (claude-3-5-sonnet-20241022).
2. ollama_chat/: Tells LiteLLM to use Ollama's chat completion endpoint, which preserves tool/function calling formats.
3. drop_params: true: Drops Anthropic-specific API parameters (like prompt caching headers) that Ollama does not support, preventing 400 Bad Request errors.
4. num_ctx: 32768: Expands Ollama's active context window to 32k tokens.

Step 3: Launch LiteLLM Proxy Server

Related: Fix Open WebUI Web Search: SearXNG Docker Guide →

Launch LiteLLM pointing to your configuration file on port 4000:

litellm --config ~/.config/claude-ollama/config.yaml --port 4000

You should see output similar to this:

INFO:     Started server process [18294]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:4000 (Press CTRL+C to quit)

Keep this terminal window open. LiteLLM must stay running in the background whenever you use Claude Code CLI.

Step 4: Configure Environment Variables and Install Claude Code CLI

Related: Open WebUI Ignoring Your Uploaded Documents? Fix It →

If you haven't installed the official Claude Code CLI yet, install it globally using npm:

npm install -g @anthropic-ai/claude-code

To route the CLI to your local LiteLLM proxy instead of api.anthropic.com, export custom environment variables in your terminal.

Create a launch script named claude-local to keep your environment organized without breaking standard Anthropic cloud access for other projects.

Save this as ~/.local/bin/claude-local:

#!/usr/bin/env bash

Override Anthropic Base API URL to point to LiteLLM

export ANTHROPIC_BASE_URL="http://localhost:4000"

Provide a dummy API key to pass SDK validation checks

export ANTHROPIC_API_KEY="sk-dummy-key-for-local-use"

Disable telemetry data collection

export CLAUDE_CODE_DISABLE_TELEMETRY=1

Execute the actual Claude Code CLI binary

claude "$@"

Make the script executable:

chmod +x ~/.local/bin/claude-local

Ensure ~/.local/bin is in your system PATH. You can now start a local session in any project directory simply by typing claude-local.

Step 5: Testing the Local Integration

Navigate to an example codebase and run your local setup:

cd ~/projects/my-python-app
claude-local

When the interactive terminal loads, ask the agent to inspect the codebase and run a basic bash command or file inspect action:

> Read package.json (or requirements.txt) and tell me what dependencies need updating.

Watch the terminal window running LiteLLM. You will see traffic streaming through:

POST /v1/messages HTTP/1.1 200 OK
Model: claude-3-5-sonnet-20241022 -> ollama_chat/qwen2.5-coder:32b
Tokens: Prompt: 1420 | Completion: 185

If set up correctly, Claude Code CLI will display its native UI components (spinners, tool confirmation prompts, diff viewers) while running off your local GPU.

Performance Realities: Cloud vs Local

While local setup is free and private, open-weight models behave differently than Anthropic's cloud-hosted Claude 3.5 Sonnet. Understanding these trade-offs will help you frame prompts effectively.

Troubleshooting Common Errors and Edge Cases

Running local models as backend drivers for a rigid cloud agent CLI introduces a few standard failure modes. Here is how to fix them.

1. Error: "Failed to parse tool call response" or JSON Format Failures

• Symptom: Claude Code prints red system errors indicating it could not interpret the response, or raw JSON spills into the chat interface.
• Cause: The local model failed to format its function call output into the exact schema expected by the Anthropic SDK.
• Fix: Lower the generation temperature to force deterministic outputs. Update your config.yaml under LiteLLM to set temperature: 0.0. Additionally, downgrade from a 7B or 14B model to a 32B model, as smaller models frequently struggle with tool call syntax.

2. Error: 400 Bad Request - Unsupported Parameter top_k or prompt_caching

• Symptom: LiteLLM receives a request from Claude Code CLI but returns a 400 status code straight back to the terminal.
• Cause: Claude Code CLI passes proprietary Anthropic headers (anthropic-beta: prompt-caching-...) that Ollama rejects.
• Fix: Ensure drop_params: true is set under general_settings: in your ~/.config/claude-ollama/config.yaml. This strips non-standard headers before passing the request to Ollama.

3. Extremely Slow Responses or Out of Memory (OOM) Errors

• Symptom: Generation crawls down to 1-2 tokens per second, or Ollama crashes with an Out-of-Memory error.
• Cause: The context window has grown larger than your available GPU VRAM, forcing Ollama to offload layers to CPU RAM.
• Fix: Reduce the context size in config.yaml from 32768 to 16384 or 8192. Additionally, clear Claude Code's session memory periodically inside the CLI using the /clear command.

4. Claude Code Freezes on Terminal Command Execution

• Symptom: Claude Code runs a local command (like ls or git status) but hangs indefinitely waiting for input.
• Cause: Local models sometimes forget to output the terminal end-of-sequence token alongside tool responses.
• Fix: Update Ollama to the latest release (ollama update). Older versions had edge-case bugs handling system stream terminations during active tool calls.

Frequently Asked Questions

Can I use Llama 3.3 70B instead of Qwen2.5-Coder?

Yes, provided you have sufficient hardware (such as 48GB+ VRAM or a Mac with 64GB+ RAM). Update your LiteLLM config.yaml model string to point to ollama_chat/llama3.3:70b. Keep in mind that while Llama 3.3 is great at general reasoning, Qwen2.5-Coder 32B generally outperforms it on structured tool calling, syntax manipulation, and code generation tasks.

Does prompt caching work with local Ollama models?

No. Claude Code CLI's native prompt caching relies specifically on Anthropic's server infrastructure. When routing through LiteLLM to Ollama, prompt caching parameters are dropped. The full system prompt and conversation history are re-evaluated by Ollama on each turn.

Why use LiteLLM instead of connecting Claude Code directly to Ollama?

Claude Code CLI specifically formats its outbound HTTP requests to match Anthropic's /v1/messages endpoint structure and header requirements. Ollama does not natively replicate the Anthropic API schema—it exposes an OpenAI-compatible endpoint (/v1/chat/completions) and a native Ollama endpoint (/api/chat). LiteLLM provides the necessary translation layer.

Is my code safe when using this local configuration?

Yes. Setting ANTHROPIC_BASE_URL to http://localhost:4000 reroutes all API requests away from Anthropic's servers directly to your local loopback address. Additionally, setting CLAUDE_CODE_DISABLE_TELEMETRY=1 prevents the CLI from sending usage metrics back to external endpoints. Your source code never leaves your local machine.