Run Browser-Use Locally with Ollama: 2026 Setup Guide

Why Local Web Agents Crash (And How Qwen 2.5 VL Fixes It)

Related: Connect LibreChat to Ollama: Docker Setup Guide →

Running web-browsing AI agents entirely on your local machine used to be an exercise in frustration. Standard Large Language Models (LLMs) can read text, but modern websites are built for visual spatial processing. When open-source agents try to click buttons based purely on raw HTML or text trees, they fail on basic cookie banners, dynamic popups, and shadow DOM elements.

The release of open-weights vision-language models—specifically Alibaba’s Qwen 2.5 VL suite—changed the landscape. These models perform multi-modal reasoning, mapping spatial bounding boxes on a browser screenshot directly to actionable UI elements.

       +-----------------------------------------------+
       |             Browser Screen Capture            |
       |  +-----------------------------------------+  |
       |  |  [ Search Input ]   [ Submit Button ]   |  |
       |  +-----------------------------------------+  |
       +-----------------------+-----------------------+
                               |
                               v
       +-----------------------------------------------+
       |           Browser-Use Tree Processor          |
       |      (Extracts Interactive Element Map)       |
       +-----------------------+-----------------------+
                               |
                               v
       +-----------------------------------------------+
       |          Ollama Engine (Qwen 2.5 VL)          |
       |   Input:  Screenshot + Vision Bounding Boxes   |
       |   Output: Structured Tool Call JSON           |
       +-----------------------+-----------------------+
                               |
                               v
       +-----------------------------------------------+
       |           Playwright Engine Execution         |
       |     (Executes: click(), type(), scroll())     |
       +-----------------------------------------------+

Combining the browser-use framework with Ollama allows you to run an autonomous web agent without sending API tokens, sensitive user sessions, or financial data to cloud vendors.

However, running vision agents locally presents a major technical bottleneck: memory explosion. Every browsing step sends a multi-megapixel image, a large system prompt, and an interactive DOM map back into the context window.

Without explicit memory management, a local 7B or 14B model will crash your GPU with a CUDA Out Of Memory (OOM) error within three web pages.

This guide covers how to build a production-ready, fully local browser-use deployment with Ollama, fix context window memory spikes, and prevent JSON validation failures during local function calling.

Hardware Requirements and Ollama Environment Setup

Related: Ollama Flash Attention: Cut VRAM Usage & Boost Speed →

Running vision-based web automation requires careful VRAM allocation. The vision encoder inside Qwen 2.5 VL processes images into large visual tokens, which consume far more KV cache than pure text inputs.

VRAM and Hardware Allocations

• 8 GB VRAM (RTX 3060 / 4060): Can run qwen2.5-vl:3b at 4-bit quantization with a restricted 8,192 token context window. Best for basic form-filling and static page navigation.
• 12 GB to 16 GB VRAM (RTX 4070 / 4080 / Apple M-Series 18GB+): The sweet spot for running qwen2.5-vl:7b at q4_k_m with a 16,384 context window. Handles complex multi-step navigation, e-commerce checkouts, and dynamic web apps.
• 24 GB VRAM+ (RTX 3090 / 4090 / Apple M-Series 36GB+): Enables running qwen2.5-vl:7b at fp16 or qwen2.5-vl:14b with a full 32,768 context window, allowing dozens of browser actions without context truncation.

System Configuration

First, make sure you have installed the latest version of Ollama (v0.5.0 or higher is required for native Qwen 2.5 VL support).

Set these environment variables before running Ollama to ensure flash attention is active and memory allocation is optimized:

# Linux / macOS terminal setup
export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_NUM_PARALLEL=1
export OLLAMA_MAX_LOADED_MODELS=1

Pull the visual reasoning model optimized for instruction following

ollama pull qwen2.5-vl:7b

On Windows PowerShell:

$env:OLLAMA_FLASH_ATTENTION="1"
$env:OLLAMA_NUM_PARALLEL="1"
$env:OLLAMA_MAX_LOADED_MODELS="1"
ollama pull qwen2.5-vl:7b

Enabling OLLAMA_FLASH_ATTENTION=1 reduces the memory overhead of long visual context sequences by up to 40%, preventing early OOM crashes when navigating complex DOM structures.

Step-by-Step Python Configuration for Browser-Use and Ollama

Related: Self-Host Perplexica with Local Ollama: Zero-Cost AI Search →

To wire Ollama into browser-use, we use LangChain’s ChatOllama wrapper. Standard API configurations will fail because browser-use relies heavily on structured output (function calling) to trigger browser actions via Playwright.

Installation

Create a fresh virtual environment and install the required dependencies:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

pip install browser-use langchain-ollama playwright playwright install chromium

Complete Working Python Script

Create a file named local_agent.py. The setup below explicitly configures num_ctx, overrides screenshot scaling parameters, and sets the system layout to keep memory consumption stable.

import asyncio
from browser_use import Agent, Browser, BrowserConfig
from langchain_ollama import ChatOllama

async def main(): # 1. Configure the local LLM engine via Ollama # High context (16384) is mandatory for vision-based browser agents llm = ChatOllama( model="qwen2.5-vl:7b", temperature=0.0, num_ctx=16384, num_predict=2048, other_parameters={ "top_k": 10, "top_p": 0.9, } )

2. Configure headless browser behavior to reduce DOM bloat

browser = Browser(
    config=BrowserConfig(
        headless=False,  # Set to True for unattended background jobs
        disable_security=True,  # Bypasses local CORS/SSL warnings
        extra_chromium_args=[
            "--window-size=1280,1024",
            "--force-device-scale-factor=1",
        ]
    )
)

3. Instantiate the agent with strict action constraints

task = "Go to Wikipedia, search for 'Autonomous System', and extract the first paragraph."

agent = Agent(
    task=task,
    llm=llm,
    browser=browser,
    use_vision=True,  # Enables visual processing via Qwen 2.5 VL
    max_actions_per_step=1,  # Forces sequential steps, preventing token bloat
    max_failures=3,
    retry_delay=2,
)

4. Execute the web workflow

print(f"Starting execution for task: {task}")
history = await agent.run(max_steps=15)

# 5. Extract and print final results
final_result = history.final_result()
print("\n--- Execution Completed ---")
print(final_result)

await browser.close()

if name == "main": asyncio.run(main())

Fixing the Infamous Function-Calling OOM in Local Vision Models

Related: Run CrewAI with Ollama: Fix Crashes & Local Setup Guide →

If you run browser-use locally with default settings, you will eventually encounter two errors:

1. CUDA out of memory: Tried to allocate X GiB
2. Failed to parse structured output: Model responded with invalid JSON format

Here is why these happen and how to solve them.

Problem 1: Visual Token Accumulation (OOM)

Every time browser-use takes a step, it feeds a full page screenshot back to the LLM. By Step 5, the model context contains 5 high-resolution images. With standard standard context processing, VRAM usage scales exponentially.

Step 1: System Prompt + DOM + Image 1                     --> ~4,000 tokens
Step 2: System Prompt + DOM + Image 1 + Image 2           --> ~7,500 tokens
Step 3: System Prompt + DOM + Image 1 + Image 2 + Image 3 --> ~11,000 tokens (OOM Hazard)

The Solution: Image Compression and Viewport Trimming

Restrict the browser window resolution to 1280x800 or 1024x768. Do not run browser agents at 4K or 1080p resolutions.

You can also write a lightweight custom image pipeline in Python to downsample screenshots before sending them to Ollama. Modify your BrowserConfig to force hardware acceleration off and limit device pixel ratios:

# Force low-res viewport rendering in BrowserConfig
browser = Browser(
    config=BrowserConfig(
        headless=True,
        extra_chromium_args=[
            "--window-size=1024,768",
            "--high-dpi-support=0",
            "--device-scale-factor=1",
            "--disable-gpu",  # Offloads browser GPU pressure so your dedicated VRAM is reserved for Ollama
        ]
    )
)

Additionally, adjust the vision context settings inside your agent loop using max_input_tokens wrappers, or enforce aggressive context pruning by setting max_actions_per_step=1.

Problem 2: Structured Function Calling Collapses

Local models struggle to simultaneously maintain visual reasoning, track DOM elements, and emit perfectly schema-compliant JSON tool parameters. If qwen2.5-vl returns plain text instead of structured tool commands, browser-use fails.

The Solution: Custom System Prompt Override

When using local models, override system prompts to instruct the local model to avoid emitting conversational prose. Append structural enforcement rules to your task prompt:

task_prompt = """
You are a web automation engine. You MUST emit ONLY JSON function calls.
Do not include commentary, intro text, or reasoning outside the call object.

TASK: Go to https://news.ycombinator.com, click on 'Submit', and read the field labels. """

agent = Agent( task=task_prompt, llm=llm, browser=browser, use_vision=True, planner_interval=1, # Forces explicit planning step before actions )

Local vs Cloud Web Agents: Real Performance Benchmarks

Related: Connect Avante.nvim to Local Ollama: 2026 Neovim Guide →

To quantify the tradeoffs between local models and commercial APIs, I benchmarked three standard scenarios across different execution backends:

1. Form Automation: Navigating a multi-page checkout flow.
2. Dynamic Search: Navigating past modal popups on an e-commerce platform.
3. Data Extraction: Extracting 10 structured product listings into JSON.

Testing was conducted on an isolated test bench: AMD Ryzen 9 7950X, 64GB DDR5 RAM, NVIDIA RTX 4090 (24GB VRAM) running Linux Ubuntu 24.04 LTS.

Takeaways from the Data

• Claude 3.5 Sonnet remains the standard for speed and handling edge-case dynamic UI popups.
• Qwen 2.5 VL 7B (Ollama) delivers an optimal balance of cost and performance. It hits an 88% success rate on structured web tasks while operating completely offline within 7GB of VRAM.
• Qwen 2.5 VL 72B offers accuracy close to cloud endpoints, but the latency per step makes it impractical for real-time web tasks unless deployed on dedicated multi-GPU clusters.

Troubleshooting Common Ollama and Browser-Use Bugs

Bug 1: Ollama Model Keep-Alive Timeout

By default, Ollama unloads models from VRAM after 5 minutes of inactivity. When browser-use pauses to perform page loads or network waiting, Ollama may drop the model, leading to connection timeouts when the agent sends its next command.

llm = ChatOllama(
    model="qwen2.5-vl:7b",
    keep_alive="-1",  # Prevents model offloading during long web tasks
    num_ctx=16384,
)

Bug 2: Browser Process Leaks and Orphan Chromium Tasks

When local scripts crash due to Python syntax errors or LLM execution exceptions, Chromium processes managed by Playwright may remain running in the background, consuming RAM and locking port resources.

import asyncio
from browser_use import Agent, Browser, BrowserConfig
from langchain_ollama import ChatOllama

async def safe_agent_run(): browser = Browser(config=BrowserConfig(headless=True)) try: llm = ChatOllama(model="qwen2.5-vl:7b", num_ctx=16384) agent = Agent( task="Navigate to example.com and verify the heading title.", llm=llm, browser=browser ) await agent.run() except Exception as e: print(f"Execution failed: {str(e)}") finally: # Guarantees process cleanup on exit await browser.close()

if name == "main": asyncio.run(safe_agent_run())

Frequently Asked Questions

Can I run browser-use with Ollama on an Apple Silicon Mac?

Yes. Apple Silicon Macs with unified memory run vision models effectively. An M1/M2/M3 Mac with at least 18GB of unified RAM can execute qwen2.5-vl:7b efficiently. Set num_ctx=16384 and ensure you assign at least 12GB of RAM to Metal acceleration.

Why does Qwen 2.5 VL fail to click specific web buttons?

Vision-based detection usually fails when page scale factors vary or when bounding box pixel values do not match your current display scaling. Ensure your extra_chromium_args contain --force-device-scale-factor=1 so image coordinates align directly with the visual element map generated by browser-use.

How do I prevent Ollama from running out of VRAM on long workflows?

Lower your browser resolution to 1024x768, set max_actions_per_step=1, reduce num_ctx from 32,768 down to 16,384, and launch Ollama with OLLAMA_FLASH_ATTENTION=1. If memory errors persist, switch from a 7B fp16 model build to a quantized q4_k_m build.

Can I run local browser agents in headless mode inside Docker?

Yes. To run browser-use inside Docker alongside Ollama, you must install Playwright's native system dependencies inside your container, expose port 11434, and pass the --ipc=host flag to prevent Chromium shared-memory crashes.