Skip to content
dhii.ai

self-hosted LLM

Ollama vs vLLM: Choosing an Engine for Self-Hosted LLMs

A practical comparison of Ollama and vLLM for running open-weight LLMs on your own servers — ease of use, throughput, hardware, quantisation, APIs and production operations.

By Amit Valecha · · 6 min read

Once you decide to run language models on your own infrastructure — for privacy, cost or control — you need an inference engine: the software that loads the model and serves requests. Two of the most popular open-source options are Ollama and vLLM. They’re often mentioned together, but they’re designed for different situations. Picking the wrong one leads either to unnecessary complexity or to a server that collapses under real load.

We run self-hosted models ourselves — our Dhii Chat workspace uses Open WebUI on top of Ollama — and deploy both engines for clients. Here’s how we choose.

Ollama in brief

Ollama is designed for simplicity. One command installs it; another downloads and runs a model. It manages a library of pre-packaged models, mostly in quantised GGUF format built on the llama.cpp engine, and exposes a local API including OpenAI-compatible endpoints.

Strengths:

  • Extremely easy setup on Linux, macOS and Windows.
  • Runs on modest hardware, including CPUs, Apple Silicon and consumer GPUs, thanks to quantised models.
  • Simple model management — pull, run, switch and remove models easily.
  • Integrates widely: Open WebUI, n8n, coding tools and many frameworks support it out of the box.
  • Automatic memory management, loading and unloading models as needed.

Limitations:

  • Throughput under concurrency. Ollama can handle parallel requests, but it isn’t optimised for serving many simultaneous users at maximum efficiency the way vLLM is.
  • Fewer production knobs for fine-grained control of batching, scheduling and multi-GPU deployments.
  • Quantised models by default. Great for memory, but you should evaluate the quality impact for your task.

Best for: individual developers, small teams, internal tools with light concurrent use, prototypes, edge deployments and any situation where simplicity matters more than peak throughput.

vLLM in brief

vLLM is a high-performance inference and serving engine built for throughput. Its core innovations — efficient memory management for the attention key-value cache (PagedAttention) and continuous batching of incoming requests — let it serve many concurrent requests on the same GPU far more efficiently than naive serving.

Strengths:

  • High throughput with many concurrent users, thanks to continuous batching.
  • OpenAI-compatible server that drops into existing applications.
  • Broad model support directly from Hugging Face formats, plus many quantisation formats.
  • Multi-GPU and multi-node deployment with tensor and pipeline parallelism.
  • Advanced features such as prefix caching, speculative decoding, structured output support and serving multiple LoRA adapters on one base model.
  • Production observability with metrics endpoints for monitoring.

Limitations:

  • GPU-focused. It’s designed primarily for data-centre and high-end GPUs.
  • More configuration. Memory utilisation, context length, parallelism and quantisation settings need tuning.
  • Heavier operations. Containers, drivers and versions must be managed carefully.

Best for: production APIs, company-wide assistants, applications with many simultaneous users and high-volume batch processing.

Side-by-side

Ollama vLLM
Setup Minutes Hours, with tuning
Hardware CPU, Apple Silicon, consumer and data-centre GPUs Primarily NVIDIA/AMD data-centre GPUs
Model format Mostly GGUF (quantised) Hugging Face weights plus quantised formats
Concurrency Light to moderate High
Throughput per GPU Good for single users Excellent under load
Multi-GPU Basic Advanced (tensor/pipeline parallel)
LoRA adapters Via model files Native multi-adapter serving
API Native plus OpenAI-compatible OpenAI-compatible
Operations Minimal Moderate

How to choose

Ask four questions.

1. How many people or processes will use it at the same time? If it’s a handful of users chatting occasionally, Ollama is fine. If it’s an application serving hundreds of requests per minute, or a company-wide assistant at 9am on a Monday, vLLM’s batching makes a large difference to latency and hardware cost.

2. What hardware do you have? No GPU, or a consumer GPU? Ollama with quantised models is often the practical option. Dedicated data-centre GPUs? vLLM will use them far more efficiently under load.

3. Who will operate it? A team without infrastructure experience will find Ollama much easier to keep running. vLLM rewards teams comfortable with containers, GPU drivers and monitoring.

4. What models and features do you need? If you need the full-precision model, specific quantisation formats, very long contexts, multi-LoRA serving or multi-GPU parallelism, vLLM is the stronger fit.

A common path

Many organisations follow a natural progression: start with Ollama to explore models and prove value; then, as usage grows, move production traffic to vLLM while keeping Ollama for development and experimentation. If your applications call the models through an OpenAI-compatible gateway such as LiteLLM, this migration is mostly configuration.

Operational tips for either engine

  • Benchmark on your own workload. Measure latency and throughput with realistic prompt lengths and concurrency, not just a single request.
  • Evaluate quality after quantisation. Lower-bit models save memory but may lose accuracy on your task.
  • Put it behind authentication. Never expose an inference server directly to the internet without access control; place it behind a reverse proxy with auth and TLS.
  • Monitor. Track GPU memory, utilisation, queue length, latency and errors.
  • Pin versions. Upgrade deliberately, test first and keep a rollback path.
  • Plan capacity for peaks, not averages.

A sizing example

Sizing is where many self-hosting plans go wrong, so it helps to reason it through. Suppose a company-wide assistant for 200 staff, of whom perhaps 20 might send a request in the same minute at peak, with prompts of around 2,000 tokens (including retrieved context) and answers of around 400 tokens.

With Ollama on a single GPU, requests are processed with limited parallelism, so at peak some users will wait noticeably while others’ answers are generated. With vLLM on the same GPU, continuous batching lets many requests share the hardware efficiently, so latency stays far more stable as concurrency rises.

Memory matters as much as compute. The model’s weights must fit, and so must the key-value cache that grows with context length and concurrent requests. A model that fits comfortably for one user may run out of cache memory for twenty. That is why we load-test with realistic prompt sizes and concurrency before recommending hardware, rather than relying on a model’s parameter count alone.

Security checklist for self-hosted inference

Whichever engine you choose, a self-hosted model is a server holding potentially sensitive prompts and documents. At a minimum: bind the inference server to a private interface and expose it only through an authenticated reverse proxy with TLS; keep the host patched and the container images pinned; restrict who can pull or load new models; log access without logging full prompt content unless you have a reason and a retention policy; and back up configuration, not just data. For chat front ends such as Open WebUI, disable open sign-up unless you intend it, enforce strong authentication and review admin roles regularly.

Beyond these two

Other good engines exist: llama.cpp’s own server for efficient CPU and small-GPU inference, Hugging Face TGI, SGLang for high-performance structured generation, and vendor-optimised stacks such as TensorRT-LLM. The same questions — concurrency, hardware, operations and features — guide the choice.

Summary

Ollama optimises for simplicity; vLLM optimises for throughput. Choose Ollama for small-scale, easy-to-run deployments and vLLM for production workloads with real concurrency. Many teams use both. Whichever you choose, put authentication, monitoring and evaluation around it.

See our private and self-hosted LLM service and our comparison of self-hosting versus APIs.

Services related to this article

Keep reading

Start a project

Have a workflow that should run itself?

Tell us what you want automated or built. You get a written assessment of feasibility, risks and a fixed quote — usually within two business days.