Self-Hosted Inference: Choosing Between Ollama and vLLM

Lab vs production, air-gapped or not — a working framework, not a benchmark chart.

As open-weight models keep closing the gap with frontier-class systems, more teams are asking whether they can self-host instead of calling an API. The honest answer is: usually yes, but “yes” hides a real decision — which engine actually serves that model once it’s on your hardware.

This post is about that decision. Not a benchmark showdown, not a “vLLM is objectively better” hot take — a working framework for choosing between Ollama and vLLM based on concurrency, hardware, and how much operational overhead you’re willing to carry. That framework changes depending on where the deployment sits: a lab standing up a self-hosted model for a handful of engineers has different constraints than a self-hosted production service that people actually depend on, and both look different again once you add an air gap. I’ll cover all three, and because a lot of this blog’s audience runs self-hosted infrastructure specifically because it has to — air-gapped, classified, or otherwise disconnected — I’ll walk through the parts that change when there’s no internet to lean on: pulling models from Hugging Face for offline transfer, and real Docker and Kubernetes configs for both engines.

And no, this isn’t a pitch for a $2M platform that promises to make your air gap disappear for the low, low price of a multi-year enterprise agreement (you know exactly who I mean). Ollama and vLLM are both free, both open, and both things you can stand up yourself this afternoon. If a vendor’s pitch deck for “AI in your SCIF” is longer than this blog post, that’s a tell, not a feature.

If you’ve been following Part 3: Operationalizing or the kmcp scaffolding walkthrough, this is the layer underneath both of those: the thing that actually serves the model your MCP tools and agents talk to.

1. The real fork: concurrency, not raw speed

Every comparison post opens with a benchmark chart, so let’s get it out of the way: on identical hardware serving a single user, the difference between Ollama and vLLM is small enough not to matter for most lab work. Where they diverge sharply is concurrency.

Ollama, built on llama.cpp, processes requests sequentially. It caps parallelism at a small number by default and queues everything past that. For one engineer at a terminal, that’s invisible. For a handful of people hitting the same endpoint, it starts to show. Push far enough — 15, 20 concurrent requests — and most of them are just sitting in a queue waiting their turn (which they will absolutely notice, and which they will absolutely bring up in the next standup).

vLLM takes a different architectural approach: PagedAttention manages the KV cache in non-contiguous memory pages instead of pre-allocating contiguous blocks, and continuous batching slots new requests into the GPU pipeline as soon as capacity frees up, rather than waiting for an entire batch to finish. The practical effect is that vLLM keeps the GPU saturated under load in a way Ollama’s sequential model structurally can’t.

One user waiting is fine. Five users queuing is fine. Twenty users queuing is a production outage in a lab that doesn’t have production support.

So the first real question isn’t “which is faster” — it’s “how many concurrent requests does this actually need to survive,” and the honest answer depends heavily on which side of the lab/production line you’re on.

A lab deployment — one engineer or a small team standing up a self-hosted model to prototype against, feed an MCP server, or validate a workflow before it goes anywhere real — usually never crosses the inflection point (roughly 5–8 concurrent users) where vLLM’s advantage becomes the deciding factor. Queuing is annoying there, not disqualifying. Nobody outside the team is depending on the response time.

A production deployment — self-hosted because policy, data sensitivity, or the air gap itself requires it, but still serving real users or downstream systems with an expectation of availability — is a different animal. Queuing under load isn’t an inconvenience there, it’s an incident. That’s true whether “production” means a connected internal service or a fully air-gapped one; the air gap changes how you troubleshoot a concurrency problem, not whether concurrency matters. If the workload has an SLA, a lab-grade concurrency assumption is the wrong starting point regardless of network posture.

2. Tensor parallelism: what it costs you self-hosted

Tensor parallelism is the other axis people reach for immediately, usually because a model is too large to fit on one GPU. It’s worth being precise about what it actually buys you and what it costs operationally, because the cost is a lot higher once you’re disconnected from the internet.

vLLM supports true tensor parallelism: pass --tensor-parallel-size 2 (or 4, or 8) and it splits the model’s layers across matching GPUs over NVLink or PCIe, coordinated through NCCL. Ollama’s multi-GPU story is different — OLLAMA_NUM_PARALLEL lets you run multiple inference contexts simultaneously, but that’s concurrent single-GPU-style execution, not layer-splitting. It increases VRAM pressure without the throughput gain you’d get from real tensor parallelism.

The practical guidance: don’t reach for tensor parallelism because a model is technically too large for one GPU if quantization gets it under the line instead. A quantized model on one GPU with Ollama is operationally simpler than a tensor-parallel deployment across two GPUs with vLLM, and “simpler” has real value when you’re the only one who can fix it and the fix has to happen without internet access.

Granted, quantization isn’t free — you’re trading some accuracy and context headroom for that simplicity, and for the wrong workload that trade is a bad one. To be fair, most lab workloads never actually need the accuracy they think they need. Nobody’s demoing a Q4 quant to a general officer and getting asked to explain the perplexity numbers.

That said, “simpler” isn’t the only variable, and lab and production deployments will often land on opposite sides of this tradeoff on purpose. A production self-hosted service, air-gapped or not, may have already committed to the larger model and the multi-GPU footprint because the accuracy requirement isn’t negotiable. In that case the NCCL and driver-matching discipline above isn’t optional overhead to avoid — it’s a one-time cost you pay deliberately, then bake into your build and provisioning process so it’s not a surprise at deploy time.

3. Pulling a model from Hugging Face for offline transfer

Regardless of which engine you land on, the model has to get onto the air-gapped side somehow. This is the step where the engine choice actually forces a decision, because the two engines want different file formats.

Ollama consumes GGUF. vLLM consumes safetensors — full precision, or a pre-quantized format like AWQ, GPTQ, or FP8. Decide which engine you’re targeting before you pull, because it determines which repo variant you download.

The full flag reference for downloading and filtering repos lives in the Hugging Face Hub download guide — worth a skim before your first pull, since allow_patterns and --include syntax differ slightly between the CLI and the Python API.

Install the Hugging Face CLI (connected machine)

pip install "huggingface_hub[cli]"

# authenticate if pulling a gated model (e.g. Llama family)
hf auth login

Pull for Ollama — GGUF, real files, no cache symlinks

hf download TheBloke/Mistral-7B-Instruct-v0.2-GGUF \
  --include "*Q4_K_M.gguf" \
  --local-dir ./transfer/mistral-7b-gguf

--local-dir writes real files instead of the default symlinked cache under ~/.cache/huggingface. That matters for transfer: symlinks and hardlinks don’t survive a copy across removable media or an NFS/SMB share cleanly. You want plain files sitting in a directory you can hand straight to the transfer process.

Pull for vLLM — safetensors snapshot

python3 - <<'PY'
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="meta-llama/Llama-3.1-8B-Instruct",
    local_dir="./transfer/llama-3.1-8b-instruct",
    allow_patterns=["*.safetensors", "*.json", "tokenizer*"],
)
PY

Trim what you pull with allow_patterns — you don’t need original PyTorch bins if safetensors are present, and you don’t need every tokenizer variant. Every extra gigabyte is extra time on whatever media is doing the actual crossing.

Before that copy leaves the connected machine, sanity-check that nothing in your pipeline is going to try to phone home once it’s inside the wire:

export HF_HUB_OFFLINE=1
# any tool that still tries to hit huggingface.co will now fail loudly
# instead of hanging on a DNS timeout inside the air gap

From here, the transfer follows the same pattern covered in the kmcp post’s Harbor workflow — connected machine builds or stages the artifact, physical or mediated transfer moves it across, and it lands in internal storage (or a registry, for container images) on the other side.

4. Example configs — Ollama

Docker

docker run -d \
  --name ollama \
  --gpus all \
  -p 11434:11434 \
  -v /data/models/ollama:/root/.ollama \
  -e OLLAMA_NUM_PARALLEL=4 \
  -e OLLAMA_MAX_LOADED_MODELS=2 \
  ollama/ollama:latest

Mount the model directory as a volume rather than relying on ollama pull at container start — on the air-gapped side there’s nothing for it to pull from. Load the GGUF you transferred earlier with a Modelfile:

# Modelfile
FROM ./transfer/mistral-7b-gguf/mistral-7b-instruct-v0.2.Q4_K_M.gguf

docker exec ollama ollama create mistral-7b -f /root/.ollama/Modelfile

Full environment variable reference — parallelism, model keep-alive, context length defaults — is in the Ollama FAQ docs.

Kubernetes

Consistent with the deployment pattern from the kmcp post — a plain Deployment and Service, no operator required for a single-engine inference server:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      containers:
        - name: ollama
          image: ollama/ollama:latest
          ports:
            - containerPort: 11434
          env:
            - name: OLLAMA_NUM_PARALLEL
              value: "4"
            - name: OLLAMA_MAX_LOADED_MODELS
              value: "2"
          resources:
            requests:
              nvidia.com/gpu: 1
            limits:
              nvidia.com/gpu: 1
          volumeMounts:
            - name: model-storage
              mountPath: /root/.ollama
          readinessProbe:
            httpGet:
              path: /api/tags
              port: 11434
            initialDelaySeconds: 15
            periodSeconds: 10
      volumes:
        - name: model-storage
          persistentVolumeClaim:
            claimName: ollama-models-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: ollama
  namespace: inference
spec:
  selector:
    app: ollama
  ports:
    - port: 11434
      targetPort: 11434

5. Example configs — vLLM

Docker

docker run -d \
  --name vllm \
  --gpus all \
  --shm-size=8g \
  -p 8000:8000 \
  -v /data/models/llama-3.1-8b-instruct:/models/llama-3.1-8b-instruct \
  vllm/vllm-openai:latest \
  --model /models/llama-3.1-8b-instruct \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192

--shm-size matters more than it looks like it should — vLLM’s multiprocessing setup will silently degrade or crash under the small default shared memory Docker allocates. Set it explicitly. Full flag reference, including every quantization and parallelism option, is in the vLLM OpenAI-compatible server docs.

Kubernetes — single GPU

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
  namespace: inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args:
            - "--model=/models/llama-3.1-8b-instruct"
            - "--tensor-parallel-size=1"
            - "--gpu-memory-utilization=0.90"
            - "--max-model-len=8192"
          ports:
            - containerPort: 8000
          resources:
            requests:
              nvidia.com/gpu: 1
            limits:
              nvidia.com/gpu: 1
          volumeMounts:
            - name: model-storage
              mountPath: /models
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 30
            periodSeconds: 10
      volumes:
        - name: model-storage
          persistentVolumeClaim:
            claimName: vllm-models-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: vllm
  namespace: inference
spec:
  selector:
    app: vllm
  ports:
    - port: 8000
      targetPort: 8000

Kubernetes — tensor-parallel across multiple GPUs, single node

          args:
            - "--model=/models/llama-3.1-70b-instruct"
            - "--tensor-parallel-size=4"
            - "--gpu-memory-utilization=0.92"
          resources:
            requests:
              nvidia.com/gpu: 4
            limits:
              nvidia.com/gpu: 4
          env:
            - name: NCCL_DEBUG
              value: "WARN"
            - name: NCCL_P2P_DISABLE
              value: "0"

6. Where each one actually wins: lab vs production

The toll booth vs the metered tollway

Here’s my actual take, not the diplomatic vendor-neutral one: Ollama is a single toll booth. vLLM is a multi-lane tollway with metered on-ramps. One booth processes every car exactly one at a time — doesn’t matter how fast any individual car is, the booth is the bottleneck, and everyone behind car one is just waiting. A metered tollway is built for throughput: multiple lanes running simultaneously, on-ramps that meter cars in as capacity actually frees up instead of admitting a fixed batch and waiting for all of them to clear before letting the next batch on. That’s continuous batching, more or less, with asphalt instead of a GPU.

I prototype at the toll booth. Every time. New model, new workflow, new MCP tool I want to sanity-check before it touches anything real — Ollama, every time, no exceptions. It’s not a compromise, it’s the correct tool for that job. One booth is plenty when it’s just me driving through. I only reach for vLLM once I actually know the workload is going to see real traffic, because standing up a multi-lane tollway for a road nobody’s driving on yet is civil engineering nobody asked for.

Granted, that’s a lab habit talking. If I already know something’s headed to production with real concurrency on day one, I skip the single booth entirely and go straight to building the tollway — no point prototyping infrastructure you’re about to tear out.

The two aren’t mutually exclusive across a deployment’s lifecycle, either. Prototype against Ollama’s OpenAI-compatible API while you’re still figuring out which model and prompt shape you actually need, then graduate the workload to vLLM once it’s headed toward production and concurrency or latency requirements justify the extra operational surface. Don’t pick the harder tool before you need it — but also don’t let “it worked fine in the lab” be the whole justification once real load and an SLA are attached to it, because that’s a different problem with a different answer.

Air-gapped raises the stakes on both ends of that lifecycle. In the lab, it’s the reason to stay on Ollama a little longer than you might connected — every extra moving part is one more thing you can’t quickly patch or roll back if it breaks. In production, it’s the reason to get the vLLM tensor-parallel discipline right the first time, because “redeploy and see what happens” isn’t a real option when a fix depends on a package that isn’t on this side of the wire.

Zoom out past inference engines and the same pattern shows up everywhere in self-hosted infrastructure: the right answer depends on which environment you’re actually in, and pretending lab and production are the same environment is where most of the pain comes from. A lab rewards simplicity — fewer moving parts, faster iteration, tolerance for a queue. Production rewards discipline — the concurrency model, the hardware topology, and the failure mode all have to be decided on purpose, not inherited from whatever got the lab prototype working.

Self-hosted because you chose to is already a different risk profile than self-hosted because you have to, and air-gapped production sits at the intersection of both pressures at once: production’s demand for real discipline, with none of the connected side’s ability to patch, roll back, or download your way out of a mistake. Know which environment you’re building for before you pick the engine. Start from the load you actually have. Earn the complexity.

And you don’t need a seven-figure platform contract to get there. You need a GPU, an afternoon, and the willingness to read the docs instead of the sales deck. Run the single booth until you actually have enough traffic to justify the tollway.