The Weight on One Side of the Wall

Where we left off

Last post covered the why — no frontier model, no API key, no internet, and why that’s still worth building agentic AI inside of anyway. This post is the how. Specifically, it’s about air-gapped LLM deployment in full: getting both the MCP server and the inference layer from an idea on a whiteboard to something running inside a wall that doesn’t talk to the outside world.

Most writing about this topic picks one or the other. Either it’s a tutorial on deploying an MCP server, or it’s a tutorial on running a local LLM. In an air-gapped environment, you can’t separate them — you need both, you need them to talk to each other, and you need to get them both across the boundary first. So this post covers both pipelines.

The asymmetry up front, because it’s the whole point: almost everything that requires judgment happens on the internet-facing side. Almost nothing does on the air-gapped side. That’s not a limitation I’m working around. That’s the design.

Two pipelines, one connected side

Everything starts on a system with internet access. There are two distinct pipelines running in parallel — the MCP server and the inference layer — and while they follow similar patterns, they’re not the same thing and shouldn’t be treated like they are.

// pipeline 1: the MCP server

1. Scaffold the MCP server project using kmcp.
2. Write the actual tool logic in Python (FastMCP), dropped into the scaffold kmcp generated.
3. Push the project into CI/CD. Code verification and vulnerability/dependency scanning run automatically on push — this is where trust gets established.
4. Build the container image from the verified, scanned code.
5. Push the approved image to the internet-facing registry — Harbor, in my setup.

// pipeline 2: the inference layer

The inference layer is two separate things that often get conflated: the serving container and the model weights. They travel differently, and treating them as the same artifact is how you end up with a 70GB container image and a confused ops team.

1. Pull the vLLM base image and verify it in CI/CD — same pipeline as the MCP server. Build, scan, push to Harbor as a standard container image.
2. On the connected side, download model weights from HuggingFace using huggingface-cli snapshot_download. Stick to quantized formats — GGUF, AWQ, or GPTQ depending on your hardware — because full-precision weights for anything useful are enormous, and quantization is honestly what makes air-gapped LLM deployment viable at all.
3. Package and transfer the weights. More on this in the next section, because it’s its own problem.

Look at where the line falls. Steps one through seven — everything requiring judgment, everything that establishes trust — happen on one side of the wall. Steps eight through fourteen are mechanical. Move the artifacts. Stage the weights. Run what already passed the gate. In fact, that’s the whole air-gapped side of the job, and if it feels anticlimactic, good. It’s supposed to.

The model weight problem

The MCP server and the vLLM container are just container images — they follow the same pipeline, cross the wall the same way, and land in Harbor the same way. Model weights are a completely different artifact class. They’re not code. They’re not images. They’re data, they can be enormous, and they need their own transfer strategy.

First, quantization — because if you’re not already using it, it’s the thing that makes air-gapped LLM deployment actually viable in constrained environments. Full-precision weights for a 13B parameter model can run 26GB or more. The same model in AWQ or GPTQ quantization lands somewhere between 7 and 10GB. GGUF can go lower depending on the quantization level. That difference matters a lot when you’re physically moving data across a boundary and you’re working with limited storage on the other side.

For the transfer itself, two approaches depending on the model and your setup:

// raw file transfer

The simpler path. Use huggingface-cli snapshot_download on the connected side to pull the full model repository — weights, tokenizer, config, the works — then tar the directory and move it as a file bundle. On the air-gapped side, extract to a persistent volume or NFS share and mount it into your vLLM container at runtime. This works for any model, requires no special tooling on the receiving end, and keeps the weights completely separate from the serving container.

// OCI artifact via Harbor

Harbor 2.x supports OCI artifacts natively, which means you can push model weights as OCI layers and pull them the same way you’d pull a container image. This approach integrates the weights into your existing registry workflow and gives you versioning and audit history essentially for free. The tradeoff is setup complexity — you need tooling like oras to push and pull non-image OCI artifacts, and it’s more to explain to whoever’s managing the internal registry on the air-gapped side. For shops that are already heavily Harbor-centric, it’s worth the investment. For everyone else, the tar bundle is usually fine.

Why a narrow crossing is the feature

Here’s the part worth sitting with, because it’s easy to look at a 14-step table and think the air-gapped side is getting shortchanged. In fact, it got the simple end, and simple is the goal. The heavier the connected-side pipeline, the thinner and more auditable the crossing becomes. That’s not a coincidence — that’s the whole reason to build it this way.

fig. 1 — two pipelines converge at the wall; two slim artifacts cross; one deployment on the other side

Name what actually crosses the wall and the point makes itself: two container images, a weight bundle or OCI artifact, and maybe a manifest or two. Not the source. Not the build tools. Not a dependency cache. Just the finished, already-approved results of everything that happened on the connected side. Less crossing the wall isn’t a compromise you make for security. It’s the security.

What air-gapped LLM deployment actually looks like

Once both artifacts are inside the wall, the job is straightforward by design. The container images go into the internal Harbor instance — same tool, separate from anything internet-facing, never touched the outside world. The weights go to persistent storage, either a PVC on Kubernetes or an NFS share, depending on your setup.

From there, two deployment patterns for vLLM depending on model size and what makes operational sense:

// weights mounted at runtime

The more common pattern for anything non-trivial. The vLLM container image stays lean — it’s just the serving layer. The weights live on a PVC or NFS share and get mounted in at deploy time. This keeps the container image manageable, lets you swap or update weights without rebuilding and re-crossing the image, and works well with Kubernetes volume claims. After all, it’s the same image regardless of which model you’re serving — only the mount changes.

// weights baked into the image

Simpler to reason about and sometimes the right call for smaller quantized models where the weight size isn’t prohibitive. One image, no external mounts to manage, no volume claim to get wrong. The tradeoff is image size and the fact that updating the model means building and re-crossing a new image. For a handful of tools running one specific model, that’s often an acceptable cost. For a more dynamic environment, it gets unwieldy fast.

Once vLLM is running, kmcp deploy and the Kubernetes controller bring up the MCP server, reconciling the image against the MCPServer CRD. Depending on your setup, you may also want a gateway layer sitting between MCP and the vLLM endpoint — something like LiteLLM or Bifrost (there are others) to handle routing, load balancing across multiple models, or to normalize the API surface if you’re swapping serving backends. If you’re running one model and one serving container, a gateway is probably more overhead than it’s worth upfront. If you’re juggling multiple models or need rate limiting and observability at the API layer, it’s worth the extra container crossing the wall. Either way, the gateway follows the same pipeline as everything else — build, scan, push, export, import, deploy.

From the MCP server’s perspective, the local inference layer is just an API endpoint — OpenAI-compatible, whether vLLM is serving directly or a gateway is in front of it. The wall doesn’t change how they communicate. It just changes where the thing they’re talking to lives.

In fact, most of the public conversation about LLM deployment assumes you can keep iterating — pull a new model version, restart the container, done. We don’t get that. Going back to update anything means doing the whole pipeline again. So the work doesn’t get lighter because you’re doing air-gapped LLM deployment. It just moves to where it can actually be checked. That’s the design. Heavy in the open, simple behind the wall.