Operationalizing

Where we left off

Post 1 covered the why — what makes agentic AI worth building inside an air-gap at all. Post 2 covered the how — the two pipelines (MCP server and inference layer) that get everything built on the connected side and landed as approved artifacts on the air-gapped side.

This post is what comes after all of that. Both pipelines are running. The MCP server is up, vLLM is serving, they’re talking to each other. Now what does “production” actually mean when you can’t SSH into a connected system to grab a new package, can’t push a hotfix in five minutes, and can’t route an alert to an external webhook?

Operationalizing air-gapped AI infrastructure is where the implicit-deny discipline has to extend beyond the pipeline and into the running system itself. Getting things in was the hard part of Post 2. Keeping them honest once they’re in is what this post is about.

Secrets without an external vault

In a connected environment, secrets management usually means a vault — HashiCorp Vault, AWS Secrets Manager, something with an external dependency and a live API to call against. None of that exists inside the boundary, so you’re working with what the cluster provides natively.

// kubernetes native secrets

Native K8s secrets are the baseline and honestly a reasonable starting point. They handle the mechanics: mount a secret as an environment variable or a volume, reference it from a pod spec, let Kubernetes manage the delivery. For MCP tool credentials, vLLM endpoint config, and gateway auth, this works fine in practice.

The honest limitation worth naming: Kubernetes secrets are base64 encoded, not encrypted at rest by default. If someone gets read access to etcd, they get your secrets. In a properly locked-down cluster with namespace RBAC and no direct etcd exposure, this risk is manageable — but it’s a gap, and it belongs in your threat model documentation, not silently assumed away.

# Fetch the public key from the in-cluster controller
kubeseal --fetch-cert \
  --controller-name=sealed-secrets \
  --controller-namespace=kube-system \
  > sealed-secrets-public.pem

# Create a standard K8s secret manifest (dry-run, no cluster write)
kubectl create secret generic mcp-tool-creds \
  --from-literal=endpoint=http://internal-api.cluster.local \
  --from-literal=token=your-token-here \
  --dry-run=client -o yaml > mcp-secret.yaml

# Seal it with the public key — this output is safe to commit to git
kubeseal --format yaml \
  --cert sealed-secrets-public.pem \
  < mcp-secret.yaml > mcp-sealed-secret.yaml

# Apply to the cluster — controller decrypts, creates the real secret
kubectl apply -f mcp-sealed-secret.yaml

The public key fetch in step one is the only operation that needs cluster access — and since it’s a read-only operation against an in-cluster service, it can be done from the air-gapped side without any connected-side involvement. Everything else is just file manipulation and a standard kubectl apply.

RBAC and the MCPServer CRD

The MCPServer CRD that kmcp installs is an RBAC surface that’s easy to overlook because it doesn’t look like a traditional privileged resource. It is. A user or service account with the ability to create or modify MCPServer objects can define what tools your agent has access to. That’s capability definition, and it deserves the same RBAC treatment you’d give anything else that defines what a workload can do.

The practical application of implicit deny here: start by asking who actually needs to create or modify MCPServer objects in production. The answer is almost certainly a very small set of identities — your deployment pipeline service account, and maybe one or two operators. Everyone else gets read-only or nothing.

• Namespace isolation — MCP servers almost certainly don’t need cluster-wide permissions. Scope them to a dedicated namespace and write RBAC accordingly. A Role scoped to ai-workloads namespace is meaningfully safer than a ClusterRole even if the permissions look identical on paper.
• Service account scoping — the pods running your MCP server need to reach the vLLM endpoint (or gateway). They do not need to read other namespaces, modify cluster resources, or do anything else. Create a dedicated service account per MCP server with the minimum permissions that service actually requires, and bind it only to what it needs.
• Deployment pipeline service account — separate from runtime service accounts. The account that runs kmcp deploy needs write access to MCPServer objects. That account should not be the same account the running pod uses at runtime. Keep deployment identity and runtime identity separate.

Network policy as implicit deny for air-gapped AI infrastructure

Running MCP servers over HTTP/SSE means your server is a network-reachable service inside the cluster. That’s a different exposure profile than stdio, where the communication is entirely in-process. It’s not a reason to avoid HTTP/SSE — it’s just a reason to actually write network policy, because without it, any pod in the cluster can reach your MCP server endpoint by default.

Kubernetes NetworkPolicy is the right tool here, and it maps almost directly onto the blog’s core thesis: default deny everything, then explicitly permit only what’s earned. Here’s a policy that does that for a typical MCP server setup:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: mcp-server-policy
  namespace: ai-workloads
spec:
  podSelector:
    matchLabels:
      app: mcp-server
  policyTypes:
    - Ingress
    - Egress

    # Only allow inbound from designated clients (e.g. your AI agent pod)
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: ai-agent
      ports:
        - protocol: TCP
          port: 8080  # MCP server HTTP port
    - from:
        - podSelector:
            matchLabels:
              app: prometheus
      ports:
        - protocol: TCP
          port: 8080  # Prometheus scrape

    # Only allow outbound to the vLLM endpoint (or gateway) and DNS
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: vllm  # or litellm/bifrost if using a gateway
      ports:
        - protocol: TCP
          port: 8000
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53  # DNS resolution

A few things worth calling out in that policy. The DNS egress rule is the one people most often forget — block all egress and your pods can’t resolve internal service names, which means silent failures rather than clean errors. Always include it. The Prometheus scrape ingress rule is there because without it, once you lock down ingress, your metrics disappear. And if you’re running a gateway layer like LiteLLM or Bifrost, swap app: vllm in the egress block for the gateway’s label — the MCP server should only know about whatever’s directly in front of it, not the inference backend behind the gateway.

Update cadence — the whole pipeline, every time

This is the operationalizing reality that catches people off guard: there’s no kubectl set image with a live tag pull, no helm upgrade from a connected chart repo, no quick model swap without a crossing. Updating anything in your air-gapped AI infrastructure means doing the full connected-side pipeline again. Build, scan, approve, export, import, deploy. Every time.

That’s not a bug. That’s the point — every running artifact earned its way in and any update has to earn its way in too. But it does mean update cadence has to be a deliberate decision, not an afterthought.

There are two distinct update cycles to manage and they have different drivers:

• MCP server updates — driven by code changes: new tools, bug fixes, dependency patches. These follow the same pipeline as the initial build. The cost per update is relatively low since the image is small and the pipeline is established. The risk is tool behavior drift — if you’re updating tool logic, make sure the agent’s expectations of what a tool does still match what it actually does after the update.
• Model weight updates — driven by new quantized releases, better performing versions, or swapping to a different model entirely. These are heavier: you’re moving potentially multi-gigabyte weight bundles back through the transfer process. Worth batching if you can. Worth having a clear rollback plan since swapping a model can silently change agent behavior in ways that aren’t immediately obvious from metrics alone.

Observability inside the boundary

Prometheus and Grafana both follow the same container pipeline as everything else — they cross the wall the same way the MCP server and vLLM did, and they run entirely inside the boundary. No external metrics backend, no cloud-hosted dashboards, no alertmanager webhook routing to PagerDuty or Slack. What you build inside stays inside.

That constraint matters for alerting design specifically: your alertmanager needs to route to something internal. Email if you have a mail relay inside the boundary, an internal chat system if you’re running one, or just log-based alerting if you’re not. The tooling is the same; the destinations are different.

// instrumenting vLLM

vLLM exposes a Prometheus-compatible metrics endpoint natively at /metrics on its serving port — no additional configuration needed to enable it. The metrics worth watching are token throughput, request queue depth, and time-to-first-token, since those tell you whether the model is keeping up with demand and where latency is actually coming from. Here’s a scrape config that covers both vLLM and the MCP server:

scrape_configs:

  - job_name: 'vllm'
    static_configs:
      - targets:
          - 'vllm-service.ai-workloads.svc.cluster.local:8000'
    metrics_path: '/metrics'
    scrape_interval: 15s
      # Key metrics: vllm:num_requests_running, vllm:num_requests_waiting,
      # vllm:kv_cache_usage_perc, vllm:e2e_request_latency_seconds_bucket,
      # vllm:request_prompt_tokens, vllm:request_generation_tokens

  - job_name: 'mcp-server'
    static_configs:
      - targets:
          - 'mcp-server-service.ai-workloads.svc.cluster.local:8080'
    metrics_path: '/metrics'
    scrape_interval: 15s

  - job_name: 'litellm-gateway'  # omit if not running a gateway
    static_configs:
      - targets:
          - 'litellm-service.ai-workloads.svc.cluster.local:4000'
    metrics_path: '/metrics'
    scrape_interval: 15s

// what to actually watch

Metrics tell you the system is running. They don’t tell you it’s running correctly. For air-gapped AI infrastructure specifically, the failure mode you’re most likely to miss on metrics alone is silent tool failure — the MCP server is up, vLLM is responding, token throughput looks normal, but a specific tool is returning garbage because an internal API it depends on changed its response format. That won’t show up in a latency graph.

Logs matter here in a way they don’t always in more observable environments. Structured logging from your MCP server tools — logging the tool name, the input, and the output at a minimum — gives you the audit trail to catch silent failures before they turn into a confused operator wondering why the agent keeps making bad decisions. Granted, this adds logging overhead, but in an environment where you can’t lean on external observability tooling, it’s the thing that saves you when something goes wrong at an inconvenient time.

We talk about air-gapped AI infrastructure like the hard part is getting it running. And it is hard. But a system you can’t observe is a system you can’t trust, and a system you can’t trust has no business touching anything that matters. We see the pipeline. We see the crossing. We see the deployment. What ties it together is knowing — actually knowing, not assuming — what it’s doing once it’s running. That’s the job metrics and logs exist to do, and inside an air-gap they’re the only tools you have.