LLMKube is a free, open-source Kubernetes operator for self-hosted LLM inference. Read our 2026 review of its features, architecture, and alternatives.
LLMKube is an open-source Kubernetes operator built by Defilan Technologies LLC that turns large language model inference into a native Kubernetes workload. It introduces custom resources for models and inference services so that deploying, scaling, and monitoring self-hosted LLMs uses the same kubectl, Helm, and GitOps workflows teams already use for everything else in their cluster.
The project launched in 2025 and is distributed under the Apache 2.0 license, making it free to adopt and modify. It is aimed at platform engineering and ML infrastructure teams who want production-grade LLM serving without hand-building a bespoke stack around vLLM, llama.cpp, or TGI.
LLMKube supports multiple inference runtimes through a pluggable backend interface, including vLLM for throughput, llama.cpp for efficiency, Hugging Face TGI, and Apple Silicon-native runtimes like mlx-server, so teams can pick the right engine per workload rather than being locked into one.
Hardware support spans NVIDIA CUDA (including Blackwell-generation GPUs), AMD GPUs via a Vulkan and RADV backend that skips a full ROCm install, and Apple Silicon via Metal, with automatic multi-GPU layer sharding and hybrid GPU and CPU offloading for large mixture-of-experts models.
Every deployment includes Prometheus metrics, Grafana dashboards, and OpenTelemetry tracing by default, plus an OpenAI-compatible API surface so existing client integrations require minimal rework when moving from a hosted API to self-hosted infrastructure.
A newer ModelRouter feature adds policy-aware routing across local inference services and external providers like Anthropic, OpenAI, and Bedrock, with budget enforcement and fail-closed semantics for hybrid local-plus-cloud strategies.
LLMKube itself is free and open source under Apache 2.0; there is no paid product tier, subscription, or license fee published on the project's site. Costs come entirely from the underlying compute (owned GPUs, cloud GPU instances, or Apple Silicon hardware) that the operator schedules workloads onto.
The project's own positioning emphasizes that self-hosting inference on consumer or owned GPU hardware through LLMKube can be dramatically cheaper than paying for equivalent throughput from managed cloud AI APIs, though actual savings depend heavily on utilization and existing hardware investment.
LLMKube is an open-source Kubernetes operator that lets teams deploy, scale, and monitor self-hosted large language model inference using native Kubernetes resources and workflows.
Yes. LLMKube is released under the Apache 2.0 open-source license and has no license fees; you only pay for the underlying compute infrastructure you run it on.
LLMKube supports vLLM, llama.cpp, Hugging Face TGI, Apple Silicon runtimes like mlx-server and vllm-swift, and custom containers through a pluggable backend interface.
Yes. LLMKube supports NVIDIA CUDA, AMD GPUs via Vulkan/RADV, and Apple Silicon via Metal, allowing heterogeneous clusters that mix hardware types.
LLMKube is developed and maintained by Defilan Technologies LLC, a Washington State-based company that launched the project in 2025.
Yes. LLMKube is built for teams already operating Kubernetes clusters; it assumes familiarity with kubectl, Helm, and standard cluster administration.