Multi-model inference
on Kubernetes.
One Helm chart per runtime — vLLM, TGI, TRT-LLM, SGLang.
A Temporal saga drives every deploy: pull weights, canary, smoke, promote or rollback.
An InferenceModel CRD makes kubectl apply the entire trigger.
plnt is a mini inference-factory for Kubernetes. Helm charts per runtime, a Temporal saga per deploy, a CRD per model. The pieces that turn "I trained a model" into "the model is serving traffic on a cluster."
plnt CLI · Python API
│
▼
┌─────────────────────────────────┐
│ InferenceModel (K8s CRD) │
│ kubectl apply -f llama-70b.yaml│
└───────────────┬─────────────────┘
│ watch(create|update|delete)
▼
┌─────────────────────────────────┐
│ plnt operator (Python · kopf) │
└───────────────┬─────────────────┘
│ start_workflow
▼
┌──────────────────────────────────────────────────┐
│ Temporal · DeployModelWorkflow │
│ │
│ validate → pull → helm_install → smoke → promote│
│ │ │ │ │
│ └───────┴──────────────┴──▶ rollback │
└───────────────────────┬──────────────────────────┘
│ helm install / rollback
▼
┌────────┬────────┬─────────┬─────────┬─────────┐
│ vLLM │ TGI │ TRT-LLM │ SGLang │ router │
│ chart │ chart │ chart │ chart │ (envoy) │
└────────┴────────┴─────────┴─────────┴─────────┘
│
▼
┌─────────────────────────────────┐
│ plnt-cloud (bookings product) │
│ consumes /v1/chat/completions │
└─────────────────────────────────┘ Three steps. No custom control plane. No hand-rolled deploy scripts. The Kubernetes primitives do the work.
Write an InferenceModel resource: model name, runtime (vllm/tgi/trt-llm/sglang), GPU class, replicas, canary budget. kubectl apply.
The plnt operator picks up the resource and starts a Temporal DeployModelWorkflow. The workflow is durable — a restart resumes from the last completed step.
Once the canary passes smoke tests and gets promoted, the model is reachable at <name>.plnt.svc/v1/chat/completions. OpenAI-shape API, streaming via SSE.
LAYER FOR
MULTI-MODEL
INFERENCE.
Built on primitives platform teams already trust — Helm, Kubernetes, Temporal.
No proprietary control plane. No cloud lock-in. Same YAML runs on
kind, GKE, EKS, on-prem.
Four inference backends, one control API. Each ships as a Helm chart; each speaks OpenAI wire format at the pod boundary.
Paged attention · continuous batching · prefix cache. Default runtime for general LLM serving. Chart: plnt/charts/vllm-runtime.
Hugging Face Text Generation Inference. Flash-attention 2, tensor parallel, GPTQ / AWQ / FP8. Chart: plnt/charts/tgi-runtime.
Precompiled TensorRT engines on Triton. Highest per-GPU throughput; engine build runs as a workflow step. Chart: plnt/charts/trt-llm-runtime.
RadixAttention prefix caching. Strong for multi-turn and structured generation. Chart: plnt/charts/sglang-runtime.
Every deploy is five steps of a Temporal workflow. Any step failing triggers compensation. Retries are per-activity with budgets — not per hop.
values.yaml is schema-valid, storage URI reachable, image pullable, GPU class available on the target node pool.
Weights fetched from the CAS-backed registry. Hash checked against the manifest. Idempotent — a partial pull resumes.
helm install at 5% traffic via Envoy VirtualService weights. Canary replica scheduled on the target GPU class.
N test prompts. TTFT / TPOT / tokens-per-sec measured against baseline. KPI budget from InferenceModel.spec.canary.
KPIs pass → scale to replicas.min at 100%. KPIs fail → helm rollback. Compensation on any earlier step failure.
Six decisions that shape the platform. Each is a hill the project will die on.
InferenceModel is the entire deploy contract. One kubectl apply triggers pull, deploy, smoke, promote. Reconciliation is the operator's job — not a bash script.
Every deploy is a Temporal workflow. Compensation on failure, retries with budgets (not per-hop), resume-across-restart. The saga is inspectable in the Temporal UI.
A Python kopf operator watches the CRD. Create/update/delete on the resource is the trigger. Nothing else runs kubectl on your behalf.
Every deploy is a canary first. 5% traffic → smoke test → promote. KPI budget in the manifest. Failed canaries roll back automatically.
One RuntimeAdapter interface, four backends. vLLM / TGI / TRT-LLM / SGLang share a control API. The chart per runtime is the deployment shape; the adapter is the code shape.
Every runtime exposes /v1/chat/completions. Existing OpenAI clients work unmodified. Streaming via SSE. No proprietary SDK.
One chart. One CRD. One curl.