Clarity for AI buildersSponsored by AI Routing
availability

Buoyant introduces Kinference, an open-source inference control plane for vLLM and Kubernetes

Buoyant introduces Kinference, an open-source inference control plane for vLLM and Kubernetes
Buoyant has introduced Kinference, an open-source control plane for building inference platforms with vLLM and Kubernetes. The project adds request routing, token rate limiting, tenant usage and cost tracking, and policy-based controls around vLLM’s inference engine, according to an announcement on the vLLM blog. Kinference is available under the Apache v2 license. Buoyant says it developed the project after seeing customers struggle to run open-weight models in their own environments. Clients send OpenAI- or Anthropic-compatible inference traffic to Kinference’s HTTP endpoint. Kinference then routes requests across the available models and backends, allowing platform operators to manage a changing mix of models, GPUs and clients. Cache-aware routing and reported performance Kinference’s load balancer considers GPU queue sizes, historical performance and the contents of workers’ key-value (KV) caches. The aim is to select a vLLM worker that needs to do less work for a particular request, rather than distributing traffic without inference-specific context. In benchmarks reported by Buoyant, Kinference reduced median time to first token by 95% and median end-to-end latency by 45% on one workload compared with standard Kubernetes load balancing. The announcement describes that test as an unloaded-latency benchmark intended to isolate the effect of KV-cache-aware routing. In a separate capacity benchmark, Buoyant reports 7.4 times the goodput on a shared-prefix workload and about 3.5 times the goodput on a multiturn workload. Here, goodput measures requests served within a latency service-level objective, or SLO. Kinference met the selected SLO at concurrency 32—the highest level measured—for the shared-prefix workload, and at concurrency 8 for the multiturn workload. According to the announcement, the kube-proxy approach did not meet that SLO at any tested concurrency level. These are project-reported results for specific workloads, not evidence of equivalent gains across all deployments. The supplied article does not include the hardware configuration or the numerical latency SLO, and refers readers elsewhere for the full benchmark setup and findings. Policies for model selection, overflow and data handling Kinference uses inference policies to express routing decisions based on factors including the tenant, requested model, GPU-pool state and semantic classification of prompts. One example considers routing a low-complexity request to a cheaper model, but selects between candidate models using predicted time to first token. That allows existing cache state to affect the decision: switching to a cheaper model may not be preferable if another worker can reuse more of the prompt. Another example excludes local workers with more than two queued prompts, prefers eligible local backends, and falls back to configured provider endpoints. A separate policy example excludes third-party providers for prompts tagged as containing personally identifiable information and rejects the request if no eligible backend remains. These examples describe configurable routing behavior, rather than independently verified classification accuracy or privacy guarantees. Buoyant says policies can be reviewed, committed to Git, checked and rolled back independently. Metrics and deployment Kinference exposes metrics for token rates, input and output token sizes, time to first token, inter-token latency, end-to-end latency, cached and uncached prompt tokens, model usage and GPU saturation. Almost every metric is also available per tenant, according to the announcement, supporting tenant-specific usage tracking and performance objectives. The supplied installation walkthrough uses Kubernetes Gateway API custom resource definitions and Helm charts to install a traffic gateway and inference handler, with separate charts for their custom resource definitions. Buoyant notes that the installation steps will evolve and directs users to the latest project documentation.