We are excited to announce Kthena v1.0.0, a major milestone for Kubernetes-native LLM inference. This release focuses on production readiness across the serving stack: more accurate Gateway API routing, first-class role-level autoscaling for prefill/decode disaggregated workloads, safer role-level rolling updates, better router scheduling signals, session boost for multi-turn conversation workloads, richer cache-aware router observability with Prometheus metrics and example dashboards, and a more complete CLI experience.

Kthena v1.0.0 also includes an important autoscaling API consolidation. AutoscalingPolicyBinding has been removed, and target configuration now lives directly in AutoscalingPolicy through homogeneousTarget, heterogeneousTarget, and disaggregatedTarget.

Release Highlights​

Key Features Overview​

• AutoscalingPolicy consolidation and P/D coordinated disaggregated autoscaling: AutoscalingPolicyBinding is removed, and autoscaling target configuration is consolidated into AutoscalingPolicy; disaggregatedTarget enables coordinated role-level autoscaling for prefill/decode workloads, allowing each role to scale from its own metrics while keeping P/D replica ratios within healthy bounds through optional ratio constraints.
• Session boost for multi-turn conversations: The router can prioritize follow-up requests from recently completed sessions, improving the chance of reusing warm KV cache under concurrent agentic and chat workloads.
• Router scheduling and observability: Per-pod in-flight request tracking, Redis-backed cross-router synchronization, configurable pod metrics scraping, and cache-aware Prometheus metrics improve scheduling accuracy and operational visibility.
• Role-level rolling update availability control: Enhancements to the RoleRollingUpdate feature. Now each role can set maxUnavailable to control the pace of the upgrade.
• Gateway API and HTTPRoute correctness: Kthena router now honors HTTPRoute hostnames, keeps matched route rules consistent for backend selection and URL rewrites, fixes PathPrefix semantics, and respects Gateway listener allowedRoutes.
• CLI and OpenAI-compatible API improvements: The CLI now exposes richer status output and supports ModelRoute and ModelServer, while the router adds an OpenAI-compatible GET /v1/models endpoint.

AutoscalingPolicy Consolidation and P/D Coordinated Disaggregated Autoscaling​

Kthena v1.0.0 introduces a simpler and more powerful autoscaling API. Users now configure what to scale, how to collect metrics, and scaling boundaries in a single AutoscalingPolicy resource.

The new disaggregatedTarget mode is designed for coordinated P/D autoscaling in role-based ModelServing deployments, especially prefill/decode disaggregated inference. Prefill and decode roles can make scaling decisions from their own metrics, while the autoscaler applies shared constraints so the two sides grow and shrink together instead of drifting independently. Each role can define its own replica range, metrics, and metric sources, and operators can configure a ratioConstraint to keep the P/D replica ratio within a healthy range.

Example disaggregatedTarget configuration:

spec:
  disaggregatedTarget:
    targetRef:
      apiVersion: workload.serving.volcano.sh/v1alpha1
      kind: ModelServing
      name: vllm-qwen-pd-ms
    roles:
      prefill:
        minReplicas: 1
        maxReplicas: 8
        metrics:
          - name: prefill_waiting_requests
            targetValue: "1"
        metricSources:
          prefill_waiting_requests:
            prometheus:
              serverURL: http://kube-prometheus-stack-prometheus.test.svc.cluster.local:9090
              query: sum(vllm:num_requests_waiting{namespace="autoscale-demo", service="vllm-prefill"})
      decode:
        minReplicas: 1
        maxReplicas: 16
        metrics:
          - name: decode_gpu_cache_usage
            targetValue: "0.75"
        metricSources:
          decode_gpu_cache_usage:
            prometheus:
              serverURL: http://kube-prometheus-stack-prometheus.test.svc.cluster.local:9090
              query: sum(vllm:gpu_cache_usage_perc{namespace="autoscale-demo", service="vllm-decode"})
    ratioConstraint:
      numeratorRole: prefill
      denominatorRole: decode
      minRatio: "0.25"
      maxRatio: "1"

Related changes:

• Proposal:
  • Proposal of merge autoscalingPolicybingding into autoscalingPolicy #1172
• PRs:
  • merge autoscalingpolicybinding to autoscalingpolicy #1203
  • Implementation of PD disaggregation auto-scaler #1258
• Contributors: @LiZhenCheng9527

Session Boost for Multi-Turn Conversation Workloads​

Kthena v1.0.0 adds session boost support to improve multi-round chat, agentic workflows, and RAG chains where each request depends on previous responses. In these workloads, follow-up requests often reuse a large shared prefix. If they wait behind unrelated traffic for too long, corresponding backend KV cache entries may be evicted and time-to-first-token can increase.

Session boost lets the router track recently completed sessions and prioritize follow-up requests from those sessions in the waiting queue. The implementation keeps this behavior separate from user-fairness scheduling and uses dedicated session-boost configuration, including session header selection, bounded recent-session tracking, inflight admission limits, and an optional grace period for advanced cache-hit optimization.

The feature is designed to improve cache reuse opportunities under concurrent multi-turn traffic without claiming pod affinity by itself. For the strongest KV cache benefit, operators should also ensure that session-aware or cache-aware routing can place follow-up requests on backends that still hold the warm cache.

Example Helm configuration:

networking:
  kthenaRouter:
    sessionBoost:
      enabled: true
      header: X-Session-ID
      maxSessions: 4096
      inflightPerPod: 16
      gracePeriod: 0s

Related changes:

• Issue: Improve multi-round conversation case #1190
• PRs:
  • session boost queue to optimize multi conversation scenario #1183
• Contributors: @YaoZengzeng, @hzxuzhonghu, @FAUST-BENCHOU, @LiZhenCheng9527

Smarter Router Scheduling and Cache-Aware Observability​

The router now has better load signals for scheduling decisions. Kthena tracks per-pod in-flight requests and can synchronize these counters across router replicas through Redis, allowing the least-request plugin to make decisions based on real-time load instead of only local router state.

Cache-aware scheduling is also much more observable. The prefix-cache and kvcache-aware score plugins now export Prometheus metrics on the router's existing /metrics endpoint, replacing klog-only visibility with queryable, model-labelled time series for load testing and production tuning.

For cache effectiveness, Kthena records match-ratio histograms instead of simple hit/miss counters. kthena_router_prefix_cache_match_ratio and kthena_router_kvcache_aware_match_ratio report the fraction of a prompt's blocks already available on the best-matching candidate pod, with 0 representing a real miss. This makes hit rate derivable from the le="0.0" bucket while also showing how much of the prefix was reused.

Related changes:

• Proposal:
  • Observability for prefix-cache and kvcache-aware Score Plugins
• PRs:
  • feat(router): add per-pod in-flight request tracking with Redis sync #962
  • Add SGLang tokenizer support for KV-cache-aware scheduling #997
  • router: add observability metrics for prefix-cache and kvcache-aware score plugins #1194
  • feat(router): make pod metrics update interval configurable #1151
  • perf(router): cache parsed prompt to avoid redundant ParsePrompt call #1123
  • fix: parallelize pod metrics scraping loop with bounded concurrency #1255
• Contributors: @hzxuzhonghu, @blenbot, @kube-gopher, @rajnish-jais, @nabrahma

Role RollingUpdate Availability Control​

Kthena v0.4.0 introduced RoleRollingUpdate, but role updates could still delete all outdated Role replicas in a ServingGroup at once. When spec.replicas was 1, this could make the service temporarily unavailable during a role-level rollout.

Kthena v1.0.0 adds per-Role maxUnavailable support for RoleRollingUpdate. Operators can now control the step size of each Role update independently, using either an absolute number or a percentage. This gives role-level rolling updates the same availability-oriented control that ServingGroup-level updates already had, while keeping Role-level and ServingGroup-level rollout budgets separate.

Example Role-level rollout configuration:

spec:
  rolloutStrategy:
    type: RoleRollingUpdate
  template:
    roles:
    - name: prefill
      replicas: 2
      maxUnavailable: 1
      # entryTemplate and workerTemplate omitted
    - name: decode
      replicas: 4
      maxUnavailable: 25%
      # entryTemplate and workerTemplate omitted

Related changes:

• Issue: Control the number of unavailable Role replicas in RoleRollingUpdate #1188
• PRs:
  • Role rollingupdate support maxUnavailable settings #1239
• Contributors: @hzxuzhonghu, @LiZhenCheng9527

Gateway API and HTTPRoute Correctness​

Kthena-router now handles Gateway API traffic with stronger correctness guarantees. The router honors HTTPRoute.spec.hostnames, keeps the matched HTTPRoute rule associated with backend selection and URL rewrite filters, and chooses more specific path rules within a route. This avoids accidentally routing a request through a backend or filter from a different rule than the one that matched the request.

This release also fixes Gateway API PathPrefix matching semantics and makes the router respect Gateway listener allowedRoutes before accepting HTTPRoutes.

Related changes:

• PRs:
  • feat: honor HTTPRoute hostnames and matched rule selection #1174
  • Fix HTTPRoute PathPrefix matching #1119
  • fix(router): respect Gateway allowedRoutes #1263
• Contributors: @zhy76, @Monti-27, @avinxshKD

CLI and OpenAI-Compatible API Improvements​

The kthena CLI now surfaces more useful status information and supports more resource types:

• kthena get model-servings now shows READY and STATUS columns.
• kthena get model-boosters now shows a STATUS column.
• kthena get model-routes and kthena get model-servers are now supported.
• kthena describe model-route and kthena describe model-server are now supported.

The router also adds an OpenAI-compatible GET /v1/models endpoint, returning available model names in the standard list response shape.

Related changes:

• PRs:
  • feat: add STATUS and READY columns to kthena get output #978
  • feat: add CLI support for ModelRoute and ModelServer resources #981
  • feat: support /v1/models endpoint #996
• Contributors: @anirudh240, @madmecodes

Additional Enhancements​

• Added KEDA/HPA compatibility for ModelServing by setting the scale subresource label selector. #839
• Added a controller-manager debug port for dumping cached ServingGroup and Role configuration. #900
• Added SGLang Dynamo mocker coverage and SGLang inference simulator integration. #920, #1231
• Added router pprof endpoint support. #1057
• Added debugPort Helm chart support for controller-manager. #1032
• Improved ModelBooster GPU and offline environment support. #972, #1141, #1146, #945
• Added GPU usage plugin E2E coverage. #1199
• Refreshed the quick-start documentation and recommends starting with ModelServing. #1260
• Added DeepSeek-v4 model-serving examples. #936, #937
• Added KV-cache-aware scheduler plugin documentation. #910

Stability and Correctness Highlights​

• Router request handling correctness: Fixes in JWT auth, empty request validation, streaming content-type parsing, retry behavior, mid-stream error propagation, IPv6 backend URLs, and HTTPRoute matching make the router more predictable under production traffic.
• Gateway API reconciliation correctness: The router now respects HTTPRoute hostnames, matched rule selection, path-prefix semantics, and Gateway listener namespace admission rules, reducing mismatches between Gateway API configuration and runtime routing behavior.
• Autoscaler and metrics reliability: Autoscaler metric collection now uses the target-referenced namespace first, checks HTTP response status before parsing metrics, stabilizes scale-down with the maximum sliding window, and supports bounded concurrent pod metric scraping.
• Controller concurrency safety: ModelBooster and ModelServing controller fixes address data races, concurrent map writes, stale pod bindings, duplicate role deletion, and unsafe PodInfo access.
• Webhook and deployment stability: Webhook certificate generation, cert-manager CA injection, webhook names, validator messages, and quick-start example defaults were corrected to reduce installation and onboarding failures.
• Test and CI hardening: Additional unit and E2E coverage was added for controllers, autoscaler, router debug modules, ModelBooster lifecycle operations, SGLang PD routing, and status-aware scale-down behavior.

Bug Fixes​

Router​

• Fixed router auth panic when the JWKS cache is empty. #1219
• Required the Bearer scheme for JWT auth. #1035
• Rejected empty router model requests. #1036
• Corrected streaming content-type detection when parameters are present. #1145
• Fixed retries sending empty bodies to fallback pods in the aggregated proxy path. #1031
• Returned mid-stream and copy errors from proxy requests. #1049
• Added IPv6 pod backend URL support. #1071
• Fixed stale KV cache ownership. #1224
• Removed an unnecessary goroutine in the ModelPrefixStore LRU eviction callback. #1243

Autoscaler and Metrics​

• Used the target-referenced namespace first in autoscaler metric collection. #1068
• Checked HTTP status before parsing metrics responses. #1142
• Used the maximum sliding window for autoscaler scale-down stabilization. #946
• Used gauge values for SGLang metrics. #976
• Treated zero TTFT/TPOT as uninitialized in least-latency scoring. #1040
• Used ModelServer's configured workload port for metrics instead of hardcoded defaults. #1205

ModelServing and ModelBooster Controllers​

• Used controller owner references for ModelBooster children. #1054
• Fixed a ModelBoosterController data race and concurrent-map-write panic. #1085
• Fixed stale ModelServer pod bindings. #1126
• Fixed concurrent grace-map access in error-pod handling. #1157
• Guarded PodInfo access against data races. #1167
• Avoided duplicate role deletion while a role is already deleting. #1269
• Fixed ModelServing webhook panic on missing replicas. #1055
• Synced PodGroup NetworkTopology when ModelServing topology changes. #1088

Connectors and PD Paths​

• Rebuilt NIXL prefill/decode request bodies on every proxy call. #947
• Rebuilt SGLang prefill/decode request bodies on every proxy call. #984
• Added streaming error handling improvements. #1236

Helm, Webhooks, and Examples​

• Fixed cert-manager CA injection annotation for controller-manager webhooks. #1018
• Used random serial numbers for webhook certificates. #1160
• Corrected webhook names and validator messages. #1152, #1159
• Fixed quick-start ModelBooster cacheURI and Hugging Face endpoint examples. #1246

API Changes and Upgrade Notes​

Breaking Change: AutoscalingPolicyBinding Removed​

AutoscalingPolicyBinding has been removed from CRDs, generated clients, informers, listers, apply configurations, and Helm chart embedded CRDs.

Users should migrate autoscaling target configuration into one of the following AutoscalingPolicy.spec fields:

• homogeneousTarget
• heterogeneousTarget
• disaggregatedTarget

For further details, please refer to the CRD documentation and the Proposal.

New ModelBooster and Router Configuration​

• ModelBackend.runtimeClassName configures the Kubernetes RuntimeClass used by generated backend pods.
• ModelWorker.tolerations configures worker pod tolerations.
• KTHENA_SKIP_ENGINE_DEPENDENCY_INSTALL=true skips startup-time connector dependency installation for offline or prebuilt engine images.
• METRICS_SCRAPE_INTERVAL controls the router pod metrics scrape interval.
• Controller-manager Helm values now support debugPort.

Build Environment​

The project toolchain, Dockerfiles, CI workflows, and development documentation have been upgraded to Go 1.26.4. See feat: comprehensive upgrade to Go 1.26.4 #1244.

Tests, Docs, and Infrastructure​

Kthena v1.0.0 includes broad test, documentation, and infrastructure improvements:

• Added unit tests for ModelRouteController and GatewayController. #992
• Added E2E coverage for updating and deleting ModelBooster. #1000
• Added E2E coverage for status-aware scale-down behavior. #982
• Added SGLang PD router E2E coverage. #994
• Added autoscaler, config, and router debug unit tests. #903
• Fixed flaky controller and router tests. #950, #1102, #1162
• Used the PodGroup informer cache when listing existing PodGroups. #1081

Upgrade Instructions​

To upgrade to Kthena v1.0.0 after the release is published:

1. Review Breaking API Changes​

Before upgrading, review any existing autoscaling resources in your cluster. AutoscalingPolicyBinding is removed in v1.0.0, so clusters using the old two-resource autoscaling model must migrate to the new single-resource AutoscalingPolicy model.

Check whether your cluster still has old binding resources:

kubectl get autoscalingpolicybindings.workload.serving.volcano.sh --all-namespaces

If any resources are returned, migrate their target and metric-source configuration into one of the following AutoscalingPolicy.spec fields before or during the upgrade:

• homogeneousTarget
• heterogeneousTarget
• disaggregatedTarget

For prefill/decode disaggregated workloads, use disaggregatedTarget.roles and, when needed, disaggregatedTarget.ratioConstraint.

2. Back Up Existing Kthena Resources​

Back up Kthena custom resources before applying the new CRDs and controllers:

kubectl get modelservings.workload.serving.volcano.sh --all-namespaces -o yaml > modelservings-backup.yaml
kubectl get modelboosters.workload.serving.volcano.sh --all-namespaces -o yaml > modelboosters-backup.yaml
kubectl get autoscalingpolicies.workload.serving.volcano.sh --all-namespaces -o yaml > autoscalingpolicies-backup.yaml
kubectl get modelroutes.networking.serving.volcano.sh --all-namespaces -o yaml > modelroutes-backup.yaml
kubectl get modelservers.networking.serving.volcano.sh --all-namespaces -o yaml > modelservers-backup.yaml

If you still have AutoscalingPolicyBinding resources, back them up before removing or migrating them:

kubectl get autoscalingpolicybindings.workload.serving.volcano.sh --all-namespaces -o yaml > autoscalingpolicybindings-backup.yaml

3. Upgrade Kthena​

Using Helm​

For OCI-based Helm installation from GHCR:

helm upgrade kthena oci://ghcr.io/volcano-sh/charts/kthena \
  --version v1.0.0 \
  --namespace kthena-system

If this is a fresh installation instead of an upgrade:

helm install kthena oci://ghcr.io/volcano-sh/charts/kthena \
  --version v1.0.0 \
  --namespace kthena-system \
  --create-namespace

If you install from the release chart package:

curl -L -o kthena.tgz https://github.com/volcano-sh/kthena/releases/download/v1.0.0/kthena.tgz
helm upgrade kthena kthena.tgz --namespace kthena-system

4. Verify the Upgrade​

Check that all Kthena components are running:

kubectl get pods -n kthena-system
kubectl get svc -n kthena-system
kubectl get crd | grep serving.volcano.sh

Verify workload, networking, and autoscaling resources:

kubectl get modelservings.workload.serving.volcano.sh --all-namespaces
kubectl get modelboosters.workload.serving.volcano.sh --all-namespaces
kubectl get autoscalingpolicies.workload.serving.volcano.sh --all-namespaces
kubectl get modelroutes.networking.serving.volcano.sh --all-namespaces
kubectl get modelservers.networking.serving.volcano.sh --all-namespaces

Upgrade Notes​

• AutoscalingPolicyBinding is removed. Migrate to AutoscalingPolicy.spec.homogeneousTarget, spec.heterogeneousTarget, or spec.disaggregatedTarget before relying on v1.0.0 autoscaling behavior.
• disaggregatedTarget supports role-level autoscaling and optional role ratio constraints. Fixed roles can use minReplicas == maxReplicas.
• Existing ModelBooster behavior remains backward compatible by default. Set KTHENA_SKIP_ENGINE_DEPENDENCY_INSTALL=true only when using offline or prebuilt engine images that already include connector dependencies.
• GPU clusters that require custom runtimes can set ModelBackend.runtimeClassName; tainted GPU nodes can be targeted with ModelWorker.tolerations.
• Router pod metrics scraping can be tuned with METRICS_SCRAPE_INTERVAL. Keep the default unless you need fresher scheduling signals or lower scrape overhead.
• Development and build prerequisites now use Go 1.26.4; this affects contributors and image builders, not normal Helm or manifest-based runtime upgrades.

Thank You, Contributors​

Thank you to everyone who contributed to Kthena v1.0.0 across controllers, router, autoscaler, CLI, Helm charts, examples, docs, CI, generated clients, and tests.

Special thanks to contributors including @Abirdcfly, @Alivestars04, @anirudh240, @avinxshKD, @blenbot, @FAUST-BENCHOU, @hzxuzhonghu, @JagjeevanAK, @katara-Jayprakash, @kube-gopher, @LiZhenCheng9527, @madmecodes, @nabrahma, @nXtCyberNet, @rajnish-jais, @Sanchit2662, @verma-garv, @WHOIM1205, @xrwang8, @zhy76, and @YaoZengzeng.

We warmly invite developers, operators, and AI infrastructure teams to try Kthena v1.0.0 and help shape the next generation of cloud native LLM serving.

Beyond rock-solid stability, Kthena v0.4.0 introduces a wave of exciting new features designed to streamline your LLM workloads and empower your AI infrastructure.

Improved Observability​

Role Status Visibility​

To minimize kube-apiserver load, Kthena's ModelServing utilizes a local store to cache the status of ServingGroups and Roles. While highly efficient, we realized this limited our observability during debugging.

In v0.4.0, we've broken the black box. We now expose role status directly via Kubernetes Events, dramatically enhancing the observability of ModelServing. Looking ahead, we plan to directly embed this crucial Role information into the ModelServing Status, giving you complete, transparent control over your deployments.

Comprehensive Access Logging​

Router observability is now easier than ever. The Router now generates a detailed access log, capturing rich routing metadata for every single request. Here is an example of a Router log:

[2026-04-16T07:33:08.435627146Z] "POST /v1/chat/completions HTTP/1.1" 200 model_name=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B model_router=deepseek-r1-1-1.5b model_server=deepseek-r1 selected_pod=deepseek-r1-1-1.5b-6989c66877-p6jvv request_id=ad683d1b-6011-4b0f-b9b5-cbb18d43c57b gateway=dev/default http_route=kthena-e2e-gie-8eoas/llm-route inference_pool=kthena-e2e-gie-8eoas/deepseek-r1-1-1.5b tokens=10/38 timings=3ms(0+2+0)

Compared to previous versions, we have introduced the gateway, http_route, and inference_pool fields to provide deeper visibility into Gateway and Gateway Inference Extension traffic.

A Faster, Smarter Router​

Deterministic & Efficient Model Selection​

Previously, mapping multiple ModelRoute resources to a single model could trigger route conflicts—leading to ambiguous rule matching and inconsistent target selection. Because Kubernetes' built-in CRD validation cannot enforce global cross-object uniqueness, we tackled this gracefully at the routing layer.

Kthena v0.4.0 introduces a robust conflict-resolution mechanism. When duplicate ModelRoutes exist, the router deterministically prioritizes the oldest (typically prebuilt) route, treating newer duplicates as lower priority. Predictable, rock-solid routing every time.

Configurable Prefix-Caching​

One size doesn't fit all. That's why Kthena replaced hardcoded prefix-cache parameters with a fully configurable prefix-matching system. You now have granular control over prefix-cache behavior with the following parameters:

• Block Size (for hash processing): Controls the granularity of the prefix match. Smaller blocks offer more precise matching but increase CPU overhead, while larger blocks process faster.
• Max Block Limits: Sets a ceiling on how much of a given prompt is hashed. This protects the router from computational bottlenecks and latency spikes when handling excessively long incoming prompts.
• Cache Capacity: Defines how many prefix entries the router can remember. Increasing this improves cache hit rates for highly diverse workloads, at the cost of slightly higher memory footprint.
• Top-K Results: Determines how many candidate instances are considered when a match is found. Tuning this allows for better load balancing, ensuring traffic is distributed smoothly across multiple nodes instead of overwhelming a single active instance.

By fine-tuning these settings, you can tailor Kthena's routing performance to match your specific models and business LLM workloads.

Granular, Resource-Efficient Rolling Updates​

Historically, Kthena executed rolling updates at the entire ServingGroup level. For massive LLM applications, completely rebuilding a ServingGroup is an incredibly resource-intensive and time-consuming process.

To solve this, we are introducing role-based rolling updates. You no longer need to update an entire ServingGroup when only a specific Role requires changes (which is also why we introduced RoleRecreate in the recovery policy). Starting from v0.4.0, you can dynamically adjust your rolloutStrategy—drastically cutting down resource consumption, speeding up deployment times.

An Open Ecosystem​

We are deeply committed to building Kthena as an open, inclusive, and thriving project alongside the wider open-source community.

ModelScope Support​

In v0.4.0, we have extended Kthena's model downloader with ModelScope protocol support. This allows users and operators to choose a model repository that suits their needs.

Support for A Variety of Inference Engines​

Furthermore, we are proud that ModelServing is now thoroughly verified to support leading inference engines like vLLM and SGLang.

To help user get started quickly, Kthena v0.4.0 includes SGLang deployment examples and documentation. This guide enables developers to use SGLang in Kthena with ease.

By seamlessly integrating with diverse AI technologies rather than locking you into a single solution, Kthena continues to deepen its roots within the vibrant Cloud Native AI landscape. We warmly invite developers worldwide to join us in building this inclusive and dynamic future together!

Summary​

Release v0.3.0 establishes Kthena as a more robust and scalable platform for AI inference workloads. This release introduces significant enhancements in ModelServing, Router, and ModelBooster. Key highlights include seamless integration with LeaderWorkerSet, advanced network topology-aware scheduling for PD disaggregation, and a comprehensive Router Observability framework. Additionally, this version brings native ModelServing version control, support for vLLM data parallel deployment, and a complete E2E test suite for the router, ensuring high stability and reliability for production environments.

What's New​

Key Features Overview​

• LeaderWorkerSet Support: Integration with the LeaderWorkerSet (LWS) API allows for sophisticated management of distributed inference workloads.
• Role-Level Gang Scheduling & Topology Awareness: Leverages Volcano's new subGroupPolicy feature to enable fine-grained, role-based gang scheduling and network topology awareness.
• ModelServing Partition Revision Control: Introduced a native revision-based version control system for ModelServing.
• Router Observability & Debugging: Comprehensive documentation and framework for router observability, plus a dedicated debug port.
• Enhanced Rolling Updates: Support for maxUnavailable allows tuning the velocity of updates for faster rollouts.
• Plugin Support: Flexible plugin architecture for ModelServing to inject custom configuration logic.

LeaderWorkerSet Support for ModelServing Role​

Background and Motivation: Distributed inference workloads often require complex topologies where a leader pod manages multiple worker pods. Configuring these relationships manually can be error-prone. By integrating with the Kubernetes LeaderWorkerSet (LWS) API, Kthena simplify the deployment and management of these workloads.

Key Capabilities:

• Direct Integration: ModelServing Roles can now leverage LWS to automatically manage leader-worker groups.
• Simplified Topology: Reduces the complexity of defining distributed inference services requiring strict coordination.

Related:

• PR: #609, #683
• Contributors: @zhiweideren

Role-Level Gang Scheduling & Topology Awareness​

Background and Motivation: In Prefill-Decode (PD) separation scenarios, the communication overhead between prefill and decode instances is critical. Ensuring these instances are scheduled closer together (e.g., on the same switch or rack) significantly improves performance. Kthena now enables fine-grained, role-level control over both gang scheduling and network topology awareness by leveraging Volcano's subGroupPolicy.

Key Capabilities:

• Declarative Topology Policies: Configure distinct network topology constraints for the entire ServingGroup (groupPolicy) and for individual Roles (rolePolicy) directly in the ModelServing spec.
• Automatic Pod Grouping: The controller automatically labels Pods with modelserving.volcano.sh/role and modelserving.volcano.sh/role-id, enabling Volcano to form subGroups for precise topology-aware placement.
• Performance Optimization: Minimizes inter-role communication latency and maximizes bandwidth utilization for intensive distributed inference jobs by co-locating related tasks on network-proximal nodes.
• Role-Level Gang Scheduling: The subGroupPolicy also enforces gang scheduling at the role level, ensuring that all Pods belonging to a specific role (e.g., all prefill-0 Pods) are scheduled together as an atomic unit. This guarantees that partial deployments of a role do not occur, which is critical for correctness in distributed inference workloads.

Note: This feature requires Volcano v1.14+ for subGroupPolicy support.

Related:

• Proposal: Network Topology
• PR: #587
• Contributors: @LiZhenCheng9527

ModelServing Partition Revision Control​

Background and Motivation: The partition field in a Kthena ModelServing defines a boundary for rolling updates, allowing you to partition the update process so that only a subset of ServingGroups are updated while others remain on the previous version. It is primarily used for canary deployments, phased rollouts, and staging updates in stateful applications where strict control over update order is necessary.

Key Capabilities:

• Revision Tracking: Automatically tracks changes to ModelServing configurations.
• Partition Protection: Supports partition-based updates to ensure service continuity during rollouts.
• Rollback: Easily revert to a previous stable revision.

Related:

• PR: #590, #653, #671
• Contributors: @FAUST-BENCHOU, @LiZhenCheng9527

Router Observability & Debugging​

Background and Motivation: Deep visibility into the inference router is essential for diagnosing latency issues and ensuring SLA compliance. The new observability framework and debug port provide the necessary tools for operators.

Key Capabilities:

• Debug Port: A dedicated port (default 15000) for real-time inspection of routing tables and upstream health.
• Comprehensive Metrics: Detailed documentation and setup for monitoring request latency, throughput, and error rates.
• E2E Testing: A robust E2E test framework covering most routing scenarios ensures reliability.

Related:

• PR: #599, #622
• Contributors: @yashisrani, @FAUST-BENCHOU

Other Notable Changes​

Features and Improvements​

• [ModelServing] Support maxUnavailable in modelserving rolling update #640 (@LiZhenCheng9527)
• [ModelServing] Implement extension plugin framework #588 (@hzxuzhonghu)
• [ModelServing] Support vLLM data parallel deployment and Expert Parallel modes
• [CLI] Add templates for PD disaggregation use cases #571 (@huntersman)
• [Client] Make client QPS and Burst customizable #686 (@FAUST-BENCHOU)
• [Webhooks] Enable ModelServing webhooks by default in Helm charts #694 (@VanderChen)
• [Infra] One-click deploy from source via hack/local-up-kthena.sh #613 (@FAUST-BENCHOU)

Bug Fixes​

• [Scheduler] Fix divide-by-zero in LeastRequest scoring #723 (@WHOIM1205)
• [Controller] Fix role status transition to Running to restore scale-down protection #706 (@WHOIM1205)
• [Controller] Fix panic in PD scheduler when no prefill pods are available #714 (@WHOIM1205)
• [Controller] Fix silent recovery of failed pods after ModelServing controller restart #697 (@WHOIM1205)
• [Controller] Fix recovering headless services after deletion #598 (@LiZhenCheng9527)
• [Controller] Fix validate gangpolicy minRoleReplicas #699 (@VanderChen)
• [Controller] Fix controllerrevision data warping #698 (@VanderChen)
• [Controller] Fix modelserving controller panic #688 (@LiZhenCheng9527)
• [Controller] Fix restart during modelserving create: pod number mismatch #689 (@hzxuzhonghu)
• [Controller] Check role.Name in ModelServing validator #684 (@FAUST-BENCHOU)
• [Controller] Fix bug where role deletion did not trigger reconstruction #629 (@LiZhenCheng9527)
• [Router] Protect Headless Services Created by ModelServing #598 (@LiZhenCheng9527)

Contributors​

Thank you to all contributors who made this release possible:

@hzxuzhonghu, @LiZhenCheng9527, @YaoZengzeng, @git-malu, @FAUST-BENCHOU, @katara-Jayprakash, @zhiweideren, @aaradhychinche-alt, @WHOIM1205, @yashisrani, @huntersman, @VanderChen

As Kubernetes becomes the de facto standard for deploying AI/ML workloads, the need for standardized, interoperable traffic management APIs has become increasingly important. The Kubernetes Gateway API represents a significant evolution from the traditional Ingress API, providing a more expressive, role-oriented, and extensible model for managing north-south traffic in Kubernetes clusters.

Building on top of Gateway API, the Gateway API Inference Extension introduces specialized resources and capabilities designed specifically for AI/ML inference workloads. This extension standardizes how inference services are exposed and routed through gateway implementations, enabling seamless integration across different gateway providers.

Kthena Router now supports both Gateway API and Gateway API Inference Extension, providing users with flexible routing options while maintaining compatibility with industry standards. This blog post explores why these APIs matter, how to enable them, and demonstrates practical usage examples.

What is Gateway API and Gateway API Inference Extension?​

Gateway API is a Kubernetes project that provides a standardized, role-oriented API for managing service networking. It separates concerns into distinct roles (Infrastructure Provider, Cluster Operator, and Application Developer) and supports advanced routing capabilities including cross-namespace routing, multiple protocols, and traffic splitting. Gateway API Inference Extension builds upon Gateway API to provide inference-specific capabilities for AI/ML workloads. It introduces specialized resources like InferencePool and InferenceObjective, enabling model-aware routing and OpenAI API compatibility for standardized inference service exposure and routing.

Why Support Gateway API and Inference Extension?​

There are several compelling reasons why Kthena Router should support these APIs:

1. Resolving Global ModelName Conflicts​

In traditional routing configurations, the modelName field in ModelRoute resources is global. When multiple ModelRoute resources use the same modelName, conflicts occur, leading to undefined routing behavior. This limitation becomes problematic in multi-tenant environments where different teams or applications might want to use the same model name for different purposes.

Gateway API solves this by introducing the concept of Gateway resources that define independent routing spaces. Each Gateway can listen on different ports, and ModelRoutes bound to different Gateways are completely isolated, even if they share the same modelName. This enables:

• Multi-tenant isolation: Different teams can use the same model names without conflicts
• Environment separation: Separate routing configurations for dev, staging, and production
• Port-based routing: Different applications can access different backends through different ports

2. Industry Standard Compatibility​

Gateway API is becoming the industry standard for Kubernetes service networking. By supporting Gateway API, Kthena Router:

• Improves interoperability: Works seamlessly with other Gateway API-compatible tools and infrastructure
• Reduces vendor lock-in: Users can migrate between different gateway implementations more easily
• Leverages ecosystem: Benefits from the broader Gateway API community and tooling

3. Supporting Gateway API Inference Extension​

Gateway API Inference Extension provides a standardized way to expose AI/ML inference services. By supporting this extension, Kthena Router:

• Enables standardized inference routing: Works with InferencePool and InferenceObjective resources
• Facilitates multi-gateway deployments: Can work alongside other gateway implementations using the same API

4. Flexible Deployment Options​

With Gateway API support, users can choose between:

• Native ModelRoute/ModelServer: Kthena's custom CRDs that provide advanced features like PD disaggregation, weighted routing, and sophisticated scheduling algorithms
• Gateway API + Inference Extension: Standard Kubernetes APIs that provide interoperability and compatibility with other gateway implementations

This flexibility allows users to select the approach that best fits their specific requirements and infrastructure constraints.

Enabling Gateway API Support​

Prerequisites​

Before enabling Gateway API support, ensure you have:

• Kubernetes cluster with Kthena installed (see Installation Guide)
• Basic understanding of Kubernetes Gateway API concepts
• kubectl configured to access your cluster

Configuration​

Enable Gateway API support by setting the --enable-gateway-api=true parameter when deploying Kthena Router:

# Configure during Helm installation
helm install kthena \
  --set networking.kthenaRouter.gatewayAPI.enabled=true \
  --version v0.2.0 \
  oci://ghcr.io/volcano-sh/charts/kthena

Or modify the configuration in an already deployed Kthena Router:

kubectl edit deployment kthena-router -n kthena-system

Ensure the container arguments include --enable-gateway-api=true.

Default Gateway​

When Gateway API support is enabled, Kthena Router automatically creates a default Gateway with the following characteristics:

• Name: default
• Namespace: Same as the Kthena Router's namespace (typically kthena-system)
• GatewayClass: kthena-router
• Listening Port: Kthena Router's default service port (defaults to 8080)
• Protocol: HTTP

View the default Gateway:

kubectl get gateway

# Example output:
# NAME      CLASS           ADDRESS   PROGRAMMED   AGE
# default   kthena-router             True         5m

Using Gateway API with Native ModelRoute/ModelServer​

This example demonstrates how to use Gateway API with Kthena's native ModelRoute and ModelServer CRDs, resolving the modelName conflict problem.

Step 1: Deploy Mock Model Servers​

Deploy mock LLM services and their corresponding ModelServer resources:

# Deploy DeepSeek 1.5B mock service
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/kthena/main/examples/kthena-router/LLM-Mock-ds1.5b.yaml

# Deploy DeepSeek 7B mock service
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/kthena/main/examples/kthena-router/LLM-Mock-ds7b.yaml

# Create ModelServer for DeepSeek 1.5B
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/kthena/main/examples/kthena-router/ModelServer-ds1.5b.yaml

# Create ModelServer for DeepSeek 7B
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/kthena/main/examples/kthena-router/ModelServer-ds7b.yaml

Wait for the pods to be ready:

kubectl wait --for=condition=ready pod -l app=deepseek-r1-1-5b --timeout=300s
kubectl wait --for=condition=ready pod -l app=deepseek-r1-7b --timeout=300s

Step 2: Create a New Gateway​

Create and apply a new Gateway listening on a different port:

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kthena-gateway-8081
  namespace: default
spec:
  gatewayClassName: kthena-router
  listeners:
  - name: http
    port: 8081  # Using a different port
    protocol: HTTP
EOF

# Verify Gateway status
kubectl get gateway kthena-gateway-8081 -n default

Important Note: The newly created Gateway listens on port 8081, but you need to manually configure the Kthena Router's Service to expose this port:

# Edit the kthena-router Service
kubectl edit service kthena-router -n kthena-system

Add the new port in spec.ports:

spec:
  ports:
  - name: http
    port: 80
    targetPort: 8080
    protocol: TCP
  - name: http-81  # Add new port
    port: 81
    targetPort: 8081
    protocol: TCP

Step 3: Create ModelRoutes Bound to Different Gateways​

Create and apply a ModelRoute bound to the default Gateway:

cat <<EOF | kubectl apply -f -
apiVersion: networking.serving.volcano.sh/v1alpha1
kind: ModelRoute
metadata:
  name: deepseek-default-route
  namespace: default
spec:
  modelName: "deepseek-r1"
  parentRefs:
  - name: "default"      # Bind to the default Gateway
    namespace: "kthena-system"
    kind: "Gateway"
  rules:
  - name: "default"
    targetModels:
    - modelServerName: "deepseek-r1-1-5b"  # Backend ModelServer
EOF

Create and apply another ModelRoute using the same modelName but bound to the new Gateway:

cat <<EOF | kubectl apply -f -
apiVersion: networking.serving.volcano.sh/v1alpha1
kind: ModelRoute
metadata:
  name: deepseek-route-8081
  namespace: default
spec:
  modelName: "deepseek-r1"  # Same modelName as the default Gateway's ModelRoute
  parentRefs:
  - name: "kthena-gateway-8081"  # Bind to the new Gateway
    namespace: "default"
    kind: "Gateway"
  rules:
  - name: "default"
    targetModels:
    - modelServerName: "deepseek-r1-7b"  # Using a different backend
EOF

Note: When Gateway API is enabled, the parentRefs field is required. ModelRoutes without parentRefs will be ignored and will not route any traffic.

Step 4: Verify the Configuration​

Now you have two independent routing configurations:

1. Default Gateway (Port 8080)

   • ModelRoute: deepseek-default-route
   • ModelName: deepseek-r1
   • Backend: deepseek-r1-1-5b (DeepSeek-R1-Distill-Qwen-1.5B)

2. New Gateway (Port 8081)

   • ModelRoute: deepseek-route-8081
   • ModelName: deepseek-r1 (same modelName)
   • Backend: deepseek-r1-7b (DeepSeek-R1-Distill-Qwen-7B)

Test the default Gateway (port 8080):

# Get the kthena-router IP or hostname
ROUTER_IP=$(kubectl get service kthena-router -n kthena-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

# If LoadBalancer is not available, use NodePort or port-forward
# kubectl port-forward -n kthena-system service/kthena-router 80:80 81:81

# Test the default port
curl http://${ROUTER_IP}:80/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-r1",
    "prompt": "What is Kubernetes?",
    "max_tokens": 100,
    "temperature": 0
  }'

# Expected output from deepseek-r1-1-5b:
# {"choices":[{"finish_reason":"length","index":0,"logprobs":null,"text":"This is simulated message from deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B!"}],...}

Test the new Gateway (port 8081):

# Test port 81
curl http://${ROUTER_IP}:81/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-r1",
    "prompt": "What is Kubernetes?",
    "max_tokens": 100,
    "temperature": 0
  }'

# Expected output from deepseek-r1-7b:
# {"choices":[{"finish_reason":"length","index":0,"logprobs":null,"text":"This is simulated message from deepseek-ai/DeepSeek-R1-Distill-Qwen-7B!"}],...}

Although both requests use the same modelName (deepseek-r1), they are routed to different backend model services because they access through different ports (corresponding to different Gateways). This demonstrates how Gateway API resolves the global modelName conflict problem.

Using Gateway API with Inference Extension​

This example demonstrates how to use Gateway API Inference Extension with Kthena Router, providing a standardized way to expose and route inference services.

Step 1: Install the Inference Extension CRDs​

Install the Gateway API Inference Extension CRDs in your cluster:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/latest/download/manifests.yaml

Step 2: Deploy Sample Model Server​

Deploy a model that will serve as the backend for the Gateway Inference Extension. Follow the Quick Start guide to deploy a model in the default namespace and ensure it's in Active state.

After deployment, identify the labels of your model pods:

# Get the model pods and their labels
kubectl get pods -l workload.serving.volcano.sh/managed-by=workload.serving.volcano.sh --show-labels

# Example output shows labels like:
# modelserving.volcano.sh/name=demo-backend1
# modelserving.volcano.sh/group-name=demo-backend1-0
# modelserving.volcano.sh/role=leader
# workload.serving.volcano.sh/model-name=demo
# workload.serving.volcano.sh/backend-name=backend1
# workload.serving.volcano.sh/managed-by=workload.serving.volcano.sh

Step 3: Deploy the InferencePool​

Kthena Router natively supports Gateway Inference Extension and does not require the Endpoint Picker Extension. Create an InferencePool resource that selects your Kthena model endpoints:

cat <<EOF | kubectl apply -f -
apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
  name: kthena-demo
spec:
  targetPorts:
    - number: 8000  # Adjust based on your model server port
  selector:
    matchLabels:
      workload.serving.volcano.sh/model-name: demo
  # Kthena Router natively supports Gateway Inference Extension and does not require the Endpoint Picker Extension.
  # It's just a placeholder for API validation.
  endpointPickerRef:
    name: kthena-demo
    port:
      number: 8000
EOF

Step 4: Enable Gateway API Inference Extension in Kthena Router​

Enable the Gateway API Inference Extension flag in your Kthena Router deployment:

kubectl patch deployment kthena-router -n kthena-system --type='json' -p='[
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/args/-",
    "value": "--enable-gateway-api=true"
  },
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/args/-",
    "value": "--enable-gateway-api-inference-extension=true"
  }
]'

Wait for the deployment to roll out:

kubectl rollout status deployment/kthena-router -n kthena-system

Step 5: Deploy the Gateway and HTTPRoute​

Create a Gateway resource that uses the kthena-router GatewayClass:

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: inference-gateway
spec:
  gatewayClassName: kthena-router
  listeners:
  - name: http
    port: 8080
    protocol: HTTP
EOF

Create and apply the HTTPRoute configuration that connects the gateway to your InferencePool:

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: kthena-demo-route
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: inference-gateway
  rules:
  - backendRefs:
    - group: inference.networking.k8s.io
      kind: InferencePool
      name: kthena-demo
    matches:
    - path:
        type: PathPrefix
        value: /
    timeouts:
      request: 300s
EOF

Step 6: Verify and Test​

Confirm that the Gateway was assigned an IP address and reports a Programmed=True status:

kubectl get gateway inference-gateway

# Expected output:
# NAME                CLASS           ADDRESS         PROGRAMMED   AGE
# inference-gateway   kthena-router   <GATEWAY_IP>    True         30s

Verify that all components are properly configured:

# Check Gateway status
kubectl get gateway inference-gateway -o yaml

# Check HTTPRoute status - should show Accepted=True and ResolvedRefs=True
kubectl get httproute kthena-demo-route -o yaml

# Check InferencePool status
kubectl get inferencepool kthena-demo -o yaml

Test inference through the gateway:

# Get the kthena-router IP or hostname
ROUTER_IP=$(kubectl get service kthena-router -n kthena-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
# If LoadBalancer is not available, use NodePort or port-forward
# kubectl port-forward -n kthena-system service/kthena-router 80:80

# Test the completions endpoint
curl http://${ROUTER_IP}:80/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen2.5-0.5B-Instruct",
    "prompt": "Write as if you were a critic: San Francisco",
    "max_tokens": 100,
    "temperature": 0
  }'

Native ModelRoute/ModelServer: Advanced Features​

While Gateway API and Gateway API Inference Extension provide standardized, interoperable routing capabilities, Kthena's native ModelRoute and ModelServer CRDs offer more experimental and advanced features specifically designed for AI/ML inference workloads:

Prefill-Decode (PD) Disaggregation​

Native ModelRoute/ModelServer supports PD disaggregation, where the compute-intensive prefill phase is separated from the token generation decode phase. This enables:

• Hardware optimization: Use specialized hardware for each phase
• Better resource utilization: Match workload characteristics to hardware capabilities
• Reduced latency: Optimize each phase independently

Weighted-Based Routing​

Native ModelRoute supports sophisticated weighted routing across multiple ModelServers, enabling:

• Traffic splitting: Distribute traffic across backends based on weights
• A/B testing: Gradually shift traffic between different model versions
• Capacity-based routing: Route based on backend capacity and availability

These advanced features make native ModelRoute/ModelServer ideal for production environments that require sophisticated traffic management and optimization strategies. However, Gateway API and Gateway API Inference Extension provide better interoperability and compatibility with other gateway implementations, making them suitable for multi-gateway deployments and standardized infrastructure.

Conclusion​

Kthena Router's support for Gateway API and Gateway API Inference Extension provides users with flexible routing options that balance standardization and advanced capabilities. Gateway API resolves the modelName conflict problem and enables multi-tenant isolation, while Gateway API Inference Extension provides standardized inference routing capabilities.

Users can choose between:

• Gateway API + Inference Extension: For standardized, interoperable routing that works across different gateway implementations
• Native ModelRoute/ModelServer: For advanced features like PD disaggregation, weighted routing, and sophisticated scheduling algorithms

Both approaches are fully supported and can be used together in the same cluster, providing maximum flexibility for different use cases and requirements.

For more information, please refer to the Gateway API Support Guide and Gateway Inference Extension Support Guide. All example files referenced in this blog are available in the kthena/examples/kthena-router directory.

As Large Language Models (LLMs) become increasingly central to modern applications, the infrastructure supporting them must evolve to meet demanding performance, scalability, and cost requirements. Deploying LLMs in production presents unique challenges: models are resource-intensive, inference workloads vary significantly, and users expect low latency with high throughput. Traditional load balancers and API gateways, while excellent for conventional web services, lack the awareness needed to intelligently route AI inference traffic.

Kthena Router addresses these challenges head-on. It is a Kubernetes-native, standalone inference router purpose-built for LLM serving workloads. Unlike generic proxies or load balancers, Kthena Router is model-aware, making intelligent routing decisions based on real-time metrics from inference engines. This enables sophisticated traffic management strategies that significantly improve throughput, reduce latency, and lower operational costs.

The router seamlessly integrates with existing API gateway infrastructure while providing advanced capabilities specifically designed for AI workloads:

• Model-Aware Routing: Leverages real-time metrics from inference engines (vLLM, SGLang, TGI) to make intelligent routing decisions
• LoRA-Aware Load Balancing: Intelligently route to pods that have already loaded the desired LoRA adapter to reduce adapter swap latency from hundreds of milliseconds to near-zero
• Advanced Scheduling Algorithms: Includes Prefix Cache Aware, KV Cache Aware and Fairness Scheduling, etc.
• Prefill-Decode Disaggregation: Native support for xPyD (x-prefill/y-decode) deployment patterns

Kthena Router is deployed as a standalone binary with minimal dependencies, ensuring lightweight operation and straightforward deployment. It continuously monitors inference engine metrics to obtain real-time information about model status, including currently loaded LoRA adapters, KV cache utilization, request queue lengths, and latency metrics (TTFT/TPOT). This real-time awareness enables the router to make optimal routing decisions that traditional load balancers simply cannot achieve.

2. Architecture​

Kthena Router implements a clean, modular architecture designed for performance and extensibility. The system consists of several core components working together to provide intelligent request routing.

2.1 Core Components Overview​

Router: The core execution framework responsible for receiving, processing, and forwarding requests. It orchestrates the interaction between all other components and maintains the request lifecycle from initial reception to final response.

Listener: Manages HTTP/HTTPS listeners and handles incoming traffic on specified ports. It provides flexible configuration for different protocols and can bind to multiple addresses to serve various types of requests. The listener ensures efficient connection handling and supports both streaming and non-streaming request patterns.

Controller: A Kubernetes-native component that synchronizes and processes Pods and Custom Resources (CRs) such as ModelRoute and ModelServer. The controller watches for changes in the cluster and updates the router's internal state accordingly, ensuring routing decisions are always based on current cluster topology.

Filters: Contains two critical sub-modules that process requests before they reach the backend:

• Auth: Handles traffic authentication and authorization, supporting API Key, JWT
• RateLimit: Manages comprehensive rate limiting strategies including input tokens and output tokens limiting

Backend: Provides an abstraction layer for accessing various inference engines. It masks differences in metrics interface access methods and metric naming conventions across frameworks like vLLM, SGLang, and TGI, presenting a unified interface to the scheduler.

Metrics Fetcher: Continuously collects real-time metrics from inference engine endpoints running on model pods. It gathers critical performance data including:

• KV cache utilization
• Current LoRA model status
• Request queue lengths
• Latency metrics (TTFT - Time To First Token, TPOT - Time Per Output Token)

Datastore: A unified data storage layer that provides efficient access to ModelServer-to-Pod associations, Base Model/LoRA configurations, and runtime metrics. It serves as the central repository for all routing-relevant information and supports callbacks for real-time updates.

Scheduler: The brain of the router, implementing sophisticated traffic scheduling algorithms. It consists of a scheduling framework and various pluggable scheduling algorithm plugins. The framework integrates and runs different scheduling plugins to filter and score pod collections corresponding to ModelServers, selecting the globally optimal pod as the final access target.

3. Router API​

Kthena Router's routing behavior is controlled by two key Custom Resource Definitions (CRDs): ModelServer and ModelRoute. These declarative APIs allow you to define sophisticated routing strategies using familiar Kubernetes patterns.

3.1 ModelRoute​

ModelRoute defines traffic routing rules based on request characteristics. It determines which ModelServer(s) should handle requests based on model names, LoRA adapters, HTTP headers, and other criteria.

Key fields include:

• ModelName: The model name to match in incoming requests
• LoRAAdapters: List of LoRA adapter names this route supports
• Rules: An ordered list of routing rules, each containing:
  • ModelMatch: Conditions for matching requests (headers, URIs, etc.)
  • TargetModels: List of ModelServers to route to, with optional weights
• RateLimit: Token-based rate limiting configuration

More details about ModelRoute can be found in the definition.

3.2 ModelServer​

ModelServer defines an inference service instance and its access policies. It identifies the pods running your models, specifies the inference framework being used, and defines how traffic should be handled.

Key fields include:

• WorkloadSelector: Identifies pods via labels and supports PD (Prefill-Decode) group specifications
• Model: Specifies the base model name that the server hosts
• InferenceFramework: Indicates the inference engine (vLLM, SGLang, TGI, etc.)
• WorkloadPort: Defines the port on which the inference service listens
• TrafficPolicy: Configures timeout, retry policies, and other traffic handling behaviors
• KVConnector: Specifies the KV connector type for PD disaggregated deployments (HTTP, Nixl, LMCache, Mooncake)

More details about ModelServer can be found in the definition.

3.3 Example: Header-Based Multi-Model Routing​

For tiered service offerings, route users to different model sizes based on headers:

apiVersion: networking.serving.volcano.sh/v1alpha1
kind: ModelRoute
metadata:
  name: deepseek-multi-models
  namespace: default
spec:
  modelName: "deepseek-multi-models"
  rules:
  - name: "premium"
    modelMatch:
      headers:
        user-type:
          exact: premium
    targetModels:
    - modelServerName: "deepseek-r1-7b"
  - name: "default"
    targetModels:
    - modelServerName: "deepseek-r1-1-5b"
---
apiVersion: networking.serving.volcano.sh/v1alpha1
kind: ModelServer
metadata:
  name: deepseek-r1-7b
  namespace: default
spec:
  workloadSelector:
    matchLabels:
      app: deepseek-r1-7b
  workloadPort:
    port: 8000
  model: "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"
  inferenceEngine: "vLLM"
  trafficPolicy:
    timeout: 10s
---
apiVersion: networking.serving.volcano.sh/v1alpha1
kind: ModelServer
metadata:
  name: deepseek-r1-1-5b
  namespace: default
spec:
  workloadSelector:
    matchLabels:
      app: deepseek-r1-1-5b
  workloadPort:
    port: 8000
  model: "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B"
  inferenceEngine: "vLLM"
  trafficPolicy:
    timeout: 10s

Traffic Flow:

1. Request arrives with model name "deepseek-multi-models"
2. Router checks if user-type: premium header exists
3. Premium users → routed to larger 7B model for better quality
4. Regular users → routed to smaller 1.5B model for cost efficiency

Testing premium routing:

curl http://$ROUTER_IP/v1/completions \
    -H "Content-Type: application/json" \
    -H "user-type: premium" \
    -d '{"model": "deepseek-multi-models", "prompt": "Explain quantum computing"}'

This example demonstrates how ModelRoute and ModelServer CRDs provide flexible, declarative control over sophisticated routing strategies, all managed through standard Kubernetes APIs.

4. Core Features​

4.1 Intelligent Scheduling Plugins​

What truly differentiates Kthena Router from traditional load balancers is its suite of model-aware scheduling plugins. These plugins leverage real-time inference engine metrics to make intelligent routing decisions that dramatically improve performance.

4.1.1 Prefix Cache Aware Scheduling​

Modern inference engines like vLLM and SGLang implement prefix caching, where commonly used prompt prefixes are cached to avoid redundant computation. The Prefix Cache Aware plugin maximizes cache hit rates by routing requests with similar prefixes to the same pods.

How it works:

• Extracts prompt prefixes from incoming requests
• Maintains a mapping of prefixes to pods that have processed them
• Routes new requests with matching prefixes to pods likely to have cached KV states
• Significantly reduces Time To First Token (TTFT) for repeated or similar prompts

4.1.2 KV Cache Aware Scheduling​

The KV Cache Aware plugin (kvcache-aware) routes requests to pods that are most likely to have matching KV cache entries, using token-block based matching with Redis-based distributed coordination. This maximizes cache hits and reduces redundant prefill computation.

How it works:

• The Kthena Runtime sidecar subscribes to vLLM ZMQ kv-events and writes token block hashes into Redis
• The router tokenizes incoming prompts, divides them into fixed-size blocks, and hashes each block
• Redis is queried to find which pods have cached each token block
• Pods are scored based on consecutive block matches from the beginning of the prompt
• Requires Redis and the Kthena Runtime sidecar deployed alongside vLLM pods

4.1.3 LoRA Affinity Scheduling​

LoRA (Low-Rank Adaptation) adapters enable fine-tuned model behavior without redeploying base models. However, loading and unloading adapters introduces latency. The LoRA Affinity plugin minimizes this overhead.

How it works:

• Tracks which LoRA adapters are currently loaded on each pod
• Routes requests requiring a specific LoRA to pods that already have it loaded
• Falls back to pods with available adapter slots if no pods have the adapter cached
• Reduces adapter swap latency from hundreds of milliseconds to near-zero

4.1.4 Least Latency Scheduling​

The Least Latency plugin routes requests to the fastest available pods based on real-time latency metrics.

Metrics considered:

• TTFT (Time To First Token): Important for streaming responses and user-perceived latency
• TPOT (Time Per Output Token): Critical for overall generation speed

4.1.5 Least Request Scheduling​

The Least Request plugin considers the number of requests waiting to be processed and the number of requests running to route to the least busy pods.

How it works:

• Monitors the num_requests_running and num_requests_waiting metrics from inference engines
• Calculates total pending work for each pod
• Routes new requests to the least busy pod
• Prevents hot-spotting and ensures even load distribution

4.1.6 Plugin Configurations​

These plugins work together through the scheduler framework. You can configure which plugins are enabled and their relative weights through the router configuration.

The scheduler framework runs enabled plugins in sequence:

1. Filtering: Plugins eliminate unsuitable pods (e.g., insufficient cache, wrong LoRA)
2. Scoring: Plugins score remaining pods based on their criteria
3. Selection: The highest-scoring pod(s) are selected for the request

This composable architecture allows you to tailor routing behavior to your specific workload requirements.

4.2 Fairness Scheduling​

The Fairness Scheduling ensures equitable resource allocation across users based on token consumption history.

How it works:

• Tracks cumulative token usage (input + output) per user per model
• Assigns request priority inversely proportional to historical usage
• Queues requests and processes them in priority order
• Prevents any single user from monopolizing resources

Use cases:

• Multi-tenant platforms with shared infrastructure
• Research clusters with fair-share policies
• SLA-driven systems requiring usage-based throttling

4.3 Prefill-Decode Disaggregation Support​

For advanced deployment patterns, Kthena Router natively supports prefill-decode disaggregation (xPyD), where the compute-intensive prefill phase is separated from the token generation decode phase.

How it works:

• Identifies PD group configurations from ModelServer CRDs
• Routes prefill requests to prefill-optimized pods
• Transfers KV cache state via configurable connectors (HTTP, Redis, etc.)
• Routes decode requests to decode-optimized pods
• Coordinates the two-phase process transparently to the client

Benefits:

• Optimizes hardware utilization by matching workload characteristics to hardware
• Reduces latency by using specialized hardware for each phase
• Improves cost-efficiency through better resource allocation

4.4 Token-Based Rate Limiting​

Kthena Router provides comprehensive rate limiting capabilities to protect your inference infrastructure from overload and ensure fair resource allocation across users.

• Input Token Limits: Control the rate of incoming prompt tokens per user or API key
• Output Token Limits: Limit generated tokens to manage compute costs
• Local Rate Limits: Enforces a limit on a per-router-instance basis.
• Global Rate Limits: Enforces a shared limit across all router instances, using a central store like Redis.

4.5 Observability​

Kthena Router provides comprehensive observability capabilities designed for production LLM serving:

• Metrics: Exposes detailed metrics including request latency, token consumption, scheduler plugin performance, and rate limiting statistics at /metrics endpoint
• Structured Access Logs: Records complete request lifecycle with routing decisions, timing breakdowns, and token tracking in JSON or text format
• Debug Endpoints: Provides /debug/config_dump/* APIs to inspect internal state, ModelRoute/ModelServer configurations, and real-time pod metrics
• Standard Integration: Seamlessly works with Prometheus, Grafana, ELK, and other observability stacks for monitoring, alerting, and troubleshooting

5. Performance​

The ScorePlugin module in Kthena Router leverages a configurable, pluggable architecture to enable multi-dimensional scoring and intelligent routing of inference requests.To demonstrate the impact of intelligent scheduling, we construct a standardized benchmarking environment based on the DeepSeek-R1-Distill-Qwen-7B model to evaluate the performance of different scheduling strategies under both long and short system prompt scenarios.

Experimental results demonstrate that in long system prompt scenarios, the KVCacheAware Plugin + Least Request Plugin combination achieves 2.73× higher throughput and reduces TTFT latency by 73.5%, significantly optimizing overall inference service performance and validating the core value of cache-aware scheduling for large-scale model inference.

5.1 Experimental Setup​

A standardized benchmarking environment was built using the DeepSeek-R1-Distill-Qwen-7B model to evaluate the performance of different scheduling strategies.

Table 1: Experimental Environment Configuration

5.2 Long System Prompt Scenario (4096 tokens)​

Table 2: Performance Metrics – Long Prompt

6. Conclusion​

Kthena Router represents a significant leap forward in LLM serving infrastructure. By moving beyond simple load balancing to model-aware, metrics-driven routing, it unlocks substantial performance improvements and cost savings that were previously unattainable.

It is open source and available today. The documentation provides comprehensive guides for installation, configuration, and deployment. The examples directory contains ready-to-use configurations for common scenarios.

Whether you're running a single model or managing a complex multi-tenant LLM platform, Kthena Router provides the intelligent routing capabilities needed to maximize performance, minimize costs, and deliver excellent user experiences.

As large models continue to grow exponentially in parameter size, the resource limits of a single virtual or physical machine can no longer meet their demands. To address this challenge, the industry has introduced innovative strategies such as PD-disaggregation deployment and hybrid deployment of large and small models. These approaches have transformed inference execution: instead of a single Pod handling an entire inference task, multiple Pods now often collaborate to complete a single prediction. This multi-Pod collaboration has become a key trend in large model inference deployment.

In practice, inference models may still run within a single Pod (as in traditional single-node scenarios), across a group of identical Pods (for larger models), or among Pods with specialized roles (as in PD-disaggregation deployments). This flexible deployment not only improves resource utilization but also enables more efficient large model inference.

ModelServing is a specialized component of Kthena designed to manage and orchestrate the lifecycle of inference model workloads. It can conveniently represent and manage multiple deployment models, such as PD-disaggregation, tensor parallelism, pipeline parallelism, and native model deployment, because of its three-tier architecture.

Three-tier Architecture​

ModelServing adopts a three-tier architecture ModelServing → ServingGroup → Role to address the limitations of Kubernetes' traditional two-tier architecture (e.g., deployment and statefulSet) in managing diverse inference workload deployment scenarios. The architecture diagram is shown below:

• ModelServing: The central component responsible for managing the lifecycle of inference model workloads. It offers a unified interface for deploying and managing inference models, as well as querying their statuses.
• ServingGroups: A ServingGroup is a collection of Roles. Each group can fulfill the whole inference; it includes both Prefill and Decode roles for PD-disaggregation deployment.
• Roles: Each Role within a ServingGroup consists of a group of Pods, which are the actual workloads responsible for executing inference tasks. Each role can be assigned different tasks. For example, in a PD-disaggregation scenario, you can configure a prefill role and a decode role, which is very convenient.

For the definition of ModelServing, please refer to the modelServing CRD reference.

After inference workloads are managed as groups, this approach aligns well with Volcano's gang scheduling and network topology-aware scheduling. However, basic capabilities required for traditional cloud-native workload management—such as rolling updates and scaling—still need to be handled additionally.

Gang Scheduling​

The Gang scheduling strategy is one of the core scheduling algorithms of the Volcano-Scheduler. It meets the scheduling requirements of “All or nothing” in the scheduling process and avoids the waste of cluster resources caused by arbitrary scheduling of Pod. The Gang scheduler algorithm is to observe whether the scheduled number of Pods meets the minimum number of runs. When the minimum number of runs of Job is satisfied, the scheduling action is executed for all Pods under Job; otherwise, it is not executed.

In Kthena, PodGroups are created based on ModelServing, leveraging Volcano's gang scheduling capabilities through PodGroups. The minTaskMember field specifies the number of pods in each Role that require gang scheduling.

Instance-Level Gang Scheduling​

Creation Process​

A single PodGroup is created for the entire ServingGroup instance.

This configuration is automatically generated by us, no need to create it manually.

PodGroup Configuration:

Here is a modelServing example:

apiVersion: workload.serving.volcano.sh/v1alpha1
kind: ModelServing
metadata:
  name: sample
  namespace: default
spec:
  schedulerName: volcano
  replicas: 1  # servingGroup replicas
  template:
    restartGracePeriodSeconds: 60
    gangPolicy:
      minRoleReplicas:
        prefill: 2
        decode: 2
    roles:
      - name: prefill
        replicas: 4 
        # ... additional role configuration
      - name: decode
        replicas: 4
        # ... additional role configuration

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: {modelserving-name}-{servinggroup-index}
  namespace: {modelserving-namespace}
  labels:
    modelserving.volcano.sh/name: {modelserving-name}
    modelserving.volcano.sh/group-name: {modelserving-name}-{servinggroup-index}
  annotations:
    scheduling.k8s.io/group-name: {modelserving-name}
spec:
  # MinTaskMember defines minimum pods required for each role replica
  minTaskMember:
    "prefill-0": 2
    "decode-0": 2
    # ... additional role replicas
  minResources:
    # Aggregated resource requirements
  ttlSecondsAfterFinished: {timeout-seconds}

Pod Count Calculation: If MinRoleReplicas is not configured, the value of minTaskMember is calculated as:

minMember = replicas × Σ(role.replicas × (1 + role.workerReplicas))

Where:

• replicas: Number of ServingGroup instances
• role.replicas: Number of role instances within each ServingGroup
• 1 + role.workerReplicas: EntryPod + WorkerPods per role instance

If MinRoleReplicas is configured, the value of minMember is calculated as:

MinTaskMember Generation Logic:

• For each role specified in MinRoleReplicas map
• Include role replicas from index 0 up to MinRoleReplicas[roleName] - 1
• Each role replica entry has value = (1 + role.workerReplicas)

Example Configuration:

gangPolicy:
  minRoleReplicas:
    prefill: 2    # Require at least 2 prefill role replicas
    decode: 1     # Require at least 1 decode role replica

Rolling Update​

Rolling updates represent a critical operational strategy for online services aiming to achieve zero downtime. In the context of LLM inference services, the implementation of rolling updates is important to reduce the risk of service unavailability.

Currently, ModelServing supports rolling upgrades at the ServingGroup level, enabling users to configure Partitions to control the rolling process.

• Partition: Indicates the ordinal at which the ModelServing should be partitioned for updates. During a rolling update, replicas with an ordinal greater than or equal to Partition will be updated. Replicas with an ordinal less than Partition will not be updated.

Here's a ModelServing configured with rollout strategy:

spec:
  rolloutStrategy:
    type: ServingGroupRollingUpdate
    rollingUpdateConfiguration:
      partition: 0

In the following we'll show how rolling update processes for a ModelServing with four replicas. Three Replica status are simulated here:

• ✅ Replica has been updated
• ❎ Replica hasn't been updated
• ⏳ Replica is in rolling update

During a rolling upgrade, the controller deletes and rebuilds the replica with the highest sequence number among the replicas need to be updated. The next replica will not be updated until the new replica is running normally.

Scaling​

In cloud-native infrastructure projects, scaling plays a crucial role in resource optimization and cost control, enhancing service availability and quickly response, and simplifying operations management.

In modelServing, as has two layers of resource descriptions, ServingGroup and Role. Therefore, we also support the scale up and scale down of the ServingGroup level and role level.

The handling of scaling and rolling updates at the servingGroup level is similar, with updates applied in reverse order within the replica set.

Role-level scaling enables fine-grained adjustment of replica counts for each Role, such as in PD-disaggregated deployment scenarios. This allows dynamic adjustment of prefill or decode replicas, enabling you to optimize the P/D ratio based on their respective workloads. For example:

1. Long prompt, short output scenarios: You can increase prefill replicas to handle the computational intensive prompt processing while maintaining fewer decode replicas.

2. Short prompt, long output scenarios: You can increase decode replicas to handle the sequential token generation while maintaining fewer prefill replicas.

This flexible scaling capability ensures optimal resource allocation based on actual workload patterns.

When the role.Replicas is modified, it triggers the Scaling of the role granularity.

When scaling is triggered, the status of the entire ServingGroup is set to scaling, and then the pod creation or deletion process is performed.

After the replicas of pods meets expectations. Then update the status of the ServingGroup based on the status of all the pods in the ServingGroup.

And because the pods in the role are carrying sequential and labeled. All scaling is processed from the last pod.

Role Scaling Process​

The overall processing flowchart is shown in the figure.

Restart Policy​

In ModelServing, the concept of grouping pods is emphasized. As a result, when an error occurs in one pod within a group, the entire group is typically restarted together. However, in production environments, restarting a whole group of pods can require significant resources and time. To address this, ModelServing also provides a restart policy that allows for individual pod restarts.

• ServingGroupRecreate: When an error occurs in pods within a group, the entire group is restarted.
• RoleRecreate: When an error occurs in pods within a group, the status of the group is updated to progressing and only the affected pod is restarted. If the serviceGroup remains in the progressing state for a certain period of time, the entire servingGroup will be deleted and recreated.

Prospect​

The current ModelServing is generally able to meet the basic requirements for model workload management and scheduling. However, support for PD-separated scenarios is still limited—for example, features such as PD instance upgrades are not yet fully supported. In future development, we plan to introduce more features specifically for PD-separated scenarios.

If you are interested, we welcome you to join the Kthena community and help build our open-source ecosystem together.

This paper analyzes the system design and implementation of the ScorePlugin module in Kthena Router, which leverages a configurable, pluggable architecture to enable multi-dimensional scoring and intelligent routing of inference requests. We provide a detailed examination of the six currently implemented ScorePlugins, and construct a standardized benchmarking environment based on the DeepSeek-R1-Distill-Qwen-7B model to evaluate the performance of different scheduling strategies under both long and short system prompt scenarios.

Experimental results demonstrate that in long system prompt scenarios, the KVCacheAware Plugin + Least Request Plugin combination achieves 2.73× higher throughput and reduces TTFT latency by 73.5%, significantly optimizing overall inference service performance and validating the core value of cache-aware scheduling for large-scale model inference.

1. Introduction​

In LLM inference services, intelligent request scheduling strategies have a decisive impact on overall system performance. Kthena Router adopts a score-based scheduling framework that enables multi-strategy load balancing through an extensible plugin system. This paper systematically analyzes the architectural design, core algorithms, and performance characteristics of this scheduling system.

2. System Architecture​

2.1 Core Interface Design​

The scheduling system in Kthena Router is built around a unified ScorePlugin interface:

type ScorePlugin interface {
    Name() string
    Score(ctx *Context, pods []*datastore.PodInfo) map[*datastore.PodInfo]int
}

Each plugin generates a normalized score in the range [0, 100] for candidate Pods, where higher scores indicate better suitability for the incoming inference request.

2.2 Scheduling Execution Pipeline​

The scheduler uses a multi-stage pipeline architecture consisting of the following steps:

1. Filtering Phase - Executes resource constraint checks and availability validation through Filter Plugins
2. Scoring Phase - Multiple Score Plugins calculate Pod suitability scores in parallel
3. Weighted Aggregation - Combines multi-plugin results according to configurable weights
4. Selection Decision - Selects the top-N Pods based on the final aggregated score

3. ScorePlugin Implementations​

3.1 GPU Cache Usage Plugin​

A resource-aware scheduling strategy based on GPU cache utilization.

Algorithm Principles:

• Monitors the GPU cache usage rate (GPUCacheUsage) of each Pod in real time
• Scoring function:

  Score = (1.0 - GPUCacheUsage) × 100
  

• Prioritizes Pods with lower GPU cache utilization, reducing the risk of memory overflow

Use Case: Environments with limited GPU memory availability.

3.2 Least Request Plugin​

A load-balancing strategy based on active request queue length.

Algorithm Principles:

• Metrics:
  • Running request count: RequestRunningNum
  • Waiting queue length: RequestWaitingNum
• Base load calculation:

  base = RequestRunningNum + 100 × RequestWaitingNum
  

• Normalized score:

  score = ((maxScore - baseScores[info]) / maxScore) × 100
  

Key Design Details:

• Waiting queue weight factor set to 100, strongly penalizing Pods with backlog accumulation
• Enables dynamic load-aware balancing

3.3 Least Latency Plugin​

A performance-oriented scheduling strategy driven by inference latency metrics.

Algorithm Principles:

• Metrics Monitored:
  • TTFT (Time To First Token)
  • TPOT (Time Per Output Token)
• Normalization:

  ScoreTTFT = (MaxTTFT - CurrentTTFT) / (MaxTTFT - MinTTFT) × 100
  ScoreTPOT = (MaxTPOT - CurrentTPOT) / (MaxTPOT - MinTPOT) × 100
  

• Final weighted score:

  Score = α × ScoreTTFT + (1 - α) × ScoreTPOT
  

Configuration:

TTFTTPOTWeightFactor: 0.5  # TTFT weight α

3.4 KVCacheAware Plugin​

An advanced cache-aware scheduling strategy leveraging KV cache hit rates.

Technical Highlights:

• Token-level semantic matching with model-specific tokenizers
• Distributed cache coordination via Redis for cross-Pod state synchronization
• Optimized contiguous block matching algorithm to maximize cache hits

Algorithm Workflow:

1. Tokenization Stage

   Input → Model-specific Tokenizer → Token Sequence [t₁, t₂, ..., tₙ]
   

2. Block Segmentation

   Token Sequence → Fixed-size Blocks [B₁, B₂, ..., Bₘ]
   Block size: 128 tokens (configurable)
   

3. Hash Generation

   For each Bᵢ: Hash(Bᵢ) = SHA256(Bᵢ) → hᵢ
   

4. Distributed Cache Query

   Redis pipeline query:
   For each hᵢ → {Pod₁: timestamp₁, Pod₂: timestamp₂, ...}
   

5. Contiguous Matching Score

   Score = (Number of contiguous matching blocks / Total blocks) × 100
   

Configuration Example:

blockSizeToHash: 128      # Token block size
maxBlocksToMatch: 128     # Maximum blocks to process

Redis Key Structure:

Key Pattern: "matrix:kv:block:{model}@{hash}"
Value Structure:
{
  "pod-name-1.namespace": "1703123456",
  "pod-name-2.namespace": "1703123789"
}

3.5 Prefix Cache Plugin​

A prefix-based cache-aware scheduling strategy.

Architecture Design:

• Three-level mapping: Model → Hash → Pod
• LRU-based cache eviction for efficient memory management
• Top-K prefix matching strategy to return the top Pods with the longest prefix matches

Key Features:

• Byte-level prefix matching algorithm
• Optimized in-memory LRU implementation
• Configurable cache capacity and candidate size

3.6 Random Plugin​

A lightweight random load-balancing strategy.

Implementation:

• Generates uniformly distributed random scores:

  Score ~ Uniform(0, 100)
  

• Thread-safe independent random number generator
• Stateless design ensures long-term balancing behavior

Use Cases:

• Mixed workloads without clear patterns
• Lightweight deployments avoiding complex scheduling overhead
• Establishing baseline performance comparisons

4. Experimental Design and Performance Evaluation​

4.1 Experimental Setup​

A standardized benchmarking environment was built using the DeepSeek-R1-Distill-Qwen-7B model to evaluate the performance of different scheduling strategies.

Table 1: Experimental Environment Configuration

4.2 Long System Prompt Scenario (4096 tokens)​

Table 2: Performance Metrics – Long Prompt

4.3 Short System Prompt Scenario (256 tokens)​

Table 3: Performance Metrics – Short Prompt

4.4 Visualization of Improvements​

Table 4: Performance Gains vs. Baseline (Random Plugin)

The KVCacheAware combination strategy shows significant superiority in long prefix scenarios, especially for throughput and TTFT metrics.

5. Results and Discussion​

5.1 Advantages of Cache-Aware Strategies​

Results clearly demonstrate the performance benefits of cache-aware scheduling for long prompts:

• Peak throughput of 32.22 req/s with KVCacheAware + Least Request, a 173% improvement over the Random baseline
• TTFT reduction from 2.15s to 0.57s (73.5% decrease)
• End-to-end latency reduced from 25.23s to 9.22s (63.5% improvement)

5.2 Scenario Adaptability​

Long Prefix Scenario (4096 tokens):

• Cache-aware strategies significantly outperform traditional balancing methods
• Token-level block matching maximizes cache utilization
• Distributed cache coordination prevents redundant computations across Pods

Short Prefix Scenario (256 tokens):

• Performance differences between strategies are minimal
• Limited prefix length restricts cache hit opportunities
• Traditional strategies provide stable results

5.3 Design Trade-Offs​

Computation Overhead vs. Performance Gains:

• KVCacheAware introduces additional overhead from tokenization and Redis queries
• In high cache-hit environments, performance gains far outweigh the added cost
• Recommends adaptive strategy selection based on workload characteristics

6. Deployment Recommendations​

6.1 High Cache Hit Scenarios​

Recommended Configuration: KVCacheAware Plugin + Least Request Plugin

Use Cases:

• Customer service chatbots, code generation, and similar structured workloads
• Multi-turn conversations with long system prompts
• Template-based content generation services

6.2 General Load-Balancing Scenarios​

Recommended Configuration: Least Request Plugin + Least Latency Plugin

Use Cases:

• General-purpose inference services with diverse request patterns
• Multi-tenant environments requiring fair resource distribution
• Real-time interactive applications

6.3 Resource-Constrained Environments​

Recommended Configuration: GPU Cache Usage Plugin

Use Cases:

• Edge deployments with limited GPU memory
• Cost-sensitive cloud environments
• Scenarios where multiple models share GPU resources

7. Conclusion​

This paper presents a comprehensive analysis of the Kthena Router ScorePlugin architecture and its performance characteristics. The experimental results validate the effectiveness of cache-aware scheduling strategies, particularly in scenarios with long system prompts where the KVCacheAware Plugin + Least Request Plugin combination achieves substantial performance improvements.

Key findings include:

1. Cache-aware strategies provide significant performance benefits in long prompt scenarios, with up to 173% throughput improvement
2. Adaptive strategy selection based on workload characteristics is crucial for optimal performance
3. The pluggable architecture enables flexible deployment configurations tailored to specific use cases

Future work will focus on developing adaptive scheduling algorithms that can automatically select optimal plugin combinations based on real-time workload analysis and system conditions.

Document generated on: September 9, 2025

Kthena is a cloud-native, high-performance LLM inference routing, orchestration, and scheduling system designed specifically for Kubernetes. It aims to solve the core challenges of deploying and serving LLMs at scale in production environments. Through its unique features such as KV Cache-aware scheduling and Prefill/Decode separation routing, Kthena significantly improves GPU resource utilization, reduces inference latency, and provides enterprises with unprecedented flexibility and control.

As a sub-project of Volcano, Kthena is dedicated to helping Volcano expand its boundaries beyond AI training, creating a complete integrated solution for both training and inference.

Project Address

The "Last Mile" Challenge of Production LLM Serving​

Large Language Models (LLMs) are reshaping industries at an unprecedented pace, but deploying them efficiently and cost-effectively in production environments—especially on cloud-native platforms based on Kubernetes—remains fraught with difficulties. Developers generally face the following challenges:

1. Low Resource Utilization: LLM inference, particularly its unique KV Cache mechanism, consumes dynamic and massive amounts of GPU memory. Traditional load balancing typically uses Round-Robin algorithms, which are unaware of these load characteristics, leading to high costs where GPU resources sit idle while requests are queued.
2. Difficulty Balancing Latency and Throughput: LLM inference consists of two phases: Prefill (processing input prompts) and Decode (generating tokens). The former is compute-intensive, while the latter is memory-bound. Scheduling them together often prevents targeted optimization, affecting overall service response speed and throughput. Consequently, Prefill/Decode (PD) separation has become mainstream, but efficient routing and scheduling remains a challenge.
3. Complex Multi-tenant and Multi-model Management: In enterprise environments, it is often necessary to serve multiple business units, different model versions, or LoRA-fine-tuned models simultaneously. Implementing fair scheduling, priority management, and dynamic routing is a complex engineering problem; some industry solutions even map AI gateways one-to-one with large models.
4. Lack of Native Kubernetes Integration: Many existing solutions are either external systems disconnected from the Kubernetes ecosystem or overly complex, failing to meet the simplicity and operational flexibility required for production-grade use.

Kthena: The Intelligent Brain of Cloud-Native LLM Inference​

To overcome these challenges, Kthena was born. It is not intended to replace existing LLM serving frameworks (such as vLLM or SGLang), but rather to serve as an intelligent traffic hub and scheduling center above them, deeply integrated into Kubernetes.

Kthena consists of two major components:

1. Kthena Router: An independent, high-performance multi-model router responsible for receiving inference requests and intelligently distributing them to backend ModelServer instances based on ModelRoute rules.

2. Kthena Controller Manager: A controller in the Kubernetes control plane, composed of multiple controllers responsible for orchestration and lifecycle management of LLM workloads. It continuously reconciles and links multiple CRDs (such as ModelBooster, ModelServing, AutoScalingPolicy/AutoScalingPolicyBinding, and ModelRoute/ModelServer) to transform declarative APIs into runtime resources.

The ModelServing controller orchestrates ServingGroup and Prefill/Decode role groupings; supports topology-aware scheduling, Gang scheduling, rolling upgrades, and fault recovery; and implements elastic scaling based on AutoScalingPolicy.

This architecture makes Kthena a highly programmable bridge connecting user requests with LLM inference backends.

Core Features and Advantages​

Kthena’s strength lies in its core capabilities specifically designed for LLM inference scenarios.

1) Production-Grade Inference Orchestration (ModelServing)​

• Three-layer LLM workload model: ModelServing -> ServingGroup -> Role. A single API supports multiple deployment forms, including native LLM deployment, PD separation, and large-scale EP (Expert Parallelism).
• Prefill-Decode Separation Deployment: Schedules compute-intensive Prefill instances onto nodes with high compute capability, and memory-bound Decode instances onto nodes with high-bandwidth memory, enabling better resource matching and lower end-to-end latency. Prefill and Decode can scale independently to adapt to mixed traffic patterns.
• Multiple Parallel Paradigms: Flexible configuration of TP/PP/DP/EP to maximize utilization and meet SLO targets.
• Topology-aware + Gang Scheduling: Gang scheduling ensures atomic placement of related Pods (ServingGroup/Role), reducing tail latency and avoiding wasted fragmented resources; topology-aware scheduling improves parallel communication efficiency.

2) Out-of-the-Box Model Onboarding (ModelBooster)​

• Provides deployment templates for mainstream models, including PD separation.
• Automatically generates routing policies and lifecycle resources such as ModelRoute, ModelServer, ModelServing, and Autoscaling.
• Covers common deployment scenarios while allowing fine-grained customization via ModelServing.

3) Intelligent, Model-Aware Routing (Kthena Router)​

• Multi-model routing: OpenAI API compatible; routes based on headers or request content.
• Pluggable scheduling: Least Request, Minimum Latency, KV Cache-aware, Prefix Cache-aware, LoRA affinity, GPU utilization-aware, Fair Scheduling, and more.
• Hot-pluggable LoRA without interruption: Detects adapters loaded by inference engines and routes without downtime.
• Traffic governance: weight-based routing, canary release, token-level rate limiting, and failover.
• All-in-one routing architecture: no need to deploy Envoy Gateway; natively supports PD-separated traffic and consolidates multi-layer routing for simpler operations.

4) Cost-Driven Auto-Scaling (Autoscaler)​

• Homogeneous scaling: stable and burst modes, scaling by business metrics (CPU/GPU/Memory/custom).
• Heterogeneous deployment optimization: cost-capability greedy placement across multiple inference engines and heterogeneous accelerators.

5) Support for Mainstream Inference Engines and Heterogeneous Hardware​

• Supports vLLM, SGLang, Triton/TGI, etc. with unified API abstraction and standardized metrics.
• Supports hybrid deployments across GPU/NPU, working with heterogeneous autoscaling to balance cost and SLO.

6) Built-in Traffic Control and Fair Scheduling​

• Fair scheduling: based on priority and historical token consumption; prevents starvation.
• Rate limiting: fine-grained controls by user, model, and token length.

Extreme Performance Improvements​

Based on the scheduling plugin architecture of Kthena Router, in scenarios with long system prompts (e.g., 4096 tokens), the KV Cache-aware + Least Request strategy compared to a random baseline:

• Throughput can be increased by approximately 2.73×.
• TTFT (Time to First Token) is reduced by approximately 73.5%.
• End-to-end latency is reduced by more than 60%.

The gap in short-prompt scenarios converges with prompt length, but in multi-turn conversations, templated generation, and business scenarios with highly similar prefixes, KV Cache-aware routing delivers significant gains. Actual improvements depend on model size, prompt length, and hardware, but the “mix-and-match, scenario-based selection” approach has proven effective.

Future Roadmap​

Kthena received attention and support from community users in its early planning and development stages, but this is only the beginning. We plan to support more efficient scheduling algorithms, broader best practices for large model deployment, and will continue to focus on large-scale deployment and performance optimization for LLM inference.

Start Exploring Kthena Now​

• GitHub Repository: https://github.com/volcano-sh/kthena
• Official Website: https://kthena.volcano.sh/
• Community (Slack): https://cloud-native.slack.com/archives/C011GJDQS0N

Let’s work together to give LLMs cloud-native wings and release the full potential of AI.