Router Routing

This page describes the router routing features and capabilities in Kthena, based on real-world examples and configurations.

Overview

Kthena Router provides sophisticated traffic routing capabilities that enable intelligent forwarding of inference requests to appropriate backend models. The routing system is built around two core Custom Resources (CRs):

• ModelServer: Defines backend inference service instances with their associated pods, models, and traffic policies. It uses workloadSelector.matchLabels to select which pods belong to it (pods must match all given labels); this applies to every ModelServer.
• ModelRoute: Defines routing rules based on request characteristics such as model name, LoRA adapters, HTTP headers, and weight distribution

For a detailed definition of the ModelServer and ModelRoute CRs, please refer to the ModelServer and ModelRoute Reference pages.

The router supports various routing strategies, from simple model-based forwarding to complex weighted distribution, header-based routing, and PD-Disaggregated (Prefill/Decode) routing. This flexibility allows for advanced deployment patterns including canary releases, A/B testing, load balancing across heterogeneous model deployments, and PD-Disaggregated inference for lower latency and better resource utilization.

Preparation

Before diving into the routing configurations, let's set up the environment and understand the prerequisites. All the configuration examples in this document can be found in the examples/kthena-router directory of the Kthena repository.

Environment Setup

To simplify deployment and reduce the requirements for demonstration environments, we use a mock LLM server instead of deploying real models. This mock server implements the vLLM standard interface and returns mock data, making it perfect for testing and learning purposes.

Prerequisites

• Kubernetes cluster with Kthena installed
• Access to the Kthena examples repository
• Basic understanding of router CRDs (ModelServer and ModelRoute)

Getting Started

1. Deploy mock LLM inference engine if you do not have a real GPU/NPU environment at the moment. mock deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B and mock deepseek-ai/DeepSeek-R1-Distill-Qwen-7B

2. Deploy all kinds of ModelServer, such as deepseek-r1-1-5b, deepseek-r1-7b, etc., as backends of different routing strategies.

3. All routing examples in this guide use these mock services, so you can experiment with different routing strategies without the overhead of real model deployment.

Routing Scenarios

1. Simple Model-Based Routing

Scenario: Direct all requests for a specific model to a single backend service.

Traffic Processing: When a request comes in for model "deepseek-r1", the router matches this criterion and forwards all traffic to the 1.5B ModelServer. This is the most straightforward routing pattern.

apiVersion: networking.serving.volcano.sh/v1alpha1
kind: ModelRoute
metadata:
  name: deepseek-simple
  namespace: default
spec:
  modelName: "deepseek-simple"
  rules:
  - name: "default"
    targetModels:
    - modelServerName: "deepseek-r1-1-5b"

Flow Description:

1. Request arrives for model name "deepseek-r1"
2. Router matches the modelName field in the ModelRoute
3. 100% of traffic is directed to deepseek-r1-1-5b
4. The ModelServer serves requests using vLLM inference engine with 10s timeout

Try it out:

export MODEL="deepseek-simple"

curl http://$ROUTER_IP/v1/completions \
    -H "Content-Type: application/json" \
    -d "{
        \"model\": \"$MODEL\",
        \"prompt\": \"San Francisco is a\",
        \"temperature\": 0
    }"
{"choices":[{"finish_reason":"length","index":0,"logprobs":null,"text":"This is simulated message from deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B!"}],"created":1756366365,"id":"cmpl-uqkvlQyYK7bGYrRHQ0eXlWi7","model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B","object":"text_completion","system_fingerprint":"fp_44709d6fcb","usage":{"completion_tokens":218,"prompt_tokens":1,"time":0.0,"total_tokens":219}}

2. LoRA-Aware Routing

Scenario: Route requests requiring specific LoRA adapters to specialized ModelServers optimized for LoRA workloads.

Traffic Processing: When a request specifies LoRA adapters (lora-A or lora-B), the router routes it to ModelServers configured to handle these specific adapters.

apiVersion: networking.serving.volcano.sh/v1alpha1
kind: ModelRoute
metadata:
  name: deepseek-lora
  namespace: default
spec:
  loraAdapters:
  - "lora-A"
  - "lora-B"
  rules:
  - name: "lora-route"
    targetModels:
    - modelServerName: "deepseek-r1-1-5b"

Flow Description:

1. Request arrives with LoRA adapter requirement (lora-A or lora-B)
2. Router matches the LoRA adapter against the supported list
3. Routes to deepseek-r1-1-5b ModelServer configured for LoRA workloads
4. ModelServer efficiently handles LoRA adapter loading and inference

Try it out:

export MODEL="lora-A"

curl http://$ROUTER_IP/v1/completions \
    -H "Content-Type: application/json" \
    -d "{
        \"model\": \"$MODEL\",
        \"prompt\": \"San Francisco is a\",
        \"temperature\": 0
    }"
{"choices":[{"finish_reason":"length","index":0,"logprobs":null,"text":"This is simulated message from lora-A!"}],"created":1756366636,"id":"cmpl-uqkvlQyYK7bGYrRHQ0eXlWi7","model":"lora-A","object":"text_completion","system_fingerprint":"fp_44709d6fcb","usage":{"completion_tokens":120,"prompt_tokens":1,"time":0.0,"total_tokens":121}}

3. Weight-Based Traffic Distribution

Scenario: Gradually roll out new model versions by splitting traffic between different versions using weighted distribution.

Traffic Processing: The router uses weighted round-robin to distribute requests. For every 100 requests, approximately 70 will go to version 1 and 30 to version 2. This allows safe validation of new model versions with controlled risk.

apiVersion: networking.serving.volcano.sh/v1alpha1
kind: ModelRoute
metadata:
  name: deepseek-subset
  namespace: default
spec:
  modelName: "deepseek-subset"
  rules:
  - name: "deepseek-r1-route"
    targetModels:
    - modelServerName: "deepseek-r1-1-5b-v1"
      weight: 70
    - modelServerName: "deepseek-r1-1-5b-v2"
      weight: 30

Flow Description:

1. Request arrives for model "deepseek-r1"
2. Router applies weighted distribution algorithm
3. 70% of requests → deepseek-r1-1-5b-v1 (stable version)
4. 30% of requests → deepseek-r1-1-5b-v2 (new version being tested)
5. This enables controlled testing of new model versions

NOTE: This scenario need to deploy canary version of ModelServer and mock deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B to test.

Try it out:

export MODEL="deepseek-subset"

for i in $(seq 1 100);
do
    curl http://$ROUTER_IP/v1/completions \
        -H "Content-Type: application/json" \
        -d "{
            \"model\": \"$MODEL\",
            \"prompt\": \"San Francisco is a\",
            \"temperature\": 0
        }";
done
{"choices":[{"finish_reason":"length","index":0,"logprobs":null,"text":"This is simulated message from deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B-v2!"}],"created":1756371124,"id":"cmpl-uqkvlQyYK7bGYrRHQ0eXlWi7","model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B-v2","object":"text_completion","system_fingerprint":"fp_44709d6fcb","usage":{"completion_tokens":313,"prompt_tokens":1,"time":0.0,"total_tokens":314}}
{"choices":[{"finish_reason":"length","index":0,"logprobs":null,"text":"This is simulated message from deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B-v2!"}],"created":1756371124,"id":"cmpl-uqkvlQyYK7bGYrRHQ0eXlWi7","model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B-v2","object":"text_completion","system_fingerprint":"fp_44709d6fcb","usage":{"completion_tokens":313,"prompt_tokens":1,"time":0.0,"total_tokens":314}}
{"choices":[{"finish_reason":"length","index":0,"logprobs":null,"text":"This is simulated message from deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B-v1!"}],"created":1756371124,"id":"cmpl-uqkvlQyYK7bGYrRHQ0eXlWi7","model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B-v1","object":"text_completion","system_fingerprint":"fp_44709d6fcb","usage":{"completion_tokens":47,"prompt_tokens":1,"time":0.0,"total_tokens":48}}
{"choices":[{"finish_reason":"length","index":0,"logprobs":null,"text":"This is simulated message from deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B-v1!"}],"created":1756371124,"id":"cmpl-uqkvlQyYK7bGYrRHQ0eXlWi7","model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B-v1","object":"text_completion","system_fingerprint":"fp_44709d6fcb","usage":{"completion_tokens":118,"prompt_tokens":1,"time":0.0,"total_tokens":119}}
{"choices":[{"finish_reason":"length","index":0,"logprobs":null,"text":"This is simulated message from deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B-v2!"}],"created":1756371124,"id":"cmpl-uqkvlQyYK7bGYrRHQ0eXlWi7","model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B-v2","object":"text_completion","system_fingerprint":"fp_44709d6fcb","usage":{"completion_tokens":285,"prompt_tokens":1,"time":0.0,"total_tokens":286}}
{"choices":[{"finish_reason":"length","index":0,"logprobs":null,"text":"This is simulated message from deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B-v2!"}],"created":1756371124,"id":"cmpl-uqkvlQyYK7bGYrRHQ0eXlWi7","model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B-v2","object":"text_completion","system_fingerprint":"fp_44709d6fcb","usage":{"completion_tokens":409,"prompt_tokens":1,"time":0.0,"total_tokens":410}}
{"choices":[{"finish_reason":"length","index":0,"logprobs":null,"text":"This is simulated message from deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B-v1!"}],"created":1756371124,"id":"cmpl-uqkvlQyYK7bGYrRHQ0eXlWi7","model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B-v1","object":"text_completion","system_fingerprint":"fp_44709d6fcb","usage":{"completion_tokens":343,"prompt_tokens":1,"time":0.0,"total_tokens":344}}
...

4. Header-Based Multi-Model Routing

Scenario: Route traffic to different model sizes based on user tier, enabling premium users to access more powerful models.

Traffic Processing: The router evaluates incoming requests in the order rules are defined. Premium users (identified by user-type: premium header) are routed to the 7B model, while regular users fall back to the 1.5B model.

apiVersion: networking.serving.volcano.sh/v1alpha1
kind: ModelRoute
metadata:
  name: deepseek-multi-models
  namespace: default
spec:
  modelName: "deepseek-multi-models"
  rules:
  - name: "premium"
    modelMatch:
      headers:
        user-type:
          exact: premium
    targetModels:
    - modelServerName: "deepseek-r1-7b"
  - name: "default"
    targetModels:
    - modelServerName: "deepseek-r1-1-5b"

Flow Description:

1. Request arrives for model "deepseek-r1" with headers
2. Router first checks if user-type: premium header exists with exact match
3. If premium header found → Routes to deepseek-r1-7b (7B model using SGLang)
4. If no premium header → Falls back to deepseek-r1-1-5b (1.5B model using vLLM)
5. Premium users get access to the more powerful 7B model for better performance

Try it out:

export MODEL="deepseek-multi-models"

curl http://$ROUTER_IP/v1/completions \
    -H "Content-Type: application/json" \
    -H "user-type: premium" \
    -d "{
        \"model\": \"$MODEL\",
        \"prompt\": \"San Francisco is a\",
        \"temperature\": 0
    }"
{"choices":[{"finish_reason":"length","index":0,"logprobs":null,"text":"This is simulated message from deepseek-ai/DeepSeek-R1-Distill-Qwen-7B!"}],"created":1756367891,"id":"cmpl-uqkvlQyYK7bGYrRHQ0eXlWi7","model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B","object":"text_completion","system_fingerprint":"fp_44709d6fcb","usage":{"completion_tokens":71,"prompt_tokens":1,"time":0.0,"total_tokens":72}}

5. PD-Disaggregated Routing

Scenario: Route inference requests to Prefill/Decode disaggregated backends, where prefill instances handle prefill and decode instances handle decode stages independently.

Traffic Processing: When a request arrives for a PD-Disaggregated model, the router selects prefill and decode pods via pdGroup, proxies prefill to the prefill instance and decode to the decode instance, coordinating KV state exchange between them.

ModelServer (with pdGroup):

apiVersion: networking.serving.volcano.sh/v1alpha1
kind: ModelServer
metadata:
  name: deepseek-r1-1-5b-pd-disaggregation
  namespace: default
spec:
  workloadSelector:
    matchLabels:
      app: deepseek-r1-1-5b
    pdGroup:
      groupKey: "modelserving.volcano.sh/group-name"
      prefillLabels:
        modelserving.volcano.sh/rolename: "P-instance"
      decodeLabels:
        modelserving.volcano.sh/rolename: "D-instance"
  workloadPort:
    port: 8000
  model: "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B"
  inferenceEngine: "vLLM"
  trafficPolicy:
    timeout: 10s

How workloadSelector works

Two layers: matchLabels (general) selects which pods belong to this ModelServer; pdGroup (PD-only) then groups and roles them.

• ModelServer: Defines backend inference service instances with their associated pods, models, and traffic policies. It uses workloadSelector.matchLabels to select which pods belong to it (pods must match all given labels); this applies to every ModelServer.
• matchLabels: Pods must match all key-value pairs. Used by every ModelServer.
• pdGroup (only for PD-Disaggregated):
  • groupKey — Label key whose value defines a PD group. Prefill and decode pods with the same value are paired (shared KV state).
  • prefillLabels / decodeLabels — Among those pods, match these labels to classify as prefill or decode.

The router sends prefill and decode for a request to a paired prefill pod and decode pod (same groupKey value).

ModelRoute:

apiVersion: networking.serving.volcano.sh/v1alpha1
kind: ModelRoute
metadata:
  name: deepseek-r1-1-5b-pd-disaggregation
  namespace: default
spec:
  modelName: "deepseek-r1-1-5b-pd-disaggregation"
  rules:
  - name: "default"
    targetModels:
    - modelServerName: "deepseek-r1-1-5b-pd-disaggregation"

Flow Description:

1. Request arrives for the PD-Disaggregated model
2. Router matches the ModelRoute and resolves the target ModelServer
3. Via pdGroup, router selects a prefill-pod and a decode-pod (same groupKey value)
4. Prefill runs on prefill instance; decode runs on decode instance, with KV state exchanged between them (configure kvConnector for nixl/mooncake when needed)
5. Response returned to client

NOTE: Deploy ModelServing with PD roles first, then apply ModelServer and ModelRoute.

OpenAI Responses API

Kthena Router routes POST /v1/responses the same way it routes /v1/chat/completions and /v1/completions: through the same ModelRoute matching, scheduling, authentication, rate limiting, and access logging described throughout this guide. The Router forwards Responses requests as received — it does not translate a Responses request into Chat Completions or vice versa — so the selected upstream model server must natively implement POST /v1/responses. A backend that only implements /v1/chat/completions will not work here, even if the request carries a model field; that field is still required for ModelRoute matching, as with any other protocol.

Supported request forms:

• input as a plain string, or as a structured list of message items ({"role": "...", "content": [...]}).
• instructions.
• max_output_tokens — the Responses equivalent of the Chat Completions max_tokens / max_completion_tokens fields.
• Opaque or provider-specific fields such as previous_response_id, tools, metadata, and reasoning are forwarded unchanged. When a ModelServer rewrites model, only the model field changes; the rest of the request body is preserved.

Streaming: stream: true requests are forwarded as Server-Sent Events without modification. The Router recognizes response.completed, response.incomplete, and response.failed as terminal events and does not require or inject a Chat Completions data: [DONE] marker.

Usage accounting: The Router reads usage.input_tokens, usage.output_tokens, and usage.total_tokens — from the response body, or from whichever terminal event carries usage in a streamed response — and feeds them into the same output-token rate limiting, access logs, and kthena_router_tokens_total metric used for Chat Completions.

Routing coverage: Responses requests are supported through every backend path described in this guide, provided the upstream natively supports /v1/responses:

• Aggregated ModelServer targets (see Simple Model-Based Routing above).
• InferencePool / Gateway API targets — see Gateway API Support and Gateway API Inference Extension Support.
• PD-Disaggregated targets (see PD-Disaggregated Routing above), including the default HTTP connector and the SGLang and NIXL KV connectors. The prefill request caps generation with max_output_tokens instead of the Chat Completions max_tokens / max_completion_tokens fields. PD mode additionally requires the upstream's Responses response to carry the selected connector's transfer metadata: the NIXL connector needs kv_transfer_params in the prefill response to hand the KV cache off to the decode Pod. As of vLLM v0.10.0, its Responses API response does not populate kv_transfer_params, so NIXL-based PD-Disaggregated routing is not currently usable with vLLM Responses endpoints; the default HTTP connector and SGLang's bootstrap-based connector do not depend on this field and are unaffected.

Try it out (using the deepseek-simple ModelRoute from Simple Model-Based Routing):

export MODEL="deepseek-simple"

curl http://$ROUTER_IP/v1/responses \
    -H "Content-Type: application/json" \
    -d "{
        \"model\": \"$MODEL\",
        \"input\": \"San Francisco is a\"
    }"

Multi-turn conversations: Responses requests can chain turns with previous_response_id or a conversation identifier. If a backend keeps per-conversation state on the Pod that produced a response, follow-up turns need to reach that same Pod to use it. Kthena Router does not derive Pod affinity from these fields, and Session Boost affects queue ordering only. To keep a conversation's turns on one Pod, enable session sticky routing and send the configured session key (for example an X-Session-ID header) on every turn. If that Pod is no longer available, the router moves the session to another Pod, and state kept only on the old Pod is not carried over.

---

This comprehensive routing system enables flexible, scalable, and maintainable model serving infrastructure that can adapt to various deployment patterns and user requirements.