API Reference

Packages

• networking.serving.volcano.sh/v1alpha1

networking.serving.volcano.sh/v1alpha1

Resource Types

• ExternalModelProvider
• ExternalModelProviderList
• ModelRoute
• ModelRouteList
• ModelServer
• ModelServerList

BodyMatch

BodyMatch defines the predicate used to match request body content

Appears in:

• ModelMatch

Field	Description	Default	Validation
model string	Model is the name of the model or lora adapter to match. If this field is not specified, any model or lora adapter will be matched.

ConnectionPool

ConnectionPool configures the HTTP connection pool for a ModelServer.

Appears in:

• TrafficPolicy

Field	Description	Default	Validation
maxIdleConnections integer	MaxIdleConnections is the total idle connections across all endpoints. Defaults to 100 when omitted.		Minimum: 0
maxIdleConnectionsPerHost integer	MaxIdleConnectionsPerHost is the idle connections per pod/endpoint. Defaults to 64 when omitted.		Minimum: 0
maxConnectionsPerHost integer	MaxConnectionsPerHost limits dialing, active and idle connections per host. 0 means unlimited. Defaults to 0 when omitted.		Minimum: 0

Endpoint

Endpoint describes a single statically configured model serving instance.

Appears in:

• ModelServerSpec

Field	Description	Default	Validation
name string	Name uniquely identifies the endpoint within the ModelServer. Together with the ModelServer name it forms the instance identity in the router, for example in metrics and debug output.		MaxLength: 253 Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ Required: {}
address string	Address is the IP address or DNS name of the model serving instance.		MaxLength: 253 Required: {}
port integer	Port is the port the model serving instance listens on. It defaults to spec.workloadPort.port when unset.		Maximum: 65535 Minimum: 1
labels object (keys:string, values:string)	Labels are attached to the endpoint. They do not select serving instances; they are only matched against workloadSelector.pdGroup to assign the endpoint a prefill or decode role, pdGroup being the sole workloadSelector field that may be combined with endpoints.

ExternalModelProvider

ExternalModelProvider is the Schema for the externalmodelproviders API.

Appears in:

• ExternalModelProviderList

Field	Description	Default	Validation
apiVersion string	networking.serving.volcano.sh/v1alpha1
kind string	ExternalModelProvider
spec ExternalModelProviderSpec
status ExternalModelProviderStatus

ExternalModelProviderList

ExternalModelProviderList contains a list of ExternalModelProvider.

Field	Description	Default	Validation
apiVersion string	networking.serving.volcano.sh/v1alpha1
kind string	ExternalModelProviderList
items ExternalModelProvider array

ExternalModelProviderSpec

ExternalModelProviderSpec defines the desired state of ExternalModelProvider.

Appears in:

• ExternalModelProvider

Field	Description	Default	Validation
providerType ExternalProviderType	ProviderType selects the protocol adapter used for this provider.	OpenAI	Enum: [OpenAI Anthropic]
model string	Model is the actual upstream model name. When set, it overwrites the model in the request, matching ModelServer.Spec.Model behavior.		MaxLength: 256
baseURL string	BaseURL is the provider endpoint root. External providers must use HTTPS. Example: https://api.deepseek.com		MinLength: 1 Pattern: ^https://.+ Required: {}
insecureSkipVerify boolean	InsecureSkipVerify disables server certificate-chain and hostname verification for HTTPS. It does not enable plain HTTP.	false
auth ProviderAuth	Auth references a credential Secret in the same namespace.
headers object (keys:string, values:string)	Non-sensitive static headers added to upstream requests.

ExternalModelProviderStatus

ExternalModelProviderStatus defines the observed state of ExternalModelProvider.

Appears in:

• ExternalModelProvider

Field	Description	Default	Validation
observedGeneration integer

ExternalProviderType

Underlying type: string

ExternalProviderType defines the protocol adapter used for an external provider.

Validation:

• Enum: [OpenAI Anthropic]

Appears in:

• ExternalModelProviderSpec

Field	Description
OpenAI	OpenAI selects the OpenAI-compatible API adapter.
Anthropic	Anthropic selects the Anthropic-compatible Messages API adapter.

GlobalRateLimit

GlobalRateLimit contains configuration for global rate limiting

Appears in:

• RateLimit

Field	Description	Default	Validation
redis RedisConfig	Redis contains configuration for Redis-based global rate limiting.

InferenceEngine

Underlying type: string

InferenceEngine defines the inference framework used by the modelServer to serve LLM requests.

Validation:

• Enum: [vLLM SGLang]

Appears in:

• ModelServerSpec

Field	Description
vLLM	https://github.com/vllm-project/vllm
SGLang	https://github.com/sgl-project/sglang

KVConnectorSpec

KVConnectorSpec defines KV connector configuration for PD disaggregated routing

Appears in:

• ModelServerSpec

Field	Description	Default	Validation
type KVConnectorType	Type specifies the connector type. If you do not know which type to use, please use "http" as default.	http	Enum: [http lmcache nixl mooncake]

KVConnectorType

Underlying type: string

Appears in:

• KVConnectorSpec

Field	Description
http
nixl
lmcache
mooncake

ModelMatch

ModelMatch defines the predicate used to match LLM inference requests to a given TargetModels. Multiple match conditions are ANDed together, i.e. the match will evaluate to true only if all conditions are satisfied.

Appears in:

• ModelBoosterSpec
• Rule

Field	Description	Default	Validation
headers object (keys:string, values:StringMatch)	Header to match: prefix, exact, regex If unset, any header will be matched.
uri StringMatch	URI to match: prefix, exact, regex If this field is not specified, a default prefix match on the "/" path is provided.
body BodyMatch	Body contains conditions to match request body content

ModelRoute

ModelRoute is the Schema for the Modelroutes API.

Appears in:

• ModelRouteList

Field	Description	Default	Validation
apiVersion string	networking.serving.volcano.sh/v1alpha1
kind string	ModelRoute
spec ModelRouteSpec
status ModelRouteStatus

ModelRouteList

ModelRouteList contains a list of ModelRoute.

Field	Description	Default	Validation
apiVersion string	networking.serving.volcano.sh/v1alpha1
kind string	ModelRouteList
items ModelRoute array

ModelRouteSpec

ModelRouteSpec defines the desired state of ModelRoute.

Appears in:

• ModelRoute

Field	Description	Default	Validation
modelName string	model in the LLM request, it could be a base model name, lora adapter name or even a virtual model name. This field is used to match scenarios other than model adapter name and this field could be empty, but it and ModelAdapters can't both be empty.
loraAdapters string array	model in the LLM request could be lora adapter name, here is a list of Lora Adapter Names to match.		MaxItems: 10
parentRefs ParentReference array	ParentRefs references the Gateways that this ModelRoute should be attached to. If empty, the ModelRoute will be attached to all Gateways in the same namespace.
rules Rule array	An ordered list of route rules for LLM traffic. The first rule matching an incoming request will be used. If no rule is matched, an HTTP 404 status code MUST be returned.		MaxItems: 16
rateLimit RateLimit	Rate limit for the LLM request based on prompt tokens or output tokens. There is no limitation if this field is not set.

ModelRouteStatus

ModelRouteStatus defines the observed state of ModelRoute.

Appears in:

• ModelRoute

ModelServer

ModelServer is the Schema for the modelservers API.

Appears in:

• ModelServerList

Field	Description	Default	Validation
apiVersion string	networking.serving.volcano.sh/v1alpha1
kind string	ModelServer
spec ModelServerSpec
status ModelServerStatus

ModelServerList

ModelServerList contains a list of ModelServer.

Field	Description	Default	Validation
apiVersion string	networking.serving.volcano.sh/v1alpha1
kind string	ModelServerList
items ModelServer array

ModelServerSpec

ModelServerSpec defines the desired state of ModelServer.

Appears in:

• ModelServer

Field	Description	Default	Validation
model string	The real model that the modelServers are running. If the model in LLM inference request is different from this field, it should be overwritten by this field. Otherwise, the model in LLM inference request will not be mutated.		MaxLength: 256
inferenceEngine InferenceEngine	The inference engine used to serve the model.		Enum: [vLLM SGLang] Required: {}
workloadSelector WorkloadSelector	WorkloadSelector is used to match the model serving instances. Currently, they must be pods within the same namespace as modelServer object. workloadSelector.matchLabels and endpoints are mutually exclusive ways of declaring the serving instances, so exactly one of them must be used. workloadSelector.pdGroup does not select instances; it only assigns them prefill and decode roles, and therefore is the sole workloadSelector field that may also be combined with endpoints.
endpoints Endpoint array	Endpoints is a static list of model serving instances. It is intended for deployments where the serving instances are not discoverable as pods of the cluster the router runs in, for example when the router reads its configuration from local files instead of the Kubernetes API server. endpoints and workloadSelector.matchLabels are mutually exclusive; exactly one of them must be specified.		MaxItems: 1024
workloadPort WorkloadPort	WorkloadPort defines the port and protocol configuration for the model server. It may be omitted only when every entry in endpoints declares its own port; endpoints without an explicit port fall back to workloadPort.port.
trafficPolicy TrafficPolicy	Traffic Policy for accessing the model server instance.
kvConnector KVConnectorSpec	KVConnector specifies the KV connector configuration for PD disaggregated routing

ModelServerStatus

ModelServerStatus defines the observed state of ModelServer.

Appears in:

• ModelServer

PDGroup

PDGroup is used to specify the group key of PD instances. Also, the labels to match the model serving instances for prefill and decode.

Appears in:

• WorkloadSelector

Field	Description	Default	Validation
groupKey string	GroupKey is the key to distinguish different PD groups. Only PD instances with the same group key and value could be paired.
prefillLabels object (keys:string, values:string)	The labels to match the model serving instances for prefill.
decodeLabels object (keys:string, values:string)	The labels to match the model serving instances for decode.

ProviderAuth

ProviderAuth defines how a provider credential is loaded and sent upstream.

Appears in:

• ExternalModelProviderSpec

Field	Description	Default	Validation
scheme ProviderAuthScheme	Scheme overrides the protocol adapter's default authentication scheme. OpenAI defaults to Bearer and Anthropic defaults to APIKey.		Enum: [Bearer APIKey]
secretRef SecretKeySelector	SecretRef references a credential Secret in the same namespace.		Required: {}

ProviderAuthScheme

Underlying type: string

ProviderAuthScheme defines how a provider credential is sent upstream.

Validation:

• Enum: [Bearer APIKey]

Appears in:

• ProviderAuth

Field	Description
Bearer	ProviderAuthSchemeBearer sends the credential in the Authorization header.
APIKey	ProviderAuthSchemeAPIKey sends the credential in the x-api-key header.

RateLimit

Appears in:

• ModelRouteSpec

Field	Description	Default	Validation
inputTokensPerUnit integer	InputTokensPerUnit is the maximum number of input tokens allowed per unit of time. If this field is not set, there is no limit on input tokens.		Minimum: 1
outputTokensPerUnit integer	OutputTokensPerUnit is the maximum number of output tokens allowed per unit of time. If this field is not set, there is no limit on output tokens.		Minimum: 1
unit RateLimitUnit	Unit is the time unit for the rate limit.	second	Enum: [second minute hour day month]
global GlobalRateLimit	Global contains configuration for global rate limiting using distributed storage. If this field is set, global rate limiting will be used; otherwise, local rate limiting will be used.

RateLimitUnit

Underlying type: string

Validation:

• Enum: [second minute hour day month]

Appears in:

• RateLimit

Field	Description
second
minute
hour
day
month

RedisConfig

RedisConfig contains Redis connection configuration

Appears in:

• GlobalRateLimit

Field	Description	Default	Validation
address string	Address is the Redis server address in the format "host:port".		Required: {}

Retry

Appears in:

• TrafficPolicy

Field	Description	Default	Validation
attempts integer	The maximum number of times an individual inference request to a model server should be retried. If the maximum number of retries has been done without a successgful response, the request will be considered failed.

Rule

Appears in:

• ModelRouteSpec

Field	Description	Default	Validation
name string	Name is the name of the rule.
modelMatch ModelMatch	Match conditions to be satisfied for the rule to be activated. Empty modelMatch means matching all requests.
targetModels TargetModel array			MaxItems: 16 MinItems: 1

SessionKeySource

SessionKeySource defines one session key extraction rule.

Appears in:

• SessionSticky

Field	Description	Default	Validation
type SessionKeySourceType			Enum: [Header Query Cookie JWTClaim] Required: {}
name string	Name is the header name, query key, cookie name, or JWT claim name.		MinLength: 1 Required: {}

SessionKeySourceType

Underlying type: string

SessionKeySourceType identifies how a session key fragment is read.

Validation:

• Enum: [Header Query Cookie JWTClaim]

Appears in:

• SessionKeySource

Field	Description
Header
Query
Cookie
JWTClaim

SessionSticky

SessionSticky configures per-ModelServer session key extraction and binding TTL. The backing store (memory vs Redis) is configured in the router process, not here.

Appears in:

• TrafficPolicy

Field	Description	Default	Validation
sessionAffinitySeconds integer	SessionAffinitySeconds is binding TTL in seconds. Once the session has been idle for more than the specified duration, the session becomes invalid. When unset, the default is 300 (5 minutes).		Minimum: 1
sources SessionKeySource array	Sources are evaluated in order; the first non-empty extracted value is the session key.		MaxItems: 16

StringMatch

StringMatch defines the matching conditions for string fields. Only one of the fields may be set.

Appears in:

• ModelMatch

Field	Description	Default	Validation
exact string
prefix string
regex string

TargetModel

LLM inference traffic target model.

Appears in:

• Rule

Field	Description	Default	Validation
modelServerName string	ModelServerName is used to specify the correlated modelServer within the same namespace. It is mutually exclusive with ExternalModelProviderName.
externalModelProviderName string	ExternalModelProviderName is used to specify the correlated ExternalModelProvider within the same namespace. It is mutually exclusive with ModelServerName.
weight integer	Weight is used to specify the percentage of traffic should be sent to the target model. The value should be in the range of [0, 100].	100	Maximum: 100 Minimum: 0

TrafficPolicy

Appears in:

• ModelServerSpec

Field	Description	Default	Validation
retry Retry	The retry policy for the inference request.
connectionPool ConnectionPool	ConnectionPool configures the upstream HTTP connection pool used when forwarding to this ModelServer's pods. When omitted, a shared default pool is used. Each ModelServer that sets this gets its own isolated pool.
sessionSticky SessionSticky	SessionSticky pins requests with the same extracted session key to the same backend Pod of this ModelServer for a TTL. Nil or omitted disables session affinity for this ModelServer. It does not override ModelRoute weighted selection among ModelServers.

WorkloadPort

WorkloadPort defines the port and protocol configuration for the model server.

Appears in:

• ModelServerSpec

Field	Description	Default	Validation
port integer	The port of the model server. The number must be between 1 and 65535.		Maximum: 65535 Minimum: 1 Required: {}
protocol string	The protocol of the model server. Supported values are "http" and "https".	http	Enum: [http https]

WorkloadSelector

WorkloadSelector is used to match the model serving instances. Currently, they must be pods within the same namespace as modelServer object.

Appears in:

• ModelServerSpec

Field	Description	Default	Validation
matchLabels object (keys:string, values:string)	The base labels to match the model serving instances. All serving instances must match these labels.
pdGroup PDGroup	PDGroup is used to further match different roles of the model serving instances, mainly used in case like PD disaggregation.