API Reference
Packages
workload.serving.volcano.sh/v1alpha1
Resource Types
AutoscalingPolicy
AutoscalingPolicy defines the autoscaling policy configuration for model serving workloads. It specifies scaling rules, metrics, and behavior for automatic replica adjustment.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
apiVersion string | workload.serving.volcano.sh/v1alpha1 | ||
kind string | AutoscalingPolicy | ||
spec AutoscalingPolicySpec | |||
status AutoscalingPolicyStatus |
AutoscalingPolicyBehavior
AutoscalingPolicyBehavior defines the scaling behavior configuration for both scale up and scale down operations.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
scaleUp AutoscalingPolicyScaleUpPolicy | ScaleUp defines the policy configuration for scaling up (increasing replicas). | ||
scaleDown AutoscalingPolicyStablePolicy | ScaleDown defines the policy configuration for scaling down (decreasing replicas). |
AutoscalingPolicyList
AutoscalingPolicyList contains a list of AutoscalingPolicy objects.
| Field | Description | Default | Validation |
|---|---|---|---|
apiVersion string | workload.serving.volcano.sh/v1alpha1 | ||
kind string | AutoscalingPolicyList | ||
items AutoscalingPolicy array |
AutoscalingPolicyMetric
AutoscalingPolicyMetric defines a metric and its target value for scaling decisions.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
name string | Name defines the metric key used by the scaling algorithm. | MinLength: 1 | |
targetValue Quantity | TargetValue defines the target value for the metric that triggers scaling operations. |
AutoscalingPolicyPanicPolicy
AutoscalingPolicyPanicPolicy defines the emergency scaling policy for handling sudden traffic surges.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
percent integer | Percent defines the maximum percentage of current instances to scale up during panic mode. | 1000 | Maximum: 1000 Minimum: 0 |
panicThresholdPercent integer | PanicThresholdPercent defines the metric threshold percentage that triggers panic mode. When metrics exceed this percentage of target values, panic mode is activated. | 200 | Maximum: 1000 Minimum: 110 |
AutoscalingPolicyScaleUpPolicy
AutoscalingPolicyScaleUpPolicy defines the scaling up policy configuration.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
stablePolicy AutoscalingPolicyStablePolicy | StablePolicy defines the stable scaling policy that uses average metric values over time windows. This policy smooths out short-term fluctuations and avoids unnecessary frequent scaling operations. | ||
panicPolicy AutoscalingPolicyPanicPolicy | PanicPolicy defines the emergency scaling policy for handling sudden traffic spikes. This policy activates during rapid load surges to prevent service degradation or timeouts. |
AutoscalingPolicySpec
AutoscalingPolicySpec defines the desired state of AutoscalingPolicy.
At most one of HomogeneousTarget, HeterogeneousTarget, or DisaggregatedTarget may be set. When the spec is used standalone (as an AutoscalingPolicy custom resource), exactly one target must be set; this is enforced by the autoscalingpolicy validating webhook rather than a CEL rule.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
tolerancePercent integer | TolerancePercent defines the percentage of deviation tolerated before scaling actions are triggered. current_replicas represents the current number of instances, while target_replicas represents the expected number of instances calculated from monitoring metrics. Scaling operations are performed only when |current_replicas - target_replicas| >= current_replicas * TolerancePercent / 100. | 10 | Maximum: 100 Minimum: 0 |
metrics AutoscalingPolicyMetric array | Metrics defines the list of metrics used to evaluate scaling decisions. This is the default metric list applied to scalable units. For DisaggregatedTarget, role-level metrics override this list for that role. | ||
behavior AutoscalingPolicyBehavior | Behavior defines the scaling behavior configuration for both scale up and scale down operations. | ||
homogeneousTarget HomogeneousTarget | HomogeneousTarget enables traditional metric-based scaling for a single ModelServing deployment (whole-deployment granularity). | ||
heterogeneousTarget HeterogeneousTarget | HeterogeneousTarget enables optimization-based scaling across multiple ModelServing deployments with different hardware capabilities. | ||
disaggregatedTarget DisaggregatedTarget | DisaggregatedTarget enables coordinated autoscaling of roles within a single ModelServing that uses disaggregated serving. |
AutoscalingPolicyStablePolicy
AutoscalingPolicyStablePolicy defines the stable scaling policy for both scale up and scale down operations.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
instances integer | Instances defines the maximum absolute number of instances to scale per period. | 1 | Minimum: 0 |
percent integer | Percent defines the maximum percentage of current instances to scale per period. | 100 | Maximum: 1000 Minimum: 0 |
selectPolicy SelectPolicyType | SelectPolicy determines the selection strategy for scaling operations. 'Or' means scaling is performed if either the Percent or Instances requirement is met. 'And' means scaling is performed only if both Percent and Instances requirements are met. | Or | Enum: [Or And] |
AutoscalingPolicyStatus
AutoscalingPolicyStatus defines the observed state of AutoscalingPolicy.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
observedGeneration integer | ObservedGeneration is the most recent generation observed by the controller. | ||
homogeneousStatus TargetScalingStatus | HomogeneousStatus reports the observed state when HomogeneousTarget is used. | ||
disaggregatedStatus DisaggregatedScalingStatus | DisaggregatedStatus reports the observed state when DisaggregatedTarget is used. | ||
heterogeneousStatus TargetScalingStatus array | HeterogeneousStatus reports the per-target observed state when HeterogeneousTarget is used. |
DisaggregatedScalingStatus
DisaggregatedScalingStatus reports the observed state of a DisaggregatedTarget.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
roles TargetScalingStatus array | Roles reports the observed scaling state per role. | ||
ratioStatus RoleRatioStatus | RatioStatus reports the observed value of the configured ratio constraint. | ||
ratioAdjusted boolean | RatioAdjusted is true when the most recent reconcile had to override the metric-derived replica counts to satisfy the ratio constraint. |
DisaggregatedTarget
DisaggregatedTarget defines coordinated autoscaling for disaggregated serving roles within a single ModelServing deployment.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
targetRef ObjectReference | TargetRef references the ModelServing deployment that contains all scalable roles. | ||
roles object (keys:string, values:RoleScalingParam) | Roles defines per-role scaling parameters. The map key is roleName from ModelServing.spec.template.roles[].name. A single role is allowed so users can autoscale one role independently without configuring a P/D pair. RatioConstraint, when set, still requires two distinct roles. | MaxProperties: 2 MinProperties: 1 | |
ratioConstraint RoleRatioConstraint | RatioConstraint defines the acceptable ratio range of a single role pair. It enforces that replicas[numeratorRole] / replicas[denominatorRole] stays within [minRatio, maxRatio] when denominator replica is non-zero. |
EvictionStrategySpec
EvictionStrategySpec defines the protection policy during node eviction.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
protectionLevel ProtectionLevelType | ProtectionLevel defines the protection level: ServingGroup or Role. - ServingGroup: guarantees that the number of ready ServingGroups is not below the threshold. - Role: guarantees that the number of ready instances for each role is not below the threshold. | ServingGroup | Enum: [ServingGroup Role] |
minAvailable IntOrString | MinAvailable defines the minimum number of available ServingGroup instances. It is used only when protectionLevel is ServingGroup. It can be an absolute number (ex: 3) or a percentage of total instances (ex: 80%). | ||
roleMinAvailable object (keys:string, values:IntOrString) | RoleMinAvailable defines role-specific minimum available role instances. It is used only when protectionLevel is Role. Map keys must match names in spec.template.roles. If a role is absent from this map, it is not protected by the eviction budget. Values can be absolute numbers (ex: 3) or percentages of total role instances (ex: 80%). |
GangPolicy
GangPolicy defines the gang scheduling configuration.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
minRoleReplicas object (keys:string, values:integer) | MinRoleReplicas defines the minimum number of replicas required for each role in gang scheduling, pods in each role are strictly gang required. This map allows users to specify different minimum replica requirements for different roles. If this field is not set, all roles in the ServingGroup are considered gang required by default. For example if you specify a 2P(prefill) 4D(decode) serving group and set the below gangPolicy: yaml<br />gangPolicy:<br /> minRoleReplicas:<br /> prefill: 1<br /> decode: 1<br />It will result in the following behavior: At least one prefill and one decode must be scheduled before any of the pods in the serving group can run. And pods within a role must be scheduled together. |
HeterogeneousTarget
HeterogeneousTarget defines the configuration for optimization-based autoscaling across multiple deployments.
It distributes replicas across several ModelServing groups with different hardware (and therefore different Cost) to satisfy the overall demand at the lowest cost. Each group is described by one entry in Params.
Example (split capacity between an H100 group and a cheaper A100 group):
heterogeneousTarget: costExpansionRatePercent: 200 params:
- cost: 100 minReplicas: 0 maxReplicas: 4 target: targetRef: kind: ModelServing name: llama-h100
- cost: 60 minReplicas: 1 maxReplicas: 8 target: targetRef: kind: ModelServing name: llama-a100
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
params HeterogeneousTargetParam array | Params defines the configuration parameters for multiple ModelServing groups to be optimized. | MinItems: 1 | |
costExpansionRatePercent integer | CostExpansionRatePercent defines the percentage rate at which the cost expands during optimization calculations. For example, 200 allows the optimizer to spend up to 2x the minimal cost to meet performance targets before refusing to scale further. | 200 | Minimum: 0 |
HeterogeneousTargetParam
HeterogeneousTargetParam defines the configuration parameters for a specific deployment type in heterogeneous scaling.
Example (one expensive H100 group within a HeterogeneousTarget):
cost: 100 minReplicas: 0 maxReplicas: 4 target: targetRef: kind: ModelServing name: llama-h100
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
target Target | Target defines the scaling instance configuration for this deployment type. | ||
cost integer | Cost defines the relative cost factor used in optimization calculations. This factor balances performance requirements against deployment costs. Values are relative across params, e.g. 100 for an H100 group and 60 for a cheaper A100 group makes the optimizer prefer A100 replicas when adequate. | Minimum: 0 | |
minReplicas integer | MinReplicas defines the minimum number of replicas to maintain for this deployment type. | Maximum: 1e+06 Minimum: 0 | |
maxReplicas integer | MaxReplicas defines the maximum number of replicas allowed for this deployment type. | Maximum: 1e+06 Minimum: 1 |
HomogeneousTarget
HomogeneousTarget defines the configuration for traditional metric-based autoscaling of a single deployment.
Example (scale podinfo-ms between 1 and 6 replicas based on RPS):
homogeneousTarget: minReplicas: 1 maxReplicas: 6 target: targetRef: kind: ModelServing name: podinfo-ms metricSources: podinfo_rps: prometheus: serverURL: http://prometheus.monitoring.svc:9090 query: sum(rate(http_requests_total[2m]))
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
target Target | Target defines the object to be monitored and scaled. | ||
minReplicas integer | MinReplicas defines the minimum number of replicas to maintain (e.g., 1). | Maximum: 1e+06 Minimum: 0 | |
maxReplicas integer | MaxReplicas defines the maximum number of replicas allowed (e.g., 6). | Maximum: 1e+06 Minimum: 1 |
Metadata
Metadata is a simplified version of ObjectMeta in Kubernetes.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
labels object (keys:string, values:string) | Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels | ||
annotations object (keys:string, values:string) | Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations |
MetricSource
MetricSource is a discriminated union selecting the metric backend.
Exactly one backend config must be provided:
- Pod -> set the pod field only.
- Prometheus -> set the prometheus field only.
Example (scrape the metric directly from each pod's /metrics endpoint):
metricSources: gpu_cache_usage: pod: name: vllm:kv_cache_usage_perc uri: /metrics port: 8000
Example (read the metric from an external Prometheus server):
metricSources: http_rps: prometheus: serverURL: http://prometheus.monitoring.svc:9090 query: sum(rate(http_requests_total[2m]))
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
pod PodMetricSource | Pod configures direct pod endpoint scraping. | ||
prometheus PrometheusMetricSource | Prometheus configures an external Prometheus server as the metric source. |
ModelBackend
ModelBackend defines the configuration for a model backend.
Deprecated: ModelBooster is deprecated as of Kthena v1.1. Use ModelServing, ModelServer, and ModelRoute instead. ModelBooster will be removed no earlier than Kthena v1.5.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
name string | Name is the name of the backend. Can't duplicate with other ModelBackend name in the same ModelBooster CR. Note: update name will cause the old ModelServing deletion and a new ModelServing creation. | Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ | |
type ModelBackendType | Type is the type of the backend. | Enum: [vLLM vLLMDisaggregated] | |
modelURI string | ModelURI is the source from which the model is fetched by the downloader init container. Supported schemes: hf://NAMESPACE/REPO — Hugging Face Hub repository ms://NAMESPACE/REPO — ModelScope repository s3://BUCKET/PATH — S3-compatible object storage obs://BUCKET/PATH — Huawei Object Storage Service (OBS) pvc:///CLAIM_NAME/PATH — path inside a PVC already mounted via CacheURI When using pvc://, the downloader reads the given path from the container filesystem. The downloader init container only mounts the volume specified by CacheURI, so the modelURI path must be reachable through that mount. Both CacheURI and modelURI must reference the same PVC, and the modelURI path must start with the CacheURI mount point. Example: CacheURI: pvc://model-storage, ModelURI: pvc:///model-storage/models/Qwen | Pattern: ^(hf://|s3://|obs://|pvc://|ms://).+ | |
cacheURI string | CacheURI specifies where the downloaded model is stored and how the storage volume is mounted inside every pod (both the downloader init container and the inference engine). Supported schemes: pvc://CLAIM_NAME — PersistentVolumeClaim; the PVC is mounted at /CLAIM_NAME hostpath://PATH — host-local directory; mounted at /PATH (empty) — an ephemeral EmptyDir volume is used (no persistence) The downloader writes model files under a hashed sub-directory of this mount path. The inference engine reads from the same path. When ModelURI uses pvc://, CacheURI must also use pvc:// and reference the same PVC so the source path is visible. | Pattern: ^(hostpath://|pvc://).+ | |
envFrom EnvFromSource array | List of sources to populate environment variables in the container. The keys defined within a source must be a C_IDENTIFIER. All invalid keys will be reported as an event when the container is starting. When a key exists in multiple sources, the value associated with the last source will take precedence. Values defined by an Env with a duplicate key will take precedence. Cannot be updated. | ||
env EnvVar array | List of environment variables to set in the container. Supported names: "ENDPOINT": When you download model from s3, you have to specify it. "RUNTIME_URL": default is http://localhost:8000 "RUNTIME_PORT": default is 8100 "RUNTIME_METRICS_PATH": default is /metrics "HF_ENDPOINT":The url of hugging face. Default is https://huggingface.co/ "KTHENA_SKIP_ENGINE_DEPENDENCY_INSTALL": default is false. When set to true, skip startup-time pip install of engine connector dependencies. Cannot be updated. | ||
replicas integer | Replicas is the fixed number of replicas for the backend. | Maximum: 1e+06 Minimum: 0 | |
workers ModelWorker array | Workers is the list of workers associated with this backend. | MaxItems: 1000 MinItems: 1 | |
schedulerName string | SchedulerName defines the name of the scheduler used by ModelServing for this backend. | ||
runtimeClassName string | RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used to run pods generated for this backend. |
ModelBackendType
Underlying type: string
ModelBackendType defines the type of model backend.
Deprecated: ModelBooster is deprecated as of Kthena v1.1. Use ModelServing, ModelServer, and ModelRoute instead. ModelBooster will be removed no earlier than Kthena v1.5.
Validation:
- Enum: [vLLM vLLMDisaggregated]
Appears in:
| Field | Description |
|---|---|
vLLM | ModelBackendTypeVLLM represents a vLLM backend. |
vLLMDisaggregated | ModelBackendTypeVLLMDisaggregated represents a disaggregated vLLM backend. |
SGLang | ModelBackendTypeSGLang represents an SGLang backend. |
MindIE | ModelBackendTypeMindIE represents a MindIE backend. |
MindIEDisaggregated | ModelBackendTypeMindIEDisaggregated represents a disaggregated MindIE backend. |
ModelBooster
ModelBooster is the Schema for the models API.
Deprecated: ModelBooster is deprecated as of Kthena v1.1. Use ModelServing, ModelServer, and ModelRoute instead. ModelBooster will be removed no earlier than Kthena v1.5.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
apiVersion string | workload.serving.volcano.sh/v1alpha1 | ||
kind string | ModelBooster | ||
spec ModelBoosterSpec | |||
status ModelStatus |
ModelBoosterList
ModelBoosterList contains a list of ModelBooster.
Deprecated: ModelBooster is deprecated as of Kthena v1.1. Use ModelServing, ModelServer, and ModelRoute instead. ModelBooster will be removed no earlier than Kthena v1.5.
| Field | Description | Default | Validation |
|---|---|---|---|
apiVersion string | workload.serving.volcano.sh/v1alpha1 | ||
kind string | ModelBoosterList | ||
items ModelBooster array |
ModelBoosterSpec
ModelBoosterSpec defines the desired state of ModelBooster.
Deprecated: ModelBooster is deprecated as of Kthena v1.1. Use ModelServing, ModelServer, and ModelRoute instead. ModelBooster will be removed no earlier than Kthena v1.5.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
name string | Name is the name of the model. ModelBooster CR name is restricted by kubernetes, for example, can't contain uppercase letters. So we use this field to specify the ModelBooster name. | MaxLength: 64 Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ | |
owner string | Owner is the owner of the model. | ||
backend ModelBackend | Backend is the model backend associated with this model. ModelBackend is the minimum unit of inference instance. It can be vLLM or vLLMDisaggregated. | ||
modelMatch ModelMatch | ModelMatch defines the predicate used to match LLM inference requests to a given TargetModels. Multiple match conditions are ANDed together, i.e. the match will evaluate to true only if all conditions are satisfied. |
ModelServing
ModelServing is the Schema for the LLM Serving API
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
apiVersion string | workload.serving.volcano.sh/v1alpha1 | ||
kind string | ModelServing | ||
spec ModelServingSpec | |||
status ModelServingStatus |
ModelServingList
ModelServingList contains a list of ModelServing
| Field | Description | Default | Validation |
|---|---|---|---|
apiVersion string | workload.serving.volcano.sh/v1alpha1 | ||
kind string | ModelServingList | ||
items ModelServing array |
ModelServingSpec
ModelServingSpec defines the specification of the ModelServing resource.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
replicas integer | Number of ServingGroups. That is the number of instances that run serving tasks Default to 1. | 1 | |
schedulerName string | SchedulerName defines the name of the scheduler used by ModelServing | volcano | |
plugins PluginSpec array | Plugins defines optional plugins that customize serving pods or manage auxiliary resources through controller lifecycle hooks. | ||
template ServingGroup | Template defines the template for ServingGroup | ||
rolloutStrategy RolloutStrategy | RolloutStrategy defines the strategy that will be applied to update replicas | ||
recoveryPolicy RecoveryPolicy | RecoveryPolicy defines the recovery policy for the failed Pod to be rebuilt | RoleRecreate | Enum: [ServingGroupRecreate RoleRecreate None] |
revisionHistoryLimit integer | RevisionHistoryLimit is the maximum number of non-live revisions to retain. Revisions still referenced by the ModelServing or its workloads do not count toward this limit. | 10 | Minimum: 0 |
ModelServingStatus
ModelServingStatus defines the observed state of ModelServing
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
observedGeneration integer | observedGeneration is the most recent generation observed for ModelServing. It corresponds to the ModelServing's generation, which is updated on mutation by the API Server. | ||
replicas integer | Replicas track the total number of ServingGroup that have been created (updated or not, ready or not) | ||
currentReplicas integer | CurrentReplicas is the number of ServingGroup created by the ModelServing controller from the ModelServing version | ||
updatedReplicas integer | UpdatedReplicas track the number of ServingGroup that have been updated (ready or not). | ||
availableReplicas integer | AvailableReplicas track the number of ServingGroup that are in ready state (updated or not). | ||
currentRevision string | CurrentRevision, if not empty, indicates the ControllerRevision version preserved by ServingGroups that have not been updated. | ||
updateRevision string | UpdateRevision, if not empty, indicates the ControllerRevision version targeted by the current ModelServing spec. | ||
collisionCount integer | CollisionCount tracks hash collisions for ControllerRevision names. | ||
labelSelector string | LabelSelector is a label query over pods that should match the replica count. |
ModelStatus
ModelStatus defines the observed state of ModelBooster.
Deprecated: ModelBooster is deprecated as of Kthena v1.1. Use ModelServing, ModelServer, and ModelRoute instead. ModelBooster will be removed no earlier than Kthena v1.5.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
observedGeneration integer | ObservedGeneration track of generation |
ModelWorker
ModelWorker defines the model worker configuration.
Deprecated: ModelBooster is deprecated as of Kthena v1.1. Use ModelServing, ModelServer, and ModelRoute instead. ModelBooster will be removed no earlier than Kthena v1.5.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
type ModelWorkerType | Type is the type of the model worker. | server | Enum: [server prefill decode controller coordinator] |
image string | Image is the container image for the worker. | ||
replicas integer | Replicas is the number of replicas for the worker. | Maximum: 1e+06 Minimum: 0 | |
pods integer | Pods is the number of pods for the worker. | Maximum: 1e+06 Minimum: 0 | |
resources ResourceRequirements | Resources specifies the resource requirements for the worker. | ||
affinity Affinity | Affinity specifies the affinity rules for scheduling the worker pods. | ||
tolerations Toleration array | Tolerations specifies the tolerations for scheduling the worker pods. | ||
config JSON | Config contains worker-specific configuration in JSON format. You can find vLLM config here https://docs.vllm.ai/en/stable/configuration/engine_args.html |
ModelWorkerType
Underlying type: string
ModelWorkerType defines the type of model worker.
Deprecated: ModelBooster is deprecated as of Kthena v1.1. Use ModelServing, ModelServer, and ModelRoute instead. ModelBooster will be removed no earlier than Kthena v1.5.
Validation:
- Enum: [server prefill decode controller coordinator]
Appears in:
| Field | Description |
|---|---|
server | ModelWorkerTypeServer represents a server worker. |
prefill | ModelWorkerTypePrefill represents a prefill worker. |
decode | ModelWorkerTypeDecode represents a decode worker. |
controller | ModelWorkerTypeController represents a controller worker. |
coordinator | ModelWorkerTypeCoordinator represents a coordinator worker. |
NetworkTopology
NetworkTopologySpec defines the network topology affinity scheduling policy for the roles and group, it works only when the scheduler supports network topology feature.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
groupPolicy NetworkTopologySpec | GroupPolicy defines the network topology scheduling requirement of all the instances within the ServingGroup. | ||
rolePolicy NetworkTopologySpec | RolePolicy defines the fine-grained network topology scheduling requirement for instances of a role. |
PluginScope
PluginScope restricts where a plugin is applied. Roles is a whitelist; empty means all roles. Target limits to entry/worker/all pods; empty means all pods.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
roles string array | Roles limits the plugin to the specified role names. | ||
target PluginTarget | Target limits the plugin to specific pod target (Entry/Worker/All). kubebuilder:default=All kubebuilder:validation:Enum={All,Entry,Worker} |
PluginSpec
PluginSpec declares a plugin instance attached to a ModelServing.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
name string | Name uniquely identifies the plugin instance within the ModelServing. | ||
type PluginType | Type indicates plugin category. For now, only BuiltIn is supported. | BuiltIn | Enum: [BuiltIn] |
config JSON | Config is an opaque JSON blob interpreted by the plugin implementation. | ||
scope PluginScope | Scope optionally narrows where this plugin runs. By default, it runs on all pods. |
PluginTarget
Underlying type: string
PluginTarget specifies which pod kinds a plugin applies to. If empty, it defaults to All.
Appears in:
| Field | Description |
|---|---|
All | |
Entry | |
Worker |
PluginType
Underlying type: string
PluginType represents the implementation category of a plugin.
Appears in:
| Field | Description |
|---|---|
BuiltIn |
PodMetricSource
PodMetricSource configures pod-endpoint scraping for a metric.
For each matching Pod, metrics are scraped from the constructed access link and extracted from Prometheus’s text output for the metric family identified by Name.
Example (the pod exposes "vllm:num_requests_waiting" on :8000/metrics):
pod: name: vllm:num_requests_waiting uri: /metrics port: 8000 labelSelector: matchLabels: role: decode
The resulting scrape URL would look like: http://10.1.2.3:8000/metrics
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
name string | Name is the Prometheus metric name matched against labels in the pod's scraped output. Defaults to the policy metric key when omitted. For example, set it to "vllm:kv_cache_usage_perc" to read that exact series. | ||
uri string | Uri defines the HTTP path where metrics are exposed (e.g., "/metrics"). | /metrics | Pattern: ^/ |
port integer | Port defines the network port where metrics are exposed by the pods (e.g., 8000). | 8100 | Maximum: 65535 Minimum: 1 |
PodTemplateSpec
PodTemplateSpec describes the data a pod should have when created from a template
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
metadata Metadata | Refer to Kubernetes API documentation for fields of metadata. | ||
spec PodSpec | Specification of the desired behavior of the pod. |
PrometheusAuth
PrometheusAuth configures authentication when connecting to an external Prometheus server.
NOTE: This struct describes the intended configuration surface. The runtime does not honor any of these fields yet; they are reserved for a follow-up implementation. Setting them today has no effect on Prometheus requests.
Appears in:
PrometheusMetricSource
PrometheusMetricSource configures an external Prometheus server as a metric backend.
The Query is executed as an instant query and must return a single scalar or a single-sample vector; the resulting value drives the scaling decision.
Example:
prometheus: serverURL: http://kube-prometheus-stack-prometheus.monitoring.svc:9090 query: sum(rate(http_requests_total[2m]))
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
serverURL string | ServerURL is the base URL of the Prometheus HTTP API server. Example: "http://prometheus.monitoring.svc:9090". | Format: uri MinLength: 1 | |
query string | Query is a PromQL instant-query expression. It must evaluate to a single scalar or a one-element vector, e.g. "avg(rate(vllm 1m ))".More Query details refer to https://prometheus.io/docs/prometheus/latest/querying/basics | MinLength: 1 | |
auth PrometheusAuth | Auth holds optional authentication configuration for the Prometheus server. |
ProtectionLevelType
Underlying type: string
ProtectionLevelType defines the level of protection during eviction.
Appears in:
| Field | Description |
|---|---|
ServingGroup | ProtectionLevelServingGroup guarantees that the number of ready ServingGroups is not below the threshold. |
Role | ProtectionLevelRole guarantees that the number of ready instances for each role is not below the threshold. |
RecoveryPolicy
Underlying type: string
Appears in:
| Field | Description |
|---|---|
ServingGroupRecreate | ServingGroupRecreate will recreate all the pods in the ServingGroup if 1. Any individual pod in the group is recreated; 2. Any containers/init-containers in a pod is restarted. This is to ensure all pods/containers in the group will be started in the same time. |
RoleRecreate | RoleRecreate will recreate all pods in one Role if 1. Any individual pod in the group is recreated; 2. Any containers/init-containers in a pod is restarted. |
None | NoneRestartPolicy will follow the same behavior as the default pod or deployment. |
Role
Role defines the specific pod instance role that performs the inference task.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
name string | The name of a role. Name must be unique within an ServingGroup | MaxLength: 12 Pattern: ^[a-zA-Z0-9]([-a-zA-Z0-9]*[a-zA-Z0-9])?$ | |
replicas integer | The number of a certain role. For example, in Disaggregated Prefilling, setting the replica count for both the P and D roles to 1 results in 1P1D deployment configuration. This approach can similarly be applied to configure a xPyD deployment scenario. Default to 1. | 1 | |
entryTemplate PodTemplateSpec | EntryTemplate defines the template for the entry pod of a role. Required: Currently, a role must have only one entry-pod. | ||
workerReplicas integer | WorkerReplicas defines the number for the worker pod of a role. Required: Need to set the number of worker-pod replicas. | ||
workerTemplate PodTemplateSpec | WorkerTemplate defines the template for the worker pod of a role. | ||
maxUnavailable IntOrString | MaxUnavailable is the maximum number of resources that may be unavailable during an update. It can be an absolute number (for example, 5) or a percentage (for example, 10%). A percentage is calculated from ModelServing replicas for ServingGroupRollingUpdate and from the corresponding Role's replicas for RoleRollingUpdate, then rounded down. It may resolve to 0 only when MaxSurge resolves above 0. Defaults to 1. | 1 | XIntOrString: {} |
maxSurge IntOrString | MaxSurge is the maximum number of resources that may be created above the desired replica count during an update. It can be an absolute number (for example, 1) or a percentage (for example, 25%). A percentage is calculated from ModelServing replicas for ServingGroupRollingUpdate and from the corresponding Role's replicas for RoleRollingUpdate, then rounded up. It defaults to 0. | XIntOrString: {} | |
partition IntOrString | Partition protects the first N existing replicas in ascending ordinal order from updates. The remaining replicas are eligible for rolling update. For a contiguous ordinal set, this is equivalent to protecting [0, Partition). Value can be an absolute number (ex: 5) or a percentage of total replicas (ex: 10%). Absolute number is calculated from percentage by rounding up. The default value is 0. | XIntOrString: {} |
RoleCoordination
RoleCoordination enables proportional rolling updates across selected Roles in each ServingGroup. Its presence and policy are immutable after creation.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
roles string array | Roles selects the Roles participating in coordinated rolling update. An empty list selects every Role in spec.template.roles. | ||
maxSkew IntOrString | MaxSkew is the maximum percentage-point difference allowed between the target-version Ready ratios of participating Roles, using their entire formal replica count, including partition-protected capacity. Only percentage values, for example "10%", are accepted. | XIntOrString: {} | |
dependencies RoleRolloutDependency array | Dependencies defines rollout startup dependencies. A graph root waits for target-version Ready capacity throughout its dependency closure before its first target start. Internal Roles may start together; old dependency capacity is retained while old callers remain. |
RoleRatioConstraint
RoleRatioConstraint defines the acceptable ratio range between two roles.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
numeratorRole string | NumeratorRole is the role on the numerator side of the ratio. | ||
denominatorRole string | DenominatorRole is the role on the denominator side of the ratio. | ||
minRatio Quantity | MinRatio is the minimum allowed value of replicas[numeratorRole] / replicas[denominatorRole]. | ||
maxRatio Quantity | MaxRatio is the maximum allowed value of replicas[numeratorRole] / replicas[denominatorRole]. |
RoleRatioStatus
RoleRatioStatus reports the observed value for the ratio constraint.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
numeratorRole string | |||
denominatorRole string | |||
currentRatio string |
RoleRolloutDependency
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
role string | Role is the dependent Role. | ||
dependsOn string array | DependsOn lists the downstream Roles required by Role. |
RoleScalingParam
RoleScalingParam defines the scaling configuration for one role.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
minReplicas integer | MinReplicas defines the minimum number of replicas for this role. | Maximum: 1e+06 Minimum: 0 | |
maxReplicas integer | MaxReplicas defines the maximum number of replicas for this role. | Maximum: 1e+06 Minimum: 1 | |
metrics AutoscalingPolicyMetric array | Metrics defines the list of metrics used to evaluate scaling decisions for this role, allowing different roles to scale on different signals. When set, these metrics override spec.metrics for this role. When omitted, the role inherits spec.metrics. A fixed role (minReplicas == maxReplicas) may omit metrics; the autoscaler keeps it at that fixed size and does not collect metrics for it. | MinItems: 1 | |
metricSources object (keys:string, values:MetricSource) | MetricSources declares how each metric is fetched for this role. Keys must match role-level metrics when present, otherwise top-level spec.metrics[].name. Missing keys are treated as missing metrics for that reconcile loop. |
RollingUpdateConfiguration
RollingUpdateConfiguration defines availability and partition settings for the rollout granularity where it is configured.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
maxUnavailable IntOrString | MaxUnavailable is the maximum number of resources that may be unavailable during an update. It can be an absolute number (for example, 5) or a percentage (for example, 10%). A percentage is calculated from ModelServing replicas for ServingGroupRollingUpdate and from the corresponding Role's replicas for RoleRollingUpdate, then rounded down. It may resolve to 0 only when MaxSurge resolves above 0. Defaults to 1. | 1 | XIntOrString: {} |
maxSurge IntOrString | MaxSurge is the maximum number of resources that may be created above the desired replica count during an update. It can be an absolute number (for example, 1) or a percentage (for example, 25%). A percentage is calculated from ModelServing replicas for ServingGroupRollingUpdate and from the corresponding Role's replicas for RoleRollingUpdate, then rounded up. It defaults to 0. | XIntOrString: {} | |
partition IntOrString | Partition protects the first N existing replicas in ascending ordinal order from updates. The remaining replicas are eligible for rolling update. For a contiguous ordinal set, this is equivalent to protecting [0, Partition). Value can be an absolute number (ex: 5) or a percentage of total replicas (ex: 10%). Absolute number is calculated from percentage by rounding up. The default value is 0. | XIntOrString: {} |
RolloutStrategy
RolloutStrategy defines the strategy that the ModelServing controller will use to perform replica updates.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
type RolloutStrategyType | Type selects the granularity of rolling updates. Supported values are ServingGroupRollingUpdate and RoleRollingUpdate. It defaults to ServingGroupRollingUpdate. ServingGroupRollingUpdate uses rolloutStrategy.rollingUpdateConfiguration; rolling update settings on individual Roles do not take effect. RoleRollingUpdate uses the rolling update configuration on each Role; rolloutStrategy.rollingUpdateConfiguration must not be set. Kthena performs RoleRollingUpdate across all ServingGroups at the same time. Therefore, we recommend using it only in scenarios with a single ServingGroup. | ServingGroupRollingUpdate | Enum: [ServingGroupRollingUpdate RoleRollingUpdate] |
updateMethod UpdateMethod | UpdateMethod selects how outdated ServingGroups or Roles are moved to the desired revision. Supported values are Recreate (default) and InPlace. InPlace patches container images on existing Pods when only regular-container images change. Otherwise the ServingGroup, or the Role instance under RoleRollingUpdate, is recreated, and an InPlaceUpdateFallback event reports why. | Recreate | Enum: [Recreate InPlace] |
progressDeadlineSeconds integer | ProgressDeadlineSeconds is the maximum time for a ServingGroup or Role instance updated in place to become ready on the target image. After it passes, unfinished Pods are handled according to RecoveryPolicy. It only applies to in-place updates, and defaults to 1800 seconds. | Minimum: 60 | |
rollingUpdateConfiguration RollingUpdateConfiguration | RollingUpdateConfiguration configures ServingGroupRollingUpdate. It must not be set when type is RoleRollingUpdate; configure maxUnavailable maxSurge, and partition on each Role instead. | ||
roleCoordination RoleCoordination | RoleCoordination defines the cross-Role coordination parameters to be used when type is RoleRollingUpdate. | ||
evictionStrategy EvictionStrategySpec | EvictionStrategy defines the protection policy during node eviction. |
RolloutStrategyType
Underlying type: string
RolloutStrategyType defines the strategy to use to update replicas. Note that if recoveryPolicy is ServingGroupRecreate and the rollout strategy is RoleRollingUpdate, deleting an outdated Role causes its entire ServingGroup to be recreated.
Appears in:
| Field | Description |
|---|---|
ServingGroupRollingUpdate | ServingGroupRollingUpdate indicates that ServingGroup replicas will be updated one by one. |
RoleRollingUpdate | RoleRollingUpdate indicates that Role replicas will be updated one by one. |
SelectPolicyType
Underlying type: string
SelectPolicyType defines the selection strategy type for scaling operations.
Validation:
- Enum: [Or And]
Appears in:
| Field | Description |
|---|---|
Or | |
And |
ServingGroup
ServingGroup is the smallest unit to complete the inference task
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
restartGracePeriodSeconds integer | RestartGracePeriodSeconds defines the grace time for the controller to rebuild the ServingGroup when an error occurs Defaults to 0 (ServingGroup will be rebuilt immediately after an error) | 0 | |
gangPolicy GangPolicy | GangPolicy defines the gang scheduler config. | ||
networkTopology NetworkTopology | NetworkTopology defines the network topology affinity scheduling policy for the roles of the ServingGroup,it works only when the scheduler supports network topology-aware scheduling. | ||
roles Role array | MaxItems: 4 MinItems: 1 |
Target
Target defines a ModelServing deployment that can be monitored and scaled.
Example:
target: targetRef: kind: ModelServing name: podinfo-ms metricSources: podinfo_rps: prometheus: serverURL: http://prometheus.monitoring.svc:9090 query: sum(rate(http_requests_total[2m]))
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
targetRef ObjectReference | TargetRef references the target object to be monitored and scaled. Default target GVK is ModelServing. Currently supported kinds: ModelServing. Example: kind=ModelServing, name=podinfo-ms. | ||
metricSources object (keys:string, values:MetricSource) | MetricSources declares how to fetch specific metrics for this target. Keys must match AutoscalingPolicy.spec.metrics[].name. Missing keys are treated as missing metrics for that reconcile loop. For example, a key "podinfo_rps" here must correspond to a metric named "podinfo_rps" in the referenced AutoscalingPolicy. |
TargetScalingStatus
TargetScalingStatus reports the observed scaling state of a single scalable unit (a whole ModelServing, or one role within it).
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
name string | Name identifies the unit when this status appears in a list. It is required for HeterogeneousStatus entries and DisaggregatedStatus roles, and may be empty for HomogeneousStatus because the target is implied. | ||
currentReplicas integer | CurrentReplicas is the number of replicas currently observed. | ||
desiredReplicas integer | DesiredReplicas is the number of replicas the controller computed from metrics, before ratio enforcement. | ||
mode string | Mode reports whether the unit is currently in "Stable" or "Panic" mode. |
UpdateMethod
Underlying type: string
UpdateMethod defines how outdated Pods are moved to the desired revision.
Appears in:
| Field | Description |
|---|---|
Recreate | RecreateUpdateMethod deletes outdated Pods and creates new ones. |
InPlace | InPlaceUpdateMethod patches container images on existing Pods when possible, and recreates them otherwise. |