用户指南概览

📄️ Multi-Node Inference

This page describes the multi-node inference capabilities in Kthena, based on real-world examples and configurations.

In distributed AI inference, communication latency between nodes directly affects inference efficiency. By being aware of the network topology, frequently communicating tasks can be scheduled onto nodes that are closer in network distance, significantly reducing communication overhead. Since bandwidth varies across different network links, efficient task scheduling can avoid network congestion and fully utilize high-bandwidth links, thereby improving overall data transmission efficiency.

📄️ 自动扩缩容

Overview

🗃️ 工作负载

3 个项目

🗃️ 路由器

8 个项目

🗃️ 可观测性

1 个项目

📄️ Runtime

Kthena Runtime is a lightweight sidecar service designed to standardize Prometheus metrics from inference engines, provides LoRA adapter download/load/unload capabilities, supports model downloading, and publishes KV cache events to Redis for the kvcache-aware router plugin.

📄️ Binpack scale down

Binpack scaling down is designed to maximize available node capacity, helping the cluster efficiently prepare for upcoming resource-intensive tasks.

🗃️ Prefill-Decode 分离