Supported hardware
Ascend
Cambricon
NVIDIA
Iluvatar CoreX
Moore Threads
Hygon
High-performance inference. Across heterogeneous accelerators.
A unified framework for LLM, VLM, DiT, and REC models. Built for efficient inference across diverse AI hardware.
Many models. Many accelerators. One inference layer.
01 Models
- LLM
- VLM
- DiT
- REC
02 Unified engine
xLLM
Service & Engine
- Scheduling
- KV cache
- Graph execution
03 Accelerators
- NPU
- MLU
- GPU
Easy
Bring models into your applications with OpenAI-compatible and Anthropic-compatible APIs for online serving.
Fast
Keep accelerators busy with continuous batching, task pipelining, and graph execution to reduce runtime overhead and improve throughput.
Cost Efficient
Reuse prompt prefixes and extend KV caching across device memory, host memory, and distributed storage to reduce repeated computation and memory pressure.
Broad Support
Cross-Platform Model & Hardware Support
Supported models
DeepSeek
Qwen
Kimi
GLM
Flux
OneRec
Resources
Build with xLLM.
From your first deployment to a deeper look at the engine. Start here.
Community
Built in the open.
Better together.
Ask a question, share an idea, or contribute to the next improvement. Help shape the future of xLLM.