xorbitsai /inference

Open-source inference platform for language, vision, embedding, reranking, image, audio, and video models. Deploy with multiple engines across heterogeneous hardware, from a laptop to a distributed cluster. Serve models through OpenAI-compatible APIs.

★ 9,657⑂ 901👁 58⑃ 3,079● 3,012▤ 151
PythonApache-2.0Created 2023-06-14inference.readthedocs.io
下载:v4.2.0v4.2.0 ↓4.2.02026-11-07
+2
日增长
9,657
stars
901
forks
58
watchers
3,079
prs
3,012
issues
151
releases

README

!Xinference

Xinference (Xorbits Inference) is an open-source distributed inference platform.

Deploy and run models on heterogeneous hardware, from a local machine to a distributed cluster. Xinference includes 300+ built-in models for language, vision, embeddings, reranking, image, audio, video, and experimental world generation. Register your own models, including traditional ML models through the Flexible type. Manage deployments in the Web UI and use models through OpenAI-compatible APIs, the Python SDK, or the CLI.

Quick start · Key features · Comparison · Self-hosting · Documentation · Model catalog · Releases · Xinference Enterprise
















Languages · English · 日本語 · 한국어 · Deutsch · Français · Español · Italiano · Português · 繁體中文 · 简体中文

 Quick start

 1. Start the server

Xinference downloads the model and installs any needed engine dependencies on the first launch.

bash
pip install xinference
xinference-local --host 127.0.0.1


Open http://127.0.0.1:9997. On a fresh instance, use the setup token printed in the server log to create an administrator account, then sign in. First-run setup

 2. Launch a chat model

In Launch Model, search for qwen3.5 and open its launch form. Click Recommend configuration to fill in settings for your hardware. Review the recommendation and click Deploy.

!Full Qwen3.5 launch form with the recommended configuration applied

 3. Make your first request

Open the deployed model in Running Models to try a prompt in the Web UI. The first instance uses qwen3.5 as its Model UID. If your instance shows a different UID, use that value for model below. Create a key under API Key Management to call the API.

bash
export XINFERENCE_API_KEY=""
curl --fail-with-body http://127.0.0.1:9997/v1/chat/completions \
  -H "Authorization: Bearer $XINFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5",
    "messages": [{"role": "user", "content": "Translate this sentence into Spanish: The model is ready."}],
    "enable_thinking": false,
    "max_tokens": 96
  }'


!The qwen3.5 chat playground showing a completed response

 Key features

| Area | What Xinference supports |
| --- | --- |
| Model catalog | 300+ built-in models across language, vision, embeddings, reranking, images, audio, video, and experimental world generation. Register your own models, including Flexible models for traditional ML. Model catalog |
| Engines and hardware | Choose vLLM, SGLang, TRT-LLM, Transformers, llama.cpp, or MLX for supported LLMs. Hardware support varies by engine and includes CPU, GPUs, Apple Silicon, and supported MLU configurations. Backends · MLU support |
| Distributed serving | Run on one machine, add workers and replicas, or run a large model across workers with a supported engine. Compatible deployments support heterogeneous prefill/decode separation. Distributed inference · P/D separation |
| APIs and tools | OpenAI- and Anthropic-compatible endpoints, with streaming and tool calling w

Top issues

Latest release

v4.2.0 v4.2.0 2026-11-07
What's new in 4.2.0 (2026-11-07)

 Highlights

- Xavier prefix cache sharing. Reuse compatible prefixes across replicas running vLLM, SGLang, TRT-LLM, or MLX. Xavier expands LMCache storage support with Mooncake Store and can prioritize retaining KV prefixes while agents wait for tool results, reducing repeated prefill when they resume. (5917, 5905, 6018)

- Software updates and rolling model reloads. Administrators can check for and install Xinference updates in the Web UI, upgrading Workers, the Supervisor, and the API in order. Supported vLLM and SGLang deployments can retain GPU weights an

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON