+3
日增长
5,706
stars
454
forks
44
watchers
34
prs
61
issues
0
releases
README
RouteLLM
RouteLLM is a framework for serving and evaluating LLM routers.
[Blog ] [Paper ]
Our core features include:
- Drop-in replacement for OpenAI's client (or launch an OpenAI-compatible server) to route simpler queries to cheaper models.
- Trained routers are provided out of the box, which we have shown to reduce costs by up to 85% while maintaining 95% GPT-4 performance on widely-used benchmarks like MT Bench.
- Benchmarks also demonstrate that these routers achieve the same performance as commercial offerings while being 40% cheaper.
- Easily extend the framework to include new routers and compare the performance of routers across multiple benchmarks.
Installation
From PyPI
pip install "routellm[serve,eval]"
From source
git clone https://github.com/lm-sys/RouteLLM.git
cd RouteLLM
pip install -e .[serve,eval]
Quickstart
Let's walkthrough replacing an existing OpenAI client to route queries between LLMs instead of using only a single model.
1. First, let's replace our OpenAI client by initializing the RouteLLM controller with the mf router. By default, RouteLLM will use the best-performing config:
python
import os
from routellm.controller import Controller
os.environ["OPENAI_API_KEY"] = "sk-XXXXXX"
Replace with your model provider, we use Anyscale's Mixtral here.
os.environ["ANYSCALE_API_KEY"] = "esecret_XXXXXX"
client = Controller(
routers=["mf"],
strong_model="gpt-4-1106-preview",
weak_model="anyscale/mistralai/Mixtral-8x7B-Instruct-v0.1",
)
Above, we pick gpt-4-1106-preview as the strong model and anyscale/mistralai/Mixtral-8x7B-Instruct-v0.1 as the weak model, setting the API keys accordingly. You can route between different model pairs or providers by updating the model names as described in Model Support.
Want to route to local models? Check out Routing to Local Models.
2. Each routing request has a cost threshold that controls the tradeoff between cost and quality. We should calibrate this based on the types of queries we receive to maximize routing performance. As an example, let's calibrate our threshold for 50% GPT-4 calls using data from Chatbot Arena.
python -m routellm.calibrate_threshold --routers mf --strong-model-pct 0.5 --config config.example.yaml
For 50.0% strong model calls for mf, threshold = 0.11593
This means that we want to use 0.11593 as our threshold so that approximately 50% of all queries (those that require GPT-4 the most) will be routed to it (see Threshold Calibration for details).
3. Now, let's update the model field when we generate completions to specify the router and threshold to use:
python
response = client.chat.completions.create(
This tells RouteLLM to use the MF router with a cost threshold of 0.11593
model="router-mf-0.11593",
messages=[
{"role": "user", "content": "Hello!"}
]
)
That's it! Now, requests with be routed between the strong and weak model depending on what is required, saving costs while maintaining a high quality of responses.
DepTop issues
#20Support function calling?9 comments · 2024-07-10#38OpenAIError: The api_key client option must be set5 comments · 2024-07-30#29Can I use Azure open AI 5 comments · 2024-07-22#23Provide support for more than two models and provide a training guide.5 comments · 2024-07-11#21can not initializing the RouteLLM controller 4 comments · 2024-07-10#67Can BERT router source code (esp. for training) be added to the repo?3 comments · 2024-11-05#28Langchain integration3 comments · 2024-07-22#63litellm.drop_params error when running the openapi server2 comments · 2024-10-07