vibrantlabsai /ragas
Supercharge Your LLM Application Evaluations 🚀
★ 16,328⑂ 1,821👁 61⑃ 1,729● 1,366▤ 91
+8
日增长
16,328
stars
1,821
forks
61
watchers
1,729
prs
1,366
issues
91
releases
README
Supercharge Your LLM Application Evaluations 🚀
Documentation |
Quick start |
Join Discord |
Blog |
NewsLetter |
Careers
Objective metrics, intelligent test generation, and data-driven insights for LLM apps
Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications. Say goodbye to time-consuming, subjective assessments and hello to data-driven, efficient evaluation workflows.
Don't have a test dataset ready? We also do production-aligned test set generation.
Key Features
- 🎯 Objective Metrics: Evaluate your LLM applications with precision using both LLM-based and traditional metrics.
- 🧪 Test Data Generation: Automatically create comprehensive test datasets covering a wide range of scenarios.
- 🔗 Seamless Integrations: Works flawlessly with popular LLM frameworks like LangChain and major observability tools.
- 📊 Build feedback loops: Leverage production data to continually improve your LLM applications.
:shield: Installation
Pypi:
bash
pip install ragas
Alternatively, from source:
bash
pip install git+https://github.com/vibrantlabsai/ragas
:fire: Quickstart
Clone a Complete Example Project
The fastest way to get started is to use the ragas quickstart command:
bash
List available templates
ragas quickstart
Create a RAG evaluation project
ragas quickstart rag_eval
Specify where you want to create it.
ragas quickstart rag_eval -o ./my-project
Available templates:
- rag_eval - Evaluate RAG systems
Coming Soon:
- agent_evals - Evaluate AI agents
- benchmark_llm - Benchmark and compare LLMs
- prompt_evals - Evaluate prompt variations
- workflow_eval - Evaluate complex workflows
Evaluate your LLM App
ragas comes with pre-built metrics for common evaluation tasks. For example, Aspect Critique evaluates any aspect of your output using DiscreteMetric:
python
import asyncio
from openai import AsyncOpenAI
from ragas.metrics import DiscreteMetric
from ragas.llms import llm_factory
Setup your LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-4o", client=client)
Create a custom aspect evaluator
metric = DiscreteMetric(
name="summary_accuracy",
allowed_values=["accurate", "inaccurate"],
prompt="""Evaluate if the summary is accurate and captures key information.
Response: {response}
Answer with only 'accurate' or 'inaccurate'."""
)
Score your application's output
async def main():
score = await metric.ascore(
llm=llm,
response="The summary of the text is..."
)
print(f"Score: {score.value}") 'accurate' or 'inaccurate'
print(f"Reason: {score.reason}")
if __name__ == "__main__":
asyncio.run(main())
Note: Make sure your OPENAI_API_KEY environment variable is set.
Find the complete Quickstart Guide
Want help in improving your AI application using evals?
In tTop issues
#1447AttributeError29 comments · 2024-10-08#1435How to adapt language of result just exactly the same to the contexts given which is separated into several files and the files are in different languages. 23 comments · 2024-10-06#1322How does RAGAS generate ground_truth in TestSetGenerator class23 comments · 2024-09-16#1439subclass of LLM20 comments · 2024-10-07#1381Can't generate testdataset, always connection error and event loops is closed.20 comments · 2024-09-30#1121Custom prompt for test set generation20 comments · 2024-07-22#1807expectation error18 comments · 2025-01-04#1150Getting error "Failed to parse output. Returning None" on faithfulness metric15 comments · 2024-08-01
Latest release
v0.4.4 v0.4.4 2026-11-14This release restores a working install, fixes a published security vulnerability, and makes Ragas work again with current OpenAI and Anthropic models. bash pip install -U ragas Highlights - import ragas works again. A fresh install pulled in langchain-community 0.4, which removed the VertexAI modules Ragas imported at start-up, so the import failed with ModuleNotFoundError. Those imports are now optional. (2956) - Security fix for multimodal metrics (CVE-2026-6587). See the security section below. (3105) - Anthropic works on current Claude models. Ragas no longer sends the sampling defau