openai /evals
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
★ 19,746⑂ 3,133👁 271⑃ 1,449● 307▤ 0
+3
日增长
+52
周增长
+133
月增长
19,746
stars
3,133
forks
271
watchers
1,449
prs
307
issues
0
releases
README
OpenAI Evals
You can now configure and run Evals directly in the OpenAI Dashboard. Get started →
Evals provide a framework for evaluating large language models (LLMs) or systems built using LLMs. We offer an existing registry of evals to test different dimensions of OpenAI models and the ability to write your own custom evals for use cases you care about. You can also use your data to build private evals which represent the common LLMs patterns in your workflow without exposing any of that data publicly.
If you are building with LLMs, creating high quality evals is one of the most impactful things you can do. Without evals, it can be very difficult and time intensive to understand how different model versions might affect your use case. In the words of OpenAI's President Greg Brockman:
Setup
To run evals, you will need to set up and specify your OpenAI API key. After you obtain an API key, specify it using the OPENAI_API_KEY environment variable. Please be aware of the costs associated with using the API when running evals. You can also run and create evals using Weights & Biases.
Minimum Required Version: Python 3.9
Downloading evals
Our evals registry is stored using Git-LFS. Once you have downloaded and installed LFS, you can fetch the evals (from within your local copy of the evals repo) with:
sh
cd evals
git lfs fetch --all
git lfs pull
This will populate all the pointer files under evals/registry/data.
You may just want to fetch data for a select eval. You can achieve this via:
sh
git lfs fetch --include=evals/registry/data/${your eval}
git lfs pull
Making evals
If you are going to be creating evals, we suggest cloning this repo directly from GitHub and installing the requirements using the following command:
sh
pip install -e .
Using -e, changes you make to your eval will be reflected immediately without having to reinstall.
Optionally, you can install the formatters for pre-committing with:
sh
pip install -e .[formatters]
Then run pre-commit install to install pre-commit into your git hooks. pre-commit will now run on every commit.
If you want to manually run all pre-commit hooks on a repository, run pre-commit run --all-files. To run individual hooks use pre-commit run .
Running evals
If you don't want to contribute new evals, but simply want to run them locally, you can install the evals package via pip:
sh
pip install evals
You can find the full instructions to run existing evals in run-evals.md and our existing eval templates in eval-templates.md. For more advanced use cases like prompt chains or tool-using agents, you can use our Completion Function Protocol.
We provide the option for you to log your eval results to a Snowflake database, if you have one or wish to set one up. For this option, you will further have to specify the SNOWFLAKE_ACCOUNT, SNOWFLAKE_DATABASE, SNOWFLAKE_USERNAME, and SNOWFLAKE_PASSWORD environment variables.
Writing evals
We suggest getting starting by:
- Walking throughTop issues
#873Are not merged PRs the result of irrelevancy to the model?12 comments · 2023-04-30#1826False-positive "distillation" ban on $200/mo Pro subscriber — automated appeal rejected by bot, requesting human review from Trust & Safety7 comments · 2026-09-05#1166Any website where I can share evaluation results?7 comments · 2023-06-15#782Idea for Evals: Sorting numbers with repeats and negatives7 comments · 2023-04-23#1393Using different models in evaluating mode-graded eval and in generating the completion6 comments · 2023-11-03#637Idea for Evals: Emotion and sentiment analysis Evals6 comments · 2023-04-11#1839Proposal update: Independent black-box evaluation of consumer AI under deployment opacity — historical v0.1 retained5 comments · 2026-09-26#1384Eval-running often hangs on last sample5 comments · 2023-10-17
Discussions
#1858Can a model distinguish a stop signal from a stale or unrelated alert?0 comments · 2026-11-15#1855A zero-latency, local-first runtime firewall for LLMs.0 comments · 2026-10-29#1852A training eval where one score hid the side effect: rewarded to win, the model ended the world more0 comments · 2026-10-17#1849Why no recent developments?1 comments · 2026-10-12#1847CML: Machine-Verifiable Continuity and State Transitions for Stateful Agents0 comments · 2026-10-07#1841RAG Evidence Lab: offline citation and number checks for exported RAG answers0 comments · 2026-09-30#1838DeciShift: behavioral regression testing for versioned ML decision systems1 comments · 2026-09-25#1836Can a fixed mock evaluation pipeline prove its own pass? (open reproducibility trial)1 comments · 2026-09-21