Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.

★ 2,090⑂ 286👁 3⑃ 509● 97▤ 11
PythonApache-2.0Created 2026-06-24docs.nvidia.com/skills/skillevaluator/
下载:SkillEvaluator 0.9.0SkillEvaluator-0.9.0 ↓0.9.02026-11-13
+804
日增长
2,090
stars
286
forks
3
watchers
509
prs
97
issues
11
releases

README

SkillEvaluator

!SkillEvaluator wordmark








 📺 Livestream: Ask the Experts: Evaluating Agent Skills

SkillEvaluator checks AI agent skills and measures their effect on task results:
start with quality checks, then compare runs with and without a skill.

Agent skills are folders of instructions and supporting files defined by the
Agent Skills specification. SkillEvaluator is part of
the NVIDIA Verified Skills pipeline.

 Get started

Three ways in, one CLI, same results.



1. Run the CLI directly
2. Add it to your repo via CI
3. Wire it to your agent




Install (or uv tool install):

bash
pip install skillevaluator


Run all Tier 1 checks, no API key:

bash
skillevaluator validate \
  ./my-skill --tiers 1


Or run from a clone: Quickstart.




Check changed skills on every pull request. On GitHub, run:

bash
skillevaluator ci init


It opens a setup PR. On GitLab, copy the CI template.




Add the SkillEvaluator skill to Claude Code, Codex or any skill-aware agent:

bash
npx skills add \
  NVIDIA/SkillEvaluator


Then ask: "Is my skill any good?" No Node.js? Copy the folder.





Guides: installation,
CI setup (GitHub,
GitLab, report examples) and
agent skill.

 Quickstart

Both paths use bundled text-analyzer.

| Start here | Requires | First result |
| --- | --- | --- |
| Run without an LLM | Git and uv | A deterministic quality score and findings |
| Use NVIDIA Build | Git, uv, and a Build API key | Model-backed rubric scores and feedback |

Install uv and Git,
then get the example:

bash
git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator


uv provisions Python 3.13 and dependencies. For standalone installation, see
Installation.

 Run without an LLM

bash
uv run --python 3.13 skillevaluator quality-check \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-quality


No API key, scanner, GPU or Docker is needed. The example scores 86.2/100 (B),
exits 0 and saves reports/first-quality/skillevaluator-quality.json. The
passing threshold defaults to 70.

This checks static quality; publication evidence stays INCOMPLETE. Security
needs scanners; agent impact needs live evaluation. Replace the example path
with your skill directory. See the
keyless walkthrough.

 Use NVIDIA Build

Eligible NVIDIA Developer Program members can use NVIDIA-hosted endpoints for
free prototyping, subject to access terms and rate limits.

1. Sign in at build.nvidia.com and complete enrollment.
2. Open a model page, select Get API Key, and generate your key.
3. Set the provider and key in the same terminal, then run the rubric check:

bash
export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
uv run --python 3.13 --extra llm skillevaluator rubric-eval \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-build


The command sends skill documentation to the model and saves feedback in
reports/first-build/skillevaluator-rubric.json. N

Top issues

Discussions

Latest release

SkillEvaluator 0.9.0 v0.9.0 2026-11-13
SkillEvaluator 0.9.0

SkillEvaluator 0.9.0 expands how you build evaluation datasets, compare skill versions, and understand why an agent passed or failed. It adds controls for evaluation budgets, reusable baselines, and clearer evidence across terminal, HTML, JSON, Markdown, and benchmark reports.

For PyPI users upgrading from 0.7.2, this release also includes the changes shipped in the GitHub-only 0.8.0 release.

 Highlights

- Build and review better evaluation datasets. Declare deterministic checks for script execution, workspace files, and final responses. Generate datasets grounded in s

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON