- Published on
claude plugin eval: Prove Your Plugin's Value With Numbers
You built a plugin. But does it actually make Claude work better? Not a gut feeling β a number. Now that''s possible.
On September 11, 2026, Anthropic released Claude Code v2.1.269 with the claude plugin eval command β a dedicated eval harness that measures how much a plugin actually contributes.
The Core Idea: "With Plugin vs. Without Plugin"
What Ξ (Delta) Means
The design of claude plugin eval is straightforward. Every test case runs twice:
- with-arm: plugin is loaded
- without-arm: plugin is absent (baseline)
The difference between the two results β Ξ (delta) β is the plugin''s real contribution. If the score rises, the plugin helped. If it falls, the plugin hurt.
Before this, plugin development relied on intuition β "it seems to work." Now you can say Ξ = +0.23.
Six Grader Types
Anthropic ships six graders. Four are free (transcript- and file-based). Two are paid (LLM-call-based).
Free Graders (Transcript and File Based)
| Grader | What It Measures |
|---|---|
regex | Whether a pattern is present in the output text |
tool_used | Whether a specific tool was called |
tool_order | Whether tools were called in the required order |
file_exists | Whether a specific file was created |
Paid Graders (LLM Calls)
llm: An LLM reads the output and scores quality.baseline: Directly compares with-arm and without-arm outputs and picks the better one.
Integrating Into a CI Pipeline
Key Flags
claude plugin eval --threshold 0.8 --max-cost-usd 20
--threshold 0.8: Fail the build if total score falls below 0.8.--max-cost-usd 20: Stop if LLM grader costs exceed $20.
Add these two flags to a GitHub Actions workflow or any CI pipeline and code changes that drop the plugin score below the threshold can''t merge.
Report Format
Execution produces a JSON report and an HTML report. The HTML version shows a visual side-by-side comparison of with/without results for each test case.
What Changes for Plugin Developers
Before this feature, the plugin development feedback loop was long: build β try it manually β ship with a feeling of "seems fine." Now it looks like this:
- Write test cases
- Run
claude plugin eval - Check Ξ
- Improve low-scoring cases
- Wire in CI thresholds
Plugins can be quality-managed like code. Between a plugin that ships with "just try it" and one that publishes "Ξ = +0.31, accuracy 87%," which would you choose?
Implications for Educational Tool Development
From an edtech perspective, this is a meaningful inflection point. Developers building Claude plugins for education β adjusting language for student reading levels, supporting teacher lesson design, generating error feedback β can now prove numerically that the plugin produces better outcomes in an educational context.
Usage Tips
- Start with five test cases: Narrow scope to verify the pipeline works before scaling up.
- Start free with regex graders: Validate basic behavior with free graders, then deepen quality evaluation with the paid llm grader.
- Always set --max-cost-usd: LLM grader costs scale quickly with test-case count.
Sources: