SkillOpt
By Microsoft Research
MIT-licensed Python framework from Microsoft Research, around 16,800 GitHub stars, that treats an agent skill file as a trainable parameter: a frozen model runs scored task trajectories, an optimizer model proposes add, delete, and replace edits in minibatches, and a change survives only if it improves a held-out validation split, leaving one deployable best_skill.md of roughly 300 to 2,000 tokens.
Best for
- tuning agent skill files against measured results instead of by hand
- raising agent accuracy without touching model weights
- validation-gated edits that keep a record of rejected changes
- overnight offline refinement drawn from past agent sessions
- skills that deploy into Claude Code, Codex, and Copilot
- Sep 9, 2026Added to the evals category: Microsoft Research framework that optimizes an agent skill file through scored trajectories, minibatched add/delete/replace edits, and a held-out validation gate, shipping one best_skill.md artifact with no weight update.
Other Evals & observability
Braintrust
Eval platform for AI products — define test sets, run them across models, and track regressions over time. The default choice for teams shipping LLM features.
Helicone
LLM observability and logging proxy. One line of code change to log every prompt, response, cost, and latency across providers.
Langfuse
Open-source LLM engineering platform with tracing, evals, prompt management, and dataset tools. Self-hostable or cloud.
Arize Phoenix
Open-source LLM tracing and eval tool from Arize. Built around OpenTelemetry — good fit if you already use OTEL elsewhere.
Patronus AI
Eval and guardrails platform focused on enterprise safety — hallucination detection, PII checks, and policy compliance for LLM outputs.
LangSmith
Observability and eval platform from the LangChain team. Tight integration if you're building agents with LangChain or LangGraph.
Langtrace
Open-source, OpenTelemetry-based end-to-end observability tool with real-time tracing, evals and metrics for LLM apps.
Promptfoo
Open-source LLM testing and red-teaming framework that runs evals and security scans against AI apps, agents and RAG.