Developing effective AI agent skills often involves meticulous refinement. Developers typically iterate on a skill's instructions, testing changes against benchmarks to achieve desired performance. This process can be time-consuming. To address this, we are introducing skill-optimizer, an AI agent skill designed to automate this iterative improvement cycle.
skill-optimizer helps developers enhance their existing AI agent skills. It operates by treating a skill's SKILL.md file as a trainable parameter. By running iterative refinement against a defined set of benchmark tasks, it automatically identifies and integrates improvements, accepting only changes that successfully pass validation gates.
How skill-optimizer Refines Skills
The core of skill-optimizer's operation is its iterative refinement process. It begins with a baseline evaluation of the target skill against a set of benchmark tasks. This initial run establishes a performance standard. Following the baseline, the tool enters a series of training epochs. Each epoch involves several steps to identify and apply improvements to the SKILL.md file.
The process cycles through forward passes, where it generates rollouts using the current SKILL.md instructions. These rollouts are then evaluated against the benchmark. Subsequently, backward passes analyze any failures or suboptimal performance observed during the forward pass. Based on this analysis, the system ranks potential edits to the SKILL.md and applies the most promising ones. This continuous loop of evaluation, analysis, and modification allows the tool to systematically improve the skill's performance over time.
Validation and Improvement Gates
Not every proposed change to a skill's SKILL.md is immediately committed. skill-optimizer employs rigorous validation gates to ensure that only beneficial modifications are accepted. Any proposed edit must undergo a median-of-3 validation process. This means a change is tested multiple times, and its performance must consistently demonstrate an improvement. Specifically, a proposed change is only committed if it shows at least a 5% improvement margin over the current version's performance against the benchmark tasks. This strict requirement ensures that the skill evolves only through verified enhancements, preventing regressions or the introduction of ineffective changes.
Usage and Operational Controls
Users can trigger skill-optimizer with simple commands such as 'optimize this skill', 'tune the skill against the benchmark', 'make the skill better', or 'run skillopt'. The tool integrates with various internal components to execute its task. It uses LLM rollout generation to simulate skill performance, benchmark validation to measure effectiveness, and version control to track and manage SKILL.md changes. Additionally, it can incorporate optional rule-based or LLM quality judges to further assess the impact of proposed modifications.
There are important operational constraints built into skill-optimizer. It is designed not to mutate frontmatter, ensuring that important routing configurations and other metadata within the SKILL.md remain invariant. Furthermore, the tool hard-refuses to modify bundled skills by default. Such modifications are only allowed under two specific conditions: an explicit allow flag must be provided, and an independent held-out test set of at least 5 tasks must be available for validation. This safeguard prevents unintended alterations to foundational or widely used skills without proper oversight.
A Concrete Example
Consider an AI agent skill whose purpose is to extract specific data points from technical documentation. Initially, its SKILL.md might contain a general instruction like "Extract all listed parameters." However, after initial testing, it might frequently miss parameters embedded in complex sentences or tables.
A developer could initiate skill-optimizer using 'optimize this skill'. The tool would first establish a baseline by running the existing skill against a benchmark of diverse technical documents. During subsequent training epochs, the tool might generate rollouts, identifying where the skill fails to extract all required data. Analyzing these failures, it could then propose edits to the SKILL.md, perhaps adding more explicit instructions like "Also extract parameters from bullet points, numbered lists, and 'key: value' pairs within paragraphs." Each such proposed edit would then be subjected to the median-of-3 validation, requiring a 5% improvement in data extraction accuracy against the benchmark before it is integrated, leading to a more robust and precise data extraction skill.
FAQ
Q: What does skill-optimizer treat as a trainable parameter?
A: skill-optimizer treats the target skill's SKILL.md file as a trainable parameter.
Q: What improvement margin is required for a change to be accepted? A: A proposed change must demonstrate at least a 5% improvement margin, validated through a median-of-3 process, to be committed.
Q: Can skill-optimizer modify bundled skills? A: It can only modify bundled skills if an explicit allow flag is provided and an independent held-out test set of at least 5 tasks is available for validation.
skill-optimizer provides a structured way to enhance AI agent skill performance, moving beyond manual trial and error. This tool helps developers refine their skills systematically, leading to more capable agents.
