Agent Skills That Matter: Automate Verifiable Workflows End-to-End

If you use any coding agent in your day-to-day work, you’ve probably bought into the hype at this point: you’ve written your first AGENTS.md file, set up some slash commands, or maybe even built your first agent skill after retyping the same prompt for the N-th time.

Here’s the thing: almost all agent skills I’ve seen in the wild fall into a handful of recognizable patterns (e.g., the persona prompt), and most of those patterns don’t fully leverage agentic AI systems. They automate the production of output, often with minimal usage of the systems’ agency. Agent skills that matter automate the production of verified output, while scripting deterministic behaviours and exploiting AI agents’ agency for non-deterministic ones. This gap is exactly where real leverage lives.

Case Study

Earlier this year my team ran into a bug in new evaluation infrastructure that resulted in an unexpected ~10pp underperformance compared to official leaderboard results, wasting compute on runs we couldn’t trust and leaving us without a reproducible baseline to hillclimb on. This task is particularly well suited for multiple reasons. First, identifying it by hand would have meant painstakingly scouring thousands of agent rollouts for a faint, systematic signal. This naturally maps to a non-deterministic yet repetitive task within the capabilities of current LLMs. Second, we can automate repetitive, but deterministic portions by implementing them as tools provided to the agents (e.g., setup_trajectories(session_id, filter_ids) ). Third, it allows us to scale the analysis to orders of magnitude more trajectories, which more than compensates for any noise in the system’s analysis of any individual trajectory. Finally, it is much quicker for a human to verify a representative of a failure cluster, than to cluster trajectories into failure modes from scratch, especially if we design the system’s final response to reduce such a verification burden.

An initial attempt with a single agent quickly failed: inspecting full trajectories of tens of thousands of tokens floods even 1M+ context windows, and once you’re that far beyond the effective context window, analysis performance degrades badly ^{1 2} and costs explode.

However, each trajectory analysis is independent, so there’s no reason to process them sequentially in a single context. Fanning out to one sub-agent per trajectory addresses both constraints at once: each agent works within a bounded context, which eliminates attention pressure and effectively gives each trajectory its own focused analysis pass. The concurrency also means the whole sweep runs in minutes rather than hours, and because each sub-agent’s context stays small, you can use a weaker model per trajectory and simply scale to more trajectories.

I opted for an agentic analysis of each trajectory, rather than a zero-shot analysis of the full trajectory, to encourage principled exploration and enable careful context management. To accomplish this, the trajectory-analyzer subagents interact with the trajectory through a dedicated tool, unveiling the specified sub trajectory slice (i.e., turns 1-3), up to a maximum cap. This multi-turn setup acts as a natural test-time scaling mechanism, by interleaving reasoning with exploration, and ensures we can handle arbitrary length trajectories without flooding the model’s context window, by capping the maximum exploration turns.

The outputs of the subagents are structured and machine parsable, so the orchestrator can gate on them: check that a report exists, parses, and matches the expected schema, then filter, aggregate, or re-run malformed or low confidence reports with a more capable model before aggregating the rest. That gate only checks form, not correctness. Correctness verification stays a human call, but the response format keeps that call cheap: suspicious turns are flagged with coordinates that let me jump straight to the exact region in the rollout, along with a first interpretation of the suspicious behavior or pattern the sub agent observed, instead of re-reading the whole trajectory to find it. The system doesn’t decide the fix is correct, it makes deciding cheap enough that a human actually can.

That last property matters more than it might seem, especially for critical infrastructure bugs. Handing off to me with a report of the issues found serves as a natural steering opportunity before committing to a fix. By steering here, I can ensure the fix is principled and targets the actual root cause, rather than applying a localized band-aid fix (LLMs have recently been shown to default to such band-aid fixes during runtime performance optimization tasks SWE-FormulaCode ³ or SWE-Perf ⁴).

Demo run on a separate corrupted session, without the leaderboard-mismatch prefiltering used in the case study below.

I preselected ~50 trajectories that the official leaderboard solved but we didn’t, a strong prior for isolating the bug. Running them through the workflow, a Sonnet 4.5 orchestrator coordinating Haiku 4.5 subagents, found a consistent pattern in 5 minutes for $2: a dirty git staging area causing silent negative evaluations after failing to apply a code patch. Fixing it recovered the full 10pp delta and unblocked our experiments. For comparison, those same ~50 trajectories add up to a ceiling of 1-1.5M tokens raw, an upper bound since the agentic setup never loads full trajectories and benefits from prompt caching across subagent calls. Feeding that ceiling into a single non-agentic pass wouldn’t have fit in context at all, and just the raw input tokens alone would have cost more than the entire agentic run.

Core Claim

The above workflow sits at one end of a spectrum. Many agent skills in the wild cluster earlier. To see what I mean, let’s take a look at four common patterns agent skills tend to fall into, ordered by increasing ownership over verified output.

The Persona Prompt. The simplest and most common pattern: a system prompt that assigns the agent a role and provides loose high level instructions. “You are a senior software engineer, review this PR for security issues.” Such skills can be useful, especially as a prompt library, but there typically is no clear terminal state, verification, or signal the agent can loop against.

The CLI Helper. These skills offer concrete command guidance and tell the agent how and when to apply specific tools. MCPs and tool integrations belong in this bucket too. They extend what an agent can reach across platforms and services (e.g. the HuggingFace skill). But reach without a defined workflow is still reach without a destination. These skills don’t compose their tools into a goal with a terminal state, and they don’t define how to branch on failure. They are genuinely useful, but they describe tools in isolation rather than workflows end-to-end.

The Scripted Workflow. This is where most customized skills that get actual work for developers live. The agent runs a workflow resembling a shell script and returns a result (e.g., deploy a model for inference on a cluster node). The gap here is subtle: it runs the happy path reliably, but typically there is no explicit, verifiable terminal state nor a clear instruction on how to branch on failures, making limited use of the system’s agency.

The End-to-End Automation. These skills explicitly define a precise terminal state and spell out failure handling upfront. The agent works toward that goal, branches and self-corrects when something breaks, and stops only once the output is verified. The case study above is a good example of exactly this.

Conclusion

Let’s apply this taxonomy to the case study I presented earlier, to understand what makes this end-to-end automation matter. First, our system solves a concrete problem end-to-end. It goes beyond a tool definition for our trajectory store and provides concrete and explicit structure for the process to automate through a mix of scripts implementing tools and prompts. Second, the terminal state is clearly defined: a predetermined number of trajectories were analyzed for unexpected behavior, or side-effects and any found were clustered and synthesized into an easily interpretable and verifiable report. Finally, the system actually utilizes the agency LLMs buy us, by outsourcing non-deterministic decision making to them rather than relying on humans. This allows us to scale the analysis to an order of magnitude more data, giving us much more confidence that the failure modes we identify are real patterns worth acting on, not noise from a handful of trajectories.

Since then, I’ve reused the same orchestrator-subagent pattern for a different problem, qualitatively analyzing model chain-of-thought traces to understand what a model actually reasons about. It held up well there too, and has become my default tool whenever a task is qualitative, too labor intensive to do by hand at scale, but decomposable into independent units an agent can verify one at a time.

Next time you build a skill, consciously reflect on its purpose. Does it really need to be a skill, or is it perhaps partially or fully scriptable? A scripted step is cheap, deterministic, and boring in the best way, code doesn’t need convincing that a task is done. An agentic step is expensive and harder to predict, but it’s the only thing that can sit with ambiguity, notice an anomaly, or judge whether a fix is principled instead of a band-aid. A skill that matters isn’t the one that hands the most work to the model, it’s the one where every bit of agency got spent on a part of the problem that actually needed it.

References