The Hidden Policy Inside Agent Skills

I used to treat an agent skill as a bag of instructions. If the task finished and the output parsed, I counted the skill as useful. That test ignores the decisions hidden inside the instructions: which options the agent compares, which evidence it trusts, how it breaks a tie, and what data it sends to a tool.

My last post focused on the carrying cost of installed tools. The papers released since then add a sharper problem. A skill can preserve the task interface while changing whose objective the agent follows.

Completion and intent need separate tests

A Finger on the Scale studies skills that steer shopping choices and Python dependency recommendations toward an attacker's preferred option. The constructed skills keep the candidate set and output schema intact. They change comparison rules, examples, and tie-breakers inside instructions that still look relevant to the task.

Across 50 queries per domain with three runs per query, the authors report attacker-favored selections in 81.33 percent of shopping responses and 63.33 percent of dependency responses. Every response remained parseable and selected a valid candidate.

The paper's downstream code analysis weakens the phrase “utility preserving” in its abstract. Mean LLM-judged quality fell from 0.558 to 0.506 across 50 clean and 50 attacked outputs. Quality-pass rates fell by 16 and 20 percentage points under two thresholds.

The 100 percent result measures interface validity. The quality analysis measures a small judged sample without executable tests. Neither number proves that the skill preserved the user's decision policy.

I want an acceptance test for intent alongside the test for completion. If a package-selection skill recommends one library, I need to know which criteria it used and whether a clean baseline would make the same choice from the same evidence.

Tool descriptions can request data

ContextLeak treats tool metadata as part of the attack surface. The researchers optimize a malicious tool's name and description so an agent selects it and places user prompts, conversation history, or tool-list content into its arguments.

Their main experiment uses 200 constructed users across 10 domains with a Qwen-3-8B backend. The malicious-tool selection rates reached 0.92 for user prompts, 0.89 for conversation history, and 0.86 for tool lists. Those numbers come from the paper's constructed setting.

The Claude Code MCP transfer gives a narrower result. The malicious tool appeared in a 20-tool set for 100 memory-attack instances and received 22 calls. The paper reports strong extraction scores inside those 22 selected cases. The conditional extraction score cannot become a 92 percent claim about Claude Code or MCP use in general.

I used to read a tool schema as documentation for the model. Now I also read it as a proposed data-flow policy. Every argument asks the client to package some part of the current state and send it across a boundary.

A client should show which fields came from the user, conversation, filesystem, or another tool before execution. The permission prompt should describe the outbound data as well as the tool's name.

Self-improvement grants write authority

EVOMAL studies agents that create new skills from examples in a shared library. An attacker plants a reference skill with a malicious banner and payload. The agent copies those elements into a new skill and writes the artifact back into the searchable library.

With 232 benign skills and 8 planted skills, six models produced self-poisoning rates from 20.3 to 41.8 percent on 153 tool-relevant SWE-bench Verified tasks. The same generic attack reached 25.8 percent across the full 500-task distribution for the paper's selected model.

Those rates belong to an architecture that retrieves code, creates skills, and returns new artifacts to shared storage. They do not describe a normal coding session that lacks that write path.

The architecture still exposes the ownership question. An agent that learns from experience can create durable executable context for a future task. That operation deserves the same review as a dependency update or a change to repository instructions.

I would place generated skills in quarantine with their source traces and evaluation results. Another task should not inherit them until a reviewer or deterministic gate grants that trust.

Useful skills need a lifecycle

Skills can add operating knowledge that a model would otherwise reconstruct during each run. Repo-To-Skill reports more than 5,000 verified skills distilled from 1,000 machine-learning repositories.

With its GPT-5.5 backbone, harness, and downstream execution budget held fixed, the authors report gains of 134.3 percent on MLE-bench, 34.4 percent on PaperBench, 9.2 percent on FrontierCS, and 14.0 percent on PassNet relative to the same agent without skills. The construction pipeline, verification process, and score denominators need an independent audit. The experiment still supplies a strong counterexample to removing skills as a category.

I want the operational knowledge without permanent exposure or inherited trust. That requires a lifecycle from discovery through selection, invocation, review, and retirement.

My review policy

1. I record the source, version, owner, and reason for installing each skill.
2. I inspect the decision criteria and tie-breakers before trusting recommendations that affect packages, services, money, or credentials.
3. I keep global configuration small and load project skills after the task establishes a need.
4. I review the data placed in tool arguments and require a separate approval when a call crosses a sensitive boundary.
5. I quarantine agent-generated skills with provenance and regression evidence before another task can retrieve them.
6. I compare important decisions against a clean baseline so a successful output cannot hide policy steering.
7. I remove or disable a capability through a reversible trial when its measured use no longer justifies its authority and maintenance cost.

This policy asks more from a skill than correct syntax. The skill must expose the decisions it changes, the data it can move, and the evidence that keeps it enabled.

I still want agents to discover specialized operating knowledge when a task needs it. I also want to see who wrote that knowledge, how the client selected it, and which parts of my intent it can redefine.