Securing Agent Skills Before They Run

‍

AI agents and agentic systems are becoming essential computing paradigms within enterprise. The use of agent skills (skill.md), allows workflows to be programed in plain English. But despite the obvious appeal of replacing classic code with natural language, production systems can’t reliably run on instruction essays.

These agentic skills, as markdown files, can produce different outputs each time they run, varying in data structure, length, and content. In addition to inconsistency in the results, these skills, which are often open-sourced, can include unsafe patterns, vulnerabilities, and misconfigured permissions, and sometimes outright malicious code.

Solving these reliability and safety challenges is necessary to advance towards a viable practice of generative computing and wide adoption of agents within the enterprise. Towards these objectives IBM Research and Alice have collaborated to incorporate its Caterpillar skill scanner into the open-source Mellea Skills Compiler repository.

Mellea: From Prose to Programs

The Mellea Skills Compiler converts natural language agent skill specifications into typed, instrumented programs with policy-driven guardrails and auditable execution traces. The compiler includes three main components: Mellea, which structures the generative programs; Granite Guardian, which runs risk detection integrated via Mellea's hook system; and AI Atlas Nexus, which maps use cases to risks across taxonomies.

These components come together in a two-phase workflow. The compile step turns each instruction into a typed contract for the model's answer, so the response comes back as structured data instead of loose text. There are two ways to run this step: the mellea-skills compile CLI command, or the /mellea-fy command inside Claude Code.

Once compiled into programs, you can govern each one according to your own compliance posture by running mellea-skills certify. This checks your skill's use case against AI Atlas Nexus's risk taxonomies and produces a policy manifest. Guardian then uses that manifest to scan model inputs and outputs for compliance at runtime.

Caterpillar: Eating Away at the Blind Spots

Caterpillar is Alice's open-source security scanner for agent skills, built to close the blind spots that third-party skills create. It's a fast, reliable CLI tool that locates vulnerabilities and malicious code. It's powered by RabbitHole, Alice's threat-intel database drawing on over a decade of adversarial data from protecting the web's biggest platforms and frontier models.

Comprehensive safety scanning

Caterpillar has 50+ detection rules built into the repo, and allows teams to easily add their own rules as well. The built-in rules include:

• Credential theft (SSH keys, AWS credentials, API tokens)
• Data exfiltration (curl/wget calls to external servers)
• Persistence mechanisms (crontab entries, startup scripts)
• Crypto wallet theft
• Network attacks (C2 traffic, port scanning)
• Code obfuscation (base64/hex-encoded payloads)
• Overly broad permissions
• Supply chain attacks (postinstall hooks, remote script execution)

How to run Caterpillar

curl -fsSL caterpillar.alice.io/d/i.sh | sh
npm install -g @alice-io/caterpillar

Caterpillar installs with a single command and supports three scan modes. Select a mode with
‍--mode, or let Caterpillar auto-detect one from available credentials.

Alice mode

caterpillar login
caterpillar ask ./skill/ --mode alice

Alice mode sends the skill to Caterpillar’s server API for full analysis. It requires authentication through caterpillar login and is the most thorough option.

OpenAI mode

caterpillar ask ./skill/ --mode openai

OpenAI mode uses your own OpenAI API key for LLM-based analysis, together with Caterpillar’s built-in pattern scanner. Configure the key through OPENAI_API_KEY or caterpillar config set openai_api_key <key>.

Offline mode

caterpillar ask ./skill/ --mode offline

Offline mode uses built-in pattern matching and needs no Alice or OpenAI credentials. It is the fastest option and is well suited to CI/CD checks. It makes no network requests unless a VirusTotal API key is configured for artifact checks.

Bulk directory scans

caterpillar scan ./my-skills/ --mode offline

Use scan to discover and analyze skills across a directory, including skills in subdirectories and supported archives. All three modes work for bulk scans:

caterpillar scan ./my-skills/ --mode alice
caterpillar scan ./my-skills/ --mode openai
caterpillar scan ./my-skills/ --mode offline

For CI/CD, emit machine-readable results:

caterpillar scan ./my-skills/ --mode offline --json

Summary

Bundled files of natural language specifications are becoming the building blocks of agentic systems, but they still pose safety and reliability challenges for production. Mellea Skills Compiler and Caterpillar are open-source examples of steps towards a safer, more robust practice of generative computing.