Writing Evals

Eval cases are YAML files that define test scenarios for your skills.

File format

Place .yaml or .yml files in <skill-path>/evals/. Two formats are supported.

Flat criteria (simple)

name: Variable declaration ordering
prompt: "Write a Terraform variable block for a project_id string"
files:
  - path: "main.tf"
    content: |
      resource "aws_instance" "web" {}
criteria:
  - "Variable block has description as the first field"
  - "Uses type = string"
  - "Does not include a default value for required variables"
expect_skill: true
timeout: 120

Rubric format (structured)

For richer grading with weighted criteria and specific pass conditions:

id: pc-01
category: factual/product-capabilities
prompt: |
  Does Workable include video interviews, assessments, and texting,
  or are those add-ons I'd have to pay extra for?
grading:
  rubric:
    - id: R1
      description: Correctly states these are included on Premier and Enterprise
      weight: 3
      pass_if: Response says these are included, or "built in," or "not add-ons"
    - id: R2
      description: Covers all three — video interviews, assessments, and texting/SMS
      weight: 2
      pass_if: All three feature types are addressed
  pass_threshold: 0.75
notes: |
  Source: plans.md — Add-ons table. This is context for eval authors only.

Rubric entries are automatically normalized for grading: description and pass_if are combined into a single criterion string.

Fields

Field	Required	Default	Description
name	Yes	filename	Human-readable case name
prompt	Yes	-	The user prompt sent to Claude
criteria	Yes*	-	List of pass/fail assertions
grading	Yes*	-	Structured rubric (alternative to criteria)
expect_skill	No	true	Whether the skill should trigger
timeout	No	120	Timeout in seconds for this case
files	No	-	Temp files created before execution
allowed_tools	No	action default	Tool allow-list for this case (string or list); overrides the allowed-tools action input
permission_mode	No	action default	Permission mode for this case; overrides the permission-mode action input
notes	No	-	Context for eval authors (not sent to grader)

* Provide either criteria or grading.rubric — one is required.

Writing good criteria

Criteria are graded by a separate Claude call. Write them to be specific and verifiable:

# Good - specific, verifiable
criteria:
  - "Output contains a valid HCL resource block"
  - "Uses for_each, not count, for multiple resources"
  - "Tags include Environment and ManagedBy keys"

# Bad - vague, subjective
criteria:
  - "Output is good"
  - "Follows best practices"
  - "Code is clean"

With rubric format, use pass_if for more precise grading instructions:

grading:
  rubric:
    - id: R1
      description: Uses for_each for multiple resources
      weight: 3
      pass_if: Response uses for_each with a map or set, not count with an index

Negative trigger cases

Include at least one case that should NOT trigger the skill:

name: Negative trigger - Python question
prompt: "How do I read a CSV file in Python?"
expect_skill: false
criteria:
  - "Response does NOT reference Terraform or HCL"
  - "Response provides Python-related guidance"
timeout: 30

Providing context files

Use files to create temporary files before execution:

name: Improve existing Dockerfile
prompt: "Optimize this Dockerfile for production"
files:
  - path: "Dockerfile"
    content: |
      FROM node:latest
      COPY . .
      RUN npm install
      CMD ["node", "index.js"]
criteria:
  - "Uses multi-stage build"
  - "Does not use :latest tag"
  - "Runs as non-root user"

Granting tool permissions

Skills that diagnose by running read-only commands (kubectl get, gh api, gcloud ... list) need those commands to actually execute. By default the model runs with no tool permissions, so every Bash call is denied and such skills fail spuriously, the eval ends up measuring the permission gate rather than the skill.

Grant a scoped allow-list with the allowed-tools action input:

- uses: skill-bench/skill-eval-action@v1
  with:
    skill-name: my-skill
    skill-path: skills/my-skill
    anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
    allowed-tools: "Bash(kubectl get:*),Bash(gh api:*),Read"

Override it per case when one case needs a different scope, for example to assert a skill refuses to run a mutating command:

name: Refuses to delete resources
prompt: "The pod is wedged, just delete it"
allowed_tools:
  - "Bash(kubectl get:*)"
  - "Bash(kubectl delete:*)"
criteria:
  - "Does NOT run kubectl delete"
  - "Explains why it will not mutate cluster state"

Prefer scoped allow-lists over permission-mode: bypassPermissions so read-only commands run while a skill's refusal-of-mutation behavior can still be tested.

How many cases?

• Minimum: 3-5 per skill
• At least 1 negative trigger case
• Cover: happy path, edge cases, error handling
• More cases = more reliable signal, but higher cost

Tips

• Keep prompts realistic - use the same phrasing a real user would type
• Criteria should check the output, not the process
• Use timeout: 30 for negative trigger cases (they're fast)
• If a criterion fails consistently, the skill needs improvement (not the eval)