Changed Only

Only evaluate skills that were modified in the pull request. Saves cost and time.

name: Skill Eval
on:
  pull_request:
    paths:
      - 'skills/**'

permissions:
  contents: read
  pull-requests: write

jobs:
  changed:
    runs-on: ubuntu-latest
    outputs:
      skills: ${{ steps.filter.outputs.skills }}
    steps:
      - uses: actions/checkout@v6
        with:
          persist-credentials: false

      - name: Find changed skills with evals
        id: filter
        run: |
          skills=$(git diff --name-only origin/main...HEAD -- 'skills/' \
            | cut -d/ -f2 | sort -u | while read s; do
            [ -d "skills/$s/evals" ] && echo "$s"
          done | jq -R -s -c 'split("\n") | map(select(. != ""))')
          echo "skills=$skills" >> "$GITHUB_OUTPUT"

  eval:
    needs: changed
    if: needs.changed.outputs.skills != '[]'
    runs-on: ubuntu-latest
    timeout-minutes: 30
    strategy:
      fail-fast: false
      matrix:
        skill: ${{ fromJSON(needs.changed.outputs.skills) }}
    steps:
      - uses: actions/checkout@v6

      - uses: skill-bench/skill-eval-action@v1
        with:
          skill-name: ${{ matrix.skill }}
          skill-path: skills/${{ matrix.skill }}
          anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}

This pattern compares the PR branch against main to find which skill directories were touched, then only evaluates those that have an evals/ directory.