Skill Tester

Here's a complete GitHub Actions quality gate that runs on every PR touching the skills/ directory:

# .github/workflows/skill-quality-gate.yml
name: Skill Quality Gate

on:
  pull_request:
    paths:
      - 'skills/**'

jobs:
  validate-skills:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Find changed skills
        id: changed
        run: |
          changed=$(git diff --name-only origin/main...HEAD | \
            grep '^skills/' | cut -d'/' -f1-2 | sort -u)
          echo "skills=$changed" >> $GITHUB_OUTPUT

      - name: Validate structure
        run: |
          for skill in ${{ steps.changed.outputs.skills }}; do
            echo "--- Validating $skill ---"
            python3 engineering/skill-tester/scripts/skill_validator.py \
              "$skill" --json | tee /tmp/validation.json
            python3 -c "
import json, sys
result = json.load(open('/tmp/validation.json'))
if not result['structure_compliance']['checks']['skill_md_exists']:
    sys.exit(1)
"
          done

      - name: Test scripts
        run: |
          for skill in ${{ steps.changed.outputs.skills }}; do
            python3 engineering/skill-tester/scripts/script_tester.py \
              "$skill" --timeout 30
          done

      - name: Score quality (minimum 75)
        run: |
          for skill in ${{ steps.changed.outputs.skills }}; do
            score=$(python3 engineering/skill-tester/scripts/quality_scorer.py \
              "$skill" --json | jq '.overall_score')
            echo "$skill score: $score"
            if (( $(echo "$score < 75" | bc) )); then
              echo "FAIL: $skill scored $score (minimum: 75)"
              exit 1
            fi
          done

What this gates on:

• Missing SKILL.md → immediate block
• Script syntax errors or import failures → block
• --help crashes or no --json support → block
• Quality score < 75 → block with score in the error message

Feedback to contractors in the PR:

Action failed: log-analyzer scored 62/100 (minimum: 75)
Failing checks:
  - scripts/log_summarizer.py: no --json output mode
  - expected_outputs/ directory missing
Run quality_scorer.py --detailed --recommendations to see all suggestions.

This eliminates back-and-forth review cycles on structural issues — contractors get immediate automated feedback.

This skill includes executable scripts — I'll need your permission before running them.

# Batch quality report across all 167 skills
python3 scripts/quality_scorer.py skills/ \
  --batch \
  --json > reports/quality_report_$(date +%Y%m%d).json

Then extract the actionable summary:

# Distribution by grade
jq -r '[.[] | .letter_grade] | group_by(.) | 
  map({grade: .[0], count: length}) | sort_by(.grade)[]' \
  reports/quality_report_*.json

# Output:
# { "grade": "A", "count": 12 }
# { "grade": "B", "count": 43 }
# { "grade": "C", "count": 67 }   ← most of them
# { "grade": "D", "count": 38 }
# { "grade": "F", "count": 7 }

# Skills scoring < 60 (need urgent attention)
jq -r '.[] | select(.overall_score < 60) | 
  "\(.skill_path): \(.overall_score)/100 — \(.improvement_suggestions[:2] | join(", "))"' \
  reports/quality_report_*.json

Most common gaps in legacy migrations:

Prioritization approach:

• Fix the 7 failing skills first (structural issues, scripts crash)
• Batch-fix the --json issue (same pattern in all scripts — one change per script)
• Add expected_outputs/ by running each script once with sample data

Track progress over time:

# Run weekly, compare to baseline
jq '[.[] | .overall_score] | add/length' reports/quality_report_*.json
# Week 1: 61.2 average → Week 4: 74.8 average