Developers · CI Integration

Run AgentSmack on every pull request. Comment the verdict back.

First-party integrations for GitHub Actions and GitLab CI. Submit an agent, run a benchmark, post a deterministic markdown comment back to the PR — re-runs update the same comment in place. Tokens stay in headers; the formatter is byte-pinned cross-language.

What ships

  • GitHub Action

    Drop in `shivemind/agentsmack-action@v1`. Docker-based, pinned Python 3.12. Posts a markdown comment via the GitHub Issues API.

  • GitLab CI snippet

    Copy-paste `.gitlab-ci.yml` job that runs on merge_request_event. Uses GitLab Notes API directly — no `glab` CLI dependency.

  • Idempotent comments

    Every comment opens with `<!-- agentsmack:ci:v1 submission=… benchmark=… -->`. Re-running on the same PR updates the comment in place.

  • Deterministic markdown

    Pure-fn formatter pinned cross-language (TS + Python). Same benchmark input → byte-identical comment. Mandate 0.0.

  • Token-safe

    Tokens live in headers, never URLs or bodies. `redact_request_for_log` replaces secrets with `[REDACTED]` before any debug print.

  • No-CI fallback

    Run `agentsmack ci-comment --markdown-only` anywhere. Pipe into curl to POST manually. Works in self-hosted CI you can't extend.

GitHub Actions

Add a workflow file at .github/workflows/agentsmack.yml. Required secret: AGENTSMACK_API_KEY. The action posts via the GitHub Issues API using the auto-injected ${{ github.token }}; no extra PAT is needed.

name: AgentSmack
on: pull_request

jobs:
  agentsmack:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
      contents: read
    steps:
      - uses: actions/checkout@v4

      # Step 1 — produce the BenchmarkResultForComment JSON.
      # (Replace with your real `agentsmack analyze` + benchmark step.)
      - run: cp ci/fixtures/sample-benchmark-result.json result.json

      # Step 2 — submit + post comment.
      - uses: shivemind/agentsmack-action@v1
        with:
          api-key: ${{ secrets.AGENTSMACK_API_KEY }}
          submission-slug: sm_aaaaaaaaaaaaaaaa_abcdef
          benchmark-slug: prompt-injection-suite-v1
          benchmark-result-file: result.json
          post-comment: 'true'

The first push will create a new comment; subsequent runs on the same PR for the same submission-slug + benchmark-slug pair update the existing comment via the agentsmack:ci:v1 marker.

GitLab CI

Copy this job into your .gitlab-ci.yml. Required CI variables: AGENTSMACK_API_KEY and either GITLAB_TOKEN (project access token, api scope) or the runner's built-in CI_JOB_TOKEN.

stages:
  - agentsmack

agentsmack:
  stage: agentsmack
  image: python:3.12-slim
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  variables:
    AGENTSMACK_BENCHMARK_INPUT: "agentsmack-result.json"
  before_script:
    - pip install --no-cache-dir agentsmack
  script:
    - agentsmack ci-comment --provider gitlab "$AGENTSMACK_BENCHMARK_INPUT"

Dry-run mode

Before wiring credentials in, run the verb locally with --dry-run. It prints the exact POST shape (URL, headers, body) the SDK would send, with all tokens redacted.

agentsmack ci-comment \
  --provider github \
  --dry-run \
  result.json

No-CI fallback (curl)

If you can't extend your CI host, render the markdown locally and POST it yourself. The SDK's --markdown-only flag writes the comment body to stdout.

GitHub

pip install agentsmack
agentsmack ci-comment --markdown-only result.json > comment.md

curl -X POST \
  -H "Authorization: Bearer $GITHUB_TOKEN" \
  -H "Accept: application/vnd.github+json" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  -H "Content-Type: application/json" \
  --data "$(jq -Rs '{body: .}' < comment.md)" \
  "https://api.github.com/repos/$OWNER/$REPO/issues/$PR_NUMBER/comments"

GitLab

pip install agentsmack
agentsmack ci-comment --markdown-only result.json > comment.md

curl -X POST \
  -H "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  -H "Content-Type: application/json" \
  --data "$(jq -Rs '{body: .}' < comment.md)" \
  "https://gitlab.com/api/v4/projects/$PROJECT_ID/merge_requests/$MR_IID/notes"

Comment input schema

The CI surface accepts a closed-shape BenchmarkResultForComment JSON. Generate it from your benchmark pipeline; the formatter rejects unknown fields (Mandate 0.4).

{
  "schema": "agentsmack.ci.pr_comment_input.v1",
  "submissionSlug": "sm_aaaaaaaaaaaaaaaa_abcdef",
  "submissionPublicReportUrl": "https://agentsmack.com/submissions/sm_…/public",
  "agentLabel": "support-bot-v3",
  "fingerprint": "<64-hex>",
  "smackId": "<16-hex>",
  "canonicalModel": "gpt-4o",
  "benchmark": {
    "slug": "prompt-injection-suite-v1",
    "name": "Prompt Injection Suite v1",
    "score": 72,
    "maxScore": 100,
    "passed": 18,
    "failed": 7,
    "total": 25,
    "costMicrocents": "127500"
  },
  "probes": {
    "honeypot":    { "total": 15, "triggered": 2, "blocked": 13 },
    "underground": { "total": 10, "breaches": 1, "clean": 9 }
  },
  "compareDelta": null
}