CI gate CLI

The tracely CLI (shipped with the SDK) runs an agent’s promoted regression suite against a PR and gates it — exit 0 (PASS) / 1 (FAIL), plus a GitHub commit status + PR comment.

tracely gate   <agent> [--env ci] [--api …] [--key …] [--pr N] [--sha …] [--github]
tracely replay <agent> (--entrypoint module:func | --cmd "…") [--live] [--github]

`tracely gate <agent>`

Gate a PR against pre-emitted env=ci traces — your CI already ran the agent and exported traces to Tracely. Each promoted case is matched to a candidate trace by input digest, the case’s assertions are evaluated, and the run aggregates to PASS/FAIL.

tracely gate planner --env ci --github

`tracely replay <agent>`

Re-run the agent on each promoted case’s recorded input (fetched from GET /api/gate/suite), emit fresh env=ci traces, then gate — all in one step.

# Python entrypoint, called once per case input (hermetic by default):
PYTHONPATH=sdk/examples tracely replay planner --entrypoint weather_agent:run
 
# or any process — it gets TRACELY_INPUT and emits its own trace:
tracely replay planner --cmd "python my_agent.py"
 
# --live makes real tool/LLM calls instead of serving recorded fixtures

gate vs replay. Use gate when CI already produced traces (just check them). Use replay when you want Tracely to drive the agent over the suite — the hermetic path, where recorded fixtures make it deterministic and free. See Hermetic replay.

What gating checks

A promoted case is a frozen failing run. The hard gate is fail-to-pass: the fix must not reproduce the failure (no error step, and any tool the model required must actually run). On top, soft warnings flag latency / token regressions versus the last green gate (non-blocking by default; set gate_block_on_warnings to make them block).

GitHub Actions

Inside Actions (or with --github), the CLI posts a commit status (tracely/regression-gate) and upserts a PR comment with per-case results and warnings. A reusable composite action lives at .github/actions/tracely-gate/:

# .github/workflows/tracely-gate.yml
name: Tracely regression gate
on: pull_request
permissions: { contents: read, statuses: write, pull-requests: write }
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - name: Tracely replay + gate
        env:
          TRACELY_API: ${{ secrets.TRACELY_API }}
          TRACELY_KEY: ${{ secrets.TRACELY_KEY }}
          TRACELY_WEB_URL: ${{ secrets.TRACELY_WEB_URL }}
          PYTHONPATH: sdk/examples
        run: |
          pip install ./sdk
          tracely replay planner --entrypoint weather_agent:run

Configuration

Flags or environment: TRACELY_API, TRACELY_KEY, TRACELY_AGENT, TRACELY_GATE_ENV, TRACELY_WEB_URL, GITHUB_TOKEN. --dry-run prints the GitHub calls instead of sending them; --no-github never touches GitHub even inside Actions.

Hermetic replay