# Run. Measure. Improve.

Run ReviewBench in Harbor and hillclimb your reviewer on your own compute.

Get started with ReviewBench in [Harbor](https://www.harborframework.com/).

Dataset: [ReviewBench on Harbor Hub](https://hub.harborframework.com/datasets/review-bench/reviewbench-public).

## 1. Install Harbor

```bash
uv tool install --force --refresh --python 3.12 --with-executables-from harbor \
  "reviewbench-harbor @ git+https://github.com/review-bench/ReviewBench.git@refs/pull/51/head"
```

## 2. Set credentials

Configure your reviewer's credential using its own instructions. For the judge,
supply a **separate Anthropic API key** with access to **Claude Opus 5** (`claude-opus-5`):

```bash
printf 'Anthropic judge key: ' >&2; IFS= read -rs RB_HARBOR_JUDGE_KEY && export RB_HARBOR_JUDGE_KEY; printf '\n' >&2
```

The plugin forwards this key only to the verifier as `ANTHROPIC_API_KEY`.
Do not use it as your reviewer's credential, bake it into an image, or commit it.

## 3. Prepare your adapter

Use a compatible Harbor agent, or integrate your reviewer through a custom
adapter with the dependencies it needs. **Your adapter must convert review
output** into `/work/out/findings.json`, following the task instructions and
[findings contract](https://github.com/review-bench/ReviewBench/blob/main/AGENT_CONTRACT.md).

You provide a compatible Docker environment; the dataset provides the task image,
not your reviewer. Respect the endpoint allowlist in `profile.json` and keep
network isolation enabled.

## 4. Run the benchmark

Replace the agent and model placeholders with your Harbor agent name or import
path and its provider/model identifier. Harbor downloads the tasks automatically.
Omit `-m` if your adapter selects and records its own model.

The run options control **what to evaluate**, **how often**, and **how much runs at once**:

| Option | What it controls |
| --- | --- |
| `-l`, `--n-tasks` | Maximum number of PRs to evaluate. `-l 1` selects one PR; omit it to run the full dataset. |
| `-k`, `--n-attempts` | Independent attempts per PR. `-k 3` evaluates every selected PR three times. |
| `-n`, `--n-concurrent` | Maximum attempts running at once. `-n 1` runs sequentially; increasing it does not add PRs or attempts. |
| `-d`, `--dataset` | Dataset and version to run; here, `review-bench/reviewbench-public@v1`. |
| `-a`, `--agent` | Harbor agent name or your custom adapter's import path (`module:Class`). |
| `-m`, `--model` | Model used by your reviewer, not the judge. |
| `-e`, `--env` | Execution environment; `docker` runs tasks in Docker containers. |
| `--plugin` | Loads the ReviewBench scoring and result-collection plugin. |
| `--job-name` | Names the run and its artifact directory under `jobs/`. Choose a new name for each run. |
| `--yes` | Automatically accepts Harbor's confirmation prompts; omit it if you want to review them interactively. |

### Smoke test example

Run one PR to check your setup:

```bash
harbor run -d review-bench/reviewbench-public@v1 \
  -a "<agent-name-or-module:Class>" -m "<provider/model>" \
  -e docker \
  --plugin reviewbench_harbor.reviewbench_plugin:ReviewBenchPlugin \
  -n 1 -l 1 -k 1 --job-name public-smoke --yes
```

### Full benchmark example

Omit `-l` to run the full dataset; adjust `-n` and `-k` as needed.

```bash
harbor run -d review-bench/reviewbench-public@v1 \
  -a "<agent-name-or-module:Class>" -m "<provider/model>" \
  -e docker \
  --plugin reviewbench_harbor.reviewbench_plugin:ReviewBenchPlugin \
  -n 1 -k 1 --job-name public-full --yes
```

## 5. Analyze and validate results

```bash
reviewbench-check-job jobs/public-full jobs/public-full/reviewbench-metric.py \
  --output public-full-metrics.json
```

Use your job's paths. The checker rejects incomplete or failed runs.

## 6. Upload results, optionally

Review artifacts for credentials and sensitive data before uploading.

```bash
harbor auth login --no-browser && harbor upload jobs/public-full --private
```

Uploads do not publish a leaderboard entry. For that, [onboard your agent](/submit/).