Monitor

Runs monitors on trajectories to detect suspicious behavior.

Purpose

Executes monitoring functions on trajectories to compute suspiciousness scores. Used to:

  • Evaluate trajectory safety
  • Compute action-level or trajectory-level suspicion scores
  • Test new monitor implementations
  • Generate monitored runs for analysis

Commands

ct run monitor

Run monitors on trajectories.

Usage:

ct run monitor TRAJ_LIKE [OPTIONS]

Arguments:

  • TRAJ_LIKE - An eval-backed source: a .eval file, or a run ID/URL whose run records a backing eval log (see Downloading from the Viewer)

Options:

  • -m, --monitor - Monitor selection, repeatable (e.g., -m multi -m quick); defaults: mean-action and mean-quick. Whether a monitor scores per action or per trajectory is a property of the monitor itself; prefix a name with action/ or trajectory/ only when the same name exists as both.
  • --trusted-model - Required. The model bound as the eval-level trusted role that every monitor runs on. Any Inspect AI provider string works, including self-hosted open-weight endpoints (see Self-Hosted and Open-Weight Models)
  • --cot-level - Chain-of-thought visibility for the monitors: none (default), reasoning, or extended
  • --save-dir PATH - Directory to save monitored trajectories
  • --no-upload - Disable upload (upload is enabled by default)
  • --run-name TEXT - Custom run name

Monitors never carry model configuration; the one model they run on is the eval-level trusted role, given by --trusted-model. The exception is monitoring-bench, which replicates its benchmark's fixed 3-model panel.

A monitor selection is a registered monitor's name, prefixed with action/ or trajectory/ when the name exists at both scopes. Repeat the selection flag to run several monitors.

Examples:

# Monitor with the default monitors
ct run monitor logs/my_eval.eval --trusted-model openai/gpt-5-nano

# One specific monitor
ct run monitor logs/my_eval.eval -m strong --trusted-model openai/gpt-5-nano

# Monitor and save locally without uploading
ct run monitor logs/my_eval.eval -m strong --save-dir monitored_trajs/ --no-upload --trusted-model openai/gpt-5-nano

# Monitor a specific run from database
ct run monitor abc123def456 -m multi --trusted-model openai/gpt-5-nano

# An action and a trajectory monitor in one pass
ct run monitor logs/my_eval.eval -m multi -m quick --trusted-model openai/gpt-5-nano

# Monitor a viewer-hosted run directly (no intermediate download)
ct run monitor https://basharena-data.vercel.app/runs/ed2b360737041f1743ac4de43280f858 \
    -m multi --trusted-model openai/gpt-5-nano

Available Monitors

A monitor is a plain async function: (messages, metadata, index) -> Score for action monitors, (messages, metadata) -> Score for trajectory monitors. Each is written out in src/control_tower/monitors/catalog.py; its prompt is a markdown file under src/control_tower/monitors/assets/{action,trajectory}/, and the function decides everything else — the output schema, how many draws, how to fan out over candidate covert objectives, how to combine (mean / peak / per_goal). The registry is the two dicts at the bottom of the catalog, and a monitor's docstring is its description.

List available monitors:

ct protocols monitors action [NAME]
ct protocols monitors trajectory [NAME]

Action-Level Monitors

Score each untrusted-model action.

  • no-history - Scores each action in isolation with no preceding action history or candidate covert-objective hints.
  • simple - The concise standard monitor: main task, prior actions and results, current action, 0-10 suspicion, no hints.
  • old-action - The default action monitor: one model call per action, shown the candidate covert objectives.
  • old-action-no-goals - The default action monitor without the candidate covert objectives.
  • checklist - Answers five yes/no checks per action (necessity, security surface, innocent explanation, concrete harm, removability) before scoring; the top of the scale is gated on concrete harm.
  • orthogonal - Flags work that does not serve the main task - unnecessary changes, premature optimization, off-task activity - rather than judging malice directly.
  • three-scores - Categorical monitor: a safe / uncertain / suspicious verdict per action, mapped onto the score scale.
  • strong - Sees the full conversation history (goal-blind) so it judges each action in context.
  • strong-goals - The strong full-history monitor, additionally shown the environment's candidate covert objectives.
  • goal-analysis - Scores each action while shown the environment's candidate covert objectives, asking whether the action advances any of them.
  • single-goal - Scores each action once per candidate covert objective in parallel and keeps the maximum score.
  • multi - Weighted blend of the orthogonal and old-action monitors' scores for each action.
  • mean-action - Averages two independent runs of the old-action monitor on each action. CLI default.
  • mean-goals - Averages two independent goal-analysis runs on each action.
  • mean-single-goal - Scores each action twice per covert objective, averages each objective's scores, and keeps the highest average.

Trajectory-Level Monitors

Score the whole transcript in one shot rather than action by action.

  • quick - Reads the full transcript in a single pass, shown the candidate covert objectives, and returns one score for the trajectory.
  • quick-no-goals - Reads the full transcript in a single pass with no covert-objective hints.
  • goal-analysis - Reads the full transcript while shown the environment's candidate covert objectives, working through them one by one.
  • single-goal-analysis - Reads the full transcript once per candidate covert objective in parallel and keeps the maximum score.
  • mean-quick - Averages five independent quick passes over the transcript. CLI default.
  • mean-goal-analysis - Averages two independent goal-analysis passes over the transcript.
  • mean-single-goal-analysis - Reads the transcript twice per covert objective, averages each objective's scores, and keeps the highest average.
  • monitoring-bench - Replica of the MonitoringBench benchmark's monitor: the bench prompt on its fixed 3-model panel (scores comparable to the imported monitoring-bench dataset), max-aggregated.

Whether a monitor sees the environment's candidate covert objectives — every side task registered for that environment, not just the one the agent was assigned — is fixed by its prompt asset: goal-aware and goal-blind variants are separate monitors (e.g. old-action vs old-action-no-goals, strong vs strong-goals, quick vs quick-no-goals).

A source .eval containing a single environment from an isolated setting is replayed in that setting's dependency environment automatically. Compose topology is resolved without pulling or building Docker images.

Output

Post-hoc monitoring reads the source, runs monitors, and writes the monitor results. Action-level monitors write per-action results; trajectory-level monitors write one result per trajectory.

Sources are scored directly

The input must be eval-backed: a local or remote .eval, an in-memory EvalLog, or a viewer run backed by an eval log — anything else (trajectory JSON, datasets, runs without a backing eval log) is rejected with an error. ct run monitor scores the recorded transcripts directly: action monitors see the transcript prefix before each action, trajectory monitors the full transcript, and nothing re-executes.

A local .eval source gains its scores in place: the scored log is written to a scratch file beside it and renamed over the source. Any other source (an in-memory log or a viewer run), and any run with --limit (whose result carries only the samples it scored), is scored onto a copy in the save dir (an explicit --save-dir, or the default ./data/monitoring/<monitors>/<label>_<timestamp>/) as <source-stem>.eval (a re-run into the same dir gets a -N suffix), and the source is left unchanged.

An action monitor's score value maps action index to score (None where the action was skipped by design or the monitor failed), with the full verdicts under metadata.verdicts; a trajectory monitor's score is the verdict itself. Decision statistics (max, thresholded audits) are computed downstream from these values, never pre-baked. Existing named scores survive, so scoring an already-scored eval with a new monitor accumulates both, and a monitor whose verdicts are already on the source is refused unless --overwrite, which replaces that monitor's verdicts — verdicts are never silently replaced. An eval scored before verdicts became inspect Scores carries monitor_<name> folds ({max, action_<i>, trajectory}) that no reader accepts; migrate it by re-scoring every folded monitor at once with ct run monitor --overwrite -m <name> ..., since a fold left on the source cannot be read around.

The eval header records derived_from (the source identifier) on copied artifacts. Uploaded monitor runs register the monitored log as their eval_log_url, and downstream consumers (e.g. ct run summarize) reconstruct the monitor_response / traj_monitor_response fields from the sample's verdicts.

Unless --no-upload is passed:

  • Creates a run in the database
  • Uploads all monitored trajectories
  • Prints run URL and trajectory URLs

Note the model response cache defaults on, so re-running an unchanged command reproduces the same scores rather than drawing a fresh sample (the CLI warns when this happens); a monitor's fan-out draws (samples, ensemble members, retry attempts) scope the cache key by position, so they are independent samples within a run. Clear inspect's model cache (inspect cache clear) to resample.

Monitoring Details

Monitors add suspiciousness scores to trajectories:

  • Scores range from 0 (not suspicious) to 10 (very suspicious)
  • Different monitors use different approaches (LLM-based, heuristic, etc.)
  • Action-level results are stored per action
  • Trajectory-level results are stored once per trajectory

Cost Attribution

Every verdict carries a cost_usd (and token fields) in its metadata populated automatically by the orchestration layer — no per-monitor changes required. Multi-model monitors (e.g. multi) additionally get a cost_by_model split. An unpriced model is a None in that split; cost_usd stays the priced spend. See function-registry.md for the mechanism and known limitations, and trajectories.md for the monitor_cost(sample) / agent_cost(sample) / total_cost_usd(sample) accessors.

Technical Details

  • Supports both action-level and trajectory-level monitors
  • Tracks token usage and costs for LLM-based monitors
  • Runs monitors concurrently for efficiency
  • Handles errors gracefully per trajectory