Dashboard and evaluation

Two tools that sit beside a run rather than inside it: a browser UI over the session database, and an offline harness for measuring whether a change to the prompts or the pipeline actually helped.

The command-line surface for both is in Other commands. This page is what is underneath.

The dashboard

locac web
locac web --cwd /path/to/target --enable-runs

It is a single-page app served by the binary over a session database. There is no build step, no bundler, and no dependency. The HTML and the client script are embedded in the executable.

What it renders

All untrusted content, from transcript text and tool output to finding evidence and live stream frames, is rendered through text interpolation, never as HTML. The repository under audit does not get to inject markup into the page that displays it.

The API

Every route is under /api/, returns JSON, and never throws: an unexpected error becomes a 500 JSON body rather than a stack trace.

A run route with --enable-runs off answers 404 with a message rather than pretending not to exist:

{ "error": "run routes are disabled — start the dashboard with `locac web --enable-runs`" }

The guards

A run executes bash, so the dashboard is a local RCE surface and is gated accordingly.

• Every /api/ route, reads included, is Host-guarded. On a loopback bind the Host header must be loopback and must carry the port actually bound. Reads expose transcripts, findings and config, so they need the same DNS-rebinding protection the run stream does.
• Every mutation additionally needs a same-origin Origin and a CSRF token. The token is minted at startup, substituted into the page shell, and sent back in a header, so a page on another origin cannot forge a request even to localhost.
• With a password set, everything except the login handshake needs a valid session JWT.
• Static assets and the page shell are public. They contain no data.

Auth mechanics are in the Security model: argon2id hashing, stateless HS256 sessions with a constant-time compare, and an HttpOnly; SameSite=Strict cookie.

Live runs

GET /api/runs/:id/stream is a server-sent event stream. Frames carry assistant text, tool calls and results, approval requests, approval resolutions, and the terminal status. The server sets no idle timeout on the connection, because a research run has long quiet stretches while a command executes.

Approvals work the same way they do in the TUI, over the wire: a dangerous tool call publishes an approval frame and blocks. The browser answers one approval, or answers "approve all", which also flips auto-approve on for the rest of that run, the equivalent of --yes. Everything else resolves to deny: no answer, a timeout, or an abort. The fail-closed default is the same one the CLI uses.

Each run keeps a bounded replay buffer, so a browser that connects late or reconnects sees recent frames rather than nothing.

API keys

Config and provider payloads have the key replaced with a redaction placeholder before they leave the process. Saving a form back preserves the stored key: a submitted value equal to the placeholder means "keep what is on disk", not "set the key to that string".

Evaluation

The eval harness exists because prompt changes feel effective. It answers two questions, and they are separate verbs for a reason: one measures, the other explains.

The run score S

A single number per trial, computed from the trial's findings and the fixture's labels:

S = Σ severity_weight(confirmed true positives) − (number of confirmed spurious findings)

Four details define what actually counts:

1. Only confirmed findings count. An unconfirmed lead scores nothing, positive or negative.
2. Findings are deduped by file:line, with backslashes normalised to forward slashes, so double-reporting the same bug cannot inflate the score.
3. A finding matches a label when the normalised paths are equal and the lines are within the window. A finding with no line matches on the file alone.
4. Each finding claims the closest unclaimed label, greedily. That makes the assignment deterministic when several planted vulnerabilities sit near each other.

Cost is tracked but is never folded into S. A cheaper arm does not win by being cheaper; you see both numbers and decide.

eval ab: did the change help?

locac eval ab \
  --baseline prompts/current.md \
  --candidate prompts/proposed.md \
  --fixtures eval/fixtures/mixed \
  --trials 12 \
  --seed 4242

Each file becomes that arm's base system prompt. Everything else, from tools and roles to gates and fixtures, is held identical, so the only variable is the prompt text.

The comparison is a two-sample bootstrap of the difference in mean S: 10,000 resampling iterations from a seeded deterministic generator, reported as a 95% interval taken at the 2.5th and 97.5th percentiles.

A/B eval (12 candidate / 12 baseline trials):
  S: candidate 0.71 vs baseline 0.58 · ΔS 0.13 · 95% CI [0.04, 0.22]
  cost (tok): candidate 41208 vs baseline 38955
  VERDICT: candidate-better

The verdict is candidate-better, baseline-better, or no-significant-difference, and it is decided by the interval, not by which mean is larger. A difference is called significant only when each arm has at least two samples and the interval excludes zero. Two trials with a lucky split produce no-significant-difference, which is the honest answer.

Because the seed pins the resampling, the same trials produce the same verdict on any machine.

eval diagnose: where did it break?

locac eval diagnose --fixtures eval/fixtures/mixed

For every planted vulnerability the run did not confirm, and every confirmed finding that matches no label, diagnose emits four fields and names the pipeline stage responsible:

diagnose eval/fixtures/mixed/zipslip (3 planted, 2 confirmed):
  no-artifact src/extract.ts:41  bottleneck: exploit-verifier
    intended: confirm the planted path-traversal at src/extract.ts:41 as high/critical
    actual:   finding at src/extract.ts:41 recorded but has no execution artifact (P1 evidence gate)
    fix:      strengthen exploit-verifier: produce a P1 execution artifact reproducing the path-traversal at src/extract.ts:41
  roll-up: bottleneck = exploit-verifier (1 failures)

Failure categories

The roll-up names the stage with the most failures. Ties break toward the earliest stage in pipeline order (surface-mapper, exploit-verifier, cross-verify, severity-assessor), because a failure to enumerate the surface is a more fundamental fix than a failure to score what was enumerated.

diagnose reuses the same matching code as the score, so its true-positive/miss/spurious partition always agrees with S. The two verbs describe the same trial.

Fixtures

A fixture is a directory holding a labels.json:

{
  "vulns": [
    { "vulnClass": "path-traversal", "file": "src/extract.ts", "line": 41, "sink": "path.join" }
  ],
  "clean": ["src/safe-extract.ts"]
}

vulns are what the run is supposed to find. clean files are precision controls: a listed file must produce no sink hit, which is what stops a "find everything" strategy from scoring well.

Point --fixtures at either one fixture directory or a parent of several; in the parent case, every immediate subdirectory containing a labels.json is used, in sorted order, so pooled scores do not depend on filesystem enumeration order.

Fixtures are not shipped in the binary. This is a research tool; the corpus is yours.

error: no labeled fixtures under eval/fixtures/mixed (need a labels.json, or subdirs with one)

The loop

diagnose names the bottleneck → you change the prompt or the tooling → ab decides whether the change was real. Both verbs are deterministic given a seed, which is what makes the loop a measurement rather than an impression.

Next

Troubleshooting covers what to do when a run does not start, does not confine, or does not finish.