REFERENCE · DESIGN PREVIEW
The audit CLI
One command with one job: read a trace export on your disk, write an audit report next to it. This page documents the designed interface — it is the contract the build is held to, published before the build so you can hold it to the contract too.
coldworks-audit run
coldworks-audit run <PATH> [--docs DIR] [--diff] [--sample] [--open] [--format NAME]
Audits the export at PATH — a file, or a directory whose newest exports are
audited together with span-level dedup across overlapping files.
| Argument | What it does | Status |
|---|---|---|
PATH | The export file or directory. Format is detected
(Langfuse, LangSmith, OTel GenAI); override with --format. |
DESIGNED |
--docs DIR | Anchor recovered judgments to lines in your own documents. Files are hashed and matched locally; anchors carry the provenance line "matched by trial, not by export-carried reference." Multi-document traces are counted unanchorable with their own reason code. | DESIGNED |
--diff | Report what changed since the last run: new repeated judgments, from the local ledger. | DESIGNED |
--sample | Run on the bundled sample export and documents — a full report with nothing of yours involved, labeled as sample data on every screen. | DESIGNED |
--open | Open the HTML report when the run finishes. | DESIGNED |
--from langfuse | Skip the manual export: page the Langfuse
observations API with LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY
from your environment, incremental via --since. Your credential, your
machine. |
PLANNED · AFTER FILE PATH |
What the headline counts
- Tool calls are grouped by (tool name, arguments digest) — exact repetition, not similarity.
- A repeat group is a tier-0 cache candidate only when its results agree across every occurrence. Disagreeing groups are reported separately as nondeterminism findings.
- Where the export carries token usage, the headline is token-weighted; otherwise it stays in span counts and says which it is.
- What the audit could not read is printed, not dropped: judgments stated only in prose, and unanchorable spans, each with a count and a reason.
Files it reads and writes
| File | Direction | What it is |
|---|---|---|
<PATH> | reads | Your export. Never modified, never transmitted. |
--docs directory | reads | Your documents. Hashed locally for anchoring. |
./coldworks-audit.html | writes | The report. Self-contained, built to be forwarded — by you. |
./.coldworks/audit/ledger.jsonl | appends | Derived figures per run, append-only. What --diff reads. Delete it any time; you lose history, nothing else. |
Network calls made by
run: zero. The one planned
exception is --from langfuse, which calls Langfuse — your trace store — with
your key, from your machine. Nothing calls Coldworks.Exit codes
| Code | Meaning |
|---|---|
0 | Audit completed — including an honest zero with a field census. |
1 | The export could not be parsed at all; the error names the first unreadable record. |
2 | Usage error; help printed. |