Crate cove_bench 

cove-bench: the interpreter performance-gate harness for ADR 0012.

This is not a cove subcommand and makes no promise of a stable CLI surface. It loads the package under benches/ and runs each benchmark’s entry directly against cove-runtime – the same crate cove run and cove test are built on – so it measures the interpreter itself rather than shelling out and re-paying parse and resolve on every sample.

Every Host a benchmark reaches is the same deterministic fake cove test already grants by default (see crates/cove-cli/src/test.rs): a console that writes into a sink nobody reads, a clock whose VirtualTime never advances on its own, and every other host answering from empty in-memory state. Nothing here touches the network or the real filesystem, which is what keeps it hermetic and non-flaky.

The one exception is the startup benchmark, which measures a real process: it spawns the cove binary built alongside this one and times the whole exec-to-exit span, because process creation and binary loading are exactly what an in-process measurement cannot see.

Two backends

ADR 0019 says every number this harness reports must say which backend produced it, because a fuel_spent or an instructions figure carries no meaning on its own – it is only ever a fact about the backend that produced it. So every measurement below carries a backend, and every benchmark is measured on all of them.

There were four backends here at different points in this file’s history: the interpreter; the executable-IR VM ADR 0019 introduced; an experimental eight-byte-word frame over that same IR, added so the comparison issue #212 asked for could be made within one benchmark binary rather than across two builds; and the linear-memory backend ADR 0034 decided as the VM’s replacement. ADR 0034’s completion condition 8 is that the replacement becomes the production path and “the predecessor executable IR, Vm, FrameVm, admits mechanism, duplicate heap and migration machinery are deleted” – and that has now happened. Two backends are left.

ast is the tree-walking interpreter, and it is the oracle: it runs every construct the language has, straight off the checked program, with no lowering and no admission predicate of its own. vm is the linear-memory backend, and it is the production path: cove_ir has no admission predicate either, but for the opposite reason – a construct it has not been taught is a gap in the replacement rather than a program a subset declines, so a benchmark it cannot lower fails this suite instead of being reported as an experiment’s subset not having reached it yet.

cove_ir::lower_entry slices by reachability – what one entry reaches – which is what cove run --backend vm lowers, so the lowering is timed once per benchmark and reported under that benchmark’s name, apart from every execution: lowering happens once per program and execution happens for as long as the program does. That separation is the compile/lower breakdown issue #111 asked for.

Output

One JSON object per line on stdout, in the order the benchmarks are listed here — which is not necessarily the order they were timed in; see --sample-order below:

{"benchmark":"pure","kind":"lowering","backend":"vm","iterations":<u32>,"wall_ns":<series>,"functions":<usize>,"ok":<bool>}
... and one `vm` lowering line for each of the other benchmarks
{"benchmark":"pure","kind":"interpreter","backend":"ast","iterations":<u32>,"wall_ns":<series>,"fuel_spent":<u64>,"fuel_per_sec":<f64>,"heap_peak_bytes":<summary>,"host_calls":<u64>,"irreversible_writes":<u64>,"instructions":<u64|null>,"ok":<bool>}
{"benchmark":"pure","kind":"vm","backend":"vm", ...the same fields...}
{"benchmark":"pure","kind":"trace_overhead","backend":"ast","untraced_wall_ns":<u64>,"traced_wall_ns":<u64>,"overhead_ratio":<f64>}
{"benchmark":"pure","kind":"trace_overhead","backend":"vm", ...the same fields...}
{"benchmark":"hostheavy", ...the same lines...}
... and the same for each of `arith`, `arrayget`, `field`, `method`,
`call`, `chars`, and `callback`
{"benchmark":"startup","kind":"process","backend":"ast","iterations":<u32>,"wall_ns":<series>,"ok":<bool>}
{"benchmark":"startup","kind":"process","backend":"vm", ...the same fields...}

instructions is how many instructions the run executed, and null on the interpreter, which has none. It sits beside the wall time because ADR 0029 makes an exact count repeatable where an absolute is not: a wall_ns regression a rebuild did not cause and an instructions count that moved with it are one finding, and a wall_ns regression whose instructions count did not move at all is a different one – the count says whether vm got slower per instruction or started running more of them.

where <summary> and <series> are

<summary> = {"min":<u64>,"mean":<u64>,"max":<u64>,"p25":<f64>,"median":<f64>,"p75":<f64>,"iqr":<f64>}
<series>  = {...the same fields...,"samples":[<u64>, ...]}

min, mean and max are the three ADR 0012 named and they still mean what they meant. The quartiles are what issue #179 asks for: a spread a regression claim can be stated against instead of a band the reader is expected to remember. crates/cove-bench/src/stats.rs says why the median and the interquartile range rather than the mean and a standard deviation.

samples is every timing the run took, on the wall-time series alone, in the order the run took them. It is what turns a recorded run into a baseline: a summary can only be compared against another summary by arithmetic that invents the spread it needs, and the samples do not have to be invented. The order is what says when a slow sample arrived, which is the difference between a machine that drifted through a series and a benchmark that is noisy in it; nothing that compares two runs reads it, because every statistic here is an order statistic.

A benchmark the linear-memory lowering cannot lower reports that instead of its vm lines:

{"benchmark":"chars","kind":"unsupported","backend":"vm","what":"<what the lowering said>","ok":false}

and it fails the suite: cove_ir has no admission predicate, so a construct it has not been taught is a gap in the backend ADR 0034 makes the production one rather than a program a subset declines. ast never reports this line – the interpreter runs the checked program directly, with no lowering of its own to refuse anything.

kind keeps the value it has always had for the interpreter’s rows, so a reader of the older format still finds exactly the rows it was reading and does not silently start counting the others. backend is what now says which of the two produced a number.

ok is false when a benchmark’s entry returned Err, a backend itself failed, the lowering was refused, or (for startup) the spawned process exited non-zero. A caller comparing two backends, or either against a recorded baseline, should refuse numbers from a run that is not ok; this harness’s own --baseline does, and compares no row that is not ok.

Comparing against a recorded run

--baseline <path> reads a file of the output above and adds one line per row it recognizes:

{"benchmark":"field","kind":"comparison","of":"vm","backend":"vm","baseline_median_ns":<f64>,"median_ns":<f64>,"delta_pct":<f64>,"ci_low_pct":<f64|null>,"ci_high_pct":<f64|null>,"confidence":0.95,"verdict":"<verdict>"}

kind is comparison rather than the kind of the row compared, again so that a reader filtering on kind keeps finding what it was finding; of is the kind this line is about. The verdict is one of regression, improvement, inside the noise, or underpowered, and it is read off the interval: an interval that excludes zero cleared the noise and one that contains it did not. A summary of the whole comparison goes to stderr, so stdout stays one JSON object per line.

The baseline is a fixed commit, not the parent. That is the discipline issue #126 exists to enforce: three changes each individually inside the noise summed to a 19% regression, and only a comparison against a commit far enough back could have seen it.

git worktree add /tmp/base <the fixed commit>
cargo build --release -p cove-cli -p cove-bench   # in /tmp/base
/tmp/base/target/release/cove-bench --iterations 15 > /tmp/base.jsonl
cargo build --release -p cove-cli -p cove-bench   # here
./target/release/cove-bench --iterations 15 --baseline /tmp/base.jsonl

Bracket the variant, do not pair it. Run the base binary, then the variant, then the base binary again, and quote the variant against the mean of the two base runs. The two base runs’ disagreement with each other is the measurement’s own error bar, it costs one extra run, and it should be quoted beside the result – where it is as large as the effect, that is the result.

Compare one row against itself. Never the largest row of the suite. Twenty-two suites in which nothing under test changed, measured for issue #205, put a single row’s disagreement with itself at 0.5% to 0.8% in the middle and 2% to 3% at the 90th percentile. The largest disagreement over the suite’s twenty-one rows is a different statistic with a different distribution: on that same null its median is about 4% and it reaches 15%. docs/VM_ARCHITECTURE.md’s earlier “7.4% against itself” was that statistic, so it is not the error bar for any row and no row should be read against it.

Two rows are not evidence at the few-percent level, and never were. benches/lowering times a 0.13 ms lowering and startup times a spawned process; between them they carry the suite’s largest null shift two thirds of the time. Read the execution rows.

Nothing about this makes a comparison across two machines, two build profiles, or two busy afternoons meaningful. It compares the samples it is given; whether they were taken on a quiet machine is the reader’s to answer, and it is the assumption every table in docs/VM_ARCHITECTURE.md rests on.

A regression verdict does not fail the process. ADR 0012’s argument for gating no wall-clock number in CI is unaffected by this: the exit code still reports correctness alone.

The mechanism benchmarks

pure, hostheavy, and startup are ADR 0012’s, and measure a program. arith, arrayget, field, method, call, and chars are issue #104’s, and measure one mechanism each: every one of them is the same 2,000,000-iteration loop with exactly one thing added, so the difference between two of them is what that thing costs. arith is the loop alone; arrayget adds an indexed read and the Option it answers; field adds a struct field; method adds a call around that field; call adds a call with no receiver; and chars is the per-character scan examples/cq spends nearly all of its time in.

callback is issue #193’s, and belongs to the same family: 2,000,000 invocations again, but of a closure reached through a higher-order builtin — filter, over an array, with the callback a helper builds over one capture. It is read beside call, which makes the same number of entries into a body through the call instruction instead, so the difference between the two is what re-entering the evaluator from inside a builtin costs. That route had no row before, which is why the per-call argument vector #184 removed from the builtin path survived on the callback path with nothing to price it.

They exist because a wall-clock number for a whole program says how slow it is and not what is slow about it. They do not replace the application measurement in examples/cq/README.md; they are what makes it readable.

This tool asserts no thresholds of its own; see ADR 0012 for why wall-clock numbers are not gated in CI, and for the thresholds a human applies when reading a --iterations-heavy local run.

Running it

cargo build --release --workspace
./target/release/cove-bench --iterations 1      # what CI runs, for correctness
./target/release/cove-bench --iterations 15     # a real local measurement
./target/release/cove-bench --iterations 15 --sample-order blocked
./target/release/cove-bench --matrix --backend ast,vm --iterations 9

--release is the profile to measure under. The workspace also defines [profile.bench-stable], which is release with codegen-units = 1, and it is not the one to reach for: it was added to test whether one codegen unit per crate would stop module boundaries being a performance variable (issue #179), it was measured against a never-executed Inst variant, and it did not – the spurious shift came out larger under it than under plain release, for 44% to 96% more build time. docs/VM_ARCHITECTURE.md, “What codegen-units = 1 was measured to be worth”, is the round. It stays defined so that result can be reproduced; nothing selects it.

Optimized in both cases. The benchmarks are sized to be measurable in an optimized build, so an unoptimized one does not run them uniformly slower in some way that could be divided back out — it runs them for minutes.

--iterations is how many samples a benchmark’s series has, and there is deliberately no second flag beside it: the runs the spread is computed over and the runs the harness performs are the same runs. So a run at --iterations 1 reports a series of one, whose median is its only sample and whose interquartile range is zero — which is exactly what CI wants and costs it nothing, and is why it stays at one. Six is the fewest samples any comparison here will draw a conclusion from, and docs/VM_ARCHITECTURE.md takes its tables at fifteen.

--sample-order is when those samples are taken, and the default is round-robin: one sample of every row, then a second of every row, so that each row’s series is spread over the whole suite rather than taken at one instant of it. blocked is the order this harness used before issue #205 — every sample of a row before the next row starts — and it is kept so the round that changed the default can be reproduced. Neither costs more than the other: the suite takes the same 564 seconds, runs the same runs, and reports the same fields. At --iterations 1 the two are the same sequence, so CI is unaffected.

Reading one backend against another is what the output is arranged for: the wall_ns medians of one benchmark are the comparison, and the fuel_spent beside them is not, because ADR 0019 makes fuel backend-specific and says so. instructions is not either, for a simpler reason: it is null on ast, which has none, so with only ast and vm left there is no second lowered backend’s count to divide vm’s by. It stays beside wall_ns anyway, for the reason given above – an exact count is worth reading run over run even with nothing beside it to ratio it against.

Modules

stats 🔒

What a series of timings says, and whether two series differ.

Structs

Compared 🔒

One row that had a baseline to be read against.

ExecutionReport 🔒

One benchmark’s execution report on one backend: wall time, fuel spent, and the heap’s peak live bytes.

LinearLowering 🔒

What lowering one benchmark to cove_ir cost, and the program every vm measurement of that benchmark runs.

MatrixRow 🔒

Runs the matrix and prints it as a table.

NotLowered 🔒

One benchmark the linear-memory lowering could not lower, and what it said.

Row 🔒

One benchmark on one backend, and the samples taken of it so far.

RunMeasurement 🔒

What one run of a benchmark’s entry measured.

StartupReport 🔒

Process-level startup: spawns the real cove binary and times the whole exec-to-exit span, which is what an in-process measurement cannot see.

TraceOverheadReport 🔒

Compares one benchmark run untraced against the same run under a real JsonlSink writing nowhere: the difference is tracing’s own cost, not the cost of whatever the sink’s destination happens to be.

Enums

Backend 🔒

Which backend produced a number.

SampleOrder 🔒

The order the suite takes its samples in.

Constants

BENCHMARKS 🔒

The benchmarks the suite runs, in the order their rows are reported.

DEFAULT_ITERATIONS 🔒

How many times each benchmark runs when --iterations is not given.

DEFAULT_SAMPLE_ORDER 🔒

The order a run uses when --sample-order is not given.

MATRIX 🔒

The rows of the calling-convention matrix, and what each one is.

MATRIX_TURNS 🔒

How many turns each row of the matrix takes. Every entry writes the same literal, and the table below divides by it to report a cost per turn.

Functions

bench 🔒

Runs every benchmark and reports each one as a line of JSON.

bench_linear_lowering 🔒

Lowers one benchmark’s entry to cove_ir iterations times.

bench_startup 🔒

The startup a --backend vm process pays is the one this measures, which is the point of measuring it here rather than in-process: the lowering is part of what a vm run costs before it does any work, and a process is where every such cost is paid at once.

bench_trace_overhead 🔒

Times one row untraced and then traced, back to back.

benches_root 🔒

The benches/ package, rooted next to this crate.

compare 🔒

Emits the comparison for one row, when there is a baseline and it has the row.

cove_binary 🔒

The cove binary built alongside this one.

entry_err_message 🔒

Some(message) when value is the Err an entry returned; None for Ok or an entry that returns bare ().

entry_for 🔒

The module, function name, and granted capabilities for [run.<name>], looked up the way cove run looks up a run.

escape 🔒

Escapes what a JSON string may not carry literally.

fake_hosts 🔒

The same deterministic fakes cove test grants by default (see crates/cove-cli/src/test.rs), always chosen here: a Host-heavy benchmark measures dispatch, grant checks, and budget accounting through them, never real I/O latency and never the network.

finish 🔒

Reads the counters a run leaves behind and says whether it passed.

load_baseline 🔒

Reads --baseline <path>, if it was given.

load_benches 🔒

main 🔒

matrix 🔒

parse_iterations 🔒

Reads --iterations <n> from the process arguments, falling back to DEFAULT_ITERATIONS when it is absent or not a positive integer.

parse_matrix_backends 🔒

Reads --backend <list> for the matrix, defaulting to vm alone.

parse_sample_order 🔒

Reads --sample-order <blocked|round-robin> from the process arguments.

render_all 🔒

run_once 🔒

Builds a fresh registry, budget, and backend – exactly what cove run builds for one run – and calls module.entry once under trace.

summarize 🔒

Ends a compared run with the sentence the JSON above is the evidence for.

take_samples 🔒

Fills every row’s series, in the order order asks for.