Module stats 

What a series of timings says, and whether two series differ.

Issue #179 records that this repository could not write the sentence a refactor most wants to write — “no statistically meaningful regression” — because cove-bench reported {min, mean, max} and nothing else. Three numbers are not a claim; they are three numbers a reader eyeballs against a band they remember.

Why the median and the quartiles

A wall-time series has a floor and no ceiling. The machine cannot run the benchmark faster than the machine runs it, and it can always run it slower — a descheduled turn, a migration between cores, a neighbour waking up. So the failure mode of a benchmark timing is a sample that is too large, and the mean is the statistic that moves furthest when one arrives. The median does not move at all until half the samples are affected, which is the property that makes it the right summary of a series taken on a machine that is not perfectly quiet.

That is the argument from the shape of the failure, and it is the one to rely on. The argument from skew is weaker than it looks, and it is recorded here because it was checked rather than assumed. The only distributions this repository has written down are the nine rows of the calling-convention matrix, taken at nine samples a row on a quiet machine and reported as {median, min, max}. That table measured the backend ADR 0034 deleted and went with it; docs/VM_ARCHITECTURE.md restates the counts below, and the table itself is in git history at commit 6e90085. If those series were reliably right-skewed then max - median would exceed median - min on most of them. It exceeds it on three rows, falls short on five, and ties on one. So “benchmark timings are right-skewed” is not a claim this data supports and is not what justifies the median here. What justifies it is that one bad sample must not be able to move the number a decision is made on — and that argument does not need the skew, because a statistic robust to an outlier in either direction is what a decision wants either way.

Two caveats on that check, because it is weaker than a real one. Three order statistics are not a distribution, and nine samples put both extremes deep in the tails where they are noisiest. A stronger check is now possible and was not before: wall_ns.samples records every timing, so whoever next takes a run on a quiet machine can look at the actual shape rather than at its extremes.

The mean, the minimum and the maximum are all still reported. The mean because ADR 0012 says wall time is reported as {min, mean, max} and a reader of that format must keep finding it; the extremes because they are the cheapest way to see that a series went wrong, and a summary that hides the one run that took four times as long is worse than no summary.

Why a comparison, and not just a spread

A spread on one run says how noisy that run was. It does not say whether this build is slower than that build, which is the actual question, and issue #126 is the reason it has to be asked against a fixed commit rather than against the parent: three changes each individually inside the noise summed to 19%. A comparison against whatever ran last cannot see that, and a comparison against a recorded baseline can.

So Comparison takes the baseline’s samples and this run’s samples and answers with the shift between their medians and an interval around it. The verdict is read off the interval: an interval that excludes zero is a difference that cleared the noise, and one that contains zero is not — and in that case the interval’s own width is the honest bound on what the change could have cost, which is the number to quote instead of claiming there was no effect.

Structs

Baseline

A previous run of this harness, read back as the samples it recorded.

Comparison

This run’s median against a baseline’s, and how sure of the difference the two series allow anyone to be.

Rng 🔒

SplitMix64, which is four lines and needs no dependency.

RowKey

Which row of a report a comparison is about.

Stats

A series of samples and the order statistics read off it.

Enums

Verdict

What a comparison concluded.

Constants

CONFIDENCE

The confidence the reported interval carries.

MIN_SAMPLES

How many samples a side needs before a comparison will call anything.

RESAMPLES 🔒

How many resamples the interval below is built from.

SEED 🔒

The seed the resampling starts from.

Functions

percentile_f64 🔒

The pth percentile of an ascending series of f64, by the same interpolation quantile uses.

quantile

The pth quantile of an ascending series, by linear interpolation between the two order statistics that bracket it.

string_field 🔒

The value of "<name>":"..." in a line this harness wrote.

wall_samples 🔒

The samples array inside this line’s wall_ns object.