rickub
nandi/clonimpublic Fork
⑂ d48248e
Branches
main
Tags
v0.1.0 v0.1.1 v0.1.10 v0.1.11 v0.1.12 v0.1.13 v0.1.14 v0.1.15 v0.1.16 v0.1.17 v0.1.2 v0.1.3 v0.1.4 v0.1.5 v0.1.6 v0.1.7 v0.1.8 v0.1.9 View all branches → View all tags →
Commits
⬇ Clone ▾
git clone https://git.rickub.com/nandi/clonim.git
git clone ssh://git@rickub.com/nandi/clonim.git

Host key fingerprint (ed25519): SHA256:iycHnxEyq0Q7uyVpB7JlznP0G7JrTPXLYRcAU5CSLhc — verify it before your first connect.

Download ZIP

bench: store Jolt comparisons and detect regressions

nandithebull committed 2026-09-20T20:35:24-07:00 Browse files
d48248e parent: bdb2eb3
modified .gitignore +1 -0
@@ -3,3 +3,4 @@ nimcache/
33 .clj-kondo/.cache/
44 .lsp/
55 bench/bin/
6+__pycache__/
@@ -3,3 +3,4 @@ nimcache/
3 .clj-kondo/.cache/3 .clj-kondo/.cache/
4 .lsp/4 .lsp/
5 bench/bin/5 bench/bin/
6+__pycache__/
modified README.md +5 -0
@@ -18,6 +18,11 @@ nim c --hints:off -o:bin/clonim src/clonim.nim # build the compiler
1818 There is a `justfile` too: `just build`, `just test`, `just bench`,
1919 `just run <file>`, `just emit <file>`, `just accept` (re-record expectations).
2020
21+`just bench-store` benchmarks against jolt-lang, commits a timestamped result,
22+and flags slowdowns above 10% versus the latest compatible run. Requires Jolt
23+and a clean working tree. Use `just bench-store 15` to change the threshold;
24+see [benchmark history](bench/README.md) for metrics and exit codes.
25+
2126 ## Pipeline
2227
2328 | stage | file | what it does |
@@ -18,6 +18,11 @@ nim c --hints:off -o:bin/clonim src/clonim.nim # build the compiler
18 There is a `justfile` too: `just build`, `just test`, `just bench`,18 There is a `justfile` too: `just build`, `just test`, `just bench`,
19 `just run <file>`, `just emit <file>`, `just accept` (re-record expectations).19 `just run <file>`, `just emit <file>`, `just accept` (re-record expectations).
20 20
21+`just bench-store` benchmarks against jolt-lang, commits a timestamped result,
22+and flags slowdowns above 10% versus the latest compatible run. Requires Jolt
23+and a clean working tree. Use `just bench-store 15` to change the threshold;
24+see [benchmark history](bench/README.md) for metrics and exit codes.
25+
21 ## Pipeline26 ## Pipeline
22 27
23 | stage | file | what it does |28 | stage | file | what it does |
added bench/README.md +91 -0
new file mode 100644
@@ -0,0 +1,91 @@
1+# Stored benchmarks
2+
3+`just bench-store` invokes `python3 bench/store.py`, with an optional percentage
4+threshold (default **10**):
5+
6+```sh
7+just bench-store
8+just bench-store 15
9+# Equivalent direct invocation:
10+python3 bench/store.py 15
11+```
12+
13+The task builds release binaries for clonim and jolt-lang using the existing
14+`measure jolt` recipe. Recording uses only Python's standard library.
15+
16+## Requirements and recording
17+
18+- Start with a **clean Git working tree and index, including untracked files**.
19+ Commit, stash, or remove outstanding work yourself first. Ignored build output
20+ does not count as dirty.
21+- `git`, `just`, `nim`, and `jolt` must be available on `PATH`. Missing jolt is an
22+ error, not permission to fall back to a clonim-only run.
23+- Run on an otherwise idle machine. Do not edit compiler sources or run another
24+ benchmark/recording process concurrently.
25+
26+The script invokes `just measure jolt` from the repository root, setting
27+`BENCH_JSON` to an absolute path in a temporary directory. The producer must write
28+JSON with `schema_version: 1`, `rounds: 11`, and `samples_ms`, `best_ms`, and
29+`compute_ms` mappings keyed by tool (`clonim`, `jolt`) then benchmark (`hello`,
30+`fib`, `loop`, `seqs`). Each raw sample array has 11 nonnegative millisecond
31+values, `best_ms` is its minimum, and `compute_ms` is best minus that tool's hello
32+best. The identically-zero hello compute entry may be omitted. Missing, failed,
33+or malformed measurements are not recorded.
34+
35+A unique UTC timestamp plus UUID names each `bench/results/*.json` record. It
36+contains the raw results, the source Git revision, machine/OS/hostname/CPU
37+information, full `nim --version` and `jolt --version` output, SHA-256 hashes of
38+all `bench/*.clj`, `bench/measure.py`, and `justfile`, and the baseline comparison.
39+Hashing `justfile` also captures changes to the benchmark build commands.
40+
41+The newest compatible valid record by UTC creation time is the baseline.
42+Compatibility requires identical machine information, tool versions, methodology,
43+and input hashes; **source revision is deliberately not a compatibility key**,
44+so compiler changes can be compared. Malformed and incompatible records are
45+skipped. With no compatible baseline, the run establishes one without an alert.
46+
47+## Metrics and exit status
48+
49+Regression checks use clonim's **best total wall-clock milliseconds**, including
50+`hello` startup, independently for all four benchmarks. An increase **strictly
51+greater than** the threshold flags a regression. Jolt results and startup-adjusted
52+compute times are retained for context, not used to trigger alerts. The report
53+and record contain absolute and percentage deltas and a regression flag. If a
54+baseline is zero, any positive value is a regression and its undefined percentage
55+is stored as JSON `null` (zero to zero is 0%).
56+
57+These are noisy performance alerts, not statistical significance tests.
58+Best-of-11 reduces scheduling interference but cannot remove thermal, frequency,
59+background-load, or toolchain variability. Confirm unexpected changes with
60+additional runs before drawing conclusions. A recorded regression can itself
61+become the next baseline; comparisons are to the latest run, not a fixed target.
62+
63+After writing the record, the script stages **only that new file** and runs
64+`git commit --only -m ... -- path` with `GIT_EDITOR=true`. Unrelated changes that
65+appear during the run are neither staged nor included by the script; unrelated
66+index entries are preserved. Git hooks still run normally. The original clean
67+HEAD is stored as `source_revision`, not the subsequent record commit. A changed
68+HEAD or changed benchmark inputs/environment during measurement aborts recording.
69+Concurrent compiler-source edits are unsupported: the initial clean-tree check
70+is not a filesystem snapshot.
71+
72+Exit codes:
73+
74+- **0**: record committed, no regression (or no compatible baseline).
75+- **1**: regression **recorded and committed first**, then reported as failure.
76+- **2**: precondition, measurement, validation, or Git failure. If staging or
77+ committing fails, the JSON file remains for inspection/recovery; the script
78+ does not reset, delete, or roll back unrelated work. Resolve the Git failure
79+ and commit the retained record explicitly before retrying.
80+
81+The record commit provides an auditable link to measured source; it does not
82+claim reproducible timings across machines or capture every external dependency.
83+
84+## Tests
85+
86+```sh
87+python3 -m unittest discover -s bench -p 'test_store.py' -v
88+```
89+
90+Tests use temporary directories and mock benchmark/Git execution; they never run
91+real benchmarks or create real commits.
new file mode 100644
@@ -0,0 +1,91 @@
1+# Stored benchmarks
2+
3+`just bench-store` invokes `python3 bench/store.py`, with an optional percentage
4+threshold (default **10**):
5+
6+```sh
7+just bench-store
8+just bench-store 15
9+# Equivalent direct invocation:
10+python3 bench/store.py 15
11+```
12+
13+The task builds release binaries for clonim and jolt-lang using the existing
14+`measure jolt` recipe. Recording uses only Python's standard library.
15+
16+## Requirements and recording
17+
18+- Start with a **clean Git working tree and index, including untracked files**.
19+ Commit, stash, or remove outstanding work yourself first. Ignored build output
20+ does not count as dirty.
21+- `git`, `just`, `nim`, and `jolt` must be available on `PATH`. Missing jolt is an
22+ error, not permission to fall back to a clonim-only run.
23+- Run on an otherwise idle machine. Do not edit compiler sources or run another
24+ benchmark/recording process concurrently.
25+
26+The script invokes `just measure jolt` from the repository root, setting
27+`BENCH_JSON` to an absolute path in a temporary directory. The producer must write
28+JSON with `schema_version: 1`, `rounds: 11`, and `samples_ms`, `best_ms`, and
29+`compute_ms` mappings keyed by tool (`clonim`, `jolt`) then benchmark (`hello`,
30+`fib`, `loop`, `seqs`). Each raw sample array has 11 nonnegative millisecond
31+values, `best_ms` is its minimum, and `compute_ms` is best minus that tool's hello
32+best. The identically-zero hello compute entry may be omitted. Missing, failed,
33+or malformed measurements are not recorded.
34+
35+A unique UTC timestamp plus UUID names each `bench/results/*.json` record. It
36+contains the raw results, the source Git revision, machine/OS/hostname/CPU
37+information, full `nim --version` and `jolt --version` output, SHA-256 hashes of
38+all `bench/*.clj`, `bench/measure.py`, and `justfile`, and the baseline comparison.
39+Hashing `justfile` also captures changes to the benchmark build commands.
40+
41+The newest compatible valid record by UTC creation time is the baseline.
42+Compatibility requires identical machine information, tool versions, methodology,
43+and input hashes; **source revision is deliberately not a compatibility key**,
44+so compiler changes can be compared. Malformed and incompatible records are
45+skipped. With no compatible baseline, the run establishes one without an alert.
46+
47+## Metrics and exit status
48+
49+Regression checks use clonim's **best total wall-clock milliseconds**, including
50+`hello` startup, independently for all four benchmarks. An increase **strictly
51+greater than** the threshold flags a regression. Jolt results and startup-adjusted
52+compute times are retained for context, not used to trigger alerts. The report
53+and record contain absolute and percentage deltas and a regression flag. If a
54+baseline is zero, any positive value is a regression and its undefined percentage
55+is stored as JSON `null` (zero to zero is 0%).
56+
57+These are noisy performance alerts, not statistical significance tests.
58+Best-of-11 reduces scheduling interference but cannot remove thermal, frequency,
59+background-load, or toolchain variability. Confirm unexpected changes with
60+additional runs before drawing conclusions. A recorded regression can itself
61+become the next baseline; comparisons are to the latest run, not a fixed target.
62+
63+After writing the record, the script stages **only that new file** and runs
64+`git commit --only -m ... -- path` with `GIT_EDITOR=true`. Unrelated changes that
65+appear during the run are neither staged nor included by the script; unrelated
66+index entries are preserved. Git hooks still run normally. The original clean
67+HEAD is stored as `source_revision`, not the subsequent record commit. A changed
68+HEAD or changed benchmark inputs/environment during measurement aborts recording.
69+Concurrent compiler-source edits are unsupported: the initial clean-tree check
70+is not a filesystem snapshot.
71+
72+Exit codes:
73+
74+- **0**: record committed, no regression (or no compatible baseline).
75+- **1**: regression **recorded and committed first**, then reported as failure.
76+- **2**: precondition, measurement, validation, or Git failure. If staging or
77+ committing fails, the JSON file remains for inspection/recovery; the script
78+ does not reset, delete, or roll back unrelated work. Resolve the Git failure
79+ and commit the retained record explicitly before retrying.
80+
81+The record commit provides an auditable link to measured source; it does not
82+claim reproducible timings across machines or capture every external dependency.
83+
84+## Tests
85+
86+```sh
87+python3 -m unittest discover -s bench -p 'test_store.py' -v
88+```
89+
90+Tests use temporary directories and mock benchmark/Git execution; they never run
91+real benchmarks or create real commits.
modified bench/measure.py +30 -4
@@ -14,7 +14,14 @@ Compute is reported as total minus the `hello` time for the same toolchain,
1414 which is how a 0.5 ms native start is kept from flattering clonim against a
1515 runtime that pays ~105 ms to boot before it reaches main.
1616 """
17-import collections, subprocess, sys, time
17+
18+import collections
19+import json
20+import os
21+import subprocess
22+import sys
23+import time
24+from pathlib import Path
1825
1926 BENCHES = ["fib", "loop", "seqs"]
2027 ROUNDS = 11
@@ -22,7 +29,7 @@ ROUNDS = 11
2229
2330 def timed(cmd):
2431 start = time.perf_counter()
25- subprocess.run(cmd, capture_output=True)
32+ subprocess.run(cmd, capture_output=True, check=True)
2633 return (time.perf_counter() - start) * 1000
2734
2835
@@ -33,13 +40,32 @@ def main(argv):
3340 suites["jolt"] = {b: [f"bench/bin/jolt-{b}"] for b in ["hello"] + BENCHES}
3441
3542 best = collections.defaultdict(lambda: float("inf"))
43+ samples = {label: {b: [] for b in cmds} for label, cmds in suites.items()}
3644 for label, cmds in suites.items(): # warm the page cache first
3745 for cmd in cmds.values():
38- subprocess.run(cmd, capture_output=True)
46+ subprocess.run(cmd, capture_output=True, check=True)
3947 for _ in range(ROUNDS):
4048 for label, cmds in suites.items():
4149 for bench, cmd in cmds.items():
42- best[label, bench] = min(best[label, bench], timed(cmd))
50+ elapsed = timed(cmd)
51+ samples[label][bench].append(elapsed)
52+ best[label, bench] = min(best[label, bench], elapsed)
53+
54+ if output := os.environ.get("BENCH_JSON"):
55+ result = {
56+ "schema_version": 1,
57+ "rounds": ROUNDS,
58+ "samples_ms": samples,
59+ "best_ms": {
60+ label: {b: best[label, b] for b in cmds}
61+ for label, cmds in suites.items()
62+ },
63+ "compute_ms": {
64+ label: {b: best[label, b] - best[label, "hello"] for b in BENCHES}
65+ for label in suites
66+ },
67+ }
68+ Path(output).write_text(json.dumps(result, indent=2) + "\n")
4369
4470 width = max(len(l) for l in suites)
4571 print(f"{'':<{width}} {'startup':>9} " + " ".join(f"{b:>8}" for b in BENCHES))
@@ -14,7 +14,14 @@ Compute is reported as total minus the `hello` time for the same toolchain,
14 which is how a 0.5 ms native start is kept from flattering clonim against a14 which is how a 0.5 ms native start is kept from flattering clonim against a
15 runtime that pays ~105 ms to boot before it reaches main.15 runtime that pays ~105 ms to boot before it reaches main.
16 """16 """
17-import collections, subprocess, sys, time17+
18+import collections
19+import json
20+import os
21+import subprocess
22+import sys
23+import time
24+from pathlib import Path
18 25
19 BENCHES = ["fib", "loop", "seqs"]26 BENCHES = ["fib", "loop", "seqs"]
20 ROUNDS = 1127 ROUNDS = 11
@@ -22,7 +29,7 @@ ROUNDS = 11
22 29
23 def timed(cmd):30 def timed(cmd):
24 start = time.perf_counter()31 start = time.perf_counter()
25- subprocess.run(cmd, capture_output=True)32+ subprocess.run(cmd, capture_output=True, check=True)
26 return (time.perf_counter() - start) * 100033 return (time.perf_counter() - start) * 1000
27 34
28 35
@@ -33,13 +40,32 @@ def main(argv):
33 suites["jolt"] = {b: [f"bench/bin/jolt-{b}"] for b in ["hello"] + BENCHES}40 suites["jolt"] = {b: [f"bench/bin/jolt-{b}"] for b in ["hello"] + BENCHES}
34 41
35 best = collections.defaultdict(lambda: float("inf"))42 best = collections.defaultdict(lambda: float("inf"))
43+ samples = {label: {b: [] for b in cmds} for label, cmds in suites.items()}
36 for label, cmds in suites.items(): # warm the page cache first44 for label, cmds in suites.items(): # warm the page cache first
37 for cmd in cmds.values():45 for cmd in cmds.values():
38- subprocess.run(cmd, capture_output=True)46+ subprocess.run(cmd, capture_output=True, check=True)
39 for _ in range(ROUNDS):47 for _ in range(ROUNDS):
40 for label, cmds in suites.items():48 for label, cmds in suites.items():
41 for bench, cmd in cmds.items():49 for bench, cmd in cmds.items():
42- best[label, bench] = min(best[label, bench], timed(cmd))50+ elapsed = timed(cmd)
51+ samples[label][bench].append(elapsed)
52+ best[label, bench] = min(best[label, bench], elapsed)
53+
54+ if output := os.environ.get("BENCH_JSON"):
55+ result = {
56+ "schema_version": 1,
57+ "rounds": ROUNDS,
58+ "samples_ms": samples,
59+ "best_ms": {
60+ label: {b: best[label, b] for b in cmds}
61+ for label, cmds in suites.items()
62+ },
63+ "compute_ms": {
64+ label: {b: best[label, b] - best[label, "hello"] for b in BENCHES}
65+ for label in suites
66+ },
67+ }
68+ Path(output).write_text(json.dumps(result, indent=2) + "\n")
43 69
44 width = max(len(l) for l in suites)70 width = max(len(l) for l in suites)
45 print(f"{'':<{width}} {'startup':>9} " + " ".join(f"{b:>8}" for b in BENCHES))71 print(f"{'':<{width}} {'startup':>9} " + " ".join(f"{b:>8}" for b in BENCHES))
added bench/store.py +222 -0
new file mode 100644
@@ -0,0 +1,222 @@
1+
2+"""Record and commit a benchmark run; exit 1 after committing a regression."""
3+
4+import argparse
5+
6+import hashlib
7+import json
8+import math
9+import os
10+
11+import platform
12+import shutil
13+import subprocess
14+import sys
15+import tempfile
16+import uuid
17+from datetime import datetime, timezone
18+from pathlib import Path
19+
20+ROOT = Path(__file__).resolve().parent.parent
21+BENCHES = ("hello", "fib", "loop", "seqs")
22+LABELS = ("clonim", "jolt")
23+
24+
25+def command(root, *args, env=None):
26+ return subprocess.check_output(args, cwd=root, env=env, text=True).strip()
27+
28+
29+def require_clean(root):
30+ if command(root, "git", "status", "--porcelain", "--untracked-files=all"):
31+ raise ValueError("working tree must be clean, including untracked files")
32+
33+
34+def cpu_info():
35+ try:
36+ for line in Path("/proc/cpuinfo").read_text().splitlines():
37+ key, sep, value = line.partition(":")
38+ if sep and key.strip() in ("model name", "Hardware"):
39+ return value.strip()
40+ except OSError:
41+ pass
42+ return platform.processor() or "unknown"
43+
44+
45+def compatibility(root):
46+ paths = sorted((root / "bench").glob("*.clj"))
47+ paths += [root / "bench/measure.py", root / "justfile"]
48+ return {
49+ "machine": {
50+ "system": platform.system(),
51+ "release": platform.release(),
52+ "machine": platform.machine(),
53+ "hostname": platform.node(),
54+ "cpu": cpu_info(),
55+ "logical_cpus": os.cpu_count(),
56+ },
57+ "tool_versions": {
58+ tool: command(root, tool, "--version") for tool in ("nim", "jolt")
59+ },
60+ "methodology": {
61+ "schema_version": 1,
62+ "rounds": 11,
63+ "metric": "best_ms",
64+ "schedule": "round-robin",
65+ },
66+ "sha256": {
67+ str(path.relative_to(root)): hashlib.sha256(path.read_bytes()).hexdigest()
68+ for path in paths
69+ },
70+ }
71+
72+
73+def number(value):
74+ return type(value) in (int, float) and math.isfinite(value)
75+
76+
77+def validate_results(results):
78+ if not isinstance(results, dict):
79+ # Invalid JSON measurements consistently raise ValueError for callers.
80+ raise ValueError("measurement must be a JSON object") # noqa: TRY004
81+ if results.get("schema_version") != 1 or results.get("rounds") != 11:
82+ raise ValueError("expected measurement schema_version=1 and rounds=11")
83+ try:
84+ for label in LABELS:
85+ for bench in BENCHES:
86+ samples = results["samples_ms"][label][bench]
87+ best = results["best_ms"][label][bench]
88+ if (not isinstance(samples, list) or len(samples) != 11
89+ or not all(number(v) and v >= 0 for v in samples)
90+ or not number(best) or best != min(samples)):
91+ raise ValueError(f"invalid samples/best for {label}/{bench}")
92+ # Some producers omit the identically-zero hello compute metric.
93+ if bench != "hello" or bench in results["compute_ms"][label]:
94+ compute = results["compute_ms"][label][bench]
95+ expected = best - results["best_ms"][label]["hello"]
96+ if not number(compute) or not math.isclose(
97+ compute, expected, rel_tol=1e-9, abs_tol=1e-9):
98+ raise ValueError(f"invalid compute for {label}/{bench}")
99+ except (KeyError, TypeError) as exc:
100+ raise ValueError("missing or invalid measurement metrics") from exc
101+
102+
103+def latest_baseline(directory, compatible):
104+ candidates = []
105+ for path in directory.glob("*.json"):
106+ try:
107+ record = json.loads(path.read_text())
108+ if record["schema_version"] != 1 or record["compatibility"] != compatible:
109+ continue
110+ validate_results(record["results"])
111+ created = datetime.fromisoformat(record["created_at"])
112+ if created.utcoffset() is None:
113+ continue
114+ candidates.append((created, path.name, record))
115+ except (OSError, ValueError, KeyError, TypeError):
116+ continue
117+ if not candidates:
118+ return None
119+ _, name, record = max(candidates, key=lambda item: (item[0], item[1]))
120+ return name, record
121+
122+
123+def compare(results, baseline, threshold):
124+ comparison = {"baseline": None, "threshold_percent": threshold,
125+ "deltas": {}, "regression": False}
126+ if baseline is None:
127+ return comparison
128+ name, prior = baseline
129+ comparison["baseline"] = name
130+ for bench in BENCHES:
131+ old = prior["results"]["best_ms"]["clonim"][bench]
132+ new = results["best_ms"]["clonim"][bench]
133+ delta = new - old
134+ percent = 100 * delta / old if old else (0.0 if new == 0 else None)
135+ regression = delta > old * threshold / 100
136+ comparison["deltas"][bench] = {
137+ "baseline_ms": old, "current_ms": new, "delta_ms": delta,
138+ "percent": percent, "regression": regression,
139+ }
140+ comparison["regression"] |= regression
141+ return comparison
142+
143+
144+def report(comparison):
145+ if comparison["baseline"] is None:
146+ print("No compatible baseline; recording initial run.")
147+ else:
148+ print(f"Baseline: {comparison['baseline']}")
149+ for bench, delta in comparison["deltas"].items():
150+ percent = delta["percent"]
151+ change = f"{percent:+.2f}%" if percent is not None else "increase from zero"
152+ print(f" {bench}: {delta['delta_ms']:+.3f} ms ({change})"
153+ + (" REGRESSION" if delta["regression"] else ""))
154+ print(f"Regression: {comparison['regression']}")
155+
156+
157+def store(root, threshold):
158+ require_clean(root)
159+ for tool in ("jolt", "nim", "just"):
160+ if shutil.which(tool) is None:
161+ raise ValueError(f"required executable not found: {tool}")
162+ revision = command(root, "git", "rev-parse", "HEAD")
163+ compatible = compatibility(root)
164+ directory = root / "bench/results"
165+ baseline = latest_baseline(directory, compatible)
166+ with tempfile.TemporaryDirectory(prefix="clonim-bench-") as work:
167+ output = Path(work).resolve() / "measurement.json"
168+ env = dict(os.environ, BENCH_JSON=str(output))
169+ subprocess.run(["just", "measure", "jolt"], cwd=root, env=env, check=True)
170+ results = json.loads(output.read_text())
171+ validate_results(results)
172+ if command(root, "git", "rev-parse", "HEAD") != revision:
173+ raise ValueError("HEAD changed during measurement; refusing to record")
174+ if compatibility(root) != compatible:
175+ raise ValueError("benchmark inputs or environment changed during measurement")
176+ comparison = compare(results, baseline, threshold)
177+ now = datetime.now(timezone.utc)
178+ record = {
179+ "schema_version": 1,
180+ "created_at": now.isoformat(),
181+ "source_revision": revision,
182+ "compatibility": compatible,
183+ "results": results,
184+ "comparison": comparison,
185+ }
186+ directory.mkdir(parents=True, exist_ok=True)
187+ path = directory / f"{now.strftime('%Y%m%dT%H%M%S.%fZ')}-{uuid.uuid4().hex}.json"
188+ with path.open("x") as stream:
189+ json.dump(record, stream, indent=2, allow_nan=False)
190+ stream.write("\n")
191+ relative = str(path.relative_to(root))
192+ report(comparison)
193+ print(f"Record: {relative}", flush=True)
194+ subprocess.run(["git", "add", "--", relative], cwd=root, check=True)
195+ # --only excludes anything another process staged while benchmarks ran.
196+ subprocess.run(["git", "commit", "--only", "-m", f"bench: record {path.stem}",
197+ "--", relative], cwd=root,
198+ env=dict(os.environ, GIT_EDITOR="true"), check=True)
199+ return 1 if comparison["regression"] else 0
200+
201+
202+def threshold_value(text):
203+ value = float(text)
204+ if not math.isfinite(value) or value < 0:
205+ raise argparse.ArgumentTypeError("threshold must be finite and nonnegative")
206+ return value
207+
208+
209+def main(argv=None):
210+ parser = argparse.ArgumentParser(description=__doc__)
211+ parser.add_argument("threshold", nargs="?", type=threshold_value, default=10.0,
212+ help="alert above this percentage increase (default: 10)")
213+ args = parser.parse_args(argv)
214+ try:
215+ return store(ROOT, args.threshold)
216+ except (OSError, ValueError, subprocess.CalledProcessError) as exc:
217+ print(f"bench-store: {exc}", file=sys.stderr)
218+ return 2
219+
220+
221+if __name__ == "__main__":
222+ sys.exit(main())
new file mode 100644
@@ -0,0 +1,222 @@
1+
2+"""Record and commit a benchmark run; exit 1 after committing a regression."""
3+
4+import argparse
5+
6+import hashlib
7+import json
8+import math
9+import os
10+
11+import platform
12+import shutil
13+import subprocess
14+import sys
15+import tempfile
16+import uuid
17+from datetime import datetime, timezone
18+from pathlib import Path
19+
20+ROOT = Path(__file__).resolve().parent.parent
21+BENCHES = ("hello", "fib", "loop", "seqs")
22+LABELS = ("clonim", "jolt")
23+
24+
25+def command(root, *args, env=None):
26+ return subprocess.check_output(args, cwd=root, env=env, text=True).strip()
27+
28+
29+def require_clean(root):
30+ if command(root, "git", "status", "--porcelain", "--untracked-files=all"):
31+ raise ValueError("working tree must be clean, including untracked files")
32+
33+
34+def cpu_info():
35+ try:
36+ for line in Path("/proc/cpuinfo").read_text().splitlines():
37+ key, sep, value = line.partition(":")
38+ if sep and key.strip() in ("model name", "Hardware"):
39+ return value.strip()
40+ except OSError:
41+ pass
42+ return platform.processor() or "unknown"
43+
44+
45+def compatibility(root):
46+ paths = sorted((root / "bench").glob("*.clj"))
47+ paths += [root / "bench/measure.py", root / "justfile"]
48+ return {
49+ "machine": {
50+ "system": platform.system(),
51+ "release": platform.release(),
52+ "machine": platform.machine(),
53+ "hostname": platform.node(),
54+ "cpu": cpu_info(),
55+ "logical_cpus": os.cpu_count(),
56+ },
57+ "tool_versions": {
58+ tool: command(root, tool, "--version") for tool in ("nim", "jolt")
59+ },
60+ "methodology": {
61+ "schema_version": 1,
62+ "rounds": 11,
63+ "metric": "best_ms",
64+ "schedule": "round-robin",
65+ },
66+ "sha256": {
67+ str(path.relative_to(root)): hashlib.sha256(path.read_bytes()).hexdigest()
68+ for path in paths
69+ },
70+ }
71+
72+
73+def number(value):
74+ return type(value) in (int, float) and math.isfinite(value)
75+
76+
77+def validate_results(results):
78+ if not isinstance(results, dict):
79+ # Invalid JSON measurements consistently raise ValueError for callers.
80+ raise ValueError("measurement must be a JSON object") # noqa: TRY004
81+ if results.get("schema_version") != 1 or results.get("rounds") != 11:
82+ raise ValueError("expected measurement schema_version=1 and rounds=11")
83+ try:
84+ for label in LABELS:
85+ for bench in BENCHES:
86+ samples = results["samples_ms"][label][bench]
87+ best = results["best_ms"][label][bench]
88+ if (not isinstance(samples, list) or len(samples) != 11
89+ or not all(number(v) and v >= 0 for v in samples)
90+ or not number(best) or best != min(samples)):
91+ raise ValueError(f"invalid samples/best for {label}/{bench}")
92+ # Some producers omit the identically-zero hello compute metric.
93+ if bench != "hello" or bench in results["compute_ms"][label]:
94+ compute = results["compute_ms"][label][bench]
95+ expected = best - results["best_ms"][label]["hello"]
96+ if not number(compute) or not math.isclose(
97+ compute, expected, rel_tol=1e-9, abs_tol=1e-9):
98+ raise ValueError(f"invalid compute for {label}/{bench}")
99+ except (KeyError, TypeError) as exc:
100+ raise ValueError("missing or invalid measurement metrics") from exc
101+
102+
103+def latest_baseline(directory, compatible):
104+ candidates = []
105+ for path in directory.glob("*.json"):
106+ try:
107+ record = json.loads(path.read_text())
108+ if record["schema_version"] != 1 or record["compatibility"] != compatible:
109+ continue
110+ validate_results(record["results"])
111+ created = datetime.fromisoformat(record["created_at"])
112+ if created.utcoffset() is None:
113+ continue
114+ candidates.append((created, path.name, record))
115+ except (OSError, ValueError, KeyError, TypeError):
116+ continue
117+ if not candidates:
118+ return None
119+ _, name, record = max(candidates, key=lambda item: (item[0], item[1]))
120+ return name, record
121+
122+
123+def compare(results, baseline, threshold):
124+ comparison = {"baseline": None, "threshold_percent": threshold,
125+ "deltas": {}, "regression": False}
126+ if baseline is None:
127+ return comparison
128+ name, prior = baseline
129+ comparison["baseline"] = name
130+ for bench in BENCHES:
131+ old = prior["results"]["best_ms"]["clonim"][bench]
132+ new = results["best_ms"]["clonim"][bench]
133+ delta = new - old
134+ percent = 100 * delta / old if old else (0.0 if new == 0 else None)
135+ regression = delta > old * threshold / 100
136+ comparison["deltas"][bench] = {
137+ "baseline_ms": old, "current_ms": new, "delta_ms": delta,
138+ "percent": percent, "regression": regression,
139+ }
140+ comparison["regression"] |= regression
141+ return comparison
142+
143+
144+def report(comparison):
145+ if comparison["baseline"] is None:
146+ print("No compatible baseline; recording initial run.")
147+ else:
148+ print(f"Baseline: {comparison['baseline']}")
149+ for bench, delta in comparison["deltas"].items():
150+ percent = delta["percent"]
151+ change = f"{percent:+.2f}%" if percent is not None else "increase from zero"
152+ print(f" {bench}: {delta['delta_ms']:+.3f} ms ({change})"
153+ + (" REGRESSION" if delta["regression"] else ""))
154+ print(f"Regression: {comparison['regression']}")
155+
156+
157+def store(root, threshold):
158+ require_clean(root)
159+ for tool in ("jolt", "nim", "just"):
160+ if shutil.which(tool) is None:
161+ raise ValueError(f"required executable not found: {tool}")
162+ revision = command(root, "git", "rev-parse", "HEAD")
163+ compatible = compatibility(root)
164+ directory = root / "bench/results"
165+ baseline = latest_baseline(directory, compatible)
166+ with tempfile.TemporaryDirectory(prefix="clonim-bench-") as work:
167+ output = Path(work).resolve() / "measurement.json"
168+ env = dict(os.environ, BENCH_JSON=str(output))
169+ subprocess.run(["just", "measure", "jolt"], cwd=root, env=env, check=True)
170+ results = json.loads(output.read_text())
171+ validate_results(results)
172+ if command(root, "git", "rev-parse", "HEAD") != revision:
173+ raise ValueError("HEAD changed during measurement; refusing to record")
174+ if compatibility(root) != compatible:
175+ raise ValueError("benchmark inputs or environment changed during measurement")
176+ comparison = compare(results, baseline, threshold)
177+ now = datetime.now(timezone.utc)
178+ record = {
179+ "schema_version": 1,
180+ "created_at": now.isoformat(),
181+ "source_revision": revision,
182+ "compatibility": compatible,
183+ "results": results,
184+ "comparison": comparison,
185+ }
186+ directory.mkdir(parents=True, exist_ok=True)
187+ path = directory / f"{now.strftime('%Y%m%dT%H%M%S.%fZ')}-{uuid.uuid4().hex}.json"
188+ with path.open("x") as stream:
189+ json.dump(record, stream, indent=2, allow_nan=False)
190+ stream.write("\n")
191+ relative = str(path.relative_to(root))
192+ report(comparison)
193+ print(f"Record: {relative}", flush=True)
194+ subprocess.run(["git", "add", "--", relative], cwd=root, check=True)
195+ # --only excludes anything another process staged while benchmarks ran.
196+ subprocess.run(["git", "commit", "--only", "-m", f"bench: record {path.stem}",
197+ "--", relative], cwd=root,
198+ env=dict(os.environ, GIT_EDITOR="true"), check=True)
199+ return 1 if comparison["regression"] else 0
200+
201+
202+def threshold_value(text):
203+ value = float(text)
204+ if not math.isfinite(value) or value < 0:
205+ raise argparse.ArgumentTypeError("threshold must be finite and nonnegative")
206+ return value
207+
208+
209+def main(argv=None):
210+ parser = argparse.ArgumentParser(description=__doc__)
211+ parser.add_argument("threshold", nargs="?", type=threshold_value, default=10.0,
212+ help="alert above this percentage increase (default: 10)")
213+ args = parser.parse_args(argv)
214+ try:
215+ return store(ROOT, args.threshold)
216+ except (OSError, ValueError, subprocess.CalledProcessError) as exc:
217+ print(f"bench-store: {exc}", file=sys.stderr)
218+ return 2
219+
220+
221+if __name__ == "__main__":
222+ sys.exit(main())
added bench/test_measure.py +39 -0
new file mode 100644
@@ -0,0 +1,39 @@
1+import contextlib
2+import io
3+import json
4+import os
5+
6+import subprocess
7+import tempfile
8+import unittest
9+from pathlib import Path
10+from unittest.mock import patch
11+
12+import measure
13+from store import validate_results
14+
15+
16+class MeasureTests(unittest.TestCase):
17+ def test_json_matches_storage_contract(self):
18+ with tempfile.TemporaryDirectory() as directory:
19+ output = Path(directory) / "result.json"
20+ with patch.dict(os.environ, BENCH_JSON=str(output)), \
21+ patch.object(measure.subprocess, "run") as run, \
22+ patch.object(measure.time, "perf_counter", side_effect=range(176)), \
23+ contextlib.redirect_stdout(io.StringIO()):
24+ measure.main(["--with-jolt"])
25+ results = json.loads(output.read_text())
26+ validate_results(results)
27+ self.assertEqual(set(results["best_ms"]), {"clonim", "jolt"})
28+ self.assertEqual(run.call_count, 96)
29+ self.assertTrue(all(call.kwargs["check"] for call in run.call_args_list))
30+
31+ def test_failed_binary_aborts(self):
32+ with patch.object(measure.subprocess, "run",
33+ side_effect=subprocess.CalledProcessError(1, "bench")), \
34+ self.assertRaises(subprocess.CalledProcessError):
35+ measure.timed(["bench"])
36+
37+
38+if __name__ == "__main__":
39+ unittest.main()
new file mode 100644
@@ -0,0 +1,39 @@
1+import contextlib
2+import io
3+import json
4+import os
5+
6+import subprocess
7+import tempfile
8+import unittest
9+from pathlib import Path
10+from unittest.mock import patch
11+
12+import measure
13+from store import validate_results
14+
15+
16+class MeasureTests(unittest.TestCase):
17+ def test_json_matches_storage_contract(self):
18+ with tempfile.TemporaryDirectory() as directory:
19+ output = Path(directory) / "result.json"
20+ with patch.dict(os.environ, BENCH_JSON=str(output)), \
21+ patch.object(measure.subprocess, "run") as run, \
22+ patch.object(measure.time, "perf_counter", side_effect=range(176)), \
23+ contextlib.redirect_stdout(io.StringIO()):
24+ measure.main(["--with-jolt"])
25+ results = json.loads(output.read_text())
26+ validate_results(results)
27+ self.assertEqual(set(results["best_ms"]), {"clonim", "jolt"})
28+ self.assertEqual(run.call_count, 96)
29+ self.assertTrue(all(call.kwargs["check"] for call in run.call_args_list))
30+
31+ def test_failed_binary_aborts(self):
32+ with patch.object(measure.subprocess, "run",
33+ side_effect=subprocess.CalledProcessError(1, "bench")), \
34+ self.assertRaises(subprocess.CalledProcessError):
35+ measure.timed(["bench"])
36+
37+
38+if __name__ == "__main__":
39+ unittest.main()
added bench/test_store.py +197 -0
new file mode 100644
@@ -0,0 +1,197 @@
1+"""Stdlib tests; git and benchmark execution are mocked, never committed."""
2+
3+import copy
4+import importlib.util
5+import json
6+
7+import subprocess
8+import tempfile
9+import unittest
10+from pathlib import Path
11+from unittest.mock import patch
12+
13+spec = importlib.util.spec_from_file_location("bench_store", Path(__file__).with_name("store.py"))
14+assert spec is not None and spec.loader is not None
15+store = importlib.util.module_from_spec(spec)
16+spec.loader.exec_module(store)
17+
18+
19+def results(value=10):
20+ best = {label: {bench: value for bench in store.BENCHES} for label in store.LABELS}
21+ return {
22+ "schema_version": 1, "rounds": 11,
23+ "samples_ms": {label: {bench: [value] * 11 for bench in store.BENCHES}
24+ for label in store.LABELS},
25+ "best_ms": best,
26+ "compute_ms": {label: {bench: 0 for bench in store.BENCHES}
27+ for label in store.LABELS},
28+ }
29+
30+
31+def record(value=10, compatible=None, date="2026-01-01T00:00:00+00:00"):
32+ return {"schema_version": 1, "created_at": date,
33+ "source_revision": "old", "compatibility": compatible or {"machine": "a"},
34+ "results": results(value)}
35+
36+
37+class ComparisonTests(unittest.TestCase):
38+ def test_no_baseline(self):
39+ comparison = store.compare(results(), None, 10)
40+ self.assertFalse(comparison["regression"])
41+ self.assertEqual(comparison["deltas"], {})
42+
43+ def test_strict_threshold_and_all_benches(self):
44+ baseline = ("prior.json", record())
45+ self.assertFalse(store.compare(results(11), baseline, 10)["regression"])
46+ self.assertFalse(store.compare(results(9), baseline, 10)["regression"])
47+ for bench in store.BENCHES:
48+ current = results()
49+ current["best_ms"]["clonim"][bench] = 12
50+ comparison = store.compare(current, baseline, 10)
51+ self.assertTrue(comparison["regression"])
52+ self.assertEqual(comparison["deltas"][bench]["percent"], 20)
53+ current = results()
54+ current["best_ms"]["jolt"]["fib"] = 100
55+ self.assertFalse(store.compare(current, baseline, 10)["regression"])
56+
57+ def test_zero_baseline_is_json_safe(self):
58+ comparison = store.compare(results(1), ("zero.json", record(0)), 10)
59+ self.assertTrue(comparison["regression"])
60+ self.assertIsNone(comparison["deltas"]["hello"]["percent"])
61+ json.dumps(comparison, allow_nan=False)
62+ self.assertFalse(store.compare(results(0), ("zero", record(0)), 10)["regression"])
63+
64+ def test_latest_compatible_not_source_revision_or_filename(self):
65+ with tempfile.TemporaryDirectory() as work:
66+ directory = Path(work)
67+ old = record()
68+ latest = record(date="2026-02-01T00:00:00+00:00")
69+ latest["source_revision"] = "different"
70+ incompatible = record(compatible={"machine": "b"},
71+ date="2026-03-01T00:00:00+00:00")
72+ for name, data in (("z.json", old), ("a.json", latest),
73+ ("new.json", incompatible)):
74+ (directory / name).write_text(json.dumps(data))
75+ (directory / "broken.json").write_text("{")
76+ self.assertEqual(store.latest_baseline(directory, old["compatibility"])[0],
77+ "a.json")
78+ self.assertIsNone(store.latest_baseline(directory, {"machine": "c"}))
79+ # Every compatibility component matters, not just machine identity.
80+ for key in ("tool_versions", "methodology", "sha256"):
81+ changed = dict(old["compatibility"], **{key: "changed"})
82+ self.assertIsNone(store.latest_baseline(directory, changed))
83+
84+ def test_validation(self):
85+ store.validate_results(results())
86+ mutations = [lambda r: r.update(rounds=10),
87+ lambda r: r["best_ms"].pop("jolt"),
88+ lambda r: r["samples_ms"]["clonim"].update(hello=[1]),
89+ lambda r: r["best_ms"]["clonim"].update(fib=float("nan")),
90+ lambda r: r["compute_ms"]["clonim"].update(fib=1)]
91+ for mutate in mutations:
92+ data = copy.deepcopy(results())
93+ mutate(data)
94+ with self.assertRaises(ValueError):
95+ store.validate_results(data)
96+
97+ def test_non_object_measurement_raises_value_error(self):
98+ for value in (None, [], "invalid", 1, True):
99+ with self.subTest(value=value), self.assertRaisesRegex(
100+ ValueError, "measurement must be a JSON object"
101+ ):
102+ store.validate_results(value)
103+
104+ def test_threshold(self):
105+ self.assertEqual(store.threshold_value("10"), 10)
106+ for text in ("-1", "nan", "inf"):
107+ with self.assertRaises(store.argparse.ArgumentTypeError):
108+ store.threshold_value(text)
109+
110+
111+class WorkflowTests(unittest.TestCase):
112+ def test_dirty_tree_including_untracked(self):
113+ for status in (" M source.nim", "?? new.txt", "A staged.txt"):
114+ with patch.object(store, "command", return_value=status) as command:
115+ with self.assertRaisesRegex(ValueError, "clean"):
116+ store.require_clean(Path("."))
117+ self.assertIn("--untracked-files=all", command.call_args.args)
118+
119+ def test_missing_jolt_before_measurement(self):
120+ with patch.object(store, "require_clean"), \
121+ patch.object(store.shutil, "which", return_value=None), \
122+ patch.object(store.subprocess, "run") as run:
123+ with self.assertRaisesRegex(ValueError, "jolt"):
124+ store.store(Path("."), 10)
125+ run.assert_not_called()
126+
127+ def workflow(self, root, value=10, failure=None):
128+ compatible = {"machine": "a"}
129+
130+ def run(args, **kwargs):
131+ if args == ["just", "measure", "jolt"]:
132+ path = Path(kwargs["env"]["BENCH_JSON"])
133+ self.assertTrue(path.is_absolute())
134+ self.assertEqual(kwargs["cwd"], root)
135+ if failure == "measure":
136+ raise subprocess.CalledProcessError(1, args)
137+ if failure == "missing":
138+ return
139+ path.write_text("{}" if failure == "invalid" else json.dumps(results(value)))
140+ elif args[:2] == ["git", "commit"] and failure == "commit":
141+ raise subprocess.CalledProcessError(1, args)
142+
143+ with patch.object(store, "require_clean"), \
144+ patch.object(store.shutil, "which", return_value="/tool"), \
145+ patch.object(store, "command", return_value="source-sha"), \
146+ patch.object(store, "compatibility", return_value=compatible), \
147+ patch.object(store.subprocess, "run", side_effect=run) as runner:
148+ if failure:
149+ with self.assertRaises((OSError, ValueError, subprocess.CalledProcessError)):
150+ store.store(root, 10)
151+ return runner
152+ self.assertEqual(store.store(root, 10), 1 if value > 11 else 0)
153+ return runner
154+
155+ def test_regression_recorded_then_committed_only_its_path(self):
156+ with tempfile.TemporaryDirectory() as work:
157+ root = Path(work)
158+ directory = root / "bench/results"
159+ directory.mkdir(parents=True)
160+ (directory / "prior.json").write_text(json.dumps(record()))
161+ runner = self.workflow(root, 12)
162+ paths = [p for p in directory.glob("*.json") if p.name != "prior.json"]
163+ self.assertEqual(len(paths), 1)
164+ data = json.loads(paths[0].read_text())
165+ self.assertEqual(data["source_revision"], "source-sha")
166+ self.assertTrue(data["comparison"]["regression"])
167+ relative = str(paths[0].relative_to(root))
168+ add, commit = runner.call_args_list[-2:]
169+ self.assertEqual(add.args[0], ["git", "add", "--", relative])
170+ self.assertEqual(commit.args[0][:3], ["git", "commit", "--only"])
171+ self.assertEqual(commit.args[0][-2:], ["--", relative])
172+ self.assertEqual(commit.kwargs["env"]["GIT_EDITOR"], "true")
173+
174+ def test_failed_measurement_creates_no_record(self):
175+ for failure in ("measure", "missing", "invalid"):
176+ with tempfile.TemporaryDirectory() as work:
177+ root = Path(work)
178+ runner = self.workflow(root, failure=failure)
179+ self.assertFalse((root / "bench/results").exists())
180+ self.assertEqual(runner.call_count, 1)
181+
182+ def test_commit_failure_keeps_record(self):
183+ with tempfile.TemporaryDirectory() as work:
184+ root = Path(work)
185+ self.workflow(root, failure="commit")
186+ self.assertEqual(len(list((root / "bench/results").glob("*.json"))), 1)
187+
188+ def test_unique_records(self):
189+ with tempfile.TemporaryDirectory() as work:
190+ root = Path(work)
191+ self.workflow(root)
192+ self.workflow(root)
193+ self.assertEqual(len(list((root / "bench/results").glob("*.json"))), 2)
194+
195+
196+if __name__ == "__main__":
197+ unittest.main()
new file mode 100644
@@ -0,0 +1,197 @@
1+"""Stdlib tests; git and benchmark execution are mocked, never committed."""
2+
3+import copy
4+import importlib.util
5+import json
6+
7+import subprocess
8+import tempfile
9+import unittest
10+from pathlib import Path
11+from unittest.mock import patch
12+
13+spec = importlib.util.spec_from_file_location("bench_store", Path(__file__).with_name("store.py"))
14+assert spec is not None and spec.loader is not None
15+store = importlib.util.module_from_spec(spec)
16+spec.loader.exec_module(store)
17+
18+
19+def results(value=10):
20+ best = {label: {bench: value for bench in store.BENCHES} for label in store.LABELS}
21+ return {
22+ "schema_version": 1, "rounds": 11,
23+ "samples_ms": {label: {bench: [value] * 11 for bench in store.BENCHES}
24+ for label in store.LABELS},
25+ "best_ms": best,
26+ "compute_ms": {label: {bench: 0 for bench in store.BENCHES}
27+ for label in store.LABELS},
28+ }
29+
30+
31+def record(value=10, compatible=None, date="2026-01-01T00:00:00+00:00"):
32+ return {"schema_version": 1, "created_at": date,
33+ "source_revision": "old", "compatibility": compatible or {"machine": "a"},
34+ "results": results(value)}
35+
36+
37+class ComparisonTests(unittest.TestCase):
38+ def test_no_baseline(self):
39+ comparison = store.compare(results(), None, 10)
40+ self.assertFalse(comparison["regression"])
41+ self.assertEqual(comparison["deltas"], {})
42+
43+ def test_strict_threshold_and_all_benches(self):
44+ baseline = ("prior.json", record())
45+ self.assertFalse(store.compare(results(11), baseline, 10)["regression"])
46+ self.assertFalse(store.compare(results(9), baseline, 10)["regression"])
47+ for bench in store.BENCHES:
48+ current = results()
49+ current["best_ms"]["clonim"][bench] = 12
50+ comparison = store.compare(current, baseline, 10)
51+ self.assertTrue(comparison["regression"])
52+ self.assertEqual(comparison["deltas"][bench]["percent"], 20)
53+ current = results()
54+ current["best_ms"]["jolt"]["fib"] = 100
55+ self.assertFalse(store.compare(current, baseline, 10)["regression"])
56+
57+ def test_zero_baseline_is_json_safe(self):
58+ comparison = store.compare(results(1), ("zero.json", record(0)), 10)
59+ self.assertTrue(comparison["regression"])
60+ self.assertIsNone(comparison["deltas"]["hello"]["percent"])
61+ json.dumps(comparison, allow_nan=False)
62+ self.assertFalse(store.compare(results(0), ("zero", record(0)), 10)["regression"])
63+
64+ def test_latest_compatible_not_source_revision_or_filename(self):
65+ with tempfile.TemporaryDirectory() as work:
66+ directory = Path(work)
67+ old = record()
68+ latest = record(date="2026-02-01T00:00:00+00:00")
69+ latest["source_revision"] = "different"
70+ incompatible = record(compatible={"machine": "b"},
71+ date="2026-03-01T00:00:00+00:00")
72+ for name, data in (("z.json", old), ("a.json", latest),
73+ ("new.json", incompatible)):
74+ (directory / name).write_text(json.dumps(data))
75+ (directory / "broken.json").write_text("{")
76+ self.assertEqual(store.latest_baseline(directory, old["compatibility"])[0],
77+ "a.json")
78+ self.assertIsNone(store.latest_baseline(directory, {"machine": "c"}))
79+ # Every compatibility component matters, not just machine identity.
80+ for key in ("tool_versions", "methodology", "sha256"):
81+ changed = dict(old["compatibility"], **{key: "changed"})
82+ self.assertIsNone(store.latest_baseline(directory, changed))
83+
84+ def test_validation(self):
85+ store.validate_results(results())
86+ mutations = [lambda r: r.update(rounds=10),
87+ lambda r: r["best_ms"].pop("jolt"),
88+ lambda r: r["samples_ms"]["clonim"].update(hello=[1]),
89+ lambda r: r["best_ms"]["clonim"].update(fib=float("nan")),
90+ lambda r: r["compute_ms"]["clonim"].update(fib=1)]
91+ for mutate in mutations:
92+ data = copy.deepcopy(results())
93+ mutate(data)
94+ with self.assertRaises(ValueError):
95+ store.validate_results(data)
96+
97+ def test_non_object_measurement_raises_value_error(self):
98+ for value in (None, [], "invalid", 1, True):
99+ with self.subTest(value=value), self.assertRaisesRegex(
100+ ValueError, "measurement must be a JSON object"
101+ ):
102+ store.validate_results(value)
103+
104+ def test_threshold(self):
105+ self.assertEqual(store.threshold_value("10"), 10)
106+ for text in ("-1", "nan", "inf"):
107+ with self.assertRaises(store.argparse.ArgumentTypeError):
108+ store.threshold_value(text)
109+
110+
111+class WorkflowTests(unittest.TestCase):
112+ def test_dirty_tree_including_untracked(self):
113+ for status in (" M source.nim", "?? new.txt", "A staged.txt"):
114+ with patch.object(store, "command", return_value=status) as command:
115+ with self.assertRaisesRegex(ValueError, "clean"):
116+ store.require_clean(Path("."))
117+ self.assertIn("--untracked-files=all", command.call_args.args)
118+
119+ def test_missing_jolt_before_measurement(self):
120+ with patch.object(store, "require_clean"), \
121+ patch.object(store.shutil, "which", return_value=None), \
122+ patch.object(store.subprocess, "run") as run:
123+ with self.assertRaisesRegex(ValueError, "jolt"):
124+ store.store(Path("."), 10)
125+ run.assert_not_called()
126+
127+ def workflow(self, root, value=10, failure=None):
128+ compatible = {"machine": "a"}
129+
130+ def run(args, **kwargs):
131+ if args == ["just", "measure", "jolt"]:
132+ path = Path(kwargs["env"]["BENCH_JSON"])
133+ self.assertTrue(path.is_absolute())
134+ self.assertEqual(kwargs["cwd"], root)
135+ if failure == "measure":
136+ raise subprocess.CalledProcessError(1, args)
137+ if failure == "missing":
138+ return
139+ path.write_text("{}" if failure == "invalid" else json.dumps(results(value)))
140+ elif args[:2] == ["git", "commit"] and failure == "commit":
141+ raise subprocess.CalledProcessError(1, args)
142+
143+ with patch.object(store, "require_clean"), \
144+ patch.object(store.shutil, "which", return_value="/tool"), \
145+ patch.object(store, "command", return_value="source-sha"), \
146+ patch.object(store, "compatibility", return_value=compatible), \
147+ patch.object(store.subprocess, "run", side_effect=run) as runner:
148+ if failure:
149+ with self.assertRaises((OSError, ValueError, subprocess.CalledProcessError)):
150+ store.store(root, 10)
151+ return runner
152+ self.assertEqual(store.store(root, 10), 1 if value > 11 else 0)
153+ return runner
154+
155+ def test_regression_recorded_then_committed_only_its_path(self):
156+ with tempfile.TemporaryDirectory() as work:
157+ root = Path(work)
158+ directory = root / "bench/results"
159+ directory.mkdir(parents=True)
160+ (directory / "prior.json").write_text(json.dumps(record()))
161+ runner = self.workflow(root, 12)
162+ paths = [p for p in directory.glob("*.json") if p.name != "prior.json"]
163+ self.assertEqual(len(paths), 1)
164+ data = json.loads(paths[0].read_text())
165+ self.assertEqual(data["source_revision"], "source-sha")
166+ self.assertTrue(data["comparison"]["regression"])
167+ relative = str(paths[0].relative_to(root))
168+ add, commit = runner.call_args_list[-2:]
169+ self.assertEqual(add.args[0], ["git", "add", "--", relative])
170+ self.assertEqual(commit.args[0][:3], ["git", "commit", "--only"])
171+ self.assertEqual(commit.args[0][-2:], ["--", relative])
172+ self.assertEqual(commit.kwargs["env"]["GIT_EDITOR"], "true")
173+
174+ def test_failed_measurement_creates_no_record(self):
175+ for failure in ("measure", "missing", "invalid"):
176+ with tempfile.TemporaryDirectory() as work:
177+ root = Path(work)
178+ runner = self.workflow(root, failure=failure)
179+ self.assertFalse((root / "bench/results").exists())
180+ self.assertEqual(runner.call_count, 1)
181+
182+ def test_commit_failure_keeps_record(self):
183+ with tempfile.TemporaryDirectory() as work:
184+ root = Path(work)
185+ self.workflow(root, failure="commit")
186+ self.assertEqual(len(list((root / "bench/results").glob("*.json"))), 1)
187+
188+ def test_unique_records(self):
189+ with tempfile.TemporaryDirectory() as work:
190+ root = Path(work)
191+ self.workflow(root)
192+ self.workflow(root)
193+ self.assertEqual(len(list((root / "bench/results").glob("*.json"))), 2)
194+
195+
196+if __name__ == "__main__":
197+ unittest.main()
modified justfile +8 -0
@@ -57,6 +57,10 @@ clean:
5757 # is installed. Both toolchains get a `hello` binary so startup can be
5858 # subtracted; see bench/measure.py for why the numbers are best-of-N
5959 # round-robin rather than averaged.
60+# Record a Jolt comparison, flag regressions, and commit only its history record.
61+bench-store threshold="10":
62+ python3 bench/store.py {{quote(threshold)}}
63+
6064 # Time bench/*.clj as native binaries, optionally against jolt.
6165 measure mode="solo": release
6266 #!/usr/bin/env bash
@@ -70,6 +74,10 @@ measure mode="solo": release
7074 exec python3 bench/measure.py
7175 fi
7276 if ! command -v jolt >/dev/null; then
77+ if [ -n "${BENCH_JSON:-}" ]; then
78+ echo "jolt is required when storing benchmark results" >&2
79+ exit 1
80+ fi
7381 echo "jolt is not installed; running clonim only" >&2
7482 exec python3 bench/measure.py
7583 fi
@@ -57,6 +57,10 @@ clean:
57 # is installed. Both toolchains get a `hello` binary so startup can be57 # is installed. Both toolchains get a `hello` binary so startup can be
58 # subtracted; see bench/measure.py for why the numbers are best-of-N58 # subtracted; see bench/measure.py for why the numbers are best-of-N
59 # round-robin rather than averaged.59 # round-robin rather than averaged.
60+# Record a Jolt comparison, flag regressions, and commit only its history record.
61+bench-store threshold="10":
62+ python3 bench/store.py {{quote(threshold)}}
63+
60 # Time bench/*.clj as native binaries, optionally against jolt.64 # Time bench/*.clj as native binaries, optionally against jolt.
61 measure mode="solo": release65 measure mode="solo": release
62 #!/usr/bin/env bash66 #!/usr/bin/env bash
@@ -70,6 +74,10 @@ measure mode="solo": release
70 exec python3 bench/measure.py74 exec python3 bench/measure.py
71 fi75 fi
72 if ! command -v jolt >/dev/null; then76 if ! command -v jolt >/dev/null; then
77+ if [ -n "${BENCH_JSON:-}" ]; then
78+ echo "jolt is required when storing benchmark results" >&2
79+ exit 1
80+ fi
73 echo "jolt is not installed; running clonim only" >&281 echo "jolt is not installed; running clonim only" >&2
74 exec python3 bench/measure.py82 exec python3 bench/measure.py
75 fi83 fi