rickub/ci-bench · README.md

⑂ 6b67a7ef1caf6851afedfcbffa9de16df998b013

README.md · 217 lines · 11.1 KBmarkdown    Blame HistoryRaw
ci-bench: cross-platform CI race scaffold (rickub CI vs GitHub Actions)  Olivier Girardot 1# ci-bench — the CI race
3An identical, pinned workload pushed to **GitHub Actions** and **rickub CI**
4(Firecracker microVM fleet), raced to answer: who starts jobs faster, whose
5steps run faster, and what the runner hardware itself is worth?
7Everything here is bash + python3 stdlib. No package installs, no external
8services beyond the two CI platforms themselves.
10
12    .github/workflows/bench.yml    GitHub side of the race
13    .rickub/workflows/bench.yml    rickub side (same steps, same order)
14    workloads/rust-build/          pinned Cargo project (cold/warm/test)
15    workloads/docker-build/        digest-pinned multi-stage Dockerfile
16    workloads/probe/probe.sh       runner capability probe (cpu/disk/fs)
17    scripts/emit_timing.sh         sourced by steps; one JSON line per step
18    scripts/collect.sh             results.jsonl -> results/<platform>/<run>.json
19    scripts/compare.py             median/p95 table per step per platform
20    Makefile                       bootstrap / run-local / collect / compare
22Why two workflow files and not one: rickub reads `.rickub/workflows/` as its
23SOLE workflow source when that directory exists and ignores
24`.github/workflows/` entirely, while GitHub only ever reads `.github/`. One
25repo therefore carries one file per platform and neither platform sees the
26other's copy. The two files run the same steps in the same order; the only
27differences are `BENCH_PLATFORM`, `runs-on:` (see *Known asymmetries*), and
28
30## The workload (identical on both platforms)
33|-----------------|--------------------------------------------------------------------------|
34| `job-start`     | zero-duration marker emitted by the first step after checkout             |
35| `rust-cold-build` | `rm -rf target && cargo build` — full crate download + compile          |
36| `rust-warm-build` | immediate second `cargo build` — incremental no-op rebuild             |
37| `rust-test`     | `cargo test` (5 real tests) over the warm build                           |
38| `docker-build`  | multi-stage `docker build` (alpine:3.20 pinned by digest) + `docker run` |
39| `probe-*`       | cpu loop, 2 GiB sequential write/read to `$RUNNER_TEMP`, fs type + free  |
41Pins: Rust toolchain 1.85.0 (`workloads/rust-build/rust-toolchain.toml`), the
42full dependency tree (`workloads/rust-build/Cargo.lock`, generated offline
43against a real cargo registry cache, versions exact-pinned in `Cargo.toml`
44with `=`), Docker base `alpine:3.20@sha256:d9e853e87e...` (the digest rickub
45itself pins for its CI images). The workload is not bit-frozen against
46`apk`/crates.io mirrors moving — see *Variance pitfalls*.
48### Cold vs warm — definitions
50Both platforms give every job a **fresh machine** (GitHub hosted runners;
51rickub boots a new Firecracker microVM with a read-only rootfs and a fresh
52scratch disk). Therefore:
54- **cold** = the FIRST `cargo build` in a job: crates.io fetch + full compile
55  of the transitive tree. Never accelerated by any cache (there is no
56  `actions/cache` and no cargo cache persistence — on purpose).
57- **warm** = the SECOND `cargo build` in the SAME job: `target/` exists, the
58  no-op incremental path (metadata checks + link of nothing).
59- Docker build is always cold (no image cache survives the machine).
60- A "cold docker" vs "warm docker" axis would need an in-job second build of
61  the same Dockerfile; not in scope for v1.
63## Metrics — exact clock boundaries
65All records are JSON lines in `results.jsonl`:
67    {"step": "...", "platform": "...", "run_id": "...",
68     "start": "2026-09-11T12:34:56.789Z", "end": "2026-09-11T12:35:40.123Z",
69     "status": "ok|fail|skipped", "duration_ms": 43334,
70     "value": <optional measurement>, "unit": "...", "reason": "..."}
72`start`/`end` are guest wall clock, UTC, millisecond precision (GNU `date
73%3N`; python3 fallback on BSD; whole-second last resort — the fallback in use
74is visible in the timestamps themselves: `.000Z` means coarse).
761. **push→job-start** = `start` of the `job-start` record − **push timestamp
77   from the platform API** (not from the guest).
78   - GitHub: `gh api repos/:o/:r/actions/runs/<run_id>` → `created_at`
79     (when the push event was accepted server-side). Alternative, fully
80     server-side cross-check: `run_started_at − created_at` (queue +
81     provisioning, no guest clock involved).
82   - rickub: the run's created/queued timestamp from its CI API for the same
83     commit.
84   - CAVEAT: this metric subtracts a **server-side** timestamp from a
85     **guest-side** timestamp. Guest clock skew shifts both platforms' numbers
86     unpredictably (NTP inside a just-booted microVM can be off by seconds).
87     Treat it as indicative; prefer the server-side-only variant above when
88     the two disagree, and record both when possible.
892. **per-step duration** = `end − start` of the step's record, from the same
90   guest clock, so skew cancels. Millisecond precision.
913. **end-to-end pipeline time** (two views):
92   - job execution = last record's `end` − `job-start` record's `start`;
93   - run wall clock = last record's `end` − push timestamp (carries the same
94     cross-clock caveat as metric 1).
954. **cold vs warm cache** = `rust-cold-build` vs `rust-warm-build` durations
96   as defined above.
975. **disk-write throughput** = `probe-write` value (MB/s, 2 GiB `dd
98   bs=1M count=2048` to the job scratch dir, then removed); `probe-read` is
99   the same file read back (warm page cache — still comparable across
100   platforms); `probe-cpu` is shell-loop kops/s; `probe-fs-type`/`probe-fs-free`
101   describe the filesystem the runner gave the job.
103Anything unavailable degrades to `"value": null` or a `skipped` record; the
104job itself never fails from the probe, and absent docker skips
105`docker-build` with a recorded reason.
107## Running the race
109### One round
1111. Push the same commit to both remotes (e.g. an empty commit:
112   `git commit --allow-empty -m "bench round N"`; or use
113   `workflow_dispatch` on both — but record the dispatch time as the "push"
114   timestamp then).
1152. When both runs finish, download `results.jsonl` from each:
116   - GitHub: `gh run download <run-id> -n bench-results` (or the run page);
117   - rickub: the run's artifacts panel.
1183. Collect:
120   scripts/collect.sh github  <run-id> /path/to/github-results/results.jsonl
121   scripts/collect.sh rickub <run-id> /path/to/rickub-results/results.jsonl
1234. Record the push timestamps (server-side, per platform API) into a
124   `push-times.json` map `"github/<run-id>": "<iso>"` if you want the
125   push→job-start row.
127### Interleaving (mandatory)
129Run rounds **alternating platforms** — G, R, G, R, … — at least **10 rounds
130per platform**, serialized (never two benchmark runs in flight at once: on
131rickub your own queued job would inflate the other platform's queue metric).
132Cloud CI variance across hours and neighbours is well documented; a single
133run is noise. Alternation decorrelates the two platforms from time-of-day
134effects (fleet load, mirror warmth, co-tenants). Optionally discard round 1
135per platform as image/toolchain warmup for the *operators*, not the runners.
137### Reporting
140python3 scripts/compare.py [--push-times push-times.json]
143Median (p50) and p95 per step per platform, linear-interpolated percentiles
144(the numpy method) over `ok` records only; `skipped`/`fail` counts are shown
145next to each cell so a platform quietly skipping docker is visible. The
146table prints n per cell — do not compare anything with n < 10.
148### Local smoke run
150`make run-local` executes the same step sequence on your machine (macOS
151works: the emitter falls back to python3 for millisecond timestamps and the
152probe degrades gracefully). Collect with
153`make collect PLATFORM=local RUN_ID=<id> SRC=results.jsonl`.
155## Known variance pitfalls
157- **Cross-clock subtraction** (push→job-start): server vs guest clock skew;
158  prefer server-side-only `run_started_at − created_at` as cross-check.
159- **Queue contamination**: rickub queues runs; benchmark rounds must be
160  serialized or you measure your own backlog. Same for GitHub concurrency
161  groups (none set here, on purpose).
162- **Runner placement**: GitHub assigns runners across regions/hosts;
163  crates.io and the Alpine mirror are at different RTTs from each placement.
164  Only medians over many rounds are meaningful.
165- **Floating package indexes**: `apk add build-base` in the docker workload
166  floats with the mirror; crate downloads float with crates.io. A changed
167  upstream version shifts a cold build by minutes. If a round's cold build
168  moves >2x the running median, check upstream before believing it.
169- **Toolchain download**: 1.85.0 is rustup-installed at job start on both
170  platforms (inside the timed cold-build step? No — rustup resolves it when
171  `cargo` first runs, so it IS inside the cold-build timing; identical on
172  both platforms by construction).
173- **Warm build is nearly zero**: it measures scheduler/link noise; treat p95,
174  not median, as the signal.
175- **probe-read is page-cache warm** by design (comparable, not absolute).
176- **vCPU asymmetry**: rickub `large` = 4 vCPU / 8 GiB vs GitHub ubuntu-latest
177  4 vCPU / 16 GiB. See below.
179## Known asymmetries (decisions to revisit)
1811. **Runner class**: the rickub workflow uses `runs-on: large` for vCPU parity
182   with GitHub's ubuntu-latest. `ubuntu-latest` on rickub (2 vCPU / 4 GiB) is
183   the "as-consumed, 1x minutes" alternative. This is a user decision.
1842. **Where the repo lives**: needs one GitHub repo and one rickub repo (or the
185   same repo mirrored to both forges). Not created by this scaffold — no
186   network was touched.
1873. **push-times.json**: the push timestamp per run is currently recorded by
188   the operator from each platform's API; automating it needs API tokens on
189   both sides.
191## Degrade behaviour
193| missing thing | behaviour                                            |
194|---------------|------------------------------------------------------|
195| docker        | `docker-build` step records `skipped` with reason    |
196| cargo         | (local runs) rust steps record `skipped`             |
197| GNU date      | python3 millis fallback, then whole-second `.000Z`   |
198| dd / stat / df| probe records `null` values, job stays green         |
199| python3       | emitter falls back to whole-second timestamps        |
bench: document the real-world tier  Olivier Girardot 200
201## Real-world tier (added round 12+)
203Alongside the synthetic micro-workloads, the bench builds four notorious OSS
204projects at pinned tags, cloned/downloaded inside the timed window:
206| step | project | pin | toolchain |
207|---|---|---|---|
208| `rust-ripgrep` | BurntSushi/ripgrep | 15.2.0 | cargo |
209| `cpp-sqlite` | SQLite amalgamation | 3.45.1 (sqlite.org 2024 tarball) | gcc/make |
210| `node-typescript` | microsoft/TypeScript | v5.9.3 (5.x line — 7.x is the Go rewrite) | node/npm |
211| `java-guava` | google/guava | v33.7.1 | maven |
213ClickHouse is deliberately absent: a full build needs dozens of cores and
214~100 GB of disk, which no shared-runner tier (ours or GitHub's) provides — a
215ClickHouse-scale tier needs a dedicated runner class, not a benchmark step.
216SQLite's amalgamation is the C/C++ stand-in: a real configure+make of one of
217the most deployed codebases on earth, reproducible from a pinned tarball.

ci-bench: cross-platform CI race scaffold (rickub CI vs GitHub Actions) 0baf736 Olivier Girardot yesterday	1	# ci-bench — the CI race
	2
	3	An identical, pinned workload pushed to GitHub Actions and rickub CI
	4	(Firecracker microVM fleet), raced to answer: who starts jobs faster, whose
	5	steps run faster, and what the runner hardware itself is worth?
	6
	7	Everything here is bash + python3 stdlib. No package installs, no external
	8	services beyond the two CI platforms themselves.
	9
	10	## Layout
	11
	12	.github/workflows/bench.yml GitHub side of the race
	13	.rickub/workflows/bench.yml rickub side (same steps, same order)
	14	workloads/rust-build/ pinned Cargo project (cold/warm/test)
	15	workloads/docker-build/ digest-pinned multi-stage Dockerfile
	16	workloads/probe/probe.sh runner capability probe (cpu/disk/fs)
	17	scripts/emit_timing.sh sourced by steps; one JSON line per step
	18	scripts/collect.sh results.jsonl -> results/<platform>/<run>.json
	19	scripts/compare.py median/p95 table per step per platform
	20	Makefile bootstrap / run-local / collect / compare
	21
	22	Why two workflow files and not one: rickub reads `.rickub/workflows/` as its
	23	SOLE workflow source when that directory exists and ignores
	24	`.github/workflows/` entirely, while GitHub only ever reads `.github/`. One
	25	repo therefore carries one file per platform and neither platform sees the
	26	other's copy. The two files run the same steps in the same order; the only
	27	differences are `BENCH_PLATFORM`, `runs-on:` (see Known asymmetries), and
	28	comments.
	29
	30	## The workload (identical on both platforms)
	31
	32	\| step \| what it does \|
	33	\|-----------------\|--------------------------------------------------------------------------\|
	34	\| `job-start` \| zero-duration marker emitted by the first step after checkout \|
	35	\| `rust-cold-build` \| `rm -rf target && cargo build` — full crate download + compile \|
	36	\| `rust-warm-build` \| immediate second `cargo build` — incremental no-op rebuild \|
	37	\| `rust-test` \| `cargo test` (5 real tests) over the warm build \|
	38	\| `docker-build` \| multi-stage `docker build` (alpine:3.20 pinned by digest) + `docker run` \|
	39	\| `probe-*` \| cpu loop, 2 GiB sequential write/read to `$RUNNER_TEMP`, fs type + free \|
	40
	41	Pins: Rust toolchain 1.85.0 (`workloads/rust-build/rust-toolchain.toml`), the
	42	full dependency tree (`workloads/rust-build/Cargo.lock`, generated offline
	43	against a real cargo registry cache, versions exact-pinned in `Cargo.toml`
	44	with `=`), Docker base `alpine:3.20@sha256:d9e853e87e...` (the digest rickub
	45	itself pins for its CI images). The workload is not bit-frozen against
	46	`apk`/crates.io mirrors moving — see Variance pitfalls.
	47
	48	### Cold vs warm — definitions
	49
	50	Both platforms give every job a fresh machine (GitHub hosted runners;
	51	rickub boots a new Firecracker microVM with a read-only rootfs and a fresh
	52	scratch disk). Therefore:
	53
	54	- cold = the FIRST `cargo build` in a job: crates.io fetch + full compile
	55	of the transitive tree. Never accelerated by any cache (there is no
	56	`actions/cache` and no cargo cache persistence — on purpose).
	57	- warm = the SECOND `cargo build` in the SAME job: `target/` exists, the
	58	no-op incremental path (metadata checks + link of nothing).
	59	- Docker build is always cold (no image cache survives the machine).
	60	- A "cold docker" vs "warm docker" axis would need an in-job second build of
	61	the same Dockerfile; not in scope for v1.
	62
	63	## Metrics — exact clock boundaries
	64
65	All records are JSON lines in `results.jsonl`:
66
67	{"step": "...", "platform": "...", "run_id": "...",
68	"start": "2026-09-11T12:34:56.789Z", "end": "2026-09-11T12:35:40.123Z",
69	"status": "ok\|fail\|skipped", "duration_ms": 43334,
70	"value": <optional measurement>, "unit": "...", "reason": "..."}
71
72	`start`/`end` are guest wall clock, UTC, millisecond precision (GNU `date
73	%3N`; python3 fallback on BSD; whole-second last resort — the fallback in use
74	is visible in the timestamps themselves: `.000Z` means coarse).
75
76	1. push→job-start = `start` of the `job-start` record − **push timestamp
77	from the platform API** (not from the guest).
78	- GitHub: `gh api repos/:o/:r/actions/runs/<run_id>` → `created_at`
79	(when the push event was accepted server-side). Alternative, fully
80	server-side cross-check: `run_started_at − created_at` (queue +
81	provisioning, no guest clock involved).
82	- rickub: the run's created/queued timestamp from its CI API for the same
83	commit.
84	- CAVEAT: this metric subtracts a server-side timestamp from a
85	guest-side timestamp. Guest clock skew shifts both platforms' numbers
86	unpredictably (NTP inside a just-booted microVM can be off by seconds).
87	Treat it as indicative; prefer the server-side-only variant above when
88	the two disagree, and record both when possible.
89	2. per-step duration = `end − start` of the step's record, from the same
90	guest clock, so skew cancels. Millisecond precision.
91	3. end-to-end pipeline time (two views):
92	- job execution = last record's `end` − `job-start` record's `start`;
93	- run wall clock = last record's `end` − push timestamp (carries the same
94	cross-clock caveat as metric 1).
95	4. cold vs warm cache = `rust-cold-build` vs `rust-warm-build` durations
96	as defined above.
97	5. disk-write throughput = `probe-write` value (MB/s, 2 GiB `dd
98	bs=1M count=2048` to the job scratch dir, then removed); `probe-read` is
99	the same file read back (warm page cache — still comparable across
100	platforms); `probe-cpu` is shell-loop kops/s; `probe-fs-type`/`probe-fs-free`
101	describe the filesystem the runner gave the job.
102
103	Anything unavailable degrades to `"value": null` or a `skipped` record; the
104	job itself never fails from the probe, and absent docker skips
105	`docker-build` with a recorded reason.
106
107	## Running the race
108
109	### One round
110
111	1. Push the same commit to both remotes (e.g. an empty commit:
112	`git commit --allow-empty -m "bench round N"`; or use
113	`workflow_dispatch` on both — but record the dispatch time as the "push"
114	timestamp then).
115	2. When both runs finish, download `results.jsonl` from each:
116	- GitHub: `gh run download <run-id> -n bench-results` (or the run page);
117	- rickub: the run's artifacts panel.
118	3. Collect:
119	```
120	scripts/collect.sh github <run-id> /path/to/github-results/results.jsonl
121	scripts/collect.sh rickub <run-id> /path/to/rickub-results/results.jsonl
122	```
123	4. Record the push timestamps (server-side, per platform API) into a
124	`push-times.json` map `"github/<run-id>": "<iso>"` if you want the
125	push→job-start row.
126
127	### Interleaving (mandatory)
128
129	Run rounds alternating platforms — G, R, G, R, … — at least **10 rounds
130	per platform**, serialized (never two benchmark runs in flight at once: on
131	rickub your own queued job would inflate the other platform's queue metric).
132	Cloud CI variance across hours and neighbours is well documented; a single
133	run is noise. Alternation decorrelates the two platforms from time-of-day
134	effects (fleet load, mirror warmth, co-tenants). Optionally discard round 1
135	per platform as image/toolchain warmup for the operators, not the runners.
136
137	### Reporting
138
139	```
140	python3 scripts/compare.py [--push-times push-times.json]
141	```
142
143	Median (p50) and p95 per step per platform, linear-interpolated percentiles
144	(the numpy method) over `ok` records only; `skipped`/`fail` counts are shown
145	next to each cell so a platform quietly skipping docker is visible. The
146	table prints n per cell — do not compare anything with n < 10.
147
148	### Local smoke run
149
150	`make run-local` executes the same step sequence on your machine (macOS
151	works: the emitter falls back to python3 for millisecond timestamps and the
152	probe degrades gracefully). Collect with
153	`make collect PLATFORM=local RUN_ID=<id> SRC=results.jsonl`.
154
155	## Known variance pitfalls
156
157	- Cross-clock subtraction (push→job-start): server vs guest clock skew;
158	prefer server-side-only `run_started_at − created_at` as cross-check.
159	- Queue contamination: rickub queues runs; benchmark rounds must be
160	serialized or you measure your own backlog. Same for GitHub concurrency
161	groups (none set here, on purpose).
162	- Runner placement: GitHub assigns runners across regions/hosts;
163	crates.io and the Alpine mirror are at different RTTs from each placement.
164	Only medians over many rounds are meaningful.
165	- Floating package indexes: `apk add build-base` in the docker workload
166	floats with the mirror; crate downloads float with crates.io. A changed
167	upstream version shifts a cold build by minutes. If a round's cold build
168	moves >2x the running median, check upstream before believing it.
169	- Toolchain download: 1.85.0 is rustup-installed at job start on both
170	platforms (inside the timed cold-build step? No — rustup resolves it when
171	`cargo` first runs, so it IS inside the cold-build timing; identical on
172	both platforms by construction).
173	- Warm build is nearly zero: it measures scheduler/link noise; treat p95,
174	not median, as the signal.
175	- probe-read is page-cache warm by design (comparable, not absolute).
176	- vCPU asymmetry: rickub `large` = 4 vCPU / 8 GiB vs GitHub ubuntu-latest
177	4 vCPU / 16 GiB. See below.
178
179	## Known asymmetries (decisions to revisit)
180
181	1. Runner class: the rickub workflow uses `runs-on: large` for vCPU parity
182	with GitHub's ubuntu-latest. `ubuntu-latest` on rickub (2 vCPU / 4 GiB) is
183	the "as-consumed, 1x minutes" alternative. This is a user decision.
184	2. Where the repo lives: needs one GitHub repo and one rickub repo (or the
185	same repo mirrored to both forges). Not created by this scaffold — no
186	network was touched.
187	3. push-times.json: the push timestamp per run is currently recorded by
188	the operator from each platform's API; automating it needs API tokens on
189	both sides.
190
191	## Degrade behaviour
192
193	\| missing thing \| behaviour \|
194	\|---------------\|------------------------------------------------------\|
195	\| docker \| `docker-build` step records `skipped` with reason \|
196	\| cargo \| (local runs) rust steps record `skipped` \|
197	\| GNU date \| python3 millis fallback, then whole-second `.000Z` \|
198	\| dd / stat / df\| probe records `null` values, job stays green \|
199	\| python3 \| emitter falls back to whole-second timestamps \|
bench: document the real-world tier d5ea3aa Olivier Girardot 6h ago	200
	201	## Real-world tier (added round 12+)
	202
	203	Alongside the synthetic micro-workloads, the bench builds four notorious OSS
	204	projects at pinned tags, cloned/downloaded inside the timed window:
	205
	206	\| step \| project \| pin \| toolchain \|
	207	\|---\|---\|---\|---\|
	208	\| `rust-ripgrep` \| BurntSushi/ripgrep \| 15.2.0 \| cargo \|
	209	\| `cpp-sqlite` \| SQLite amalgamation \| 3.45.1 (sqlite.org 2024 tarball) \| gcc/make \|
	210	\| `node-typescript` \| microsoft/TypeScript \| v5.9.3 (5.x line — 7.x is the Go rewrite) \| node/npm \|
	211	\| `java-guava` \| google/guava \| v33.7.1 \| maven \|
	212
	213	ClickHouse is deliberately absent: a full build needs dozens of cores and
	214	~100 GB of disk, which no shared-runner tier (ours or GitHub's) provides — a
	215	ClickHouse-scale tier needs a dedicated runner class, not a benchmark step.
	216	SQLite's amalgamation is the C/C++ stand-in: a real configure+make of one of
	217	the most deployed codebases on earth, reproducible from a pinned tarball.