rickub/ci-bench · 7764675

⑂ 7764675

bench results: round 1 — rickub large vs GitHub ubuntu-latest, 10 interleaved roundsUnverified

Olivier Girardot committed 2026-09-13T10:40:32+02:00 Browse files

7764675 parent: 66a47c3

added RESULTS.md +38 -0

new file mode 100644
@@ -0,0 +1,38 @@
	1	+# CI race — round 1 results (2026-09-13)
	2	+
	3	+10 interleaved rounds per platform (G,R,G,R…, one commit per round pushed to both
	4	+mirrors, 75 s spacing; push timestamps in `push-times.jsonl`). rickub arm ran the
	5	+`large` class (4 vCPU / 8 GiB) for vCPU parity with GitHub's ubuntu-latest (4 vCPU).
	6	+Silicon is NOT matched: rickub runs a Ryzen 9700X bare-metal node, GitHub runs its
	7	+standard Azure fleet — the probe-cpu row shows what that alone is worth.
	8	+
	9	+## Medians (n=10, p95 in parens)
	10	+
	11	+\| step \| github \| rickub \| rickub advantage \|
	12	+\|---\|---\|---\|---\|
	13	+\| rust cold build \| 14.65 s (24.55) \| 5.63 s (7.38) \| 2.6× median, 3.3× p95 \|
	14	+\| rust test \| 1.51 s (2.79) \| 0.37 s (0.50) \| 4.0× \|
	15	+\| docker build \| 5.76 s (12.23) \| 3.14 s (3.36) \| 1.8× median, 3.7× p95 \|
	16	+\| push → job start \| 9.76 s (11.39) \| 4.08 s (4.75) \| 2.4× (cross-clock caveat) \|
	17	+\| end-to-end wall (run create→update) \| ~43 s \| ~16 s \| ~2.7× \|
	18	+
	19	+## Runner probe (medians)
	20	+
	21	+\| metric \| github \| rickub \|
	22	+\|---\|---\|---\|
	23	+\| single-core CPU \| 220 kops/s \| 595 kops/s (2.7×) \|
	24	+\| scratch write (2 GiB dd burst) \| 3,903 MB/s \| 3,459 MB/s \|
	25	+\| scratch read (page cache) \| 17,864 MB/s \| 30,422 MB/s \|
	26	+\| workspace fs / free \| ext-family / 87 GB \| overlayfs on scratch / 29 GB \|
	27	+
	28	+Honest readings:
	29	+- The build-step gap is dominated by single-thread CPU (2.7×) plus variance
	30	+ (GitHub's p95 tails are 2-3× its own median; rickub's are tight).
	31	+- Storage BURST throughput is on par — a 2 GiB dd on GitHub's runners is not the
	32	+ bottleneck it is on a write-worn consumer NVMe under sustained load. The zram
	33	+ ("faster" class) A/B must therefore be judged on job wall-time and sustained
	34	+ states, not on a dd burst.
	35	+- push→job-start spans two clocks (local Mac vs runner); treat as ±1-2 s.
	36	+
	37	+Runs: rickub.com/rickub/ci-bench/actions (runs 1-10) ·
	38	+github.com/ogirardot/ci-bench/actions/runs (34747021476…34747992384).

++# CI race — round 1 results (2026-09-13)
++
++10 interleaved rounds per platform (G,R,G,R…, one commit per round pushed to both
++mirrors, 75 s spacing; push timestamps in `push-times.jsonl`). rickub arm ran the
++`large` class (4 vCPU / 8 GiB) for vCPU parity with GitHub's ubuntu-latest (4 vCPU).
++Silicon is NOT matched: rickub runs a Ryzen 9700X bare-metal node, GitHub runs its
++standard Azure fleet — the probe-cpu row shows what that alone is worth.
++
++## Medians (n=10, p95 in parens)
++
++| step | github | rickub | rickub advantage |
++|---|---|---|---|
++| rust cold build | 14.65 s (24.55) | **5.63 s** (7.38) | 2.6× median, 3.3× p95 |
++| rust test | 1.51 s (2.79) | **0.37 s** (0.50) | 4.0× |
++| docker build | 5.76 s (12.23) | **3.14 s** (3.36) | 1.8× median, 3.7× p95 |
++| push → job start | 9.76 s (11.39) | **4.08 s** (4.75) | 2.4× (cross-clock caveat) |
++| end-to-end wall (run create→update) | ~43 s | **~16 s** | ~2.7× |
++
++## Runner probe (medians)
++
++| metric | github | rickub |
++|---|---|---|
++| single-core CPU | 220 kops/s | **595 kops/s** (2.7×) |
++| scratch write (2 GiB dd burst) | **3,903 MB/s** | 3,459 MB/s |
++| scratch read (page cache) | 17,864 MB/s | **30,422 MB/s** |
++| workspace fs / free | ext-family / 87 GB | overlayfs on scratch / 29 GB |
++
++Honest readings:
++- The build-step gap is dominated by single-thread CPU (2.7×) plus variance
++  (GitHub's p95 tails are 2-3× its own median; rickub's are tight).
++- Storage BURST throughput is on par — a 2 GiB dd on GitHub's runners is not the
++  bottleneck it is on a write-worn consumer NVMe under sustained load. The zram
++  ("faster" class) A/B must therefore be judged on job wall-time and sustained
++  states, not on a dd burst.
++- push→job-start spans two clocks (local Mac vs runner); treat as ±1-2 s.
++
++Runs: rickub.com/rickub/ci-bench/actions (runs 1-10) ·
++github.com/ogirardot/ci-bench/actions/runs (34747021476…34747992384).

added push-times.jsonl +20 -0

new file mode 100644
@@ -0,0 +1,20 @@
	1	+{"platform": "rickub", "round": 1, "sha": "0baf736bf04d237117e719e3f8b2c8d5860f6bd4", "pushed_at": "2026-09-13T08:10:15Z"}
	2	+{"platform": "github", "round": 1, "sha": "0baf736bf04d237117e719e3f8b2c8d5860f6bd4", "pushed_at": "2026-09-13T08:10:23Z"}
	3	+{"platform": "rickub", "round": 2, "sha": "1914eedc7f8b4c474f71c821431005bb5cc7281a", "pushed_at": "2026-09-13T08:11:49.000Z"}
	4	+{"platform": "github", "round": 2, "sha": "1914eedc7f8b4c474f71c821431005bb5cc7281a", "pushed_at": "2026-09-13T08:13:05.000Z"}
	5	+{"platform": "github", "round": 3, "sha": "09e7984e3695d820dceab05c61327b2e486c5019", "pushed_at": "2026-09-13T08:14:22.000Z"}
	6	+{"platform": "rickub", "round": 3, "sha": "09e7984e3695d820dceab05c61327b2e486c5019", "pushed_at": "2026-09-13T08:15:40.000Z"}
	7	+{"platform": "rickub", "round": 4, "sha": "da2d24baabb9dce05ce66a0e95ca7b806a6fa086", "pushed_at": "2026-09-13T08:16:56.000Z"}
	8	+{"platform": "github", "round": 4, "sha": "da2d24baabb9dce05ce66a0e95ca7b806a6fa086", "pushed_at": "2026-09-13T08:18:12.000Z"}
	9	+{"platform": "github", "round": 5, "sha": "8421335df28fbd4614c026374c4bd3896cd5f184", "pushed_at": "2026-09-13T08:19:29.000Z"}
	10	+{"platform": "rickub", "round": 5, "sha": "8421335df28fbd4614c026374c4bd3896cd5f184", "pushed_at": "2026-09-13T08:20:47.000Z"}
	11	+{"platform": "rickub", "round": 6, "sha": "9990e0f6ae4ac4c66a0251fee9c88526dd99b975", "pushed_at": "2026-09-13T08:22:03.000Z"}
	12	+{"platform": "github", "round": 6, "sha": "9990e0f6ae4ac4c66a0251fee9c88526dd99b975", "pushed_at": "2026-09-13T08:23:19.000Z"}
	13	+{"platform": "github", "round": 7, "sha": "fba04eba4dee92ce17de4847289c54c813914bcc", "pushed_at": "2026-09-13T08:24:37.000Z"}
	14	+{"platform": "rickub", "round": 7, "sha": "fba04eba4dee92ce17de4847289c54c813914bcc", "pushed_at": "2026-09-13T08:25:54.000Z"}
	15	+{"platform": "rickub", "round": 8, "sha": "35efc9cc863d18db697b419bf6386089ead68a6d", "pushed_at": "2026-09-13T08:27:10.000Z"}
	16	+{"platform": "github", "round": 8, "sha": "35efc9cc863d18db697b419bf6386089ead68a6d", "pushed_at": "2026-09-13T08:28:27.000Z"}
	17	+{"platform": "github", "round": 9, "sha": "39ba5db452f23f08907cce6b693297d70f8ec54a", "pushed_at": "2026-09-13T08:29:45.000Z"}
	18	+{"platform": "rickub", "round": 9, "sha": "39ba5db452f23f08907cce6b693297d70f8ec54a", "pushed_at": "2026-09-13T08:31:02.000Z"}
	19	+{"platform": "rickub", "round": 10, "sha": "66a47c3230a6a8269f9dc15bcb3176203bd2a225", "pushed_at": "2026-09-13T08:32:18.000Z"}
	20	+{"platform": "github", "round": 10, "sha": "66a47c3230a6a8269f9dc15bcb3176203bd2a225", "pushed_at": "2026-09-13T08:33:34.000Z"}

++{"platform": "rickub", "round": 1, "sha": "0baf736bf04d237117e719e3f8b2c8d5860f6bd4", "pushed_at": "2026-09-13T08:10:15Z"}
++{"platform": "github", "round": 1, "sha": "0baf736bf04d237117e719e3f8b2c8d5860f6bd4", "pushed_at": "2026-09-13T08:10:23Z"}
++{"platform": "rickub", "round": 2, "sha": "1914eedc7f8b4c474f71c821431005bb5cc7281a", "pushed_at": "2026-09-13T08:11:49.000Z"}
++{"platform": "github", "round": 2, "sha": "1914eedc7f8b4c474f71c821431005bb5cc7281a", "pushed_at": "2026-09-13T08:13:05.000Z"}
++{"platform": "github", "round": 3, "sha": "09e7984e3695d820dceab05c61327b2e486c5019", "pushed_at": "2026-09-13T08:14:22.000Z"}
++{"platform": "rickub", "round": 3, "sha": "09e7984e3695d820dceab05c61327b2e486c5019", "pushed_at": "2026-09-13T08:15:40.000Z"}
++{"platform": "rickub", "round": 4, "sha": "da2d24baabb9dce05ce66a0e95ca7b806a6fa086", "pushed_at": "2026-09-13T08:16:56.000Z"}
++{"platform": "github", "round": 4, "sha": "da2d24baabb9dce05ce66a0e95ca7b806a6fa086", "pushed_at": "2026-09-13T08:18:12.000Z"}
++{"platform": "github", "round": 5, "sha": "8421335df28fbd4614c026374c4bd3896cd5f184", "pushed_at": "2026-09-13T08:19:29.000Z"}
++{"platform": "rickub", "round": 5, "sha": "8421335df28fbd4614c026374c4bd3896cd5f184", "pushed_at": "2026-09-13T08:20:47.000Z"}
++{"platform": "rickub", "round": 6, "sha": "9990e0f6ae4ac4c66a0251fee9c88526dd99b975", "pushed_at": "2026-09-13T08:22:03.000Z"}
++{"platform": "github", "round": 6, "sha": "9990e0f6ae4ac4c66a0251fee9c88526dd99b975", "pushed_at": "2026-09-13T08:23:19.000Z"}
++{"platform": "github", "round": 7, "sha": "fba04eba4dee92ce17de4847289c54c813914bcc", "pushed_at": "2026-09-13T08:24:37.000Z"}
++{"platform": "rickub", "round": 7, "sha": "fba04eba4dee92ce17de4847289c54c813914bcc", "pushed_at": "2026-09-13T08:25:54.000Z"}
++{"platform": "rickub", "round": 8, "sha": "35efc9cc863d18db697b419bf6386089ead68a6d", "pushed_at": "2026-09-13T08:27:10.000Z"}
++{"platform": "github", "round": 8, "sha": "35efc9cc863d18db697b419bf6386089ead68a6d", "pushed_at": "2026-09-13T08:28:27.000Z"}
++{"platform": "github", "round": 9, "sha": "39ba5db452f23f08907cce6b693297d70f8ec54a", "pushed_at": "2026-09-13T08:29:45.000Z"}
++{"platform": "rickub", "round": 9, "sha": "39ba5db452f23f08907cce6b693297d70f8ec54a", "pushed_at": "2026-09-13T08:31:02.000Z"}
++{"platform": "rickub", "round": 10, "sha": "66a47c3230a6a8269f9dc15bcb3176203bd2a225", "pushed_at": "2026-09-13T08:32:18.000Z"}
++{"platform": "github", "round": 10, "sha": "66a47c3230a6a8269f9dc15bcb3176203bd2a225", "pushed_at": "2026-09-13T08:33:34.000Z"}