rickub/ci-bench · BLOG-draft.md

⑂ 6b67a7ef1caf6851afedfcbffa9de16df998b013

BLOG-draft.md · 111 lines · 5.4 KBmarkdown    Blame HistoryRaw
bench round 2: real-world suite results (n=11, 22/22 green) + publishable blog draft + durable push-times joiner  Olivier Girardot 1# Draft: We raced our CI against GitHub Actions — same workloads, eleven rounds, honest numbers
3*Draft for publication. Numbers are final (n=11 per platform, 2026-09-13;
4traces public in the repo). Keep the disclosures — including the row we lost —
5they are the credibility of the post.*
9Every CI provider claims to be fast. None of them publishes numbers you can
10check, because there is no official benchmark for CI platforms — nothing like
11SPEC exists for build infrastructure. So we built one, pointed it at our own
12platform and at GitHub Actions, and ran the same race eleven times.
14The harness is public: [rickub/ci-bench](https://rickub.com/rickub/ci-bench),
15mirrored on [github.com/ogirardot/ci-bench](https://github.com/ogirardot/ci-bench).
16It pushes the identical repository — identical commits, identical workflow —
17to both platforms, alternating which platform receives each push first, and
18records per-step timings for every run. All 22 runs succeeded; no step was
19skipped, no run excluded. You can run it yourself against us.
21## What we measured
23Two tiers of work, all pinned, all cold — every job on both platforms starts
24from a fresh machine, which is structural on both sides, not a tuning choice:
26- **Micro tier**: a Rust build (cold and warm) + tests, a Docker image build,
27  and a runner probe (single-core CPU loop, scratch-disk write/read).
28- **Real-world tier**: four projects any developer knows, cloned inside the
29  timed window and built at pinned tags:
30  - **ripgrep 15.2.0** (Rust) — release build of the tool you use daily
31  - **SQLite 3.45.1** (C) — a real `configure && make -j` of the amalgamation
32  - **TypeScript 5.9.3** (Node) — `npm ci` plus a compiler build
33  - **Guava 33.7.1** (Java) — a Maven build of the guava module
35We deliberately did not build ClickHouse: a full build needs dozens of cores
36and ~100 GB of disk — no shared-runner tier on any provider runs that. A
37benchmark step that times out is marketing, not measurement.
39## The honest setup
41We matched what a customer can compare: vCPU count. rickub's `large` runner
42(4 vCPU / 8 GiB) against GitHub's `ubuntu-latest` (4 vCPU / 16 GiB). We did
43NOT match silicon: rickub ran on our own bare-metal Firecracker fleet in
44Europe (Ryzen 9700X); GitHub ran on Azure's standard fleet. The probe makes
45that gap visible instead of hiding it — 594 vs 220 kops/s single-core, 2.7× —
46so you can judge how much of what follows is CPU versus platform.
48
50Medians over 11 interleaved rounds per platform (p95 in parentheses):
52| step | GitHub | rickub |
53|---|---|---|
54| ripgrep 15.2.0 (Rust) | 28.8 s | **12.7 s** (2.3×) |
55| SQLite 3.45.1 (C) | 80.9 s | **42.2 s** (1.9×) |
56| TypeScript 5.9.3 (Node) | 47.3 s | **22.9 s** (2.1×) |
57| Guava 33.7.1 (Java) | **38.0 s** | 40.3 s (GitHub +6%) |
58| Rust cold build (micro) | 13.0 s | **5.0 s** (2.6×) |
59| Docker build (micro) | 5.8 s | **3.1 s** (1.8×) |
60| push → job start | 9.8 s (p95 80 s) | **4.3 s** (p95 4.7 s) |
61| end-to-end pipeline | 243 s (197–342) | **134 s** (128–138) |
63Three of the four real projects finished about twice as fast on rickub, and
64the whole pipeline — every step, both tiers — ran **1.8× faster end to end**
65at the median. The variance story is stronger than the medians: rickub's
66eleven runs span 128–138 s wall-clock; GitHub's span 197–342 s, and one round
67waited 80 seconds in queue before a single instruction ran.
69## The row we lost — read it first
71Guava is the only step GitHub won, by 6%, and it is the most informative
72number in the table. A single-module Maven build is dominated by dependency
73download and single-threaded compilation — the least CPU-bound workload we
74tested. Where a build is not CPU-bound, the gap closes or flips. We lead
75with the probe's 2.7× single-core gap precisely so you can make this
76attribution yourself: the other three projects are what a 9700X does to a
77shared-cloud core when your build actually uses it.
79Equally: our storage-burst probe (a 2 GiB sequential write) came out
80*comparable* between platforms — we are not claiming magical disks. Fast
81single-core and a quiet, dedicated fleet are the honest summary of where
82these numbers come from.
84## What this does and doesn't prove
86It proves: for these five workloads, on this hardware, at this hour, jobs on
87rickub finished ~1.8–2.3× faster at the median, the advantage held at p95,
88and jobs started 5.5 s sooner after push — with one bare-metal node that a
89single person can rent for the price of a dinner.
91It doesn't prove: that a young fleet sustains GitHub's planetary queue depth
92(queue waits like that 80-second p95 happen everywhere, including here);
93that cached builds behave the same (we measure cold by construction — cache
94paths are a separate experiment we're running now); or that your workload
95behaves like ours — which is exactly why the harness, the traces, and every
96run id are public.
98## What's next
100The same harness now measures us continuously: a fair "as-consumed" arm on
101our 2-vCPU standard class, and — before we market it — an arm on the
102storage-accelerated runner class we're rolling out. If the numbers ever
103stop flattering us, this post is the receipts.
107*All traces: per-step timings for all 22 runs are in the public run history
108of both mirrors ([rickub](https://rickub.com/rickub/ci-bench/actions) ·
109[github](https://github.com/ogirardot/ci-bench/actions)). Methodology,
110caveats and the collection scripts live in the repo (README,
111RESULTS-realworld.md).*

bench round 2: real-world suite results (n=11, 22/22 green) + publishable blog draft + durable push-times joiner 6b67a7e Olivier Girardot 5h ago	1	# Draft: We raced our CI against GitHub Actions — same workloads, eleven rounds, honest numbers
	2
	3	*Draft for publication. Numbers are final (n=11 per platform, 2026-09-13;
	4	traces public in the repo). Keep the disclosures — including the row we lost —
	5	they are the credibility of the post.*
	6
	7	---
	8
	9	Every CI provider claims to be fast. None of them publishes numbers you can
	10	check, because there is no official benchmark for CI platforms — nothing like
	11	SPEC exists for build infrastructure. So we built one, pointed it at our own
	12	platform and at GitHub Actions, and ran the same race eleven times.
	13
	14	The harness is public: [rickub/ci-bench](https://rickub.com/rickub/ci-bench),
	15	mirrored on [github.com/ogirardot/ci-bench](https://github.com/ogirardot/ci-bench).
	16	It pushes the identical repository — identical commits, identical workflow —
	17	to both platforms, alternating which platform receives each push first, and
	18	records per-step timings for every run. All 22 runs succeeded; no step was
	19	skipped, no run excluded. You can run it yourself against us.
	20
	21	## What we measured
	22
	23	Two tiers of work, all pinned, all cold — every job on both platforms starts
	24	from a fresh machine, which is structural on both sides, not a tuning choice:
	25
	26	- Micro tier: a Rust build (cold and warm) + tests, a Docker image build,
	27	and a runner probe (single-core CPU loop, scratch-disk write/read).
	28	- Real-world tier: four projects any developer knows, cloned inside the
	29	timed window and built at pinned tags:
	30	- ripgrep 15.2.0 (Rust) — release build of the tool you use daily
	31	- SQLite 3.45.1 (C) — a real `configure && make -j` of the amalgamation
	32	- TypeScript 5.9.3 (Node) — `npm ci` plus a compiler build
	33	- Guava 33.7.1 (Java) — a Maven build of the guava module
	34
	35	We deliberately did not build ClickHouse: a full build needs dozens of cores
	36	and ~100 GB of disk — no shared-runner tier on any provider runs that. A
	37	benchmark step that times out is marketing, not measurement.
	38
	39	## The honest setup
	40
	41	We matched what a customer can compare: vCPU count. rickub's `large` runner
	42	(4 vCPU / 8 GiB) against GitHub's `ubuntu-latest` (4 vCPU / 16 GiB). We did
	43	NOT match silicon: rickub ran on our own bare-metal Firecracker fleet in
	44	Europe (Ryzen 9700X); GitHub ran on Azure's standard fleet. The probe makes
	45	that gap visible instead of hiding it — 594 vs 220 kops/s single-core, 2.7× —
	46	so you can judge how much of what follows is CPU versus platform.
	47
	48	## Results
	49
	50	Medians over 11 interleaved rounds per platform (p95 in parentheses):
	51
	52	\| step \| GitHub \| rickub \|
	53	\|---\|---\|---\|
	54	\| ripgrep 15.2.0 (Rust) \| 28.8 s \| 12.7 s (2.3×) \|
	55	\| SQLite 3.45.1 (C) \| 80.9 s \| 42.2 s (1.9×) \|
	56	\| TypeScript 5.9.3 (Node) \| 47.3 s \| 22.9 s (2.1×) \|
	57	\| Guava 33.7.1 (Java) \| 38.0 s \| 40.3 s (GitHub +6%) \|
	58	\| Rust cold build (micro) \| 13.0 s \| 5.0 s (2.6×) \|
	59	\| Docker build (micro) \| 5.8 s \| 3.1 s (1.8×) \|
	60	\| push → job start \| 9.8 s (p95 80 s) \| 4.3 s (p95 4.7 s) \|
	61	\| end-to-end pipeline \| 243 s (197–342) \| 134 s (128–138) \|
	62
	63	Three of the four real projects finished about twice as fast on rickub, and
	64	the whole pipeline — every step, both tiers — ran 1.8× faster end to end
65	at the median. The variance story is stronger than the medians: rickub's
66	eleven runs span 128–138 s wall-clock; GitHub's span 197–342 s, and one round
67	waited 80 seconds in queue before a single instruction ran.
68
69	## The row we lost — read it first
70
71	Guava is the only step GitHub won, by 6%, and it is the most informative
72	number in the table. A single-module Maven build is dominated by dependency
73	download and single-threaded compilation — the least CPU-bound workload we
74	tested. Where a build is not CPU-bound, the gap closes or flips. We lead
75	with the probe's 2.7× single-core gap precisely so you can make this
76	attribution yourself: the other three projects are what a 9700X does to a
77	shared-cloud core when your build actually uses it.
78
79	Equally: our storage-burst probe (a 2 GiB sequential write) came out
80	comparable between platforms — we are not claiming magical disks. Fast
81	single-core and a quiet, dedicated fleet are the honest summary of where
82	these numbers come from.
83
84	## What this does and doesn't prove
85
86	It proves: for these five workloads, on this hardware, at this hour, jobs on
87	rickub finished ~1.8–2.3× faster at the median, the advantage held at p95,
88	and jobs started 5.5 s sooner after push — with one bare-metal node that a
89	single person can rent for the price of a dinner.
90
91	It doesn't prove: that a young fleet sustains GitHub's planetary queue depth
92	(queue waits like that 80-second p95 happen everywhere, including here);
93	that cached builds behave the same (we measure cold by construction — cache
94	paths are a separate experiment we're running now); or that your workload
95	behaves like ours — which is exactly why the harness, the traces, and every
96	run id are public.
97
98	## What's next
99
100	The same harness now measures us continuously: a fair "as-consumed" arm on
101	our 2-vCPU standard class, and — before we market it — an arm on the
102	storage-accelerated runner class we're rolling out. If the numbers ever
103	stop flattering us, this post is the receipts.
104
105	---
106
107	*All traces: per-step timings for all 22 runs are in the public run history
108	of both mirrors ([rickub](https://rickub.com/rickub/ci-bench/actions) ·
109	[github](https://github.com/ogirardot/ci-bench/actions)). Methodology,
110	caveats and the collection scripts live in the repo (README,
111	RESULTS-realworld.md).*