Draft: We raced our CI against GitHub Actions — same workloads, eleven rounds, honest numbers

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111

# Draft: We raced our CI against GitHub Actions — same workloads, eleven rounds, honest numbers

*Draft for publication. Numbers are final (n=11 per platform, 2026-09-13;
traces public in the repo). Keep the disclosures — including the row we lost —
they are the credibility of the post.*

---

Every CI provider claims to be fast. None of them publishes numbers you can
check, because there is no official benchmark for CI platforms — nothing like
SPEC exists for build infrastructure. So we built one, pointed it at our own
platform and at GitHub Actions, and ran the same race eleven times.

The harness is public: [rickub/ci-bench](https://rickub.com/rickub/ci-bench),
mirrored on [github.com/ogirardot/ci-bench](https://github.com/ogirardot/ci-bench).
It pushes the identical repository — identical commits, identical workflow —
to both platforms, alternating which platform receives each push first, and
records per-step timings for every run. All 22 runs succeeded; no step was
skipped, no run excluded. You can run it yourself against us.

## What we measured

Two tiers of work, all pinned, all cold — every job on both platforms starts
from a fresh machine, which is structural on both sides, not a tuning choice:

- **Micro tier**: a Rust build (cold and warm) + tests, a Docker image build,
  and a runner probe (single-core CPU loop, scratch-disk write/read).
- **Real-world tier**: four projects any developer knows, cloned inside the
  timed window and built at pinned tags:
  - **ripgrep 15.2.0** (Rust) — release build of the tool you use daily
  - **SQLite 3.45.1** (C) — a real `configure && make -j` of the amalgamation
  - **TypeScript 5.9.3** (Node) — `npm ci` plus a compiler build
  - **Guava 33.7.1** (Java) — a Maven build of the guava module

We deliberately did not build ClickHouse: a full build needs dozens of cores
and ~100 GB of disk — no shared-runner tier on any provider runs that. A
benchmark step that times out is marketing, not measurement.

## The honest setup

We matched what a customer can compare: vCPU count. rickub's `large` runner
(4 vCPU / 8 GiB) against GitHub's `ubuntu-latest` (4 vCPU / 16 GiB). We did
NOT match silicon: rickub ran on our own bare-metal Firecracker fleet in
Europe (Ryzen 9700X); GitHub ran on Azure's standard fleet. The probe makes
that gap visible instead of hiding it — 594 vs 220 kops/s single-core, 2.7× —
so you can judge how much of what follows is CPU versus platform.

## Results

Medians over 11 interleaved rounds per platform (p95 in parentheses):

| step | GitHub | rickub |
|---|---|---|
| ripgrep 15.2.0 (Rust) | 28.8 s | **12.7 s** (2.3×) |
| SQLite 3.45.1 (C) | 80.9 s | **42.2 s** (1.9×) |
| TypeScript 5.9.3 (Node) | 47.3 s | **22.9 s** (2.1×) |
| Guava 33.7.1 (Java) | **38.0 s** | 40.3 s (GitHub +6%) |
| Rust cold build (micro) | 13.0 s | **5.0 s** (2.6×) |
| Docker build (micro) | 5.8 s | **3.1 s** (1.8×) |
| push → job start | 9.8 s (p95 80 s) | **4.3 s** (p95 4.7 s) |
| end-to-end pipeline | 243 s (197–342) | **134 s** (128–138) |

Three of the four real projects finished about twice as fast on rickub, and
the whole pipeline — every step, both tiers — ran **1.8× faster end to end**
at the median. The variance story is stronger than the medians: rickub's
eleven runs span 128–138 s wall-clock; GitHub's span 197–342 s, and one round
waited 80 seconds in queue before a single instruction ran.

## The row we lost — read it first

Guava is the only step GitHub won, by 6%, and it is the most informative
number in the table. A single-module Maven build is dominated by dependency
download and single-threaded compilation — the least CPU-bound workload we
tested. Where a build is not CPU-bound, the gap closes or flips. We lead
with the probe's 2.7× single-core gap precisely so you can make this
attribution yourself: the other three projects are what a 9700X does to a
shared-cloud core when your build actually uses it.

Equally: our storage-burst probe (a 2 GiB sequential write) came out
*comparable* between platforms — we are not claiming magical disks. Fast
single-core and a quiet, dedicated fleet are the honest summary of where
these numbers come from.

## What this does and doesn't prove

It proves: for these five workloads, on this hardware, at this hour, jobs on
rickub finished ~1.8–2.3× faster at the median, the advantage held at p95,
and jobs started 5.5 s sooner after push — with one bare-metal node that a
single person can rent for the price of a dinner.

It doesn't prove: that a young fleet sustains GitHub's planetary queue depth
(queue waits like that 80-second p95 happen everywhere, including here);
that cached builds behave the same (we measure cold by construction — cache
paths are a separate experiment we're running now); or that your workload
behaves like ours — which is exactly why the harness, the traces, and every
run id are public.

## What's next

The same harness now measures us continuously: a fair "as-consumed" arm on
our 2-vCPU standard class, and — before we market it — an arm on the
storage-accelerated runner class we're rolling out. If the numbers ever
stop flattering us, this post is the receipts.

---

*All traces: per-step timings for all 22 runs are in the public run history
of both mirrors ([rickub](https://rickub.com/rickub/ci-bench/actions) ·
[github](https://github.com/ogirardot/ci-bench/actions)). Methodology,
caveats and the collection scripts live in the repo (README,
RESULTS-realworld.md).*