nandi/jolt-native · 089e6f7

⑂ 089e6f7

Carry the worker's own image, rather than borrow one that had the parts

The public rbe-ubuntu24-04 was quietly supplying what this graph depends
on and never declared. Asking for a remote-only build named them one at a
time: pkg-config, without which alsa-sys panics before emitting a link
directive; python3 — and with its standard library, since the prelude's
Rust rules import json from a worker-side pass; libclang and the kernel's
headers, which v4l2r's bindgen needs to parse videodev2.h; and libc's
own, because that header includes <sys/time.h>. A cargo build takes the
last two from the justfile's environment, which names this machine's zig
headers, and no remote action inherits that.

So the image is this project's now, built by .gitlab-ci.yml from
ci/rbe.Dockerfile and published to the container registry. kaniko rather
than docker-in-docker: no privileged runner, and the layer cache is the
registry's own.

platforms/BUCK names it by digest rather than by tag, and has to: the
executor caches an image under the reference it is given, so a rebuilt
`:latest` went on serving the old layers and the build kept failing for a
package the image already carried.

//:libjoltmoq now builds with `--remote-only` — 145 commands, none of
them local — and the object it produces names libasound and has nothing
undefined left in it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

nandi committed 2026-08-31T23:13:41-07:00 Browse files

089e6f7 parent: 3b4eb6b

added .gitlab-ci.yml +33 -0 Viewed

new file mode 100644
@@ -0,0 +1,33 @@
	1	+# The remote executor's image, built and published here.
	2	+#
	3	+# buck2's remote execution names a container image by URL, and the worker pulls
	4	+# it; nothing about this pipeline runs the build itself. So this fires only when
	5	+# the image's own definition changes, and the tag it publishes is what
	6	+# platforms/BUCK names.
	7	+#
	8	+# The registry has to be readable without credentials, because the thing pulling
	9	+# is BuildBuddy's executor and not this project's CI: Settings → General →
	10	+# Visibility → Container registry, set to Everyone.
	11	+stages: [image]
	12	+
	13	+rbe-image:
	14	+ stage: image
	15	+ image:
	16	+ name: gcr.io/kaniko-project/executor:debug
	17	+ entrypoint: [""]
	18	+ rules:
	19	+ # Only on the default branch, and only when the image's definition moves.
	20	+ - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
	21	+ changes: [ci/rbe.Dockerfile, .gitlab-ci.yml]
	22	+ script:
	23	+ # kaniko rather than docker-in-docker: no privileged runner, and the layer
	24	+ # cache is the registry's own.
	25	+ - /kaniko/executor
	26	+ --context "$CI_PROJECT_DIR"
	27	+ --dockerfile "$CI_PROJECT_DIR/ci/rbe.Dockerfile"
	28	+ --destination "$CI_REGISTRY_IMAGE/rbe:$CI_COMMIT_SHORT_SHA"
	29	+ --destination "$CI_REGISTRY_IMAGE/rbe:latest"
	30	+ --digest-file /tmp/digest
	31	+ # The digest, because a tag is a moving target and an execution platform
	32	+ # should name bytes. Paste it into platforms/BUCK.
	33	+ - echo "$CI_REGISTRY_IMAGE/rbe@$(cat /tmp/digest)"

++# The remote executor's image, built and published here.
++#
++# buck2's remote execution names a container image by URL, and the worker pulls
++# it; nothing about this pipeline runs the build itself. So this fires only when
++# the image's own definition changes, and the tag it publishes is what
++# platforms/BUCK names.
++#
++# The registry has to be readable without credentials, because the thing pulling
++# is BuildBuddy's executor and not this project's CI: Settings → General →
++# Visibility → Container registry, set to Everyone.
++stages: [image]
++
++rbe-image:
++  stage: image
++  image:
++    name: gcr.io/kaniko-project/executor:debug
++    entrypoint: [""]
++  rules:
++    # Only on the default branch, and only when the image's definition moves.
++    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
++      changes: [ci/rbe.Dockerfile, .gitlab-ci.yml]
++  script:
++    # kaniko rather than docker-in-docker: no privileged runner, and the layer
++    # cache is the registry's own.
++    - /kaniko/executor
++      --context "$CI_PROJECT_DIR"
++      --dockerfile "$CI_PROJECT_DIR/ci/rbe.Dockerfile"
++      --destination "$CI_REGISTRY_IMAGE/rbe:$CI_COMMIT_SHORT_SHA"
++      --destination "$CI_REGISTRY_IMAGE/rbe:latest"
++      --digest-file /tmp/digest
++    # The digest, because a tag is a moving target and an execution platform
++    # should name bytes. Paste it into platforms/BUCK.
++    - echo "$CI_REGISTRY_IMAGE/rbe@$(cat /tmp/digest)"

added ci/rbe.Dockerfile +42 -0 Viewed

new file mode 100644
@@ -0,0 +1,42 @@
	1	+# The remote executor's image.
	2	+#
	3	+# buck2 fetches its own toolchains — zig, rustc, the Rust standard library, the
	4	+# NDK — from the pins in scripts/*.dotslash, so almost nothing here has to be
	5	+# installed. Almost: a handful of `-sys` crates ask pkg-config where a system
	6	+# library is, and one of them then links against it.
	7	+#
	8	+# That is why this image exists rather than gcr.io/flame-public/rbe-ubuntu24-04,
	9	+# which carries neither. Without pkg-config, alsa-sys's build script panics
	10	+# before it can emit a single link directive; without libasound2-dev, the flags
	11	+# it emits name a library the linker cannot find.
	12	+#
	13	+# 24.04 for the same reason the public image was pinned to it: a build script is
	14	+# compiled here and run on the worker, and 22.04's glibc 2.35 is old enough to
	15	+# fail that before a crate is compiled.
	16	+FROM ubuntu:24.04
	17	+
	18	+# ca-certificates because an action fetches over TLS, and python3 because the
	19	+# prelude's own Rust rules shell out to it — a dep-file pass runs on the worker,
	20	+# not here. The rest is the system half of what the graph's build scripts look
	21	+# for: alsa-sys asks pkg-config for libasound, and v4l2r runs bindgen over
	22	+# `linux/videodev2.h`, which needs libclang to parse it and the kernel's own
	23	+# headers to find it — and libc's own, since videodev2.h includes <sys/time.h>. A cargo build takes both from the justfile's environment,
	24	+# which names this machine's zig headers — nothing a remote action inherits, so
	25	+# the worker has to carry them.
	26	+#
	27	+# libclang1 and its builtin headers rather than libclang-dev, which drags the
	28	+# whole clang toolchain in and took the image from 130 MB to 731 MB. bindgen
	29	+# loads the library and reads those headers; it has no use for the compiler.
	30	+#
	31	+# Kept to what a build has actually asked for: an unused package here is an
	32	+# image every action pulls.
	33	+RUN apt-get update && apt-get install --no-install-recommends -y \
	34	+ ca-certificates \
	35	+ libasound2-dev \
	36	+ libc6-dev \
	37	+ libclang1-18 \
	38	+ libclang-common-18-dev \
	39	+ linux-libc-dev \
	40	+ pkg-config \
	41	+ python3 \
	42	+ && rm -rf /var/lib/apt/lists/*

++# The remote executor's image.
++#
++# buck2 fetches its own toolchains — zig, rustc, the Rust standard library, the
++# NDK — from the pins in scripts/*.dotslash, so almost nothing here has to be
++# installed. Almost: a handful of `-sys` crates ask pkg-config where a system
++# library is, and one of them then links against it.
++#
++# That is why this image exists rather than gcr.io/flame-public/rbe-ubuntu24-04,
++# which carries neither. Without pkg-config, alsa-sys's build script panics
++# before it can emit a single link directive; without libasound2-dev, the flags
++# it emits name a library the linker cannot find.
++#
++# 24.04 for the same reason the public image was pinned to it: a build script is
++# compiled here and *run* on the worker, and 22.04's glibc 2.35 is old enough to
++# fail that before a crate is compiled.
++FROM ubuntu:24.04
++
++# ca-certificates because an action fetches over TLS, and python3 because the
++# prelude's own Rust rules shell out to it — a dep-file pass runs on the worker,
++# not here. The rest is the system half of what the graph's build scripts look
++# for: alsa-sys asks pkg-config for libasound, and v4l2r runs bindgen over
++# `linux/videodev2.h`, which needs libclang to parse it and the kernel's own
++# headers to find it — and libc's own, since videodev2.h includes <sys/time.h>. A cargo build takes both from the justfile's environment,
++# which names this machine's zig headers — nothing a remote action inherits, so
++# the worker has to carry them.
++#
++# libclang1 and its builtin headers rather than libclang-dev, which drags the
++# whole clang toolchain in and took the image from 130 MB to 731 MB. bindgen
++# loads the library and reads those headers; it has no use for the compiler.
++#
++# Kept to what a build has actually asked for: an unused package here is an
++# image every action pulls.
++RUN apt-get update && apt-get install --no-install-recommends -y \
++        ca-certificates \
++        libasound2-dev \
++        libc6-dev \
++        libclang1-18 \
++        libclang-common-18-dev \
++        linux-libc-dev \
++        pkg-config \
++        python3 \
++    && rm -rf /var/lib/apt/lists/*

modified platforms/BUCK +19 -6 Viewed

@@ -23,12 +23,25 @@ execution_platform(
23	23	remote_enabled = True,
24	24	remote_execution_properties = {
25	25	"OSFamily": "Linux",
26		- # 24.04, not 22.04: a build script compiled against this machine's glibc
27		- # has to run on the worker, and 22.04's is 2.35 — old enough that
28		- # thiserror's build script died with "GLIBC_2.39 not found" before a
29		- # single crate was compiled. frq hit the same floor from the other side
30		- # (jolt itself needs 2.38) and moved for the same reason.
31		- "container-image": "docker://gcr.io/flame-public/rbe-ubuntu24-04:latest",
	26	+ # This project's own image, built by .gitlab-ci.yml from ci/rbe.Dockerfile
	27	+ # and published to its container registry. The public
	28	+ # gcr.io/flame-public/rbe-ubuntu24-04 carries neither pkg-config nor
	29	+ # libasound2-dev, and alsa-sys's build script needs the first to find the
	30	+ # second — so with that image the graph could not be built remotely at
	31	+ # all, whatever the executor was asked to do.
	32	+ #
	33	+ # Still 24.04, and for the reason that image was pinned to it: a build
	34	+ # script is compiled on the host and run on the worker, and 22.04's
	35	+ # glibc 2.35 is old enough that thiserror's died with "GLIBC_2.39 not
	36	+ # found" before a single crate was compiled. frq hit the same floor from
	37	+ # the other side (jolt itself needs 2.38) and moved for the same reason.
	38	+ #
	39	+ # By digest, not by tag. An execution platform ought to name bytes, and
	40	+ # here it has to: the executor caches an image under the reference it
	41	+ # was given, so a rebuilt `:latest` kept serving the old layers and the
	42	+ # build went on failing for a package the image already had. The
	43	+ # pipeline prints this line on every publish.
	44	+ "container-image": "docker://registry.gitlab.com/nandithebull/jolt-native/rbe@sha256:3de0e1c922b35f40023db8d50bbf606a3cb8168b5b8b4a27aac08f06cf7e0328",
32	45	},
33	46	use_windows_path_separators = host_info().os.is_windows,
34	47	visibility = ["PUBLIC"],

      remote_enabled = True,
      remote_execution_properties = {
          "OSFamily": "Linux",
--        # 24.04, not 22.04: a build script compiled against this machine's glibc
++        # This project's own image, built by .gitlab-ci.yml from ci/rbe.Dockerfile
--        # has to *run* on the worker, and 22.04's is 2.35 — old enough that
++        # and published to its container registry. The public
--        # thiserror's build script died with "GLIBC_2.39 not found" before a
++        # gcr.io/flame-public/rbe-ubuntu24-04 carries neither pkg-config nor
--        # single crate was compiled. frq hit the same floor from the other side
++        # libasound2-dev, and alsa-sys's build script needs the first to find the
--        # (jolt itself needs 2.38) and moved for the same reason.
++        # second — so with that image the graph could not be built remotely at
--        "container-image": "docker://gcr.io/flame-public/rbe-ubuntu24-04:latest",
++        # all, whatever the executor was asked to do.
++        #
++        # Still 24.04, and for the reason that image was pinned to it: a build
++        # script is compiled on the host and *run* on the worker, and 22.04's
++        # glibc 2.35 is old enough that thiserror's died with "GLIBC_2.39 not
++        # found" before a single crate was compiled. frq hit the same floor from
++        # the other side (jolt itself needs 2.38) and moved for the same reason.
++        #
++        # By digest, not by tag. An execution platform ought to name bytes, and
++        # here it has to: the executor caches an image under the reference it
++        # was given, so a rebuilt `:latest` kept serving the old layers and the
++        # build went on failing for a package the image already had. The
++        # pipeline prints this line on every publish.
++        "container-image": "docker://registry.gitlab.com/nandithebull/jolt-native/rbe@sha256:3de0e1c922b35f40023db8d50bbf606a3cb8168b5b8b4a27aac08f06cf7e0328",
      },
      use_windows_path_separators = host_info().os.is_windows,
      visibility = ["PUBLIC"],