Build the image in CI and roll it out to Railway
Railway cannot watch a GitLab repo — it deploys from a GitHub repo, a local directory, or an image — so pushes here reached production only by running `railway up` from a working tree, which shipped uncommitted changes and relied on Railway's builder rather than the flake. Build .#image in CI instead, push it to this project's registry, and point the Railway service at the new tag. The project is public, so the registry allows anonymous pulls and Railway needs no credentials. The deploy job is skipped without RAILWAY_TOKEN, so the image still builds and publishes if the token is missing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
7f2c924 parent: ceffbf8 added
.gitlab-ci.yml +78 -0 | new file mode 100644 | ||
| @@ -0,0 +1,78 @@ | ||
| 1 | +# Build the container image with Nix and hand it to Railway. | |
| 2 | +# | |
| 3 | +# The flake is the build: `nix build .#image` (flake.nix, dockerTools | |
| 4 | +# streamLayeredImage) produces the same image locally and in CI. Railway has no | |
| 5 | +# GitLab repo integration, so it cannot watch this repo directly; instead CI | |
| 6 | +# pushes the image to this project's own container registry and then points the | |
| 7 | +# Railway service at the new tag. | |
| 8 | + | |
| 9 | +stages: | |
| 10 | + - build | |
| 11 | + - deploy | |
| 12 | + | |
| 13 | +variables: | |
| 14 | + IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA | |
| 15 | + IMAGE_LATEST: $CI_REGISTRY_IMAGE:latest | |
| 16 | + RAILWAY_SERVICE_ID: 60dd4db8-32c7-434f-8b46-c3274dc76ba5 | |
| 17 | + RAILWAY_ENVIRONMENT_ID: 03c2a9fa-6cdf-4bb9-b924-4181d5a3445b | |
| 18 | + RAILWAY_API: https://backboard.railway.com/graphql/v2 | |
| 19 | + | |
| 20 | +build_image: | |
| 21 | + stage: build | |
| 22 | + image: nixos/nix:latest | |
| 23 | + variables: | |
| 24 | + NIX_CONFIG: "experimental-features = nix-command flakes" | |
| 25 | + rules: | |
| 26 | + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH | |
| 27 | + script: | |
| 28 | + # streamLayeredImage's output *is* a script that streams the tarball. | |
| 29 | + - | | |
| 30 | + nix build .#image --print-out-paths --no-link -L > store-path | |
| 31 | + "$(cat store-path)" > glean-image.tar | |
| 32 | + # skopeo pushes the docker-archive straight to the registry: no docker | |
| 33 | + # daemon, so this works on a plain shared runner. | |
| 34 | + - | | |
| 35 | + nix shell nixpkgs#skopeo -c skopeo copy \ | |
| 36 | + --dest-creds "gitlab-ci-token:$CI_JOB_TOKEN" \ | |
| 37 | + docker-archive:glean-image.tar \ | |
| 38 | + "docker://$IMAGE" | |
| 39 | + - | | |
| 40 | + nix shell nixpkgs#skopeo -c skopeo copy \ | |
| 41 | + --dest-creds "gitlab-ci-token:$CI_JOB_TOKEN" \ | |
| 42 | + "docker://$IMAGE" "docker://$IMAGE_LATEST" | |
| 43 | + | |
| 44 | +deploy_railway: | |
| 45 | + stage: deploy | |
| 46 | + image: alpine:latest | |
| 47 | + needs: [build_image] | |
| 48 | + # Without the token the pipeline still builds and publishes the image; only | |
| 49 | + # the rollout is skipped, rather than failing the whole pipeline. | |
| 50 | + rules: | |
| 51 | + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $RAILWAY_TOKEN | |
| 52 | + before_script: | |
| 53 | + - apk add --no-cache curl jq | |
| 54 | + script: | |
| 55 | + - | | |
| 56 | + call() { | |
| 57 | + curl -sS -X POST "$RAILWAY_API" \ | |
| 58 | + -H "Content-Type: application/json" \ | |
| 59 | + -H "Project-Access-Token: $RAILWAY_TOKEN" \ | |
| 60 | + --data "$1" | tee response.json | |
| 61 | + if jq -e '.errors' response.json >/dev/null; then | |
| 62 | + echo "Railway API returned errors" >&2 | |
| 63 | + exit 1 | |
| 64 | + fi | |
| 65 | + } | |
| 66 | + # Point the service at the tag this pipeline just published. | |
| 67 | + - | | |
| 68 | + call "$(jq -nc \ | |
| 69 | + --arg s "$RAILWAY_SERVICE_ID" --arg e "$RAILWAY_ENVIRONMENT_ID" --arg i "$IMAGE" \ | |
| 70 | + '{query:"mutation($s:String!,$e:String!,$in:ServiceInstanceUpdateInput!){serviceInstanceUpdate(serviceId:$s,environmentId:$e,input:$in)}", | |
| 71 | + variables:{s:$s,e:$e,in:{source:{image:$i}}}}')" | |
| 72 | + # Roll it out. | |
| 73 | + - | | |
| 74 | + call "$(jq -nc \ | |
| 75 | + --arg s "$RAILWAY_SERVICE_ID" --arg e "$RAILWAY_ENVIRONMENT_ID" \ | |
| 76 | + '{query:"mutation($s:String!,$e:String!){serviceInstanceDeploy(serviceId:$s,environmentId:$e)}", | |
| 77 | + variables:{s:$s,e:$e}}')" | |
| 78 | + - echo "Deployed $IMAGE" | |
| new file mode 100644 | |||
| @@ -0,0 +1,78 @@ | |||
| 1 | +# Build the container image with Nix and hand it to Railway. | ||
| 2 | +# | ||
| 3 | +# The flake is the build: `nix build .#image` (flake.nix, dockerTools | ||
| 4 | +# streamLayeredImage) produces the same image locally and in CI. Railway has no | ||
| 5 | +# GitLab repo integration, so it cannot watch this repo directly; instead CI | ||
| 6 | +# pushes the image to this project's own container registry and then points the | ||
| 7 | +# Railway service at the new tag. | ||
| 8 | + | ||
| 9 | +stages: | ||
| 10 | + - build | ||
| 11 | + - deploy | ||
| 12 | + | ||
| 13 | +variables: | ||
| 14 | + IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA | ||
| 15 | + IMAGE_LATEST: $CI_REGISTRY_IMAGE:latest | ||
| 16 | + RAILWAY_SERVICE_ID: 60dd4db8-32c7-434f-8b46-c3274dc76ba5 | ||
| 17 | + RAILWAY_ENVIRONMENT_ID: 03c2a9fa-6cdf-4bb9-b924-4181d5a3445b | ||
| 18 | + RAILWAY_API: https://backboard.railway.com/graphql/v2 | ||
| 19 | + | ||
| 20 | +build_image: | ||
| 21 | + stage: build | ||
| 22 | + image: nixos/nix:latest | ||
| 23 | + variables: | ||
| 24 | + NIX_CONFIG: "experimental-features = nix-command flakes" | ||
| 25 | + rules: | ||
| 26 | + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH | ||
| 27 | + script: | ||
| 28 | + # streamLayeredImage's output *is* a script that streams the tarball. | ||
| 29 | + - | | ||
| 30 | + nix build .#image --print-out-paths --no-link -L > store-path | ||
| 31 | + "$(cat store-path)" > glean-image.tar | ||
| 32 | + # skopeo pushes the docker-archive straight to the registry: no docker | ||
| 33 | + # daemon, so this works on a plain shared runner. | ||
| 34 | + - | | ||
| 35 | + nix shell nixpkgs#skopeo -c skopeo copy \ | ||
| 36 | + --dest-creds "gitlab-ci-token:$CI_JOB_TOKEN" \ | ||
| 37 | + docker-archive:glean-image.tar \ | ||
| 38 | + "docker://$IMAGE" | ||
| 39 | + - | | ||
| 40 | + nix shell nixpkgs#skopeo -c skopeo copy \ | ||
| 41 | + --dest-creds "gitlab-ci-token:$CI_JOB_TOKEN" \ | ||
| 42 | + "docker://$IMAGE" "docker://$IMAGE_LATEST" | ||
| 43 | + | ||
| 44 | +deploy_railway: | ||
| 45 | + stage: deploy | ||
| 46 | + image: alpine:latest | ||
| 47 | + needs: [build_image] | ||
| 48 | + # Without the token the pipeline still builds and publishes the image; only | ||
| 49 | + # the rollout is skipped, rather than failing the whole pipeline. | ||
| 50 | + rules: | ||
| 51 | + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $RAILWAY_TOKEN | ||
| 52 | + before_script: | ||
| 53 | + - apk add --no-cache curl jq | ||
| 54 | + script: | ||
| 55 | + - | | ||
| 56 | + call() { | ||
| 57 | + curl -sS -X POST "$RAILWAY_API" \ | ||
| 58 | + -H "Content-Type: application/json" \ | ||
| 59 | + -H "Project-Access-Token: $RAILWAY_TOKEN" \ | ||
| 60 | + --data "$1" | tee response.json | ||
| 61 | + if jq -e '.errors' response.json >/dev/null; then | ||
| 62 | + echo "Railway API returned errors" >&2 | ||
| 63 | + exit 1 | ||
| 64 | + fi | ||
| 65 | + } | ||
| 66 | + # Point the service at the tag this pipeline just published. | ||
| 67 | + - | | ||
| 68 | + call "$(jq -nc \ | ||
| 69 | + --arg s "$RAILWAY_SERVICE_ID" --arg e "$RAILWAY_ENVIRONMENT_ID" --arg i "$IMAGE" \ | ||
| 70 | + '{query:"mutation($s:String!,$e:String!,$in:ServiceInstanceUpdateInput!){serviceInstanceUpdate(serviceId:$s,environmentId:$e,input:$in)}", | ||
| 71 | + variables:{s:$s,e:$e,in:{source:{image:$i}}}}')" | ||
| 72 | + # Roll it out. | ||
| 73 | + - | | ||
| 74 | + call "$(jq -nc \ | ||
| 75 | + --arg s "$RAILWAY_SERVICE_ID" --arg e "$RAILWAY_ENVIRONMENT_ID" \ | ||
| 76 | + '{query:"mutation($s:String!,$e:String!){serviceInstanceDeploy(serviceId:$s,environmentId:$e)}", | ||
| 77 | + variables:{s:$s,e:$e}}')" | ||
| 78 | + - echo "Deployed $IMAGE" | ||
modified
docs/deploy.md +49 -37 | @@ -1,39 +1,51 @@ | ||
| 1 | 1 | # Deploying |
| 2 | 2 | |
| 3 | -The canonical build is the Nix flake (`flake.nix`): `nix build .#image` produces | |
| 4 | -the container image, and `make image-push IMAGE=<ref>` publishes it. That path | |
| 5 | -still works and is the reproducible one. | |
| 6 | - | |
| 7 | -## Railway (production) | |
| 8 | - | |
| 9 | -The `glean` service in the `glean` project builds **from source** on | |
| 10 | -`railway up`, rather than running a prebuilt image. Railway ignores in-repo | |
| 11 | -build config here — `railway.toml`, `nixpacks.toml` and a start script in | |
| 12 | -`scripts/` were all silently skipped — so the configuration lives in the | |
| 13 | -service settings and environment variables instead: | |
| 14 | - | |
| 15 | -| Setting | Value | | |
| 16 | -|---|---| | |
| 17 | -| Builder | `NIXPACKS` (the Railpack default cannot build this repo) | | |
| 18 | -| Build command | `cd web && bun install && cd .. && make build` | | |
| 19 | -| Start command | `sh -c "GLEAN_API_URL=http://127.0.0.1:8080 GLEAN_ADDR=127.0.0.1:8080 /app/glean & exec node /app/web/build/index.js"` | | |
| 20 | -| `NIXPACKS_PKGS` | `bun nodejs gnumake gcc gawk gnugrep` | | |
| 21 | -| `CGO_ENABLED` | `1` | | |
| 22 | - | |
| 23 | -Notes on why each is needed: | |
| 24 | - | |
| 25 | -- **`CGO_ENABLED=1`** — `sqlite-vec-go-bindings/cgo` and `mattn/go-sqlite3` are | |
| 26 | - cgo packages. The builder defaults to cgo off, which fails with "build | |
| 27 | - constraints exclude all Go files". | |
| 28 | -- **`NIXPACKS_PKGS`** — the Go provider installs only Go. The frontend needs | |
| 29 | - `bun` to build and `nodejs` to serve; `make build` needs make/grep/awk. This | |
| 30 | - mirrors the dependency list in `.tangled/workflows`. Use `nodejs`, not | |
| 31 | - `nodejs_22`: the pinned nixpkgs has no such attribute. | |
| 32 | -- **Build command** — the builder's default `go build -o out` skips both the | |
| 33 | - `fts5` tag and the SvelteKit build. `make build` does both. | |
| 34 | -- **Start command** — mirrors the flake's `mkEntrypoint`: the Go API on | |
| 35 | - loopback:8080 with the SvelteKit Node server in front on `$PORT`. It is | |
| 36 | - inline rather than a script file because the runtime image does not carry | |
| 37 | - `scripts/`. | |
| 38 | - | |
| 39 | -Deploy with `railway up` from the repo root. | |
| 3 | +The Nix flake is the build. `nix build .#image` (flake.nix, dockerTools | |
| 4 | +`streamLayeredImage`) produces the container image, and the same command runs | |
| 5 | +locally and in CI, so a local build and a deployed build are the same artifact. | |
| 6 | + | |
| 7 | +## Pipeline | |
| 8 | + | |
| 9 | +Railway has no GitLab repo integration — it deploys from a GitHub repo, a local | |
| 10 | +directory, or a container image — so it cannot watch this repo. `.gitlab-ci.yml` | |
| 11 | +closes that gap on every push to `main`: | |
| 12 | + | |
| 13 | +1. `build_image` builds `.#image` with Nix and pushes it to this project's own | |
| 14 | + container registry as `$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA` (and | |
| 15 | + `:latest`). skopeo pushes the docker-archive directly, so no Docker daemon | |
| 16 | + is needed on the runner. | |
| 17 | +2. `deploy_railway` points the Railway service at that tag | |
| 18 | + (`serviceInstanceUpdate`) and rolls it out (`serviceInstanceDeploy`). | |
| 19 | + | |
| 20 | +The project is public, so the registry allows anonymous pulls and Railway needs | |
| 21 | +no registry credentials. Making it private again would require registry | |
| 22 | +credentials on the Railway side, and a paid plan — private registry sources are | |
| 23 | +a Pro feature. | |
| 24 | + | |
| 25 | +### Required CI variable | |
| 26 | + | |
| 27 | +`deploy_railway` needs `RAILWAY_TOKEN`, a Railway **project token** for the | |
| 28 | +`glean` project, set under Settings → CI/CD → Variables (protected, masked). | |
| 29 | +Without it the deploy job is skipped rather than failing: the image is still | |
| 30 | +built and published, and can be rolled out by hand. | |
| 31 | + | |
| 32 | +The service and environment IDs are not secret and are inlined in | |
| 33 | +`.gitlab-ci.yml`. | |
| 34 | + | |
| 35 | +## Deploying by hand | |
| 36 | + | |
| 37 | +``` | |
| 38 | +make image-push IMAGE=registry.gitlab.com/nandithebull/glean:<tag> | |
| 39 | +``` | |
| 40 | + | |
| 41 | +Then point the service at `<tag>` in the Railway dashboard, or re-run the | |
| 42 | +`deploy_railway` job. | |
| 43 | + | |
| 44 | +## History | |
| 45 | + | |
| 46 | +The service briefly built from source on `railway up` with Nixpacks. That | |
| 47 | +needed `CGO_ENABLED=1` (the sqlite cgo packages), a `NIXPACKS_PKGS` toolchain | |
| 48 | +list, and an overridden build command, because Railway ignores in-repo | |
| 49 | +`railway.toml` / `nixpacks.toml` here. It also deployed the working tree rather | |
| 50 | +than a commit. The image pipeline above replaces it; those service settings | |
| 51 | +have been removed. | |
| @@ -1,39 +1,51 @@ | |||
| 1 | # Deploying | 1 | # Deploying |
| 2 | 2 | ||
| 3 | -The canonical build is the Nix flake (`flake.nix`): `nix build .#image` produces | 3 | +The Nix flake is the build. `nix build .#image` (flake.nix, dockerTools |
| 4 | -the container image, and `make image-push IMAGE=<ref>` publishes it. That path | 4 | +`streamLayeredImage`) produces the container image, and the same command runs |
| 5 | -still works and is the reproducible one. | 5 | +locally and in CI, so a local build and a deployed build are the same artifact. |
| 6 | - | 6 | + |
| 7 | -## Railway (production) | 7 | +## Pipeline |
| 8 | - | 8 | + |
| 9 | -The `glean` service in the `glean` project builds **from source** on | 9 | +Railway has no GitLab repo integration — it deploys from a GitHub repo, a local |
| 10 | -`railway up`, rather than running a prebuilt image. Railway ignores in-repo | 10 | +directory, or a container image — so it cannot watch this repo. `.gitlab-ci.yml` |
| 11 | -build config here — `railway.toml`, `nixpacks.toml` and a start script in | 11 | +closes that gap on every push to `main`: |
| 12 | -`scripts/` were all silently skipped — so the configuration lives in the | 12 | + |
| 13 | -service settings and environment variables instead: | 13 | +1. `build_image` builds `.#image` with Nix and pushes it to this project's own |
| 14 | - | 14 | + container registry as `$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA` (and |
| 15 | -| Setting | Value | | 15 | + `:latest`). skopeo pushes the docker-archive directly, so no Docker daemon |
| 16 | -|---|---| | 16 | + is needed on the runner. |
| 17 | -| Builder | `NIXPACKS` (the Railpack default cannot build this repo) | | 17 | +2. `deploy_railway` points the Railway service at that tag |
| 18 | -| Build command | `cd web && bun install && cd .. && make build` | | 18 | + (`serviceInstanceUpdate`) and rolls it out (`serviceInstanceDeploy`). |
| 19 | -| Start command | `sh -c "GLEAN_API_URL=http://127.0.0.1:8080 GLEAN_ADDR=127.0.0.1:8080 /app/glean & exec node /app/web/build/index.js"` | | 19 | + |
| 20 | -| `NIXPACKS_PKGS` | `bun nodejs gnumake gcc gawk gnugrep` | | 20 | +The project is public, so the registry allows anonymous pulls and Railway needs |
| 21 | -| `CGO_ENABLED` | `1` | | 21 | +no registry credentials. Making it private again would require registry |
| 22 | - | 22 | +credentials on the Railway side, and a paid plan — private registry sources are |
| 23 | -Notes on why each is needed: | 23 | +a Pro feature. |
| 24 | - | 24 | + |
| 25 | -- **`CGO_ENABLED=1`** — `sqlite-vec-go-bindings/cgo` and `mattn/go-sqlite3` are | 25 | +### Required CI variable |
| 26 | - cgo packages. The builder defaults to cgo off, which fails with "build | 26 | + |
| 27 | - constraints exclude all Go files". | 27 | +`deploy_railway` needs `RAILWAY_TOKEN`, a Railway **project token** for the |
| 28 | -- **`NIXPACKS_PKGS`** — the Go provider installs only Go. The frontend needs | 28 | +`glean` project, set under Settings → CI/CD → Variables (protected, masked). |
| 29 | - `bun` to build and `nodejs` to serve; `make build` needs make/grep/awk. This | 29 | +Without it the deploy job is skipped rather than failing: the image is still |
| 30 | - mirrors the dependency list in `.tangled/workflows`. Use `nodejs`, not | 30 | +built and published, and can be rolled out by hand. |
| 31 | - `nodejs_22`: the pinned nixpkgs has no such attribute. | 31 | + |
| 32 | -- **Build command** — the builder's default `go build -o out` skips both the | 32 | +The service and environment IDs are not secret and are inlined in |
| 33 | - `fts5` tag and the SvelteKit build. `make build` does both. | 33 | +`.gitlab-ci.yml`. |
| 34 | -- **Start command** — mirrors the flake's `mkEntrypoint`: the Go API on | 34 | + |
| 35 | - loopback:8080 with the SvelteKit Node server in front on `$PORT`. It is | 35 | +## Deploying by hand |
| 36 | - inline rather than a script file because the runtime image does not carry | 36 | + |
| 37 | - `scripts/`. | 37 | +``` |
| 38 | - | 38 | +make image-push IMAGE=registry.gitlab.com/nandithebull/glean:<tag> |
| 39 | -Deploy with `railway up` from the repo root. | 39 | +``` |
| 40 | + | ||
| 41 | +Then point the service at `<tag>` in the Railway dashboard, or re-run the | ||
| 42 | +`deploy_railway` job. | ||
| 43 | + | ||
| 44 | +## History | ||
| 45 | + | ||
| 46 | +The service briefly built from source on `railway up` with Nixpacks. That | ||
| 47 | +needed `CGO_ENABLED=1` (the sqlite cgo packages), a `NIXPACKS_PKGS` toolchain | ||
| 48 | +list, and an overridden build command, because Railway ignores in-repo | ||
| 49 | +`railway.toml` / `nixpacks.toml` here. It also deployed the working tree rather | ||
| 50 | +than a commit. The image pipeline above replaces it; those service settings | ||
| 51 | +have been removed. | ||