Build the window somewhere with room for it
A local `nix build .#frq` is killed for memory before it finishes, so CLAUDE.md now says to stop: this machine evaluates -- flake check, eval, --dry-run, repl -- and Modal realises. `containers/frq` is where that happens, and `.#frq` is the cosmic GUI, so this builds the window without ever opening one. It runs as a Sandbox on a real VM, which is what lets nix build anything out there; the ptyshim that stood in for a pty under gVisor is deprecated and does not come back. No devShell either -- `nix build` wants a store, not a shell to sit inside. `path:/app` rather than `.`: a worktree's `.git` is a file naming a gitdir back on this machine, and copying it in has nix chase a path that is not there. `path:` says plain directory and means it. The nix-cache volume is read as a substituter and written with `nix copy --all` -- fat on purpose. The result's closure alone only pays while nothing changes; keeping the intermediate outputs is what makes a changed dependency cost a partial rebuild instead of the whole graph. containers/_loader.py is a copy of the one in the modal repo, carried here so the container needs nothing but this checkout. Two copies drift; the `[build] context` and `[volumes]` keys both halves now have are the first thing that would. Also: never pipe a long build through `tail`. It shows nothing until the run ends and loses everything if the run is killed, which is how the first of these builds was watched, and why nothing was seen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1451151 parent: 63365cd modified
CLAUDE.md +48 -7 | @@ -2,23 +2,64 @@ | ||
| 2 | 2 | |
| 3 | 3 | ## Nix |
| 4 | 4 | |
| 5 | -You are already running inside the Arch distrobox, where `nix` lives, so run | |
| 6 | -nix commands directly — do not wrap them in `distrobox enter`: | |
| 5 | +**STOP BUILDING LOCALLY. Build on Modal.** This machine is for editing and for | |
| 6 | +evaluating — `nix flake check`, `nix eval`, `nix build --dry-run`, `nix repl` | |
| 7 | +— and not for realising a derivation. A local `nix build .#frq` gets killed for | |
| 8 | +memory long before it finishes, and the minutes spent finding that out are | |
| 9 | +minutes not spent on the change. So: | |
| 7 | 10 | |
| 8 | 11 | ```bash |
| 9 | -nix build .#frq | |
| 12 | +modal run containers/frq/container.py # nix build .#frq, on Modal | |
| 10 | 13 | ``` |
| 11 | 14 | |
| 12 | -A remote builder (`eu.nixbuild.net`) is already configured here. Large builds | |
| 13 | -want `--store ssh-ng://eu.nixbuild.net --eval-store auto` rather than a | |
| 14 | -`builders` entry, so the whole graph stays there and only .drv files go up — | |
| 15 | -libjoltcosmic's dependency tree is the one that makes this worth remembering. | |
| 15 | +`--dry-run` locally to see what *would* be built, then hand the build to Modal. | |
| 16 | +The one exception is a derivation you already know is trivial and already | |
| 17 | +substitutable; if you are unsure, it is not the exception. | |
| 18 | + | |
| 19 | +`modal app logs` is no substitute for watching that command: it resolves | |
| 20 | +deployed apps by name, not the ephemeral one a `modal run` creates, and carries | |
| 21 | +nothing until the Sandbox starts — the image build streams to the client and | |
| 22 | +nowhere else. | |
| 23 | + | |
| 24 | +You are already running inside the Arch distrobox, where `nix` lives, so run | |
| 25 | +the evaluating commands directly — do not wrap them in `distrobox enter`. | |
| 26 | + | |
| 27 | +The container runs as a Modal **Sandbox on a real VM**, which is what makes | |
| 28 | +`nix build` work out there at all: the ptyshim that used to stand in for a | |
| 29 | +working pty under gVisor is deprecated, and nothing here should reintroduce it. | |
| 30 | +Substitution comes from the `nix-cache` Modal Volume plus cache.nixos.org and | |
| 31 | +nix-cache.wasix.org — libjoltcosmic's dependency tree is the one that makes | |
| 32 | +that cache worth having. | |
| 33 | + | |
| 34 | +A remote builder (`eu.nixbuild.net`) is also configured here, for the case | |
| 35 | +where you want the graph built somewhere other than Modal: `--store | |
| 36 | +ssh-ng://eu.nixbuild.net --eval-store auto` rather than a `builders` entry, so | |
| 37 | +the whole graph stays there and only .drv files go up. | |
| 16 | 38 | |
| 17 | 39 | One thing this container is *not* representative of: `/etc/localtime` is a |
| 18 | 40 | regular file here rather than a symlink, so anything that reads the zone out |
| 19 | 41 | of its path sees nothing. That is a real deployment shape, not an artefact — |
| 20 | 42 | frq.clock handles it. |
| 21 | 43 | |
| 44 | +## Never pipe a long task through `tail` | |
| 45 | + | |
| 46 | +`tail` and `head` do not emit anything until their input ends, so a build, a | |
| 47 | +test run or a deploy piped through one shows nothing at all until it is over — | |
| 48 | +and if it is killed or times out first, its output is lost with it. That is the | |
| 49 | +opposite of what you want from the commands that take longest. | |
| 50 | + | |
| 51 | +Let them write to the terminal, or `tee` them if you want a copy to grep | |
| 52 | +afterwards: | |
| 53 | + | |
| 54 | +```bash | |
| 55 | +modal run containers/frq/container.py 2>&1 | tee /tmp/frq-build.log | |
| 56 | +``` | |
| 57 | + | |
| 58 | +Trim afterwards, on the file, where the whole run is still there to re-read. | |
| 59 | +The same goes for `grep` and `awk` in a live pipeline: they buffer when their | |
| 60 | +output is not a terminal, so pass `--line-buffered` / `fflush()` or watch the | |
| 61 | +file instead. | |
| 62 | + | |
| 22 | 63 | ## The three source trees |
| 23 | 64 | |
| 24 | 65 | ``` |
| @@ -2,23 +2,64 @@ | |||
| 2 | 2 | ||
| 3 | ## Nix | 3 | ## Nix |
| 4 | 4 | ||
| 5 | -You are already running inside the Arch distrobox, where `nix` lives, so run | 5 | +**STOP BUILDING LOCALLY. Build on Modal.** This machine is for editing and for |
| 6 | -nix commands directly — do not wrap them in `distrobox enter`: | 6 | +evaluating — `nix flake check`, `nix eval`, `nix build --dry-run`, `nix repl` |
| 7 | +— and not for realising a derivation. A local `nix build .#frq` gets killed for | ||
| 8 | +memory long before it finishes, and the minutes spent finding that out are | ||
| 9 | +minutes not spent on the change. So: | ||
| 7 | 10 | ||
| 8 | ```bash | 11 | ```bash |
| 9 | -nix build .#frq | 12 | +modal run containers/frq/container.py # nix build .#frq, on Modal |
| 10 | ``` | 13 | ``` |
| 11 | 14 | ||
| 12 | -A remote builder (`eu.nixbuild.net`) is already configured here. Large builds | 15 | +`--dry-run` locally to see what *would* be built, then hand the build to Modal. |
| 13 | -want `--store ssh-ng://eu.nixbuild.net --eval-store auto` rather than a | 16 | +The one exception is a derivation you already know is trivial and already |
| 14 | -`builders` entry, so the whole graph stays there and only .drv files go up — | 17 | +substitutable; if you are unsure, it is not the exception. |
| 15 | -libjoltcosmic's dependency tree is the one that makes this worth remembering. | 18 | + |
| 19 | +`modal app logs` is no substitute for watching that command: it resolves | ||
| 20 | +deployed apps by name, not the ephemeral one a `modal run` creates, and carries | ||
| 21 | +nothing until the Sandbox starts — the image build streams to the client and | ||
| 22 | +nowhere else. | ||
| 23 | + | ||
| 24 | +You are already running inside the Arch distrobox, where `nix` lives, so run | ||
| 25 | +the evaluating commands directly — do not wrap them in `distrobox enter`. | ||
| 26 | + | ||
| 27 | +The container runs as a Modal **Sandbox on a real VM**, which is what makes | ||
| 28 | +`nix build` work out there at all: the ptyshim that used to stand in for a | ||
| 29 | +working pty under gVisor is deprecated, and nothing here should reintroduce it. | ||
| 30 | +Substitution comes from the `nix-cache` Modal Volume plus cache.nixos.org and | ||
| 31 | +nix-cache.wasix.org — libjoltcosmic's dependency tree is the one that makes | ||
| 32 | +that cache worth having. | ||
| 33 | + | ||
| 34 | +A remote builder (`eu.nixbuild.net`) is also configured here, for the case | ||
| 35 | +where you want the graph built somewhere other than Modal: `--store | ||
| 36 | +ssh-ng://eu.nixbuild.net --eval-store auto` rather than a `builders` entry, so | ||
| 37 | +the whole graph stays there and only .drv files go up. | ||
| 16 | 38 | ||
| 17 | One thing this container is *not* representative of: `/etc/localtime` is a | 39 | One thing this container is *not* representative of: `/etc/localtime` is a |
| 18 | regular file here rather than a symlink, so anything that reads the zone out | 40 | regular file here rather than a symlink, so anything that reads the zone out |
| 19 | of its path sees nothing. That is a real deployment shape, not an artefact — | 41 | of its path sees nothing. That is a real deployment shape, not an artefact — |
| 20 | frq.clock handles it. | 42 | frq.clock handles it. |
| 21 | 43 | ||
| 44 | +## Never pipe a long task through `tail` | ||
| 45 | + | ||
| 46 | +`tail` and `head` do not emit anything until their input ends, so a build, a | ||
| 47 | +test run or a deploy piped through one shows nothing at all until it is over — | ||
| 48 | +and if it is killed or times out first, its output is lost with it. That is the | ||
| 49 | +opposite of what you want from the commands that take longest. | ||
| 50 | + | ||
| 51 | +Let them write to the terminal, or `tee` them if you want a copy to grep | ||
| 52 | +afterwards: | ||
| 53 | + | ||
| 54 | +```bash | ||
| 55 | +modal run containers/frq/container.py 2>&1 | tee /tmp/frq-build.log | ||
| 56 | +``` | ||
| 57 | + | ||
| 58 | +Trim afterwards, on the file, where the whole run is still there to re-read. | ||
| 59 | +The same goes for `grep` and `awk` in a live pipeline: they buffer when their | ||
| 60 | +output is not a terminal, so pass `--line-buffered` / `fflush()` or watch the | ||
| 61 | +file instead. | ||
| 62 | + | ||
| 22 | ## The three source trees | 63 | ## The three source trees |
| 23 | 64 | ||
| 24 | ``` | 65 | ``` |
added
containers/__pycache__/_loader.cpython-314.pyc +0 -0 | new file mode 100644 | ||
| Binary files /dev/null and b/containers/__pycache__/_loader.cpython-314.pyc differ | ||
| new file mode 100644 | |||
| Binary files /dev/null and b/containers/__pycache__/_loader.cpython-314.pyc differ | Binary files /dev/null and b/containers/__pycache__/_loader.cpython-314.pyc differ | ||
added
containers/_loader.py +364 -0 | new file mode 100644 | ||
| @@ -0,0 +1,364 @@ | ||
| 1 | +"""Turn a `container.toml` into a `modal.Image` and a `modal.App`. | |
| 2 | + | |
| 3 | +Every container under `containers/` is a directory with a `container.toml` and | |
| 4 | +a stub `container.py`. See `spec.md` for the keys; this file is what reads | |
| 5 | +them. Nothing here is Modal-specific configuration in its own right -- each | |
| 6 | +spec key maps onto a documented Modal argument, and the mapping is meant to | |
| 7 | +stay boring enough to read straight through. | |
| 8 | +""" | |
| 9 | + | |
| 10 | +import os | |
| 11 | +import shlex | |
| 12 | +import subprocess | |
| 13 | +import tomllib | |
| 14 | + | |
| 15 | +import modal | |
| 16 | + | |
| 17 | +REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) | |
| 18 | +PTYSHIM_C = os.path.join(REPO, "ptyshim.c") | |
| 19 | +SHIM_SO = "/opt/ptyshim.so" | |
| 20 | + | |
| 21 | + | |
| 22 | +class SpecError(Exception): | |
| 23 | + """The container.toml says something that cannot be built.""" | |
| 24 | + | |
| 25 | + | |
| 26 | +def _running_in_modal() -> bool: | |
| 27 | + """True inside a Modal container, false on the machine that launched it.""" | |
| 28 | + return bool(os.environ.get("MODAL_TASK_ID")) | |
| 29 | + | |
| 30 | + | |
| 31 | +class Container: | |
| 32 | + """One container: its spec, its image, its app, and how to run it.""" | |
| 33 | + | |
| 34 | + def __init__(self, spec: dict, directory: str): | |
| 35 | + self.is_remote = _running_in_modal() | |
| 36 | + self.dir = directory | |
| 37 | + self.spec = spec | |
| 38 | + self.name = self._require("container", "name") | |
| 39 | + self.description = spec.get("container", {}).get("description", "") | |
| 40 | + | |
| 41 | + run = spec.get("run", {}) | |
| 42 | + self.workdir = run.get("workdir", "/app") | |
| 43 | + self.command = run.get("command", "") | |
| 44 | + self.env = dict(run.get("env", {})) | |
| 45 | + | |
| 46 | + # "function" (the default) or "sandbox". Sandboxes can run on a real | |
| 47 | + # VM, which Functions cannot -- see ../README.md. | |
| 48 | + self.runtime = spec.get("container", {}).get("runtime", "function") | |
| 49 | + | |
| 50 | + nix = spec.get("nix", {}) | |
| 51 | + self.use_flake = bool(nix.get("flake", False)) | |
| 52 | + self.use_shim = bool(nix.get("shim", False)) | |
| 53 | + | |
| 54 | + # name -> mount path. Modal Volumes, mounted while the container runs | |
| 55 | + # and NOT while its image is built: a volume mount is not part of the | |
| 56 | + # resulting image, so anything written to one during a build step is | |
| 57 | + # gone by the time the container starts. Persist across runs is the | |
| 58 | + # whole point -- a nix store to substitute from, a cargo target | |
| 59 | + # directory, a dataset too big to bake in. | |
| 60 | + self.volume_spec = dict(spec.get("volumes", {})) | |
| 61 | + | |
| 62 | + # Modal re-imports this module inside the container, so everything | |
| 63 | + # below runs twice: once here, once out there. Out there the local | |
| 64 | + # tree does not exist -- no flake.nix, no ptyshim.c, no repo -- and | |
| 65 | + # the image is already built, so validating and rebuilding it would | |
| 66 | + # only fail. The App still has to exist for the decorators to bind. | |
| 67 | + if self.is_remote: | |
| 68 | + self.image = None | |
| 69 | + self.app = modal.App(self.name) | |
| 70 | + else: | |
| 71 | + self._validate() | |
| 72 | + self.image = self._build_image() | |
| 73 | + self.app = modal.App(self.name, image=self.image) | |
| 74 | + | |
| 75 | + # -- spec reading ---------------------------------------------------- | |
| 76 | + | |
| 77 | + @classmethod | |
| 78 | + def from_toml(cls, container_py: str) -> "Container": | |
| 79 | + """Load the container.toml sitting next to the given container.py.""" | |
| 80 | + directory = os.path.dirname(os.path.abspath(container_py)) | |
| 81 | + path = os.path.join(directory, "container.toml") | |
| 82 | + if not os.path.exists(path): | |
| 83 | + raise SpecError(f"no container.toml in {directory}") | |
| 84 | + with open(path, "rb") as f: | |
| 85 | + return cls(tomllib.load(f), directory) | |
| 86 | + | |
| 87 | + def _require(self, table: str, key: str): | |
| 88 | + try: | |
| 89 | + return self.spec[table][key] | |
| 90 | + except KeyError: | |
| 91 | + raise SpecError(f"container.toml needs [{table}] {key}") from None | |
| 92 | + | |
| 93 | + def _validate(self): | |
| 94 | + c = self.spec.get("container", {}) | |
| 95 | + if self.runtime not in ("function", "sandbox"): | |
| 96 | + raise SpecError( | |
| 97 | + f'[container] runtime must be "function" or "sandbox",' | |
| 98 | + f" not {self.runtime!r}" | |
| 99 | + ) | |
| 100 | + if bool(c.get("base")) == bool(c.get("registry")): | |
| 101 | + raise SpecError("set exactly one of [container] base or registry") | |
| 102 | + if self.use_flake: | |
| 103 | + if not os.path.exists(os.path.join(self.dir, "flake.nix")): | |
| 104 | + raise SpecError("[nix] flake = true but there is no flake.nix") | |
| 105 | + if not self.use_shim: | |
| 106 | + # Warming the shell builds nix-shell-env, which is the gVisor | |
| 107 | + # pty bug. Failing here beats failing ten minutes into a build. | |
| 108 | + raise SpecError( | |
| 109 | + "[nix] flake = true needs shim = true -- see ../README.md" | |
| 110 | + ) | |
| 111 | + if self.use_shim and not os.path.exists(PTYSHIM_C): | |
| 112 | + raise SpecError(f"[nix] shim = true but {PTYSHIM_C} is missing") | |
| 113 | + for name, mount in self.volume_spec.items(): | |
| 114 | + if not isinstance(mount, str) or not mount.startswith("/"): | |
| 115 | + raise SpecError( | |
| 116 | + f"[volumes] {name} must be an absolute path, not {mount!r}" | |
| 117 | + ) | |
| 118 | + # Mounting over one of these hides what the image already has | |
| 119 | + # there -- /nix in particular, where the empty volume would shadow | |
| 120 | + # the store the base image spent its build populating. | |
| 121 | + if mount.rstrip("/") in ("", "/nix", "/nix/store", "/usr", "/etc"): | |
| 122 | + raise SpecError( | |
| 123 | + f"[volumes] {name} may not mount over {mount} --" | |
| 124 | + " it would hide what the image has there" | |
| 125 | + ) | |
| 126 | + if mount.rstrip("/") == self.workdir.rstrip("/"): | |
| 127 | + raise SpecError( | |
| 128 | + f"[volumes] {name} may not mount over the workdir" | |
| 129 | + f" ({mount}) -- [build] include copies land there" | |
| 130 | + ) | |
| 131 | + | |
| 132 | + # -- image ----------------------------------------------------------- | |
| 133 | + | |
| 134 | + @property | |
| 135 | + def nix(self) -> str: | |
| 136 | + """The `nix` command as the container runs it. | |
| 137 | + | |
| 138 | + A sandbox on a real VM has a working pty, so the shim buys nothing | |
| 139 | + there and is left off even when the image was built with it. | |
| 140 | + """ | |
| 141 | + if self.use_shim and not self.vm_at_runtime: | |
| 142 | + return f"LD_PRELOAD={SHIM_SO} nix" | |
| 143 | + return "nix" | |
| 144 | + | |
| 145 | + @property | |
| 146 | + def vm_at_runtime(self) -> bool: | |
| 147 | + """True when this container actually runs on a VM rather than gVisor.""" | |
| 148 | + return (self.runtime == "sandbox" | |
| 149 | + and bool(self.experimental_options.get("vm_runtime"))) | |
| 150 | + | |
| 151 | + @property | |
| 152 | + def build_nix(self) -> str: | |
| 153 | + """The `nix` command for BUILD steps, which always run under gVisor. | |
| 154 | + | |
| 155 | + Image builds are Functions underneath, so a sandbox container still | |
| 156 | + needs the shim while its image is being built -- only its run time | |
| 157 | + gets the VM. | |
| 158 | + """ | |
| 159 | + return f"LD_PRELOAD={SHIM_SO} nix" if self.use_shim else "nix" | |
| 160 | + | |
| 161 | + def _build_image(self) -> modal.Image: | |
| 162 | + c = self.spec["container"] | |
| 163 | + build = self.spec.get("build", {}) | |
| 164 | + | |
| 165 | + if c.get("base"): | |
| 166 | + image = modal.Image.from_name(c["base"]) | |
| 167 | + else: | |
| 168 | + image = modal.Image.from_registry(c["registry"]) | |
| 169 | + | |
| 170 | + if self.use_shim: | |
| 171 | + image = image.add_local_file( | |
| 172 | + PTYSHIM_C, "/opt/ptyshim.c", copy=True | |
| 173 | + ).run_commands( | |
| 174 | + f"gcc -shared -fPIC -O2 -o {SHIM_SO} /opt/ptyshim.c -ldl" | |
| 175 | + ) | |
| 176 | + | |
| 177 | + # Warm the devShell BEFORE the source is copied in. Everything the | |
| 178 | + # shell needs is binary-cached, so the download happens once -- but | |
| 179 | + # only if this layer survives. Copy the source first and any edit to | |
| 180 | + # any file invalidates the warm, and the whole closure is fetched | |
| 181 | + # again on every build. Only flake.nix and flake.lock go in here, so | |
| 182 | + # the layer is invalidated by a dependency change and nothing else. | |
| 183 | + if self.use_flake: | |
| 184 | + for f in ("flake.nix", "flake.lock"): | |
| 185 | + path = os.path.join(self.dir, f) | |
| 186 | + if os.path.exists(path): | |
| 187 | + image = image.add_local_file( | |
| 188 | + path, f"{self.workdir}/{f}", copy=True | |
| 189 | + ) | |
| 190 | + image = image.run_commands( | |
| 191 | + f"cd {self.workdir} && {self.build_nix} develop" | |
| 192 | + " --accept-flake-config --command true", | |
| 193 | + f'echo "store paths after warming:' | |
| 194 | + f' $({self.build_nix} path-info --all | wc -l)"', | |
| 195 | + ) | |
| 196 | + | |
| 197 | + # copy=True throughout: later run_commands need these files present. | |
| 198 | + # `context` is what include paths are relative to, and it may sit above | |
| 199 | + # the container directory -- a container that builds the repo it lives | |
| 200 | + # in sets context = "../..", so include = ["."] means the whole repo. | |
| 201 | + context = os.path.normpath(os.path.join(self.dir, build.get("context", "."))) | |
| 202 | + for rel in build.get("include", ["."]): | |
| 203 | + src = os.path.normpath(os.path.join(context, rel)) | |
| 204 | + dest = self.workdir if rel == "." else f"{self.workdir}/{rel}" | |
| 205 | + if os.path.isdir(src): | |
| 206 | + image = image.add_local_dir(src, dest, copy=True) | |
| 207 | + else: | |
| 208 | + image = image.add_local_file(src, dest, copy=True) | |
| 209 | + | |
| 210 | + if commands := build.get("commands", []): | |
| 211 | + image = image.run_commands(*commands) | |
| 212 | + | |
| 213 | + # container.py does `from _loader import Container`, and Modal mounts | |
| 214 | + # the entrypoint file alone -- so without this the import that works | |
| 215 | + # locally fails in the container. copy=False adds it at startup rather | |
| 216 | + # than baking a layer, so it invalidates nothing above it, and it must | |
| 217 | + # therefore come after every build step. | |
| 218 | + image = image.add_local_python_source("_loader") | |
| 219 | + # ...and container.py reads its spec at import time, so the spec has to | |
| 220 | + # be there too. Modal re-imports the entrypoint at /root, which is the | |
| 221 | + # one directory that gets none of the workdir copies above. | |
| 222 | + image = image.add_local_file( | |
| 223 | + os.path.join(self.dir, "container.toml"), "/root/container.toml" | |
| 224 | + ) | |
| 225 | + | |
| 226 | + return image | |
| 227 | + | |
| 228 | + # -- volumes --------------------------------------------------------- | |
| 229 | + | |
| 230 | + @property | |
| 231 | + def volumes(self) -> dict: | |
| 232 | + """{mount path: Volume}, as both Sandbox.create and @app.function want. | |
| 233 | + | |
| 234 | + `from_name` is lazy, so this is safe to evaluate on the re-import | |
| 235 | + inside the container as well as out here. create_if_missing means a | |
| 236 | + spec naming a volume that does not exist yet makes it rather than | |
| 237 | + failing -- the first run of a cache is the one that fills it. | |
| 238 | + """ | |
| 239 | + return { | |
| 240 | + mount: modal.Volume.from_name(name, create_if_missing=True) | |
| 241 | + for name, mount in self.volume_spec.items() | |
| 242 | + } | |
| 243 | + | |
| 244 | + # -- function -------------------------------------------------------- | |
| 245 | + | |
| 246 | + @property | |
| 247 | + def function_kwargs(self) -> dict: | |
| 248 | + """Everything `@app.function` should be given, from [resources].""" | |
| 249 | + r = self.spec.get("resources", {}) | |
| 250 | + kwargs: dict = {"timeout": int(r.get("timeout", 900))} | |
| 251 | + if r.get("cpu"): | |
| 252 | + kwargs["cpu"] = float(r["cpu"]) | |
| 253 | + if r.get("memory"): | |
| 254 | + kwargs["memory"] = int(r["memory"]) | |
| 255 | + if r.get("gpu"): | |
| 256 | + kwargs["gpu"] = r["gpu"] | |
| 257 | + # Deliberately NOT self.experimental_options: that fills in the | |
| 258 | + # sandbox default, and vm_runtime on a Function is refused by the | |
| 259 | + # server. A sandbox container's @app.function is vestigial anyway. | |
| 260 | + if self.runtime == "function": | |
| 261 | + if experimental := dict(self.spec.get("experimental", {})): | |
| 262 | + kwargs["experimental_options"] = experimental | |
| 263 | + if volumes := self.volumes: | |
| 264 | + kwargs["volumes"] = volumes | |
| 265 | + return kwargs | |
| 266 | + | |
| 267 | + @property | |
| 268 | + def experimental_options(self) -> dict: | |
| 269 | + """Experimental options, with the sandbox default filled in. | |
| 270 | + | |
| 271 | + `vm_runtime` is Sandbox-only: the server rejects it on a Function | |
| 272 | + outright. So it is defaulted on for sandboxes and never for functions, | |
| 273 | + and an explicit [experimental] table always wins. | |
| 274 | + """ | |
| 275 | + explicit = dict(self.spec.get("experimental", {})) | |
| 276 | + if explicit: | |
| 277 | + return explicit | |
| 278 | + if self.runtime == "sandbox": | |
| 279 | + return {"vm_runtime": True} | |
| 280 | + return {} | |
| 281 | + | |
| 282 | + @property | |
| 283 | + def sandbox_kwargs(self) -> dict: | |
| 284 | + """Everything `Sandbox.create` should be given, from [resources].""" | |
| 285 | + r = self.spec.get("resources", {}) | |
| 286 | + kwargs: dict = {"timeout": int(r.get("timeout", 900))} | |
| 287 | + if r.get("cpu"): | |
| 288 | + kwargs["cpu"] = float(r["cpu"]) | |
| 289 | + if r.get("memory"): | |
| 290 | + # A VM sandbox gets exactly this much and cannot grow into more. | |
| 291 | + kwargs["memory"] = int(r["memory"]) | |
| 292 | + if opts := self.experimental_options: | |
| 293 | + kwargs["experimental_options"] = dict(opts) | |
| 294 | + if volumes := self.volumes: | |
| 295 | + kwargs["volumes"] = volumes | |
| 296 | + return kwargs | |
| 297 | + | |
| 298 | + def shell_command(self, override: str = "") -> str: | |
| 299 | + """The full shell line the container runs, devShell wrapper included.""" | |
| 300 | + command = override or self.command | |
| 301 | + if not command: | |
| 302 | + raise SpecError("container.toml has no [run] command") | |
| 303 | + if self.use_flake: | |
| 304 | + return ( | |
| 305 | + f"cd {self.workdir} && {self.nix} develop --accept-flake-config" | |
| 306 | + f" --command sh -c {shlex.quote(command)}" | |
| 307 | + ) | |
| 308 | + return f"cd {self.workdir} && {command}" | |
| 309 | + | |
| 310 | + def run_sandbox(self, override: str = "") -> str: | |
| 311 | + """Run one command in a Sandbox that dies when the command does. | |
| 312 | + | |
| 313 | + The command IS the sandbox's process, rather than something exec'd | |
| 314 | + into a `sleep infinity` box that then has to be torn down. There is no | |
| 315 | + idle window to pay for and nothing to leak if this script is killed; | |
| 316 | + the [resources] timeout is a backstop, not the mechanism. | |
| 317 | + | |
| 318 | + Modal streams a Sandbox's output into the app log as it runs -- which | |
| 319 | + is what you want for a build -- so this returns "" rather than handing | |
| 320 | + back a copy for the caller to print underneath it. | |
| 321 | + """ | |
| 322 | + line = self.shell_command(override) | |
| 323 | + sb = modal.Sandbox.create( | |
| 324 | + "sh", "-c", line, | |
| 325 | + app=self.app, | |
| 326 | + image=self.image, | |
| 327 | + workdir=self.workdir, | |
| 328 | + env={k: str(v) for k, v in self.env.items()}, | |
| 329 | + **self.sandbox_kwargs, | |
| 330 | + ) | |
| 331 | + sb.wait() | |
| 332 | + if sb.returncode != 0: | |
| 333 | + raise RuntimeError( | |
| 334 | + f"{self.name}: command failed ({sb.returncode})\n" | |
| 335 | + f"$ {line}\n{sb.stderr.read()}" | |
| 336 | + ) | |
| 337 | + return "" | |
| 338 | + | |
| 339 | + def open_sandbox(self) -> "modal.Sandbox": | |
| 340 | + """Start a Sandbox and leave it running, for `scripts/shell`.""" | |
| 341 | + return modal.Sandbox.create( | |
| 342 | + "sleep", | |
| 343 | + "infinity", | |
| 344 | + app=self.app, | |
| 345 | + image=self.image, | |
| 346 | + **self.sandbox_kwargs, | |
| 347 | + ) | |
| 348 | + | |
| 349 | + def execute(self, override: str = "") -> str: | |
| 350 | + """Run the command in this container. Called remotely, not locally.""" | |
| 351 | + line = self.shell_command(override) | |
| 352 | + result = subprocess.run( | |
| 353 | + line, | |
| 354 | + shell=True, | |
| 355 | + capture_output=True, | |
| 356 | + text=True, | |
| 357 | + env={**os.environ, **{k: str(v) for k, v in self.env.items()}}, | |
| 358 | + ) | |
| 359 | + if result.returncode != 0: | |
| 360 | + raise RuntimeError( | |
| 361 | + f"{self.name}: command failed ({result.returncode})\n" | |
| 362 | + f"$ {line}\n{result.stderr}" | |
| 363 | + ) | |
| 364 | + return result.stdout | |
| new file mode 100644 | |||
| @@ -0,0 +1,364 @@ | |||
| 1 | +"""Turn a `container.toml` into a `modal.Image` and a `modal.App`. | ||
| 2 | + | ||
| 3 | +Every container under `containers/` is a directory with a `container.toml` and | ||
| 4 | +a stub `container.py`. See `spec.md` for the keys; this file is what reads | ||
| 5 | +them. Nothing here is Modal-specific configuration in its own right -- each | ||
| 6 | +spec key maps onto a documented Modal argument, and the mapping is meant to | ||
| 7 | +stay boring enough to read straight through. | ||
| 8 | +""" | ||
| 9 | + | ||
| 10 | +import os | ||
| 11 | +import shlex | ||
| 12 | +import subprocess | ||
| 13 | +import tomllib | ||
| 14 | + | ||
| 15 | +import modal | ||
| 16 | + | ||
| 17 | +REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) | ||
| 18 | +PTYSHIM_C = os.path.join(REPO, "ptyshim.c") | ||
| 19 | +SHIM_SO = "/opt/ptyshim.so" | ||
| 20 | + | ||
| 21 | + | ||
| 22 | +class SpecError(Exception): | ||
| 23 | + """The container.toml says something that cannot be built.""" | ||
| 24 | + | ||
| 25 | + | ||
| 26 | +def _running_in_modal() -> bool: | ||
| 27 | + """True inside a Modal container, false on the machine that launched it.""" | ||
| 28 | + return bool(os.environ.get("MODAL_TASK_ID")) | ||
| 29 | + | ||
| 30 | + | ||
| 31 | +class Container: | ||
| 32 | + """One container: its spec, its image, its app, and how to run it.""" | ||
| 33 | + | ||
| 34 | + def __init__(self, spec: dict, directory: str): | ||
| 35 | + self.is_remote = _running_in_modal() | ||
| 36 | + self.dir = directory | ||
| 37 | + self.spec = spec | ||
| 38 | + self.name = self._require("container", "name") | ||
| 39 | + self.description = spec.get("container", {}).get("description", "") | ||
| 40 | + | ||
| 41 | + run = spec.get("run", {}) | ||
| 42 | + self.workdir = run.get("workdir", "/app") | ||
| 43 | + self.command = run.get("command", "") | ||
| 44 | + self.env = dict(run.get("env", {})) | ||
| 45 | + | ||
| 46 | + # "function" (the default) or "sandbox". Sandboxes can run on a real | ||
| 47 | + # VM, which Functions cannot -- see ../README.md. | ||
| 48 | + self.runtime = spec.get("container", {}).get("runtime", "function") | ||
| 49 | + | ||
| 50 | + nix = spec.get("nix", {}) | ||
| 51 | + self.use_flake = bool(nix.get("flake", False)) | ||
| 52 | + self.use_shim = bool(nix.get("shim", False)) | ||
| 53 | + | ||
| 54 | + # name -> mount path. Modal Volumes, mounted while the container runs | ||
| 55 | + # and NOT while its image is built: a volume mount is not part of the | ||
| 56 | + # resulting image, so anything written to one during a build step is | ||
| 57 | + # gone by the time the container starts. Persist across runs is the | ||
| 58 | + # whole point -- a nix store to substitute from, a cargo target | ||
| 59 | + # directory, a dataset too big to bake in. | ||
| 60 | + self.volume_spec = dict(spec.get("volumes", {})) | ||
| 61 | + | ||
| 62 | + # Modal re-imports this module inside the container, so everything | ||
| 63 | + # below runs twice: once here, once out there. Out there the local | ||
| 64 | + # tree does not exist -- no flake.nix, no ptyshim.c, no repo -- and | ||
| 65 | + # the image is already built, so validating and rebuilding it would | ||
| 66 | + # only fail. The App still has to exist for the decorators to bind. | ||
| 67 | + if self.is_remote: | ||
| 68 | + self.image = None | ||
| 69 | + self.app = modal.App(self.name) | ||
| 70 | + else: | ||
| 71 | + self._validate() | ||
| 72 | + self.image = self._build_image() | ||
| 73 | + self.app = modal.App(self.name, image=self.image) | ||
| 74 | + | ||
| 75 | + # -- spec reading ---------------------------------------------------- | ||
| 76 | + | ||
| 77 | + @classmethod | ||
| 78 | + def from_toml(cls, container_py: str) -> "Container": | ||
| 79 | + """Load the container.toml sitting next to the given container.py.""" | ||
| 80 | + directory = os.path.dirname(os.path.abspath(container_py)) | ||
| 81 | + path = os.path.join(directory, "container.toml") | ||
| 82 | + if not os.path.exists(path): | ||
| 83 | + raise SpecError(f"no container.toml in {directory}") | ||
| 84 | + with open(path, "rb") as f: | ||
| 85 | + return cls(tomllib.load(f), directory) | ||
| 86 | + | ||
| 87 | + def _require(self, table: str, key: str): | ||
| 88 | + try: | ||
| 89 | + return self.spec[table][key] | ||
| 90 | + except KeyError: | ||
| 91 | + raise SpecError(f"container.toml needs [{table}] {key}") from None | ||
| 92 | + | ||
| 93 | + def _validate(self): | ||
| 94 | + c = self.spec.get("container", {}) | ||
| 95 | + if self.runtime not in ("function", "sandbox"): | ||
| 96 | + raise SpecError( | ||
| 97 | + f'[container] runtime must be "function" or "sandbox",' | ||
| 98 | + f" not {self.runtime!r}" | ||
| 99 | + ) | ||
| 100 | + if bool(c.get("base")) == bool(c.get("registry")): | ||
| 101 | + raise SpecError("set exactly one of [container] base or registry") | ||
| 102 | + if self.use_flake: | ||
| 103 | + if not os.path.exists(os.path.join(self.dir, "flake.nix")): | ||
| 104 | + raise SpecError("[nix] flake = true but there is no flake.nix") | ||
| 105 | + if not self.use_shim: | ||
| 106 | + # Warming the shell builds nix-shell-env, which is the gVisor | ||
| 107 | + # pty bug. Failing here beats failing ten minutes into a build. | ||
| 108 | + raise SpecError( | ||
| 109 | + "[nix] flake = true needs shim = true -- see ../README.md" | ||
| 110 | + ) | ||
| 111 | + if self.use_shim and not os.path.exists(PTYSHIM_C): | ||
| 112 | + raise SpecError(f"[nix] shim = true but {PTYSHIM_C} is missing") | ||
| 113 | + for name, mount in self.volume_spec.items(): | ||
| 114 | + if not isinstance(mount, str) or not mount.startswith("/"): | ||
| 115 | + raise SpecError( | ||
| 116 | + f"[volumes] {name} must be an absolute path, not {mount!r}" | ||
| 117 | + ) | ||
| 118 | + # Mounting over one of these hides what the image already has | ||
| 119 | + # there -- /nix in particular, where the empty volume would shadow | ||
| 120 | + # the store the base image spent its build populating. | ||
| 121 | + if mount.rstrip("/") in ("", "/nix", "/nix/store", "/usr", "/etc"): | ||
| 122 | + raise SpecError( | ||
| 123 | + f"[volumes] {name} may not mount over {mount} --" | ||
| 124 | + " it would hide what the image has there" | ||
| 125 | + ) | ||
| 126 | + if mount.rstrip("/") == self.workdir.rstrip("/"): | ||
| 127 | + raise SpecError( | ||
| 128 | + f"[volumes] {name} may not mount over the workdir" | ||
| 129 | + f" ({mount}) -- [build] include copies land there" | ||
| 130 | + ) | ||
| 131 | + | ||
| 132 | + # -- image ----------------------------------------------------------- | ||
| 133 | + | ||
| 134 | + @property | ||
| 135 | + def nix(self) -> str: | ||
| 136 | + """The `nix` command as the container runs it. | ||
| 137 | + | ||
| 138 | + A sandbox on a real VM has a working pty, so the shim buys nothing | ||
| 139 | + there and is left off even when the image was built with it. | ||
| 140 | + """ | ||
| 141 | + if self.use_shim and not self.vm_at_runtime: | ||
| 142 | + return f"LD_PRELOAD={SHIM_SO} nix" | ||
| 143 | + return "nix" | ||
| 144 | + | ||
| 145 | + @property | ||
| 146 | + def vm_at_runtime(self) -> bool: | ||
| 147 | + """True when this container actually runs on a VM rather than gVisor.""" | ||
| 148 | + return (self.runtime == "sandbox" | ||
| 149 | + and bool(self.experimental_options.get("vm_runtime"))) | ||
| 150 | + | ||
| 151 | + @property | ||
| 152 | + def build_nix(self) -> str: | ||
| 153 | + """The `nix` command for BUILD steps, which always run under gVisor. | ||
| 154 | + | ||
| 155 | + Image builds are Functions underneath, so a sandbox container still | ||
| 156 | + needs the shim while its image is being built -- only its run time | ||
| 157 | + gets the VM. | ||
| 158 | + """ | ||
| 159 | + return f"LD_PRELOAD={SHIM_SO} nix" if self.use_shim else "nix" | ||
| 160 | + | ||
| 161 | + def _build_image(self) -> modal.Image: | ||
| 162 | + c = self.spec["container"] | ||
| 163 | + build = self.spec.get("build", {}) | ||
| 164 | + | ||
| 165 | + if c.get("base"): | ||
| 166 | + image = modal.Image.from_name(c["base"]) | ||
| 167 | + else: | ||
| 168 | + image = modal.Image.from_registry(c["registry"]) | ||
| 169 | + | ||
| 170 | + if self.use_shim: | ||
| 171 | + image = image.add_local_file( | ||
| 172 | + PTYSHIM_C, "/opt/ptyshim.c", copy=True | ||
| 173 | + ).run_commands( | ||
| 174 | + f"gcc -shared -fPIC -O2 -o {SHIM_SO} /opt/ptyshim.c -ldl" | ||
| 175 | + ) | ||
| 176 | + | ||
| 177 | + # Warm the devShell BEFORE the source is copied in. Everything the | ||
| 178 | + # shell needs is binary-cached, so the download happens once -- but | ||
| 179 | + # only if this layer survives. Copy the source first and any edit to | ||
| 180 | + # any file invalidates the warm, and the whole closure is fetched | ||
| 181 | + # again on every build. Only flake.nix and flake.lock go in here, so | ||
| 182 | + # the layer is invalidated by a dependency change and nothing else. | ||
| 183 | + if self.use_flake: | ||
| 184 | + for f in ("flake.nix", "flake.lock"): | ||
| 185 | + path = os.path.join(self.dir, f) | ||
| 186 | + if os.path.exists(path): | ||
| 187 | + image = image.add_local_file( | ||
| 188 | + path, f"{self.workdir}/{f}", copy=True | ||
| 189 | + ) | ||
| 190 | + image = image.run_commands( | ||
| 191 | + f"cd {self.workdir} && {self.build_nix} develop" | ||
| 192 | + " --accept-flake-config --command true", | ||
| 193 | + f'echo "store paths after warming:' | ||
| 194 | + f' $({self.build_nix} path-info --all | wc -l)"', | ||
| 195 | + ) | ||
| 196 | + | ||
| 197 | + # copy=True throughout: later run_commands need these files present. | ||
| 198 | + # `context` is what include paths are relative to, and it may sit above | ||
| 199 | + # the container directory -- a container that builds the repo it lives | ||
| 200 | + # in sets context = "../..", so include = ["."] means the whole repo. | ||
| 201 | + context = os.path.normpath(os.path.join(self.dir, build.get("context", "."))) | ||
| 202 | + for rel in build.get("include", ["."]): | ||
| 203 | + src = os.path.normpath(os.path.join(context, rel)) | ||
| 204 | + dest = self.workdir if rel == "." else f"{self.workdir}/{rel}" | ||
| 205 | + if os.path.isdir(src): | ||
| 206 | + image = image.add_local_dir(src, dest, copy=True) | ||
| 207 | + else: | ||
| 208 | + image = image.add_local_file(src, dest, copy=True) | ||
| 209 | + | ||
| 210 | + if commands := build.get("commands", []): | ||
| 211 | + image = image.run_commands(*commands) | ||
| 212 | + | ||
| 213 | + # container.py does `from _loader import Container`, and Modal mounts | ||
| 214 | + # the entrypoint file alone -- so without this the import that works | ||
| 215 | + # locally fails in the container. copy=False adds it at startup rather | ||
| 216 | + # than baking a layer, so it invalidates nothing above it, and it must | ||
| 217 | + # therefore come after every build step. | ||
| 218 | + image = image.add_local_python_source("_loader") | ||
| 219 | + # ...and container.py reads its spec at import time, so the spec has to | ||
| 220 | + # be there too. Modal re-imports the entrypoint at /root, which is the | ||
| 221 | + # one directory that gets none of the workdir copies above. | ||
| 222 | + image = image.add_local_file( | ||
| 223 | + os.path.join(self.dir, "container.toml"), "/root/container.toml" | ||
| 224 | + ) | ||
| 225 | + | ||
| 226 | + return image | ||
| 227 | + | ||
| 228 | + # -- volumes --------------------------------------------------------- | ||
| 229 | + | ||
| 230 | + @property | ||
| 231 | + def volumes(self) -> dict: | ||
| 232 | + """{mount path: Volume}, as both Sandbox.create and @app.function want. | ||
| 233 | + | ||
| 234 | + `from_name` is lazy, so this is safe to evaluate on the re-import | ||
| 235 | + inside the container as well as out here. create_if_missing means a | ||
| 236 | + spec naming a volume that does not exist yet makes it rather than | ||
| 237 | + failing -- the first run of a cache is the one that fills it. | ||
| 238 | + """ | ||
| 239 | + return { | ||
| 240 | + mount: modal.Volume.from_name(name, create_if_missing=True) | ||
| 241 | + for name, mount in self.volume_spec.items() | ||
| 242 | + } | ||
| 243 | + | ||
| 244 | + # -- function -------------------------------------------------------- | ||
| 245 | + | ||
| 246 | + @property | ||
| 247 | + def function_kwargs(self) -> dict: | ||
| 248 | + """Everything `@app.function` should be given, from [resources].""" | ||
| 249 | + r = self.spec.get("resources", {}) | ||
| 250 | + kwargs: dict = {"timeout": int(r.get("timeout", 900))} | ||
| 251 | + if r.get("cpu"): | ||
| 252 | + kwargs["cpu"] = float(r["cpu"]) | ||
| 253 | + if r.get("memory"): | ||
| 254 | + kwargs["memory"] = int(r["memory"]) | ||
| 255 | + if r.get("gpu"): | ||
| 256 | + kwargs["gpu"] = r["gpu"] | ||
| 257 | + # Deliberately NOT self.experimental_options: that fills in the | ||
| 258 | + # sandbox default, and vm_runtime on a Function is refused by the | ||
| 259 | + # server. A sandbox container's @app.function is vestigial anyway. | ||
| 260 | + if self.runtime == "function": | ||
| 261 | + if experimental := dict(self.spec.get("experimental", {})): | ||
| 262 | + kwargs["experimental_options"] = experimental | ||
| 263 | + if volumes := self.volumes: | ||
| 264 | + kwargs["volumes"] = volumes | ||
| 265 | + return kwargs | ||
| 266 | + | ||
| 267 | + @property | ||
| 268 | + def experimental_options(self) -> dict: | ||
| 269 | + """Experimental options, with the sandbox default filled in. | ||
| 270 | + | ||
| 271 | + `vm_runtime` is Sandbox-only: the server rejects it on a Function | ||
| 272 | + outright. So it is defaulted on for sandboxes and never for functions, | ||
| 273 | + and an explicit [experimental] table always wins. | ||
| 274 | + """ | ||
| 275 | + explicit = dict(self.spec.get("experimental", {})) | ||
| 276 | + if explicit: | ||
| 277 | + return explicit | ||
| 278 | + if self.runtime == "sandbox": | ||
| 279 | + return {"vm_runtime": True} | ||
| 280 | + return {} | ||
| 281 | + | ||
| 282 | + @property | ||
| 283 | + def sandbox_kwargs(self) -> dict: | ||
| 284 | + """Everything `Sandbox.create` should be given, from [resources].""" | ||
| 285 | + r = self.spec.get("resources", {}) | ||
| 286 | + kwargs: dict = {"timeout": int(r.get("timeout", 900))} | ||
| 287 | + if r.get("cpu"): | ||
| 288 | + kwargs["cpu"] = float(r["cpu"]) | ||
| 289 | + if r.get("memory"): | ||
| 290 | + # A VM sandbox gets exactly this much and cannot grow into more. | ||
| 291 | + kwargs["memory"] = int(r["memory"]) | ||
| 292 | + if opts := self.experimental_options: | ||
| 293 | + kwargs["experimental_options"] = dict(opts) | ||
| 294 | + if volumes := self.volumes: | ||
| 295 | + kwargs["volumes"] = volumes | ||
| 296 | + return kwargs | ||
| 297 | + | ||
| 298 | + def shell_command(self, override: str = "") -> str: | ||
| 299 | + """The full shell line the container runs, devShell wrapper included.""" | ||
| 300 | + command = override or self.command | ||
| 301 | + if not command: | ||
| 302 | + raise SpecError("container.toml has no [run] command") | ||
| 303 | + if self.use_flake: | ||
| 304 | + return ( | ||
| 305 | + f"cd {self.workdir} && {self.nix} develop --accept-flake-config" | ||
| 306 | + f" --command sh -c {shlex.quote(command)}" | ||
| 307 | + ) | ||
| 308 | + return f"cd {self.workdir} && {command}" | ||
| 309 | + | ||
| 310 | + def run_sandbox(self, override: str = "") -> str: | ||
| 311 | + """Run one command in a Sandbox that dies when the command does. | ||
| 312 | + | ||
| 313 | + The command IS the sandbox's process, rather than something exec'd | ||
| 314 | + into a `sleep infinity` box that then has to be torn down. There is no | ||
| 315 | + idle window to pay for and nothing to leak if this script is killed; | ||
| 316 | + the [resources] timeout is a backstop, not the mechanism. | ||
| 317 | + | ||
| 318 | + Modal streams a Sandbox's output into the app log as it runs -- which | ||
| 319 | + is what you want for a build -- so this returns "" rather than handing | ||
| 320 | + back a copy for the caller to print underneath it. | ||
| 321 | + """ | ||
| 322 | + line = self.shell_command(override) | ||
| 323 | + sb = modal.Sandbox.create( | ||
| 324 | + "sh", "-c", line, | ||
| 325 | + app=self.app, | ||
| 326 | + image=self.image, | ||
| 327 | + workdir=self.workdir, | ||
| 328 | + env={k: str(v) for k, v in self.env.items()}, | ||
| 329 | + **self.sandbox_kwargs, | ||
| 330 | + ) | ||
| 331 | + sb.wait() | ||
| 332 | + if sb.returncode != 0: | ||
| 333 | + raise RuntimeError( | ||
| 334 | + f"{self.name}: command failed ({sb.returncode})\n" | ||
| 335 | + f"$ {line}\n{sb.stderr.read()}" | ||
| 336 | + ) | ||
| 337 | + return "" | ||
| 338 | + | ||
| 339 | + def open_sandbox(self) -> "modal.Sandbox": | ||
| 340 | + """Start a Sandbox and leave it running, for `scripts/shell`.""" | ||
| 341 | + return modal.Sandbox.create( | ||
| 342 | + "sleep", | ||
| 343 | + "infinity", | ||
| 344 | + app=self.app, | ||
| 345 | + image=self.image, | ||
| 346 | + **self.sandbox_kwargs, | ||
| 347 | + ) | ||
| 348 | + | ||
| 349 | + def execute(self, override: str = "") -> str: | ||
| 350 | + """Run the command in this container. Called remotely, not locally.""" | ||
| 351 | + line = self.shell_command(override) | ||
| 352 | + result = subprocess.run( | ||
| 353 | + line, | ||
| 354 | + shell=True, | ||
| 355 | + capture_output=True, | ||
| 356 | + text=True, | ||
| 357 | + env={**os.environ, **{k: str(v) for k, v in self.env.items()}}, | ||
| 358 | + ) | ||
| 359 | + if result.returncode != 0: | ||
| 360 | + raise RuntimeError( | ||
| 361 | + f"{self.name}: command failed ({result.returncode})\n" | ||
| 362 | + f"$ {line}\n{result.stderr}" | ||
| 363 | + ) | ||
| 364 | + return result.stdout | ||
added
containers/frq/README.md +16 -0 | new file mode 100644 | ||
| @@ -0,0 +1,16 @@ | ||
| 1 | +# `frq` | |
| 2 | + | |
| 3 | + scripts/deploy frq | |
| 4 | + modal run containers/frq/container.py | |
| 5 | + | |
| 6 | +Defined by `container.toml`; see `../spec.md` for the keys. | |
| 7 | +Built on the published `arch-nix` image. | |
| 8 | + | |
| 9 | +Runs as a Sandbox on a real VM (kernel 6.x, not gVisor). The command | |
| 10 | +is the sandbox's own process, so it dies when the command exits -- | |
| 11 | +no idle window and nothing to tear down. Note the VM restrictions: | |
| 12 | +no GPU, and memory is exactly what `[resources] memory` asks for. | |
| 13 | + | |
| 14 | +No nix at run time: nothing is substituted at build time and nothing | |
| 15 | +is evaluated at start. Add a `flake.nix` and set `[nix] flake`/`shim` | |
| 16 | +together if you want a devShell, knowing what it costs. | |
| new file mode 100644 | |||
| @@ -0,0 +1,16 @@ | |||
| 1 | +# `frq` | ||
| 2 | + | ||
| 3 | + scripts/deploy frq | ||
| 4 | + modal run containers/frq/container.py | ||
| 5 | + | ||
| 6 | +Defined by `container.toml`; see `../spec.md` for the keys. | ||
| 7 | +Built on the published `arch-nix` image. | ||
| 8 | + | ||
| 9 | +Runs as a Sandbox on a real VM (kernel 6.x, not gVisor). The command | ||
| 10 | +is the sandbox's own process, so it dies when the command exits -- | ||
| 11 | +no idle window and nothing to tear down. Note the VM restrictions: | ||
| 12 | +no GPU, and memory is exactly what `[resources] memory` asks for. | ||
| 13 | + | ||
| 14 | +No nix at run time: nothing is substituted at build time and nothing | ||
| 15 | +is evaluated at start. Add a `flake.nix` and set `[nix] flake`/`shim` | ||
| 16 | +together if you want a devShell, knowing what it costs. | ||
added
containers/frq/__pycache__/container.cpython-314.pyc +0 -0 | new file mode 100644 | ||
| Binary files /dev/null and b/containers/frq/__pycache__/container.cpython-314.pyc differ | ||
| new file mode 100644 | |||
| Binary files /dev/null and b/containers/frq/__pycache__/container.cpython-314.pyc differ | Binary files /dev/null and b/containers/frq/__pycache__/container.cpython-314.pyc differ | ||
added
containers/frq/container.py +22 -0 | new file mode 100644 | ||
| @@ -0,0 +1,22 @@ | ||
| 1 | +"""Generated stub -- the container is defined by container.toml. | |
| 2 | + | |
| 3 | +Edit container.toml, not this file. | |
| 4 | +""" | |
| 5 | + | |
| 6 | +import os | |
| 7 | +import sys | |
| 8 | + | |
| 9 | +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) | |
| 10 | + | |
| 11 | +from _loader import Container # noqa: E402 | |
| 12 | + | |
| 13 | +c = Container.from_toml(__file__) | |
| 14 | +image, app = c.image, c.app | |
| 15 | + | |
| 16 | + | |
| 17 | +# No @app.function here: a Sandbox runs its command as its own process and | |
| 18 | +# nothing of this module is imported into it. Registering a Function would be | |
| 19 | +# dead weight, and its kwargs are where vm_runtime would be wrongly applied. | |
| 20 | +@app.local_entrypoint() | |
| 21 | +def main(command: str = ""): | |
| 22 | + c.run_sandbox(command) | |
| new file mode 100644 | |||
| @@ -0,0 +1,22 @@ | |||
| 1 | +"""Generated stub -- the container is defined by container.toml. | ||
| 2 | + | ||
| 3 | +Edit container.toml, not this file. | ||
| 4 | +""" | ||
| 5 | + | ||
| 6 | +import os | ||
| 7 | +import sys | ||
| 8 | + | ||
| 9 | +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) | ||
| 10 | + | ||
| 11 | +from _loader import Container # noqa: E402 | ||
| 12 | + | ||
| 13 | +c = Container.from_toml(__file__) | ||
| 14 | +image, app = c.image, c.app | ||
| 15 | + | ||
| 16 | + | ||
| 17 | +# No @app.function here: a Sandbox runs its command as its own process and | ||
| 18 | +# nothing of this module is imported into it. Registering a Function would be | ||
| 19 | +# dead weight, and its kwargs are where vm_runtime would be wrongly applied. | ||
| 20 | +@app.local_entrypoint() | ||
| 21 | +def main(command: str = ""): | ||
| 22 | + c.run_sandbox(command) | ||
added
containers/frq/container.toml +96 -0 | new file mode 100644 | ||
| @@ -0,0 +1,96 @@ | ||
| 1 | +[container] | |
| 2 | +name = "frq" | |
| 3 | +description = "nix build .#frq -- the cosmic GUI" | |
| 4 | +base = "arch-nix" | |
| 5 | +# A Sandbox, not a Function: runs on a real VM, and the command is | |
| 6 | +# the sandbox's own process so it dies when the command does. | |
| 7 | +runtime = "sandbox" | |
| 8 | + | |
| 9 | +[build] | |
| 10 | +# The container lives inside the repo it builds, so the copy is rooted two | |
| 11 | +# levels up and `.` is the whole tree -- uncommitted edits included, which is | |
| 12 | +# the point of copying rather than fetching. | |
| 13 | +context = "../.." | |
| 14 | +include = ["."] | |
| 15 | + | |
| 16 | +# The Modal Volume that makes a second build cheap. Mounted at run time, which | |
| 17 | +# is when the build happens here -- nothing is written to it while the image is | |
| 18 | +# built, because a volume mount is not part of the resulting image. | |
| 19 | +[volumes] | |
| 20 | +nix-cache = "/nix-cache" | |
| 21 | + | |
| 22 | +[resources] | |
| 23 | +cpu = 8 | |
| 24 | +memory = 16384 | |
| 25 | +# libjoltcosmic's tree is the long pole even when most of it substitutes. | |
| 26 | +timeout = 3600 | |
| 27 | + | |
| 28 | +[run] | |
| 29 | +workdir = "/app" | |
| 30 | +# The build, not the window: `#frq` is the cosmic GUI (its frqScript runs | |
| 31 | +# `-m frq.cosmic`), and nothing here tries to open it -- there is no GL and | |
| 32 | +# no display on a build box, which is exactly why this only builds. | |
| 33 | +# | |
| 34 | +# `path:/app` and not `.`: a checkout copied in here brings its `.git` with | |
| 35 | +# it, and in a worktree that is a *file* naming a gitdir back on the host. | |
| 36 | +# Nix believes it, tries to open a repository that is not there, and fails | |
| 37 | +# before it evaluates anything. `path:` says plain directory and means it. | |
| 38 | +# | |
| 39 | +# Two halves. The build reads from /nix-cache when there is something there to | |
| 40 | +# read, and `nix copy` writes what it produced back. Naming the substituter | |
| 41 | +# unconditionally would break the first run -- a directory with no | |
| 42 | +# nix-cache-info is not a binary cache yet -- so the test for it is the reason | |
| 43 | +# this is a script rather than a line. | |
| 44 | +# | |
| 45 | +# `--all`, deliberately fat: every path in the container's store goes up, not | |
| 46 | +# just the result's runtime closure. The closure alone is the cheap cache, and | |
| 47 | +# it only pays off while nothing changes -- nix substitutes libjoltcosmic and | |
| 48 | +# never looks at its inputs. Change a dependency, though, and the intermediate | |
| 49 | +# outputs are gone: crane's deps-only artifact, the toolchain, the vendored | |
| 50 | +# crates, all rebuilt from nothing. Those are exactly the paths a fat cache | |
| 51 | +# keeps, so a partial invalidation costs a partial rebuild rather than a whole | |
| 52 | +# one. It buys that with volume size and with upload time on every run. | |
| 53 | +# | |
| 54 | +# `--max-jobs auto` on the command line and not only in nix.conf: the base | |
| 55 | +# image carries that setting now, but only from the next `modal run | |
| 56 | +# arch_nix.py` onwards, and the flag costs nothing once it is redundant. | |
| 57 | +# Nix's default is 1 -- the whole graph end to end, one derivation at a time. | |
| 58 | +# | |
| 59 | +# `set -e` earns its place: the last command here is `du`, so without it a | |
| 60 | +# failed `nix build` would still leave the sandbox exiting 0 and the run would | |
| 61 | +# report success. The substituter test is an `if` rather than `&&` for the same | |
| 62 | +# reason -- under `set -e` a false `&&` would abort the whole script on the | |
| 63 | +# first run, when there is legitimately nothing in the cache yet. | |
| 64 | +command = """ | |
| 65 | +set -e | |
| 66 | +mkdir -p /nix-cache | |
| 67 | +subs="" | |
| 68 | +if [ -f /nix-cache/nix-cache-info ]; then | |
| 69 | + subs="--extra-substituters file:///nix-cache" | |
| 70 | + echo "cache: reading from /nix-cache" | |
| 71 | +else | |
| 72 | + echo "cache: empty, this run fills it" | |
| 73 | +fi | |
| 74 | +# One setting per invocation -- `nix config show a b` is an argument error, | |
| 75 | +# and under `set -e` that kills the run before it builds anything. | |
| 76 | +nix config show max-jobs | |
| 77 | +nix config show cores | |
| 78 | +nix build path:/app#frq --accept-flake-config $subs \ | |
| 79 | + --max-jobs auto --cores 0 \ | |
| 80 | + --print-out-paths --print-build-logs | |
| 81 | +nix copy --no-check-sigs --all --to file:///nix-cache | |
| 82 | +echo "cache size: $(du -sh /nix-cache | cut -f1)" | |
| 83 | +""" | |
| 84 | +env = { } | |
| 85 | + | |
| 86 | +[nix] | |
| 87 | +# No devShell: we want `nix build`, not a shell to run something inside, so | |
| 88 | +# there is nothing to warm at build time and no reason for the ptyshim. The | |
| 89 | +# build happens at run time, in the Sandbox, on a real VM -- which is the | |
| 90 | +# whole reason it can realise a derivation at all. | |
| 91 | +flake = false | |
| 92 | +shim = false | |
| 93 | + | |
| 94 | +# [experimental] overrides the sandbox default of vm_runtime = true. | |
| 95 | +# Setting it here turns that default off, so leave it alone unless you | |
| 96 | +# want gVisor. | |
| new file mode 100644 | |||
| @@ -0,0 +1,96 @@ | |||
| 1 | +[container] | ||
| 2 | +name = "frq" | ||
| 3 | +description = "nix build .#frq -- the cosmic GUI" | ||
| 4 | +base = "arch-nix" | ||
| 5 | +# A Sandbox, not a Function: runs on a real VM, and the command is | ||
| 6 | +# the sandbox's own process so it dies when the command does. | ||
| 7 | +runtime = "sandbox" | ||
| 8 | + | ||
| 9 | +[build] | ||
| 10 | +# The container lives inside the repo it builds, so the copy is rooted two | ||
| 11 | +# levels up and `.` is the whole tree -- uncommitted edits included, which is | ||
| 12 | +# the point of copying rather than fetching. | ||
| 13 | +context = "../.." | ||
| 14 | +include = ["."] | ||
| 15 | + | ||
| 16 | +# The Modal Volume that makes a second build cheap. Mounted at run time, which | ||
| 17 | +# is when the build happens here -- nothing is written to it while the image is | ||
| 18 | +# built, because a volume mount is not part of the resulting image. | ||
| 19 | +[volumes] | ||
| 20 | +nix-cache = "/nix-cache" | ||
| 21 | + | ||
| 22 | +[resources] | ||
| 23 | +cpu = 8 | ||
| 24 | +memory = 16384 | ||
| 25 | +# libjoltcosmic's tree is the long pole even when most of it substitutes. | ||
| 26 | +timeout = 3600 | ||
| 27 | + | ||
| 28 | +[run] | ||
| 29 | +workdir = "/app" | ||
| 30 | +# The build, not the window: `#frq` is the cosmic GUI (its frqScript runs | ||
| 31 | +# `-m frq.cosmic`), and nothing here tries to open it -- there is no GL and | ||
| 32 | +# no display on a build box, which is exactly why this only builds. | ||
| 33 | +# | ||
| 34 | +# `path:/app` and not `.`: a checkout copied in here brings its `.git` with | ||
| 35 | +# it, and in a worktree that is a *file* naming a gitdir back on the host. | ||
| 36 | +# Nix believes it, tries to open a repository that is not there, and fails | ||
| 37 | +# before it evaluates anything. `path:` says plain directory and means it. | ||
| 38 | +# | ||
| 39 | +# Two halves. The build reads from /nix-cache when there is something there to | ||
| 40 | +# read, and `nix copy` writes what it produced back. Naming the substituter | ||
| 41 | +# unconditionally would break the first run -- a directory with no | ||
| 42 | +# nix-cache-info is not a binary cache yet -- so the test for it is the reason | ||
| 43 | +# this is a script rather than a line. | ||
| 44 | +# | ||
| 45 | +# `--all`, deliberately fat: every path in the container's store goes up, not | ||
| 46 | +# just the result's runtime closure. The closure alone is the cheap cache, and | ||
| 47 | +# it only pays off while nothing changes -- nix substitutes libjoltcosmic and | ||
| 48 | +# never looks at its inputs. Change a dependency, though, and the intermediate | ||
| 49 | +# outputs are gone: crane's deps-only artifact, the toolchain, the vendored | ||
| 50 | +# crates, all rebuilt from nothing. Those are exactly the paths a fat cache | ||
| 51 | +# keeps, so a partial invalidation costs a partial rebuild rather than a whole | ||
| 52 | +# one. It buys that with volume size and with upload time on every run. | ||
| 53 | +# | ||
| 54 | +# `--max-jobs auto` on the command line and not only in nix.conf: the base | ||
| 55 | +# image carries that setting now, but only from the next `modal run | ||
| 56 | +# arch_nix.py` onwards, and the flag costs nothing once it is redundant. | ||
| 57 | +# Nix's default is 1 -- the whole graph end to end, one derivation at a time. | ||
| 58 | +# | ||
| 59 | +# `set -e` earns its place: the last command here is `du`, so without it a | ||
| 60 | +# failed `nix build` would still leave the sandbox exiting 0 and the run would | ||
| 61 | +# report success. The substituter test is an `if` rather than `&&` for the same | ||
| 62 | +# reason -- under `set -e` a false `&&` would abort the whole script on the | ||
| 63 | +# first run, when there is legitimately nothing in the cache yet. | ||
| 64 | +command = """ | ||
| 65 | +set -e | ||
| 66 | +mkdir -p /nix-cache | ||
| 67 | +subs="" | ||
| 68 | +if [ -f /nix-cache/nix-cache-info ]; then | ||
| 69 | + subs="--extra-substituters file:///nix-cache" | ||
| 70 | + echo "cache: reading from /nix-cache" | ||
| 71 | +else | ||
| 72 | + echo "cache: empty, this run fills it" | ||
| 73 | +fi | ||
| 74 | +# One setting per invocation -- `nix config show a b` is an argument error, | ||
| 75 | +# and under `set -e` that kills the run before it builds anything. | ||
| 76 | +nix config show max-jobs | ||
| 77 | +nix config show cores | ||
| 78 | +nix build path:/app#frq --accept-flake-config $subs \ | ||
| 79 | + --max-jobs auto --cores 0 \ | ||
| 80 | + --print-out-paths --print-build-logs | ||
| 81 | +nix copy --no-check-sigs --all --to file:///nix-cache | ||
| 82 | +echo "cache size: $(du -sh /nix-cache | cut -f1)" | ||
| 83 | +""" | ||
| 84 | +env = { } | ||
| 85 | + | ||
| 86 | +[nix] | ||
| 87 | +# No devShell: we want `nix build`, not a shell to run something inside, so | ||
| 88 | +# there is nothing to warm at build time and no reason for the ptyshim. The | ||
| 89 | +# build happens at run time, in the Sandbox, on a real VM -- which is the | ||
| 90 | +# whole reason it can realise a derivation at all. | ||
| 91 | +flake = false | ||
| 92 | +shim = false | ||
| 93 | + | ||
| 94 | +# [experimental] overrides the sandbox default of vm_runtime = true. | ||
| 95 | +# Setting it here turns that default off, so leave it alone unless you | ||
| 96 | +# want gVisor. | ||
added
containers/frq/main.py +15 -0 | new file mode 100644 | ||
| @@ -0,0 +1,15 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +"""The app. Replace this with something that earns its container.""" | |
| 3 | + | |
| 4 | +import os | |
| 5 | +import platform | |
| 6 | + | |
| 7 | + | |
| 8 | +def main(): | |
| 9 | + print(os.environ.get("GREETING", "hello")) | |
| 10 | + print(f"python {platform.python_version()}") | |
| 11 | + print(f"host {platform.node()} ({platform.machine()})") | |
| 12 | + | |
| 13 | + | |
| 14 | +if __name__ == "__main__": | |
| 15 | + main() | |
| new file mode 100644 | |||
| @@ -0,0 +1,15 @@ | |||
| 1 | +#!/usr/bin/env python3 | ||
| 2 | +"""The app. Replace this with something that earns its container.""" | ||
| 3 | + | ||
| 4 | +import os | ||
| 5 | +import platform | ||
| 6 | + | ||
| 7 | + | ||
| 8 | +def main(): | ||
| 9 | + print(os.environ.get("GREETING", "hello")) | ||
| 10 | + print(f"python {platform.python_version()}") | ||
| 11 | + print(f"host {platform.node()} ({platform.machine()})") | ||
| 12 | + | ||
| 13 | + | ||
| 14 | +if __name__ == "__main__": | ||
| 15 | + main() | ||