nandi/rustnim · DESIGN.md

⑂ 3d4a7d283b83d3a5305dfc6654c290bd4ee162ae
DESIGN.md · 800 lines · 38.6 KBmarkdown    Blame HistoryRaw
Scaffold rustnim: Rust->Nim transpiler, type mapping   1# rustnim — a Rust → Nim transpiler
Add display.rs and the alloc half: all of base16ct now goes through  nandithebull 5**Milestone 1 is reached: all of `base16ct` goes through.** Every one of its
6source files transpiles byte-for-byte as published, `alloc` half included, and
7its decode and encode output is byte-identical to rustc's. 33 differential
8cases, 29 behavioural and 4 rejections, plus 6 unit/integration tests. All
9green. Run `cargo test`.
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 10
11Passing today: functions, `impl` methods, trait impls (formatting traits and
12`From`), structs, enums (C-like and data-carrying), `Option`/`Result` with
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 13`?`, closures, `unsafe`, slice iterators (`iter`/`iter_mut`/`enumerate`/`zip`/`chunks_exact`/
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 14`chunks_exact_mut`/`windows`), borrowed slices as values and return types,
15`let`/`let mut`, the full integer
Add enums, Option/Result, `?`, and the machinery base16ct needs around them  nandithebull 16and float operator set at exact widths, `as` casts, `if`/`while`/`loop`/`for`,
17`match` including patterns that bind, `Vec`/slices/arrays, type aliases
18(including generic ones), function-typed parameters (`impl Fn(A) -> B`),
19multi-file input, `#[cfg]` evaluation, and `println!`/`format!` with `{}`,
20`{:?}`, `{:x}`, `{:b}`, positional and inline-named arguments, and
Add the differential test runner, and a lowering to measure with it  nandithebull 21zero/space padding.
Scaffold rustnim: Rust->Nim transpiler, type mapping   22
23## Why this exists
25We tried [tarekwasfy01/Code-Transpiler](https://github.com/tarekwasfy01/Code-Transpiler),
26which advertises `rust` as a source language, on the `base16ct` crate. It emits
27empty files and exits 0. The full investigation is in [`findings/`](findings/)
28and is published at
29https://rickub.com/nandi/code-transpiler-rust-frontend-findings
31The decisive finding, and the reason this is a new project rather than a patch:
32its Universal AST cannot represent Rust. `defaultSemanticTypeContract()` in
33`internal/backend/semantic_program.go:85` is hardcoded to
36numeric: binary64, integer_width: unknown, truth: r_compatible,
37ownership: unknown, index_base: 1
40and `semantic_document.go:1014` *validates* that every contract equals exactly
41that, while `typed_operation.go:46` rejects any value model that is not
42`tagged_dynamic_binary64`. There is no integer width and no ownership in the
43model at all. Code like `base16ct`'s constant-time decoder —
46ret += (((0x2fi16 - byte) & (byte - 0x3a)) >> 8) & (byte - 47);
49— depends on exact 16-bit signed wrapping and arithmetic shift. Lowering that
50into a 1-indexed dynamic float64 model produces silently wrong answers. So the
51first rule of this project is the one that codebase broke:
53> **Never approximate a semantic you cannot represent. Fail loudly instead.**
55`src/ty.rs` already does this: `i128`/`u128` are rejected with a reason rather
56than widened or truncated.
58## Architecture
61Rust source ──syn──> syn AST ──lower──> Nim source ──nim c──> binary
64**The frontend is `syn`, deliberately.** Hand-rolling a Rust grammar is how the
65other project went wrong; a correct parser is not the interesting part of this
66problem. The interesting part is the lowering, which is where all the work goes.
68Planned modules:
70| file | role | state |
71|---|---|---|
72| `src/ty.rs` | Rust type → Nim type, exact widths, explicit rejections | written |
Add the differential test runner, and a lowering to measure with it  nandithebull 73| `src/lower.rs` | items, statements, expressions → Nim | written |
74| `src/fmt.rs` | `println!`/`format!` format-string handling | written |
75| `src/prelude.nim` | `Option`/`Result`/panic/`Display`/`Debug` runtime | written |
76| `src/main.rs` | CLI: `rustnim <in.rs> -o <out.nim>` | written |
77| `tests/differential.rs` | the runner described below | written |
Add enums, Option/Result, `?`, and the machinery base16ct needs around them  nandithebull 79### Enums, `Option` and `Result`
81A C-like enum becomes a plain Nim `enum`, which compares, orders and
82`case`-checks the way Rust's does. A data-carrying enum becomes a Nim object
83variant — a discriminant enum plus one branch per variant — which is the same
84shape the prelude already uses for `Option` and `Result`. Nim requires the
85branches of a variant object to have distinct field names, so each payload
86field is prefixed with its variant.
88`match` takes one of two forms. Arms that neither bind nor destructure become
89a Nim `case`, which is exhaustiveness-checked the way Rust's is. Arms that do
90bind become an `if`/`elif` chain with the bindings emitted as `let`s, because
91Nim's `case` cannot destructure. The chain always ends in an arm that panics:
92Rust proved it unreachable, but Nim cannot see that, and leaving the chain
93open would silently fall through instead.
95`Ok`, `Err` and `Some` are emitted with their full type arguments
96(`rsOk[T, E](v)`), because Nim cannot infer `E` from an `Ok(v)` alone. That is
97why the expected type has to reach a `match` arm as well as a `let`.
99`?` expands to statements — a temporary, a discriminant test, and an early
100`return` — which are emitted ahead of the line being built. Rust inserts a
101`From::from` on the error there; we accept only the case where the two error
102types already agree, rather than assume a conversion is the identity. `?` in a
103`while` condition is rejected: the early return would run once before the
104loop rather than on each iteration.
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 106### Trait impls
108A `Display` impl becomes `proc rsDisplay(self: T): string`. Rust's `Formatter`
109is a sink and the observable result of `{}` is exactly the bytes written into
Add display.rs and the alloc half: all of base16ct now goes through  nandithebull 110it, so a write through the formatter **appends** to that string — a `fmt` body
111may write repeatedly, and `UpperHex` writes once per byte in a loop. A body
112that does anything else with the formatter — padding, precision,
113`debug_struct` — is rejected, because those change the output and this model
114does not carry them. `Debug`, `LowerHex`, `UpperHex`, `Binary` and `Octal`
115work the same way.
117Writing into a string cannot fail, so `?` on a formatter write is a no-op. `?`
118on anything else inside a `fmt` body *can* fail, and `format!` panics when a
119formatting impl returns an error — so that is what the error branch does, with
120std's own message.
122`{:x}` on an integer formats its two's-complement bit pattern; on any other
123type it calls that type's own `LowerHex` impl. Those are different operations,
124so a radix format on an argument of unknown type is rejected rather than
125
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 126
127`impl From<A> for B` becomes a conversion proc that `.into()` resolves
128through. A marker trait with no items generates nothing: we do not model trait
129resolution anywhere, so there is nothing for it to affect; a use that actually
130needed the trait (a `dyn`, a bound) is rejected where it appears. Any other
131trait impl is rejected.
133Methods are keyed by `(receiver type, name)`, not by name alone — two types
134may define the same method, and Nim tells them apart by overload resolution on
135the first parameter.
137`fmt::Error` is *not* the same type as a crate's own `Error`. Collapsing a
138qualified path to its last segment merged them, which was a real soundness
139bug; `core::fmt`'s types are now recognised by their qualified name.
141### Slice iterators are resolved to one index loop
143Rust's slice iterators are lazy and compose. Nim's `for` is over one sequence,
144so a chain of adaptors is resolved into a small IR and emitted as a single
145index loop in which **each binding is an lvalue into the original container**.
146That is what makes `*d = v` through `iter_mut()` write back to the caller's
147slice instead of to a copy, and what lets `chunks_exact(2)` hand out a window
148that indexes straight into the source with an offset.
150Only adaptors with an exact index-loop equivalent are accepted. `map`,
151`filter` and `take_while` are rejected rather than partially honoured:
152silently dropping an adaptor would change which elements the loop visits.
154`zip` stops at the shorter side, as Rust's does — that is a test, not an
155assumption (`tests/cases/023`).
157### Borrowed slices are views, not copies
159`&[T]` is a borrow. Nim's experimental view types model exactly that,
160including returning one from a proc: writing through the returned view is
161visible in the original buffer. That was probed against Nim 2.2.4 before being
162relied on, because copying into a `seq` would print the right bytes while
163silently changing aliasing.
Add display.rs and the alloc half: all of base16ct now goes through  nandithebull 165Nim does allow a view inside an object and inside an object *field* — both
166probed, both preserving aliasing — so `Result<&[u8], E>` and
167`HexDisplay<'a>(&'a [u8])` both work. (An earlier version of this document
168claimed otherwise; that was wrong.)
170Two real constraints remain. Nim will not let a `let` borrow out of a local,
171so `.unwrap()`/`.expect()` on a `Result` holding a view is expanded inline and
172the binding becomes an alias — a view is a reference, so there is nothing to
173materialise, and the substituted expression is a plain field access that
174re-evaluates nothing. And `s.get(a..b)` is an `Option` of a view whose
175*validity* is what matters: the view and its condition travel together through
176`ok_or` until a `?` or `unwrap` resolves them into a bounds check plus a
177binding. Keeping such an `Option` in a variable is rejected with a message
178
180A `let` binding a borrow keeps the view rather than copying into a `seq`:
181`let res = encode(..)?` names the caller's buffer, and copying would print the
182right bytes while silently breaking the aliasing.
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 183
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 184### Closures and `unsafe`
186`unsafe` is a permission marker, not a semantic change: it does not alter what
187the enclosed operations mean. So the block is transparent, and every operation
188inside still goes through the ordinary lowering and is still rejected if it has
189no faithful mapping. `unsafe fn` lowers like any other proc.
191A closure becomes a Nim anonymous proc. Nim's closures capture by reference, as
192Rust's non-`move` closures do; a `move` closure captures by value, which is a
193different thing, so it is rejected rather than lowered to the same construct.
194`impl Fn(A) -> B` is left at Nim's default calling convention, which accepts
195both a plain top-level proc and a capturing closure — as Rust's `impl Fn` does.
197`.map`/`.and_then` over an `Option`/`Result` are expanded inline with the
198closure's parameter aliased to the payload, rather than handed to a generic
199proc. That keeps the whole thing an expression and keeps a view a view.
201`&str` is a borrowed view of someone else's bytes, so it maps to
202`openArray[char]`, not to an owned `string`. Nim accepts a `string` argument
203for an `openArray[char]` parameter, so a literal still passes straight through.
204`from_utf8_unchecked` reinterprets a byte view as a character view over the
205same memory — no copy, no validation, and writes through the original are
206visible, as in Rust.
208### Modules
210Rust keeps `lower::decode` and `mixed::decode` apart by module; flattening into
211one Nim module would merge them — they are *different functions*. So the first
212input is the crate root and each later one is a module named by its file stem,
213items are emitted as `<module>_<name>`, and a call resolves through an explicit
214qualifier, then the current module, then what `use` brought into scope, then
215
Add generics; transpile the part of cosmic-theme that is reachable  nandithebull 217### Generics
219Rust type parameters become Nim's. Nim instantiates a generic structurally at
220the call site much as Rust does, so `fn f<T>(x: T) -> T` has a direct target in
221`proc f[T](x: T): T` and no monomorphisation pass is needed.
223**Trait bounds and `where` clauses are dropped.** That is sound in the
224direction that matters: an operation the bound permitted either exists for the
225instantiated type or is a compile error at that instantiation site. Dropping a
226bound cannot make an accepted program mean something different — it only makes
227rustnim accept some programs rustc would reject, which does not matter when
228the input is known-good Rust. (Where it *would* matter is bound-directed
229method selection, e.g. blanket impls choosing between candidates. We do not
230model trait resolution at all, so such a program is rejected elsewhere.)
232Const generic parameters have no Nim equivalent and are still rejected.
234Two things need more than a rename. Nim cannot infer an object's generic
235parameters from a constructor's field values, so `Pair { a: 1, b: 2 }` is
236emitted as `Pair[int32](...)` using the expected type — and a generic enum's
237unit variant (`Holder::Empty`) likewise. And a binding's annotation cannot
238name a parameter Nim is still inferring, so call sites run a small unifier:
239the callee's declared parameter types are matched against the actual argument
240types to bind `T`, and the result is substituted into the return type.
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 242### Declaration order
244Rust has no declaration-before-use rule and Nim does, so every proc is
245forward-declared between the type definitions and the bodies. Reordering the
246input instead would not handle mutual recursion.
Add the differential test runner, and a lowering to measure with it  nandithebull 248### Type propagation is load-bearing
250Rust infers an unsuffixed integer literal's type from context and falls back
251to `i32`; Nim falls back to 64-bit `int`. So `lower.rs` threads an *expected
252type* down through every expression — into `let` annotations, call arguments,
253`match` patterns, compound assignments and both operands of a binary — and
254annotates every binding it emits. Without that, `let x: u8 = 200; x + 100`
255means two different things in the two languages. With it, a width the lowering
256gets wrong becomes a Nim compile error (a loud failure, reported by the
257runner) rather than a wrong answer.
Scaffold rustnim: Rust->Nim transpiler, type mapping   258
259## Mapping decisions made so far
261- **Integers**: exact width. `i32`→`int32`, `usize`→`uint`, etc. `i128`/`u128`
262  rejected.
263- **Indexing**: both 0-based. Direct.
264- **`&T`** → plain value. **`&mut T`** → `var T` parameter.
265- **`&[T]`** → `openArray[T]` in parameter position, `seq[T]` when owned.
266  `Nim::owned()` performs that conversion.
267- **Ownership/borrowck**: ignored. Nim is GC'd; for safe Rust this is sound.
268- **`Option`/`Result`** → object variants in the prelude.
269- **`match`** → Nim `case` where the arms are simple, `if`/`elif` when arms have
270  guards or bindings.
271- **Rust's expression-orientation** maps well: Nim `if`/`case` are expressions
272  too, and a proc's trailing expression is its return value.
Settle signed-shr and unsigned-wrap semantics against both compilers   274### Settled empirically (Nim 2.2.4 vs rustc 1.98.1, both run)
2761. **Nim's `shr` on a signed integer is arithmetic**, matching Rust.
277   `int16(-256) shr 8` = `-1` in Nim; `(-256i16) >> 8` = `-1` in Rust.
278   `base16ct`'s decoder depends on this, so it maps directly with no helper.
2792. **Nim's fixed-width unsigned arithmetic wraps silently**, matching Rust's
280   `wrapping_*`. `uint8(200) + 100` = `44` in Nim; `200u8.wrapping_add(100)`
281   = `44` in Rust. So `wrapping_add` on an unsigned type is just `+`.
Add the differential test runner, and a lowering to measure with it  nandithebull 2833. **We model rustc's debug profile.** Rust debug builds panic on signed
284   integer overflow; Nim's default build raises `OverflowDefect` on it. Those
285   are the matching pair, so the runner invokes `rustc` without `-O` and `nim
286   c` with its defaults, and `tests/cases/016` pins the behaviour. A Rust
287   panic exits 101 where a Nim Defect exits 1, so every generated module ends
288   with a handler that maps one to the other — otherwise the runner's
289   exit-status comparison would be vacuous. `wrapping_*` is therefore an
290   explicit operation on both sides: unsigned maps to the bare operator (item
291   2), signed is routed through the unsigned view of the same width.
2924. **`char` round-trips.** Rust `char` → Nim `Rune`, confirmed for ASCII and
293   non-ASCII scalars in both `{}` and `{:?}`, and across `as u32`
294   (`tests/cases/014`).
Prove byte-identity for base16ct by enumerating whole input domains  nandithebull 2955. **Nim's integer conversion `T(x)` truncates; it does not range-check.**
296   `uint8(511'u16)` is `255`, `uint8(300'i32)` is `44`, `uint8(-1'i32)` is
297   `255` — the same answers as `cast[uint8]`. An earlier version of this
298   document asserted that `T(x)` range-checks, and used that to justify
299   `cast`. The conclusion stands — `cast` is the clearer spelling of
300   "truncate" — but the stated reason was wrong, and it had been assumed
301   rather than probed. Found by sabotaging the cast lowering and watching the
302   exhaustive proof *not* fail, which is what a sabotage test is for.
Add the differential test runner, and a lowering to measure with it  nandithebull 303
Settle signed-shr and unsigned-wrap semantics against both compilers   304### Still open
Add generics; transpile the part of cosmic-theme that is reachable  nandithebull 3066. Const generic parameters, `move` closures, closure bodies with statements,
Lower `extern "C"` blocks to Nim importc; do not transpile libc  nandithebull 307   trait objects and `macro_rules!` definitions are rejected with a reason. A `trait` declaration lowers to nothing — trait resolution
Add associated types and consts, and test whether the blocker stack bottoms out  nandithebull 308   is not modelled — but one giving a method a *default body* is rejected,
309   since that body is code with no impl to be emitted into.
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 310   Lifetime parameters are *not* a rejection: they carry no runtime meaning
311   and Nim is GC'd, so `fn encode<'a>(..)` lowers fine.
Add generics; transpile the part of cosmic-theme that is reachable  nandithebull 3127. `saturating_*` and `checked_*` are implemented, detecting overflow on the
313   unsigned view of the same width rather than with a range check that would
314   itself trap. `wrapping_*`, `overflowing_*` and `strict_*` are not all
315   covered: only add, sub and mul have the saturating and checked forms.
Prove byte-identity for base16ct by enumerating whole input domains  nandithebull 3168. Float formatting matches Rust for ordinary values and for `inf`/`NaN`, but
Add the differential test runner, and a lowering to measure with it  nandithebull 317   the exponent-form thresholds have only been checked at `1e21`.
Prove byte-identity for base16ct by enumerating whole input domains  nandithebull 3189. Functions are scoped by module now, but *types* are still global: two
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 319   modules declaring the same type name would collide. Relatedly, a crate's
320   own `type Result<T>` is told apart from the builtin `Result<T, E>` by
321   arity, which is not how Rust resolves it.
Evaluate host cfg predicates, and measure what that actually buys  nandithebull 32210. Host `#[cfg]` predicates — `unix`, `windows`, `target_os`, `target_arch`,
Lower the `log` facade, and add explicit enum discriminants  nandithebull 323   `target_family`, `target_pointer_width`, `target_endian`,
324   `target_has_atomic` — are evaluated
Evaluate host cfg predicates, and measure what that actually buys  nandithebull 325   against the machine, since the generated Nim is compiled for it. That makes
326   the output host-shaped: a crate branching on platform has had that branch
327   decided at transpile time. `doc`/`doctest`/`miri` are false. A custom or
328   build-script `cfg` (`crossbeam_loom`, `target_has_atomic`) has no value we
329   could know and is rejected.
Transpile a second crate, adler2, to test whether any of this generalises  nandithebull 33011. Associated types (`impl Iterator { type Item = .. }`) and `mod`
331   directories (`specialized/mod.rs`) are not implemented.
33212. `String::from_utf8_unchecked` copies, because Nim's `string` is an owned
Add display.rs and the alloc half: all of base16ct now goes through  nandithebull 333   value. Rust's consumes the `Vec` without copying. Observably the same from
334   the caller, but it is a copy where Rust has none.
Scaffold rustnim: Rust->Nim transpiler, type mapping   335
Transpile a second crate, adler2, to test whether any of this generalises  nandithebull 336## A second crate: does this generalise, or is it fitted to `base16ct`?
338`base16ct` is the crate this was built toward, so passing it proves less than
339it looks. `adler2` 2.0.1 was picked as a deliberately different shape —
340a stateful struct with methods, operator-overload trait impls, a hand-unrolled
341four-lane inner loop — and it now works: `tests/cases/029-adler2-crate/`
342transpiles `algo.rs` byte-for-byte as published, with `lib.rs`'s items and a
343driver, and its checksums are byte-identical to rustc's across every single
344byte, every length to 600 (crossing the 4-byte unrolling boundary and the
3455552-chunk path), and 144 incremental-write splits.
347It needed real work, which is the honest part of the answer. Ten features:
348trait impls generalised beyond formatting and `From` (any trait's methods
349become procs on the type, with the operator traits wired into `+=`/`+`
350dispatch), `Self`, `Type::method()` static calls, `u32::from` between
351primitives, tuple-destructuring `let`, `split_at`, iterators bound to
352variables and `.remainder()`, `[0; 4]` as an array rather than a `seq`, and
353the bare `#[cfg]` flags.
355It also caught a **regression I had introduced**: the three-phase emission
356added for forward declarations was silently dropping `const` items declared
357*inside* a function body. `base16ct` has none, so 33 passing cases said
358nothing about it.
360### What the other crates did
362Run without fixing anything, to see where the wall is rather than to move it:
364| crate | outcome |
365
366| `adler2` 2.0.1 | **works**, byte-identical |
367| `siphasher` 1.0.1 | rejected: `u128` |
368| `rustc-hash` 2.1.1 | rejected: `u128` |
369| `hex` 0.4.3 | rejected: `impl Iterator` needs an associated type |
370| `crc32fast` 1.5.0 | rejected: directory modules (`specialized/mod.rs`), then SIMD intrinsics |
372Two of the five stop at `u128`, which is the founding rule doing its job
373rather than a gap: they are told they cannot be translated instead of being
374handed a silently truncated hasher. The other two are honest missing
375features — associated types, and `mod` directories.
Evaluate host cfg predicates, and measure what that actually buys  nandithebull 377## How far off is a crate like `libcosmic`?
379Measured, not guessed. Running rustnim over `libcosmic`'s own `src/`:
3820 of 164 files produce any translation
38350,633 lines, 112 direct dependencies
386with 66 generic-parameter blockers, 20 trait objects, 51 `async` uses, 235
387`where` clauses, 73 associated types and 1,104 lifetime annotations. Those are
388not features the crate happens to use; they are its architecture. `libcosmic`
389is a north star, not a next step.
391### The blocker survey, and what it says about roadmaps
393400 crates from the local registry (under 4,000 lines each), all their module
394files passed together, first blocker recorded:
Add associated types and consts, and test whether the blocker stack bottoms out  nandithebull 396| blocker | start | +host-`cfg` | +generics | +assoc types |
397|---|---|---|---|---|
398| unevaluable `#[cfg]` | 124 | 34 | 34 | 34 |
399| generic type parameter | 69 | 91 | **1** | 1 |
400| associated types in an `impl` | 54 | 63 | 87 | **2** |
401| unsupported type | 20 | 29 | 60 | 71 |
402| trait object | 17 | 25 | 30 | 35 |
403| macro definition | 21 | 24 | 25 | 32 |
404| raw pointer | 12 | 22 | 24 | 28 |
405| **crates fully transpiled** | **2** | **2** | **2** | **3** |
Add generics; transpile the part of cosmic-theme that is reachable  nandithebull 406
407This has now happened twice. Evaluating the host `#[cfg]` predicates cleared
40890 of 124 blockers and moved the fully-working count by zero. Generics then
409cleared 90 of 91 and moved it by zero again. Every crate each unblocked simply
410hit its next blocker.
412That is the shape of the problem: blockers are **deep, not wide**. A frequency
413ranking of *first* blockers is not a roadmap — it says which feature is most
414often first, not which one finishes a crate. `base16ct`, `adler2` and
415`cosmic-theme`'s spacing model work because their whole stack was ground
416through, one blocker at a time.
Evaluate host cfg predicates, and measure what that actually buys  nandithebull 417
Add associated types and consts, and test whether the blocker stack bottoms out  nandithebull 418### Does the stack have a bottom?
420If the stacks are drawn from one shared finite pool of language features, then
421clearing the pool clears every stack at once, and the flat counts above are
422just what progress looks like before a step change. That is a real possibility
423and it is testable, so it was tested. Sampling every *file* of the 400 crates
424rather than only each crate's first blocker — 1,766 blocker observations:
426| | count | bounded? |
427|---|---|---|
428| distinct normalised blocker kinds | **63** | **yes — this is the language-feature pool** |
429| distinct unsupported std methods | 80 | no — this is `std`'s surface |
430| distinct unknown functions | 75 | no — these are calls into dependencies |
431| distinct unsupported macros | 13 | no |
433So the answer is *both*, split by class:
435- **Language features do bottom out.** 63 distinct kinds, and each one cleared
436  is cleared for every crate forever. This part is finite and the step-change
437  intuition is correct for it. Associated types produced the first net gain
438  (2 → 3), which is what that dynamic looks like starting.
439- **The API surface does not.** 80 distinct `std` methods appeared in 1,766
440  samples of *small* crates; `std` has thousands of items, and each needs a
441  verified Nim equivalent rather than a guess. That is enumerable but it has
442  no bottom you reach by clearing features.
443- **Dependencies are not a pool at all.** 315 of the 400 crates depend on
444  other crates, which have to be transpiled too, recursively, until the graph
445  ends at `libc`, proc macros or SIMD intrinsics — which it does, and those do
446  not translate.
448And among the 85 dependency-free crates, 1 of 84 currently transpiles in full.
449So even with dependencies removed from the picture, features alone are not the
450only remaining gate.
Evaluate host cfg predicates, and measure what that actually buys  nandithebull 452So the ranking above is not a roadmap — it says which feature is most often
453*first*, which is not the same as which feature finishes a crate. The only
454honest way to add a crate is to pick it and clear its stack, as was done
Add associated types and consts, and test whether the blocker stack bottoms out  nandithebull 457Generics and associated types are now done. By frequency the next are
458unsupported types (71), trait objects (35) and `macro_rules!` definitions (32)
459— but see above before treating that as a plan.
Add generics; transpile the part of cosmic-theme that is reachable  nandithebull 460
461### `cosmic-theme`: what was reachable
463`tests/cases/032-cosmic-theme-spacing/` transpiles `corner.rs`, `spacing.rs`
464and `layout.rs` from `cosmic-theme` 1.0.0 — the spacing scale, corner radii
465and density model a COSMIC-native UI needs to match the desktop — with output
466byte-identical to rustc's, including the `Density`/`Spacing` and
467`Roundness`/`CornerRadii` round trips.
469Those files are the crate's own, with one mechanical change recorded here: the
470`use serde::{Deserialize, Serialize}` line and the `Serialize, Deserialize`
471entries in two `derive` lists were removed, because the oracle is plain
472`rustc` with no dependencies available. Nothing else was touched; rustnim
473ignores both anyway.
475The rest of `cosmic-theme` — `theme.rs` (1,830 lines), `color.rs`,
476`cosmic_palette.rs`, `derivation.rs`, `steps.rs`, `composite.rs` — is colour
477work built on `palette` (40,874 lines across 122 files, plus a proc-macro
478crate). `mode.rs` needs `cosmic-config` and its derive macro. Those are
479dependency walls, not language gaps.
Evaluate host cfg predicates, and measure what that actually buys  nandithebull 480
Lower `bitflags!` directly, checked against the real crate  nandithebull 481## `bitflags!`, and why it is lowered rather than expanded
483`bitflags` is the most-depended-on translatable crate in `libcosmic`'s
484resolved tree — 79 of its 741 crates — so it is the highest-leverage target
485there. It is also 26 `macro_rules!` definitions across five files, which is
486the one thing the lowering cannot represent.
488Expanding the macro does not rescue this. `RUSTC_BOOTSTRAP=1 cargo rustc --
489-Zunpretty=expanded` on a single-flag user produces 869 lines that still call
490`bitflags::{Bits, Flag, Flags, iter::Iter, iter::IterNames, parser::from_str,
491parser::to_writer}` — items defined by those same macros. The chain does not
492end in code we could lower.
494What the macro *means*, though, is small and stable: a newtype over an integer
495with named constants and set operations. So `src/macros.rs` parses the
496invocation and the lowering emits that directly. This is a deliberate
497exception to "a macro whose expansion is not known is rejected", and the
498argument is that the expansion *is* known here — it is documented, stable, and
499now pinned by a test.
501`tests/cases/034-bitflags.rs` is that test, and it is unusual: `//@ extern:
502bitflags` makes the **oracle** compile against the real crate while rustnim
503gets no such crate. rustnim has to reproduce bitflags' behaviour without it,
504and the outputs are compared byte for byte. Two behaviours it pins that a
505reimplementation would get wrong:
507- `!x` is complemented and then **masked to `all()`** — `!(READ|WRITE)` is
508  `EXEC`, not `0xFFFFFFFC`.
509- `from_bits` returns `None` for any bit outside `all()`;
510  `from_bits_truncate` masks instead.
512plus the `Debug` spelling, which is `Perms(READ | WRITE)` and `Perms(0x0)`.
514The risk this carries is version drift: a future `bitflags` could change what
515the macro generates, and the shim would not know. The test is what would catch
516it, which is why it links the real crate rather than a copy of its
517documentation.
Lower the `log` facade, and add explicit enum discriminants  nandithebull 519## `log`, on the same argument
521`log` is second by dependents in `libcosmic`'s tree (61 of 741), and it has
522the same shape as `bitflags`: `src/macros.rs` is 20 `macro_rules!`, and that
523is what the dependents use. `src/lib.rs` is the facade — `Level`,
524`LevelFilter`, the `Log` trait behind a `&'static dyn`, atomics and
525`set_logger`.
527So the macros are lowered directly, against behaviour pinned from the real
528crate by `tests/cases/035-log.rs` (again `//@ extern: log`, so the oracle
529links it and rustnim does not). What that test pins:
531- `max_level()` starts at `Off`.
532- `log_enabled!` is **false even after `set_max_level`** when no logger is
533  installed, because the facade consults the logger as well as the level.
534- `Display` for a level is upper-case (`WARN`), `Debug` is not (`Warn`).
535- `Level::Error as usize` is 1 through `Trace` as 5; `LevelFilter::Off` is 0.
537A record's arguments are not evaluated when the level is disabled, so the
538lowering emits `if rsLogEnabled(l): rsLog(l, ..)` rather than computing the
539message first.
541**The boundary is deliberate: rustnim models log's *emitting* side, not its
542installing side.** A transpiled library's `info!` calls work and, with no
543logger, do nothing — which is exactly Rust's behaviour. Installing a logger
544is an application's job and is done from Nim:
547rsLogSetLogger(proc (level: RsLogLevel, target, msg: string) =
548  echo "[", rsDisplay(level), "] ", msg)
549rsLogMaxLevel = int(rsLvlTrace)
552Modelling `impl Log` instead would mean reproducing `Record` and `Metadata`,
553which is more shim surface for something a Nim application would not write in
554Rust anyway.
556The facade's types are emitted as `RsLogLevel` and `RsLogFilter`, not `Level`
557and `LevelFilter`. The first attempt used Rust's names and broke six existing
558cases, because a crate's own `Error` type and an enum field named `Error`
559cannot coexist in one Nim module.
Lower `extern "C"` blocks to Nim importc; do not transpile libc  nandithebull 561## `libc`: not transpiled, and should not be
563`libc` is third by dependents in `libcosmic`'s tree (60 of 741), but it is not
564a crate to translate. Measured:
567129,594 lines across 387 files
568 54,544 `pub const`
569  7,660 `pub fn`, nearly all inside `extern "C"` blocks -- declarations
570  1,926 `pub type`
571    121 actual function bodies in the entire crate
574There is essentially no code in it. It is a set of declarations binding to the
575platform's C library, and **Nim reaches those same symbols natively** — the
576symbols are the same objects, not two implementations of one idea.
578So what was built is the general capability instead: an `extern "C"` block
579lowers to Nim `importc` declarations. Both are statements *about* a symbol
580someone else defines, and both are bound by the C ABI, so the two declarations
581describe one symbol rather than one being a translation of the other. Raw
582pointers map too (`*mut T` to `ptr T`), which cleared all 38 raw-pointer
583blockers in the survey.
585### `const` matters at the C level even though it does not at the Rust one
587Rust emits no C prototype; Nim emits a real one. So declaring `strlen` as
588taking `*const u8` produces:
591error: conflicting types for 'strlen'; have 'NU(char *)'
592note: previous declaration with type 'size_t(const char *)'
595which is the C compiler catching a declaration that does not match the symbol
596— a loud failure, and a better outcome than Rust's silence. To make honest
597declarations expressible, `*const T` maps to a generated const-qualified
601type RsConstPtrcchar* {.importc: "const char *", nodecl.} = distinct pointer
602proc strlen*(s: RsConstPtrcchar): uint {.importc: "strlen", cdecl.}
605one per element type actually used. Nim's generic form (`importc: "const $1*"`)
606was tried first and does not work in 2.2.4. A `*const T` whose C spelling we
607do not know is rejected rather than declared without the `const`.
609`tests/cases/036-extern-c.rs` calls `abs`, `labs`, `strlen` and `atoi` through
610this path, byte-identical to rustc.
Expand `macro_rules!` rather than translating it to a Nim template  nandithebull 612## `macro_rules!`: expanded, not translated
614Nim has `template` and `macro`, so a shape-level correspondence with
615`macro_rules!` exists. Across 1,833 definitions in a 400-crate sample:
617| shape | share | Nim equivalent |
618|---|---|---|
619| single rule, no repetition | 36% | a `template` |
620| multiple rules | 12% | a `macro` dispatching on shape |
621| `$(..)` repetition | 28% | `varargs` in a `macro` |
622| `:tt` token-tree munching | 22% | an interpreter; not mechanical |
624So ~76% has a translatable shape. **We expand instead, and the reason is the
625type-directed lowering.** A Nim template body is *untyped*: substituted first,
626type-checked after. This lowering needs a type at nearly every step — to
627choose `div` over `/`, to size a `cast`, to pick an integer literal's width.
628Translating a macro body would mean lowering Rust with no type information,
629which is exactly the guessing the project refuses. Expanding at the call site
630yields ordinary Rust in a context where the types are known, so it lowers like
631anything else. Same applicability, faithful output.
Expand `$(..)` repetition in macro_rules  nandithebull 633`src/mrules.rs` implements single-rule macros, **with `$(..)` repetition**.
634Recounting the 1,833 definitions by the expander's actual boundary — one rule,
635any repetition, no `:tt`:
637| | count |
638
639| single rule, no repetition | 661 |
640| single rule, with repetition | 271 |
641| **expandable** | **932 (51%)** |
642| multiple rules | 481 |
643| `:tt` | 420 |
645A definition it cannot handle is recorded *with its reason*, so a call site
646says "`foo!` cannot be expanded: nested `$(..)` repetition is not implemented
647yet" rather than "unknown macro". Captured fragments are parenthesised on
648substitution, so `square!(2 + 3)` is 25 and not 11.
650Repetition needed two things that are easy to get wrong, both caught by
651`tests/cases/038`:
653- `>` closes `=>` and `->` as well as a generic argument list, so the nesting
654  depth used to find a fragment's end must not go negative. Without that,
655  `$($a:expr => $b:expr),*` swallows the whole invocation.
656- A repetition's separator may be *inside* the pattern rather than between
657  iterations — `$first:expr $(, $rest:expr)*` has no separator, its comma
658  leads each iteration. A trailing fragment therefore stops at the separator
659  when there is one and at whatever starts the next iteration when there is
662A block expression with statements (`{{ let mut m = ..; m }}`, which is how
663these macros are usually written) now lowers to Nim's `block:` expression
664rather than being hoisted, so the same macro expanded at two call sites does
665not collide in one scope.
667**Expansion moved the survey more than everything before it combined: 3 → 18
668of 400 crates accepted.** A `macro_rules!` used to be a hard stop at item
669level, failing a whole crate on sight.
671Repetition then moved it by **zero** — 18 before, 18 after — for the reason
672every previous feature did: the crates it unblocked hit their next blocker.
673That is now seven features running. The capability is real and tested; the
674crate count is gated by something else.
Expand `macro_rules!` rather than translating it to a Nim template  nandithebull 675
676### A stricter number
678"rustnim exits 0" is not "the output is real". Of those 18, **12 produce Nim
679that the Nim compiler accepts**:
682adler2  arrayref  cfg_aliases ×3  cfg-if ×2  ctor-lite  darling ×4
685Compiling is still not behaving: only the cases in `tests/cases/` are checked
686against rustc for identical output. Three numbers, in increasing strength —
687accepted 18, compiles 12, behaviourally verified only the corpus.
689## `serde`: assessed, not attempted
691`serde` is 49 dependents in `libcosmic`'s tree and the most generic crate
692looked at here: 17,237 lines, 369 `impl<`, 922 `where` clauses, 849 uses of
693`'de`, 324 associated types. `serde_derive` is another 8,975 lines of *proc
694macro* — a program that runs at compile time, so it can be expanded (as
695`bitflags!` was, with `RUSTC_BOOTSTRAP=1`) but never translated.
697The derive's expansion is small and clean — thirteen lines for a two-field
698struct. But it is **generic over a `Serializer`**, so unlike `bitflags!`
699(self-contained) and `log` (a facade with a defined no-op default), it has no
700observable behaviour at all until a format crate supplies one. A shim would
701therefore have to pick a format and implement *that*, which is a narrower and
702much larger commitment than either previous shim.
704`serde_json`'s exact output was pinned for whenever that is attempted:
705declaration order, no whitespace, `null` for `None`, and a float keeps the
706`.0` that `Display` drops — `{"x":-3,"ratio":2.0,"maybe":null}`.
Prove byte-identity for base16ct by enumerating whole input domains  nandithebull 708## Proof of byte-identity for `base16ct`
710[`PROOF.md`](PROOF.md) sets out what is actually established: exhaustive
711agreement over every two-byte decode input (65,536), every two-byte encode
712input (65,536), every single byte through `encode_str` and `HexDisplay`, and
713every length to 128 — plus a compositional argument extending those to inputs
714of any length, and 20,000 pseudorandom multi-chunk cases attacking the one
715step in that argument that is inspection rather than enumeration. Run it with
716`cargo test --test proof`. It is explicit about the difference between the
717exhaustive parts and the sampled ones.
Scaffold rustnim: Rust->Nim transpiler, type mapping   719## Testing: differential, not golden
721The bar is **behavioural equivalence with rustc**, not that the output looks
722plausible. For each case in `tests/cases/`:
725rustc case.rs && ./case            > expected
726rustnim case.rs -o case.nim && nim c -r case.nim > actual
727diff expected actual
730A case only counts as passing when both binaries build *and* produce identical
Add the differential test runner, and a lowering to measure with it  nandithebull 731stdout *and* exit with the same status.
733`tests/differential.rs` implements this, and checks each stage separately so a
734failure says where it went wrong: `rustnim`, `rustc`, `nim`, or `diff`. Three
735guards exist specifically because of how the other transpiler failed:
737- `rustnim` exiting 0 while writing **no output file** is a failure.
738- `rustnim` exiting 0 while writing an **empty output file** is a failure.
739- An **empty corpus** is a failure, so the runner cannot pass by finding
740  nothing to do.
742All three have been verified by deliberately breaking the transpiler and
743confirming the runner goes red.
745Cases carry directives in leading `//@` comments:
747| directive | meaning |
748
749| `//@ reject: <substring>` | `rustnim` must *fail*, with this in its message |
750| `//@ skip: <reason>` | not run; reported as skipped |
751| `//@ args: <argv>` | passed to both binaries |
752| `//@ stdin: <line>` | fed to both binaries |
Lower `bitflags!` directly, checked against the real crate  nandithebull 753| `//@ cfg: feature=<name>` | passed to rustnim, and to rustc as `--cfg feature="<name>"` |
754| `//@ extern: <crate>` | the **oracle** links this crate; rustnim does not get it |
Add the differential test runner, and a lowering to measure with it  nandithebull 755
756`reject` cases are how the "fail loudly" rule is tested rather than merely
757stated: `900`–`904` pin the rejections of `i128`, an unmapped standard-library
758method, a float→int cast, an unimplemented format spec, and a closure.
760Run one case with `RUSTNIM_CASE=005 cargo test --test differential --
761--nocapture`. Nim is found at `.nim-toolchain/bin/nim` in the repository root
762or any parent, or via `RUSTNIM_NIM`.
Scaffold rustnim: Rust->Nim transpiler, type mapping   763
764## Toolchain
766- `rustc` / `cargo` 1.98.1 — system.
767- Nim 2.2.4 — vendored at `.nim-toolchain/` (gitignored; downloaded from
768  nim-lang.org, not installed system-wide). Binary: `.nim-toolchain/bin/nim`.
770## Milestone 1
772Transpile `base16ct` 1.0.0 — the crate the other transpiler failed on — and
Add enums, Option/Result, `?`, and the machinery base16ct needs around them  nandithebull 773have its decoder produce byte-identical output to the Rust original.
Add display.rs and the alloc half: all of base16ct now goes through  nandithebull 775**Reached.** `tests/cases/026-base16ct-crate/` transpiles **every source file
776of base16ct 1.0.0** — `error.rs`, `lower.rs`, `upper.rs`, `mixed.rs` and
777`display.rs`, each byte-for-byte as published on crates.io, verified with
778`cmp` rather than by eye — together with `lib.rs`'s `decoded_len`,
779`encoded_len` and `decode_inner` verbatim. The `alloc` half is on, via
780`--cfg feature=alloc`. Output is byte-identical to rustc's:
Add enums, Option/Result, `?`, and the machinery base16ct needs around them  nandithebull 781
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 782
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 783lower ok abcd1234 len=4              decode: lower, upper, mixed
784upper-rej err InvalidEncoding ...    upper correctly rejects lowercase
785oddlen err InvalidLength / invalid Base16 length     <- Debug and Display
786encode ok 6162636431323334 len=8     encode, both cases
787encode_str ok abcd1234 len=8         closure over unsafe, borrowed &str
Add display.rs and the alloc half: all of base16ct now goes through  nandithebull 788Ok([171, 205, 18, 52])               decode_vec       \
789abcd1234                             encode_string     > the alloc half
790ABCD1234 abcd1234                    HexDisplay {:X} {:x}
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 791
Add enums, Option/Result, `?`, and the machinery base16ct needs around them  nandithebull 792
Add display.rs and the alloc half: all of base16ct now goes through  nandithebull 793Everything lowers as written: `dst.get_mut(..decoded_len(src)?)`,
794`src.chunks_exact(2).zip(dst.iter_mut())`, `*dst = byte as u8`, the returned
795`&'a [u8]` view into the caller's buffer, `encode(src, dst).map(|r| unsafe {
796core::str::from_utf8_unchecked(r) })`, and `HexDisplay`'s `UpperHex` impl
797writing once per byte into the formatter.
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 798
Add display.rs and the alloc half: all of base16ct now goes through  nandithebull 799This is the crate whose six files the transpiler in `findings/` emitted empty
800output for, while exiting 0.