nandi/rustnim · DESIGN.md

⑂ afb2a6e152356d5a78d536ad75f005bb1fc7ed63
DESIGN.md · 369 lines · 17.8 KBmarkdown    Blame HistoryRaw
Scaffold rustnim: Rust->Nim transpiler, type mapping   1# rustnim — a Rust → Nim transpiler
Add display.rs and the alloc half: all of base16ct now goes through  nandithebull 5**Milestone 1 is reached: all of `base16ct` goes through.** Every one of its
6source files transpiles byte-for-byte as published, `alloc` half included, and
7its decode and encode output is byte-identical to rustc's. 33 differential
8cases, 29 behavioural and 4 rejections, plus 6 unit/integration tests. All
9green. Run `cargo test`.
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 10
11Passing today: functions, `impl` methods, trait impls (formatting traits and
12`From`), structs, enums (C-like and data-carrying), `Option`/`Result` with
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 13`?`, closures, `unsafe`, slice iterators (`iter`/`iter_mut`/`enumerate`/`zip`/`chunks_exact`/
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 14`chunks_exact_mut`/`windows`), borrowed slices as values and return types,
15`let`/`let mut`, the full integer
Add enums, Option/Result, `?`, and the machinery base16ct needs around them  nandithebull 16and float operator set at exact widths, `as` casts, `if`/`while`/`loop`/`for`,
17`match` including patterns that bind, `Vec`/slices/arrays, type aliases
18(including generic ones), function-typed parameters (`impl Fn(A) -> B`),
19multi-file input, `#[cfg]` evaluation, and `println!`/`format!` with `{}`,
20`{:?}`, `{:x}`, `{:b}`, positional and inline-named arguments, and
Add the differential test runner, and a lowering to measure with it  nandithebull 21zero/space padding.
Scaffold rustnim: Rust->Nim transpiler, type mapping   22
23## Why this exists
25We tried [tarekwasfy01/Code-Transpiler](https://github.com/tarekwasfy01/Code-Transpiler),
26which advertises `rust` as a source language, on the `base16ct` crate. It emits
27empty files and exits 0. The full investigation is in [`findings/`](findings/)
28and is published at
29https://rickub.com/nandi/code-transpiler-rust-frontend-findings
31The decisive finding, and the reason this is a new project rather than a patch:
32its Universal AST cannot represent Rust. `defaultSemanticTypeContract()` in
33`internal/backend/semantic_program.go:85` is hardcoded to
36numeric: binary64, integer_width: unknown, truth: r_compatible,
37ownership: unknown, index_base: 1
40and `semantic_document.go:1014` *validates* that every contract equals exactly
41that, while `typed_operation.go:46` rejects any value model that is not
42`tagged_dynamic_binary64`. There is no integer width and no ownership in the
43model at all. Code like `base16ct`'s constant-time decoder —
46ret += (((0x2fi16 - byte) & (byte - 0x3a)) >> 8) & (byte - 47);
49— depends on exact 16-bit signed wrapping and arithmetic shift. Lowering that
50into a 1-indexed dynamic float64 model produces silently wrong answers. So the
51first rule of this project is the one that codebase broke:
53> **Never approximate a semantic you cannot represent. Fail loudly instead.**
55`src/ty.rs` already does this: `i128`/`u128` are rejected with a reason rather
56than widened or truncated.
58## Architecture
61Rust source ──syn──> syn AST ──lower──> Nim source ──nim c──> binary
64**The frontend is `syn`, deliberately.** Hand-rolling a Rust grammar is how the
65other project went wrong; a correct parser is not the interesting part of this
66problem. The interesting part is the lowering, which is where all the work goes.
68Planned modules:
70| file | role | state |
71|---|---|---|
72| `src/ty.rs` | Rust type → Nim type, exact widths, explicit rejections | written |
Add the differential test runner, and a lowering to measure with it  nandithebull 73| `src/lower.rs` | items, statements, expressions → Nim | written |
74| `src/fmt.rs` | `println!`/`format!` format-string handling | written |
75| `src/prelude.nim` | `Option`/`Result`/panic/`Display`/`Debug` runtime | written |
76| `src/main.rs` | CLI: `rustnim <in.rs> -o <out.nim>` | written |
77| `tests/differential.rs` | the runner described below | written |
Add enums, Option/Result, `?`, and the machinery base16ct needs around them  nandithebull 79### Enums, `Option` and `Result`
81A C-like enum becomes a plain Nim `enum`, which compares, orders and
82`case`-checks the way Rust's does. A data-carrying enum becomes a Nim object
83variant — a discriminant enum plus one branch per variant — which is the same
84shape the prelude already uses for `Option` and `Result`. Nim requires the
85branches of a variant object to have distinct field names, so each payload
86field is prefixed with its variant.
88`match` takes one of two forms. Arms that neither bind nor destructure become
89a Nim `case`, which is exhaustiveness-checked the way Rust's is. Arms that do
90bind become an `if`/`elif` chain with the bindings emitted as `let`s, because
91Nim's `case` cannot destructure. The chain always ends in an arm that panics:
92Rust proved it unreachable, but Nim cannot see that, and leaving the chain
93open would silently fall through instead.
95`Ok`, `Err` and `Some` are emitted with their full type arguments
96(`rsOk[T, E](v)`), because Nim cannot infer `E` from an `Ok(v)` alone. That is
97why the expected type has to reach a `match` arm as well as a `let`.
99`?` expands to statements — a temporary, a discriminant test, and an early
100`return` — which are emitted ahead of the line being built. Rust inserts a
101`From::from` on the error there; we accept only the case where the two error
102types already agree, rather than assume a conversion is the identity. `?` in a
103`while` condition is rejected: the early return would run once before the
104loop rather than on each iteration.
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 106### Trait impls
108A `Display` impl becomes `proc rsDisplay(self: T): string`. Rust's `Formatter`
109is a sink and the observable result of `{}` is exactly the bytes written into
Add display.rs and the alloc half: all of base16ct now goes through  nandithebull 110it, so a write through the formatter **appends** to that string — a `fmt` body
111may write repeatedly, and `UpperHex` writes once per byte in a loop. A body
112that does anything else with the formatter — padding, precision,
113`debug_struct` — is rejected, because those change the output and this model
114does not carry them. `Debug`, `LowerHex`, `UpperHex`, `Binary` and `Octal`
115work the same way.
117Writing into a string cannot fail, so `?` on a formatter write is a no-op. `?`
118on anything else inside a `fmt` body *can* fail, and `format!` panics when a
119formatting impl returns an error — so that is what the error branch does, with
120std's own message.
122`{:x}` on an integer formats its two's-complement bit pattern; on any other
123type it calls that type's own `LowerHex` impl. Those are different operations,
124so a radix format on an argument of unknown type is rejected rather than
125
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 126
127`impl From<A> for B` becomes a conversion proc that `.into()` resolves
128through. A marker trait with no items generates nothing: we do not model trait
129resolution anywhere, so there is nothing for it to affect; a use that actually
130needed the trait (a `dyn`, a bound) is rejected where it appears. Any other
131trait impl is rejected.
133Methods are keyed by `(receiver type, name)`, not by name alone — two types
134may define the same method, and Nim tells them apart by overload resolution on
135the first parameter.
137`fmt::Error` is *not* the same type as a crate's own `Error`. Collapsing a
138qualified path to its last segment merged them, which was a real soundness
139bug; `core::fmt`'s types are now recognised by their qualified name.
141### Slice iterators are resolved to one index loop
143Rust's slice iterators are lazy and compose. Nim's `for` is over one sequence,
144so a chain of adaptors is resolved into a small IR and emitted as a single
145index loop in which **each binding is an lvalue into the original container**.
146That is what makes `*d = v` through `iter_mut()` write back to the caller's
147slice instead of to a copy, and what lets `chunks_exact(2)` hand out a window
148that indexes straight into the source with an offset.
150Only adaptors with an exact index-loop equivalent are accepted. `map`,
151`filter` and `take_while` are rejected rather than partially honoured:
152silently dropping an adaptor would change which elements the loop visits.
154`zip` stops at the shorter side, as Rust's does — that is a test, not an
155assumption (`tests/cases/023`).
157### Borrowed slices are views, not copies
159`&[T]` is a borrow. Nim's experimental view types model exactly that,
160including returning one from a proc: writing through the returned view is
161visible in the original buffer. That was probed against Nim 2.2.4 before being
162relied on, because copying into a `seq` would print the right bytes while
163silently changing aliasing.
Add display.rs and the alloc half: all of base16ct now goes through  nandithebull 165Nim does allow a view inside an object and inside an object *field* — both
166probed, both preserving aliasing — so `Result<&[u8], E>` and
167`HexDisplay<'a>(&'a [u8])` both work. (An earlier version of this document
168claimed otherwise; that was wrong.)
170Two real constraints remain. Nim will not let a `let` borrow out of a local,
171so `.unwrap()`/`.expect()` on a `Result` holding a view is expanded inline and
172the binding becomes an alias — a view is a reference, so there is nothing to
173materialise, and the substituted expression is a plain field access that
174re-evaluates nothing. And `s.get(a..b)` is an `Option` of a view whose
175*validity* is what matters: the view and its condition travel together through
176`ok_or` until a `?` or `unwrap` resolves them into a bounds check plus a
177binding. Keeping such an `Option` in a variable is rejected with a message
178
180A `let` binding a borrow keeps the view rather than copying into a `seq`:
181`let res = encode(..)?` names the caller's buffer, and copying would print the
182right bytes while silently breaking the aliasing.
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 183
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 184### Closures and `unsafe`
186`unsafe` is a permission marker, not a semantic change: it does not alter what
187the enclosed operations mean. So the block is transparent, and every operation
188inside still goes through the ordinary lowering and is still rejected if it has
189no faithful mapping. `unsafe fn` lowers like any other proc.
191A closure becomes a Nim anonymous proc. Nim's closures capture by reference, as
192Rust's non-`move` closures do; a `move` closure captures by value, which is a
193different thing, so it is rejected rather than lowered to the same construct.
194`impl Fn(A) -> B` is left at Nim's default calling convention, which accepts
195both a plain top-level proc and a capturing closure — as Rust's `impl Fn` does.
197`.map`/`.and_then` over an `Option`/`Result` are expanded inline with the
198closure's parameter aliased to the payload, rather than handed to a generic
199proc. That keeps the whole thing an expression and keeps a view a view.
201`&str` is a borrowed view of someone else's bytes, so it maps to
202`openArray[char]`, not to an owned `string`. Nim accepts a `string` argument
203for an `openArray[char]` parameter, so a literal still passes straight through.
204`from_utf8_unchecked` reinterprets a byte view as a character view over the
205same memory — no copy, no validation, and writes through the original are
206visible, as in Rust.
208### Modules
210Rust keeps `lower::decode` and `mixed::decode` apart by module; flattening into
211one Nim module would merge them — they are *different functions*. So the first
212input is the crate root and each later one is a module named by its file stem,
213items are emitted as `<module>_<name>`, and a call resolves through an explicit
214qualifier, then the current module, then what `use` brought into scope, then
215
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 217### Declaration order
219Rust has no declaration-before-use rule and Nim does, so every proc is
220forward-declared between the type definitions and the bodies. Reordering the
221input instead would not handle mutual recursion.
Add the differential test runner, and a lowering to measure with it  nandithebull 223### Type propagation is load-bearing
225Rust infers an unsuffixed integer literal's type from context and falls back
226to `i32`; Nim falls back to 64-bit `int`. So `lower.rs` threads an *expected
227type* down through every expression — into `let` annotations, call arguments,
228`match` patterns, compound assignments and both operands of a binary — and
229annotates every binding it emits. Without that, `let x: u8 = 200; x + 100`
230means two different things in the two languages. With it, a width the lowering
231gets wrong becomes a Nim compile error (a loud failure, reported by the
232runner) rather than a wrong answer.
Scaffold rustnim: Rust->Nim transpiler, type mapping   233
234## Mapping decisions made so far
236- **Integers**: exact width. `i32`→`int32`, `usize`→`uint`, etc. `i128`/`u128`
237  rejected.
238- **Indexing**: both 0-based. Direct.
239- **`&T`** → plain value. **`&mut T`** → `var T` parameter.
240- **`&[T]`** → `openArray[T]` in parameter position, `seq[T]` when owned.
241  `Nim::owned()` performs that conversion.
242- **Ownership/borrowck**: ignored. Nim is GC'd; for safe Rust this is sound.
243- **`Option`/`Result`** → object variants in the prelude.
244- **`match`** → Nim `case` where the arms are simple, `if`/`elif` when arms have
245  guards or bindings.
246- **Rust's expression-orientation** maps well: Nim `if`/`case` are expressions
247  too, and a proc's trailing expression is its return value.
Settle signed-shr and unsigned-wrap semantics against both compilers   249### Settled empirically (Nim 2.2.4 vs rustc 1.98.1, both run)
2511. **Nim's `shr` on a signed integer is arithmetic**, matching Rust.
252   `int16(-256) shr 8` = `-1` in Nim; `(-256i16) >> 8` = `-1` in Rust.
253   `base16ct`'s decoder depends on this, so it maps directly with no helper.
2542. **Nim's fixed-width unsigned arithmetic wraps silently**, matching Rust's
255   `wrapping_*`. `uint8(200) + 100` = `44` in Nim; `200u8.wrapping_add(100)`
256   = `44` in Rust. So `wrapping_add` on an unsigned type is just `+`.
Add the differential test runner, and a lowering to measure with it  nandithebull 2583. **We model rustc's debug profile.** Rust debug builds panic on signed
259   integer overflow; Nim's default build raises `OverflowDefect` on it. Those
260   are the matching pair, so the runner invokes `rustc` without `-O` and `nim
261   c` with its defaults, and `tests/cases/016` pins the behaviour. A Rust
262   panic exits 101 where a Nim Defect exits 1, so every generated module ends
263   with a handler that maps one to the other — otherwise the runner's
264   exit-status comparison would be vacuous. `wrapping_*` is therefore an
265   explicit operation on both sides: unsigned maps to the bare operator (item
266   2), signed is routed through the unsigned view of the same width.
2674. **`char` round-trips.** Rust `char` → Nim `Rune`, confirmed for ASCII and
268   non-ASCII scalars in both `{}` and `{:?}`, and across `as u32`
269   (`tests/cases/014`).
Settle signed-shr and unsigned-wrap semantics against both compilers   271### Still open
Add the differential test runner, and a lowering to measure with it  nandithebull 2735. `checked_*` and `saturating_*` are not mapped yet; they are currently
274   rejected as unsupported methods rather than approximated.
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 2756. Generics (type and const parameters), `move` closures, closure bodies with
276   statements, and trait impls other than the formatting traits and `From` are
277   rejected with a reason.
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 278   Lifetime parameters are *not* a rejection: they carry no runtime meaning
279   and Nim is GC'd, so `fn encode<'a>(..)` lowers fine.
Add the differential test runner, and a lowering to measure with it  nandithebull 2807. Float formatting matches Rust for ordinary values and for `inf`/`NaN`, but
281   the exponent-form thresholds have only been checked at `1e21`.
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 2828. Functions are scoped by module now, but *types* are still global: two
283   modules declaring the same type name would collide. Relatedly, a crate's
284   own `type Result<T>` is told apart from the builtin `Result<T, E>` by
285   arity, which is not how Rust resolves it.
Add display.rs and the alloc half: all of base16ct now goes through  nandithebull 2869. `String::from_utf8_unchecked` copies, because Nim's `string` is an owned
287   value. Rust's consumes the `Vec` without copying. Observably the same from
288   the caller, but it is a copy where Rust has none.
Scaffold rustnim: Rust->Nim transpiler, type mapping   289
290## Testing: differential, not golden
292The bar is **behavioural equivalence with rustc**, not that the output looks
293plausible. For each case in `tests/cases/`:
296rustc case.rs && ./case            > expected
297rustnim case.rs -o case.nim && nim c -r case.nim > actual
298diff expected actual
301A case only counts as passing when both binaries build *and* produce identical
Add the differential test runner, and a lowering to measure with it  nandithebull 302stdout *and* exit with the same status.
304`tests/differential.rs` implements this, and checks each stage separately so a
305failure says where it went wrong: `rustnim`, `rustc`, `nim`, or `diff`. Three
306guards exist specifically because of how the other transpiler failed:
308- `rustnim` exiting 0 while writing **no output file** is a failure.
309- `rustnim` exiting 0 while writing an **empty output file** is a failure.
310- An **empty corpus** is a failure, so the runner cannot pass by finding
311  nothing to do.
313All three have been verified by deliberately breaking the transpiler and
314confirming the runner goes red.
316Cases carry directives in leading `//@` comments:
318| directive | meaning |
319
320| `//@ reject: <substring>` | `rustnim` must *fail*, with this in its message |
321| `//@ skip: <reason>` | not run; reported as skipped |
322| `//@ args: <argv>` | passed to both binaries |
323| `//@ stdin: <line>` | fed to both binaries |
325`reject` cases are how the "fail loudly" rule is tested rather than merely
326stated: `900`–`904` pin the rejections of `i128`, an unmapped standard-library
327method, a float→int cast, an unimplemented format spec, and a closure.
329Run one case with `RUSTNIM_CASE=005 cargo test --test differential --
330--nocapture`. Nim is found at `.nim-toolchain/bin/nim` in the repository root
331or any parent, or via `RUSTNIM_NIM`.
Scaffold rustnim: Rust->Nim transpiler, type mapping   332
333## Toolchain
335- `rustc` / `cargo` 1.98.1 — system.
336- Nim 2.2.4 — vendored at `.nim-toolchain/` (gitignored; downloaded from
337  nim-lang.org, not installed system-wide). Binary: `.nim-toolchain/bin/nim`.
339## Milestone 1
341Transpile `base16ct` 1.0.0 — the crate the other transpiler failed on — and
Add enums, Option/Result, `?`, and the machinery base16ct needs around them  nandithebull 342have its decoder produce byte-identical output to the Rust original.
Add display.rs and the alloc half: all of base16ct now goes through  nandithebull 344**Reached.** `tests/cases/026-base16ct-crate/` transpiles **every source file
345of base16ct 1.0.0** — `error.rs`, `lower.rs`, `upper.rs`, `mixed.rs` and
346`display.rs`, each byte-for-byte as published on crates.io, verified with
347`cmp` rather than by eye — together with `lib.rs`'s `decoded_len`,
348`encoded_len` and `decode_inner` verbatim. The `alloc` half is on, via
349`--cfg feature=alloc`. Output is byte-identical to rustc's:
Add enums, Option/Result, `?`, and the machinery base16ct needs around them  nandithebull 350
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 351
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 352lower ok abcd1234 len=4              decode: lower, upper, mixed
353upper-rej err InvalidEncoding ...    upper correctly rejects lowercase
354oddlen err InvalidLength / invalid Base16 length     <- Debug and Display
355encode ok 6162636431323334 len=8     encode, both cases
356encode_str ok abcd1234 len=8         closure over unsafe, borrowed &str
Add display.rs and the alloc half: all of base16ct now goes through  nandithebull 357Ok([171, 205, 18, 52])               decode_vec       \
358abcd1234                             encode_string     > the alloc half
359ABCD1234 abcd1234                    HexDisplay {:X} {:x}
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 360
Add enums, Option/Result, `?`, and the machinery base16ct needs around them  nandithebull 361
Add display.rs and the alloc half: all of base16ct now goes through  nandithebull 362Everything lowers as written: `dst.get_mut(..decoded_len(src)?)`,
363`src.chunks_exact(2).zip(dst.iter_mut())`, `*dst = byte as u8`, the returned
364`&'a [u8]` view into the caller's buffer, `encode(src, dst).map(|r| unsafe {
365core::str::from_utf8_unchecked(r) })`, and `HexDisplay`'s `UpperHex` impl
366writing once per byte into the formatter.
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 367
Add display.rs and the alloc half: all of base16ct now goes through  nandithebull 368This is the crate whose six files the transpiler in `findings/` emitted empty
369output for, while exiting 0.