nandi/rustnim · DESIGN.md

⑂ 0a375d8ec0bfffe1baadd340864ab1ffa92fdb98
DESIGN.md · 354 lines · 16.9 KBmarkdown    Blame HistoryRaw
Scaffold rustnim: Rust->Nim transpiler, type mapping   1# rustnim — a Rust → Nim transpiler
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 5**Milestone 1 is reached, and then some.** Four of base16ct's six modules go
6through byte-for-byte. 32 differential cases, 28 behavioural and 4 rejections,
7plus 6 unit/integration tests. All green. Run `cargo test`.
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 8
9Passing today: functions, `impl` methods, trait impls (formatting traits and
10`From`), structs, enums (C-like and data-carrying), `Option`/`Result` with
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 11`?`, closures, `unsafe`, slice iterators (`iter`/`iter_mut`/`enumerate`/`zip`/`chunks_exact`/
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 12`chunks_exact_mut`/`windows`), borrowed slices as values and return types,
13`let`/`let mut`, the full integer
Add enums, Option/Result, `?`, and the machinery base16ct needs around them  nandithebull 14and float operator set at exact widths, `as` casts, `if`/`while`/`loop`/`for`,
15`match` including patterns that bind, `Vec`/slices/arrays, type aliases
16(including generic ones), function-typed parameters (`impl Fn(A) -> B`),
17multi-file input, `#[cfg]` evaluation, and `println!`/`format!` with `{}`,
18`{:?}`, `{:x}`, `{:b}`, positional and inline-named arguments, and
Add the differential test runner, and a lowering to measure with it  nandithebull 19zero/space padding.
Scaffold rustnim: Rust->Nim transpiler, type mapping   20
21## Why this exists
23We tried [tarekwasfy01/Code-Transpiler](https://github.com/tarekwasfy01/Code-Transpiler),
24which advertises `rust` as a source language, on the `base16ct` crate. It emits
25empty files and exits 0. The full investigation is in [`findings/`](findings/)
26and is published at
27https://rickub.com/nandi/code-transpiler-rust-frontend-findings
29The decisive finding, and the reason this is a new project rather than a patch:
30its Universal AST cannot represent Rust. `defaultSemanticTypeContract()` in
31`internal/backend/semantic_program.go:85` is hardcoded to
34numeric: binary64, integer_width: unknown, truth: r_compatible,
35ownership: unknown, index_base: 1
38and `semantic_document.go:1014` *validates* that every contract equals exactly
39that, while `typed_operation.go:46` rejects any value model that is not
40`tagged_dynamic_binary64`. There is no integer width and no ownership in the
41model at all. Code like `base16ct`'s constant-time decoder —
44ret += (((0x2fi16 - byte) & (byte - 0x3a)) >> 8) & (byte - 47);
47— depends on exact 16-bit signed wrapping and arithmetic shift. Lowering that
48into a 1-indexed dynamic float64 model produces silently wrong answers. So the
49first rule of this project is the one that codebase broke:
51> **Never approximate a semantic you cannot represent. Fail loudly instead.**
53`src/ty.rs` already does this: `i128`/`u128` are rejected with a reason rather
54than widened or truncated.
56## Architecture
59Rust source ──syn──> syn AST ──lower──> Nim source ──nim c──> binary
62**The frontend is `syn`, deliberately.** Hand-rolling a Rust grammar is how the
63other project went wrong; a correct parser is not the interesting part of this
64problem. The interesting part is the lowering, which is where all the work goes.
66Planned modules:
68| file | role | state |
69|---|---|---|
70| `src/ty.rs` | Rust type → Nim type, exact widths, explicit rejections | written |
Add the differential test runner, and a lowering to measure with it  nandithebull 71| `src/lower.rs` | items, statements, expressions → Nim | written |
72| `src/fmt.rs` | `println!`/`format!` format-string handling | written |
73| `src/prelude.nim` | `Option`/`Result`/panic/`Display`/`Debug` runtime | written |
74| `src/main.rs` | CLI: `rustnim <in.rs> -o <out.nim>` | written |
75| `tests/differential.rs` | the runner described below | written |
Add enums, Option/Result, `?`, and the machinery base16ct needs around them  nandithebull 77### Enums, `Option` and `Result`
79A C-like enum becomes a plain Nim `enum`, which compares, orders and
80`case`-checks the way Rust's does. A data-carrying enum becomes a Nim object
81variant — a discriminant enum plus one branch per variant — which is the same
82shape the prelude already uses for `Option` and `Result`. Nim requires the
83branches of a variant object to have distinct field names, so each payload
84field is prefixed with its variant.
86`match` takes one of two forms. Arms that neither bind nor destructure become
87a Nim `case`, which is exhaustiveness-checked the way Rust's is. Arms that do
88bind become an `if`/`elif` chain with the bindings emitted as `let`s, because
89Nim's `case` cannot destructure. The chain always ends in an arm that panics:
90Rust proved it unreachable, but Nim cannot see that, and leaving the chain
91open would silently fall through instead.
93`Ok`, `Err` and `Some` are emitted with their full type arguments
94(`rsOk[T, E](v)`), because Nim cannot infer `E` from an `Ok(v)` alone. That is
95why the expected type has to reach a `match` arm as well as a `let`.
97`?` expands to statements — a temporary, a discriminant test, and an early
98`return` — which are emitted ahead of the line being built. Rust inserts a
99`From::from` on the error there; we accept only the case where the two error
100types already agree, rather than assume a conversion is the identity. `?` in a
101`while` condition is rejected: the early return would run once before the
102loop rather than on each iteration.
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 104### Trait impls
106A `Display` impl becomes `proc rsDisplay(self: T): string`. Rust's `Formatter`
107is a sink and the observable result of `{}` is exactly the bytes written into
108it, so every write through the formatter produces that string and the existing
109`match`/trailing-expression machinery assembles it. A `fmt` body that does
110anything else with the formatter — padding, precision, `debug_struct` — is
111rejected, because those change the output and this model does not carry them.
112`Debug`, `LowerHex`, `UpperHex`, `Binary` and `Octal` work the same way.
114`impl From<A> for B` becomes a conversion proc that `.into()` resolves
115through. A marker trait with no items generates nothing: we do not model trait
116resolution anywhere, so there is nothing for it to affect; a use that actually
117needed the trait (a `dyn`, a bound) is rejected where it appears. Any other
118trait impl is rejected.
120Methods are keyed by `(receiver type, name)`, not by name alone — two types
121may define the same method, and Nim tells them apart by overload resolution on
122the first parameter.
124`fmt::Error` is *not* the same type as a crate's own `Error`. Collapsing a
125qualified path to its last segment merged them, which was a real soundness
126bug; `core::fmt`'s types are now recognised by their qualified name.
128### Slice iterators are resolved to one index loop
130Rust's slice iterators are lazy and compose. Nim's `for` is over one sequence,
131so a chain of adaptors is resolved into a small IR and emitted as a single
132index loop in which **each binding is an lvalue into the original container**.
133That is what makes `*d = v` through `iter_mut()` write back to the caller's
134slice instead of to a copy, and what lets `chunks_exact(2)` hand out a window
135that indexes straight into the source with an offset.
137Only adaptors with an exact index-loop equivalent are accepted. `map`,
138`filter` and `take_while` are rejected rather than partially honoured:
139silently dropping an adaptor would change which elements the loop visits.
141`zip` stops at the shorter side, as Rust's does — that is a test, not an
142assumption (`tests/cases/023`).
144### Borrowed slices are views, not copies
146`&[T]` is a borrow. Nim's experimental view types model exactly that,
147including returning one from a proc: writing through the returned view is
148visible in the original buffer. That was probed against Nim 2.2.4 before being
149relied on, because copying into a `seq` would print the right bytes while
150silently changing aliasing.
152`s.get(a..b)` is the one place this leaks. It is an `Option<&[T]>`, and Nim
153cannot put a view inside an object, so there is no value to hand back. Instead
154the view and its validity condition travel together through `ok_or` until a
155`?` or `unwrap` resolves them into a bounds check plus a binding. Keeping such
156an `Option` in a variable is rejected with a message saying so.
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 158### Closures and `unsafe`
160`unsafe` is a permission marker, not a semantic change: it does not alter what
161the enclosed operations mean. So the block is transparent, and every operation
162inside still goes through the ordinary lowering and is still rejected if it has
163no faithful mapping. `unsafe fn` lowers like any other proc.
165A closure becomes a Nim anonymous proc. Nim's closures capture by reference, as
166Rust's non-`move` closures do; a `move` closure captures by value, which is a
167different thing, so it is rejected rather than lowered to the same construct.
168`impl Fn(A) -> B` is left at Nim's default calling convention, which accepts
169both a plain top-level proc and a capturing closure — as Rust's `impl Fn` does.
171`.map`/`.and_then` over an `Option`/`Result` are expanded inline with the
172closure's parameter aliased to the payload, rather than handed to a generic
173proc. That keeps the whole thing an expression and keeps a view a view.
175`&str` is a borrowed view of someone else's bytes, so it maps to
176`openArray[char]`, not to an owned `string`. Nim accepts a `string` argument
177for an `openArray[char]` parameter, so a literal still passes straight through.
178`from_utf8_unchecked` reinterprets a byte view as a character view over the
179same memory — no copy, no validation, and writes through the original are
180visible, as in Rust.
182### Modules
184Rust keeps `lower::decode` and `mixed::decode` apart by module; flattening into
185one Nim module would merge them — they are *different functions*. So the first
186input is the crate root and each later one is a module named by its file stem,
187items are emitted as `<module>_<name>`, and a call resolves through an explicit
188qualifier, then the current module, then what `use` brought into scope, then
189
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 191### Declaration order
193Rust has no declaration-before-use rule and Nim does, so every proc is
194forward-declared between the type definitions and the bodies. Reordering the
195input instead would not handle mutual recursion.
Add the differential test runner, and a lowering to measure with it  nandithebull 197### Type propagation is load-bearing
199Rust infers an unsuffixed integer literal's type from context and falls back
200to `i32`; Nim falls back to 64-bit `int`. So `lower.rs` threads an *expected
201type* down through every expression — into `let` annotations, call arguments,
202`match` patterns, compound assignments and both operands of a binary — and
203annotates every binding it emits. Without that, `let x: u8 = 200; x + 100`
204means two different things in the two languages. With it, a width the lowering
205gets wrong becomes a Nim compile error (a loud failure, reported by the
206runner) rather than a wrong answer.
Scaffold rustnim: Rust->Nim transpiler, type mapping   207
208## Mapping decisions made so far
210- **Integers**: exact width. `i32`→`int32`, `usize`→`uint`, etc. `i128`/`u128`
211  rejected.
212- **Indexing**: both 0-based. Direct.
213- **`&T`** → plain value. **`&mut T`** → `var T` parameter.
214- **`&[T]`** → `openArray[T]` in parameter position, `seq[T]` when owned.
215  `Nim::owned()` performs that conversion.
216- **Ownership/borrowck**: ignored. Nim is GC'd; for safe Rust this is sound.
217- **`Option`/`Result`** → object variants in the prelude.
218- **`match`** → Nim `case` where the arms are simple, `if`/`elif` when arms have
219  guards or bindings.
220- **Rust's expression-orientation** maps well: Nim `if`/`case` are expressions
221  too, and a proc's trailing expression is its return value.
Settle signed-shr and unsigned-wrap semantics against both compilers   223### Settled empirically (Nim 2.2.4 vs rustc 1.98.1, both run)
2251. **Nim's `shr` on a signed integer is arithmetic**, matching Rust.
226   `int16(-256) shr 8` = `-1` in Nim; `(-256i16) >> 8` = `-1` in Rust.
227   `base16ct`'s decoder depends on this, so it maps directly with no helper.
2282. **Nim's fixed-width unsigned arithmetic wraps silently**, matching Rust's
229   `wrapping_*`. `uint8(200) + 100` = `44` in Nim; `200u8.wrapping_add(100)`
230   = `44` in Rust. So `wrapping_add` on an unsigned type is just `+`.
Add the differential test runner, and a lowering to measure with it  nandithebull 2323. **We model rustc's debug profile.** Rust debug builds panic on signed
233   integer overflow; Nim's default build raises `OverflowDefect` on it. Those
234   are the matching pair, so the runner invokes `rustc` without `-O` and `nim
235   c` with its defaults, and `tests/cases/016` pins the behaviour. A Rust
236   panic exits 101 where a Nim Defect exits 1, so every generated module ends
237   with a handler that maps one to the other — otherwise the runner's
238   exit-status comparison would be vacuous. `wrapping_*` is therefore an
239   explicit operation on both sides: unsigned maps to the bare operator (item
240   2), signed is routed through the unsigned view of the same width.
2414. **`char` round-trips.** Rust `char` → Nim `Rune`, confirmed for ASCII and
242   non-ASCII scalars in both `{}` and `{:?}`, and across `as u32`
243   (`tests/cases/014`).
Settle signed-shr and unsigned-wrap semantics against both compilers   245### Still open
Add the differential test runner, and a lowering to measure with it  nandithebull 2475. `checked_*` and `saturating_*` are not mapped yet; they are currently
248   rejected as unsupported methods rather than approximated.
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 2496. Generics (type and const parameters), `move` closures, closure bodies with
250   statements, and trait impls other than the formatting traits and `From` are
251   rejected with a reason.
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 252   Lifetime parameters are *not* a rejection: they carry no runtime meaning
253   and Nim is GC'd, so `fn encode<'a>(..)` lowers fine.
Add the differential test runner, and a lowering to measure with it  nandithebull 2547. Float formatting matches Rust for ordinary values and for `inf`/`NaN`, but
255   the exponent-form thresholds have only been checked at `1e21`.
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 2568. Functions are scoped by module now, but *types* are still global: two
257   modules declaring the same type name would collide. Relatedly, a crate's
258   own `type Result<T>` is told apart from the builtin `Result<T, E>` by
259   arity, which is not how Rust resolves it.
Scaffold rustnim: Rust->Nim transpiler, type mapping   260
261## Testing: differential, not golden
263The bar is **behavioural equivalence with rustc**, not that the output looks
264plausible. For each case in `tests/cases/`:
267rustc case.rs && ./case            > expected
268rustnim case.rs -o case.nim && nim c -r case.nim > actual
269diff expected actual
272A case only counts as passing when both binaries build *and* produce identical
Add the differential test runner, and a lowering to measure with it  nandithebull 273stdout *and* exit with the same status.
275`tests/differential.rs` implements this, and checks each stage separately so a
276failure says where it went wrong: `rustnim`, `rustc`, `nim`, or `diff`. Three
277guards exist specifically because of how the other transpiler failed:
279- `rustnim` exiting 0 while writing **no output file** is a failure.
280- `rustnim` exiting 0 while writing an **empty output file** is a failure.
281- An **empty corpus** is a failure, so the runner cannot pass by finding
282  nothing to do.
284All three have been verified by deliberately breaking the transpiler and
285confirming the runner goes red.
287Cases carry directives in leading `//@` comments:
289| directive | meaning |
290
291| `//@ reject: <substring>` | `rustnim` must *fail*, with this in its message |
292| `//@ skip: <reason>` | not run; reported as skipped |
293| `//@ args: <argv>` | passed to both binaries |
294| `//@ stdin: <line>` | fed to both binaries |
296`reject` cases are how the "fail loudly" rule is tested rather than merely
297stated: `900`–`904` pin the rejections of `i128`, an unmapped standard-library
298method, a float→int cast, an unimplemented format spec, and a closure.
300Run one case with `RUSTNIM_CASE=005 cargo test --test differential --
301--nocapture`. Nim is found at `.nim-toolchain/bin/nim` in the repository root
302or any parent, or via `RUSTNIM_NIM`.
Scaffold rustnim: Rust->Nim transpiler, type mapping   303
304## Toolchain
306- `rustc` / `cargo` 1.98.1 — system.
307- Nim 2.2.4 — vendored at `.nim-toolchain/` (gitignored; downloaded from
308  nim-lang.org, not installed system-wide). Binary: `.nim-toolchain/bin/nim`.
310## Milestone 1
312Transpile `base16ct` 1.0.0 — the crate the other transpiler failed on — and
Add enums, Option/Result, `?`, and the machinery base16ct needs around them  nandithebull 313have its decoder produce byte-identical output to the Rust original.
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 315**Reached, for four of the crate's six modules.**
316`tests/cases/026-base16ct-crate/` transpiles base16ct's `error.rs`,
317`lower.rs`, `upper.rs` and `mixed.rs` **byte-for-byte as published on
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 318crates.io** — verified with `cmp`, not by eye — together with `lib.rs`'s
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 319`decoded_len`, `encoded_len` and `decode_inner` verbatim. Output is
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 320byte-identical to rustc's:
Add enums, Option/Result, `?`, and the machinery base16ct needs around them  nandithebull 321
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 322
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 323lower ok abcd1234 len=4              decode: lower, upper, mixed
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 324mixed-m ok abcd1234 len=4
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 325upper ok abcd1234 len=4
326upper-rej err InvalidEncoding ...    upper correctly rejects lowercase
327oddlen err InvalidLength / invalid Base16 length     <- Debug and Display
328encode ok 6162636431323334 len=8     encode, both cases
329encode-up ok 4142434431323334 len=8
330encode_str ok abcd1234 len=8         closure over unsafe, borrowed &str
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 331
Add enums, Option/Result, `?`, and the machinery base16ct needs around them  nandithebull 332
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 333`decode_inner` goes through as written: `dst.get_mut(..decoded_len(src)?)`,
334`src.chunks_exact(2).zip(dst.iter_mut())`, `*dst = byte as u8`, and the
335returned `&'a [u8]` view into the caller's buffer. The `Display` line in that
336output comes from the crate's own `impl fmt::Display for Error`.
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 338`encode_str` goes through as written — `encode(src, dst).map(|r| unsafe {
339core::str::from_utf8_unchecked(r) })` — returning a `&str` view of the bytes
340just written, not a copy.
Add trait impls and slice iterators; base16ct's decoder now goes through  nandithebull 342### Still to do for the whole crate
Add closures and unsafe; base16ct's lower.rs and upper.rs go through  nandithebull 344Two modules remain, and both are close:
346- `display.rs` needs the formatter to *accumulate* writes. Our model maps a
347  formatting impl to a proc returning the string it wrote, which handles one
348  write or a `match` over several; `UpperHex` here calls `f.write_str(..)?` in
349  a loop, so the writes have to append instead of being the value. It also
350  needs `HexDisplay<'a>(pub &'a [u8])` — a struct *field* of view type, the
351  same wall as `Option<&[T]>`.
352- The `alloc` half (`--cfg feature=alloc`) now gets as far as
353  `debug_assert_eq!`, then needs `String::from_utf8_unchecked` to produce an
354  owned `String` from a `Vec<u8>`.