Rust → Nim in Code-Transpiler: what actually happens

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151

# Rust → Nim in Code-Transpiler: what actually happens

A reproducible bug report for [tarekwasfy01/Code-Transpiler](https://github.com/tarekwasfy01/Code-Transpiler),
which lists `rust` as a supported source language.

**Summary: Rust cannot be used as a source language, and the CLI does not say so.**
Most real Rust fails in the lexer. The rest parses into a tree containing no
functions, and the CLI writes an empty-but-valid target file and exits `0`.

Nothing here is a criticism of the project's R-derived core, which works. The
issue is specifically the Rust frontend, and the silent failure that hides it.

## Reproduce

```bash
./repro.sh
```

Clones upstream, builds `cmd/r2many`, runs the minimal cases through the CLI,
then re-runs them through a prober that reports which UAST node kinds the
frontend actually produced. Needs Go 1.26+ and network access. Recorded output:
[`EXPECTED.md`](EXPECTED.md).

## What was tried first

A random real crate: `base16ct` 1.0.0 — 613 lines, pure Rust, `no_std`, no
dependencies. Five of its six source files failed to parse. The sixth, which
contains three functions, transpiled to this complete Nim program:

```nim
{
}
```

For the minimal cases in `cases/`, the emitted Nim file is even smaller: one
byte, a newline. Exit code 0 in both situations.

Full transcript: [`outputs/base16ct-1.0.0-transcript.txt`](outputs/base16ct-1.0.0-transcript.txt).

## The four defects

### 0. `//` line comments are not supported

[`internal/backend/parser.go:157`](https://github.com/tarekwasfy01/Code-Transpiler/blob/main/internal/backend/parser.go#L157)

This tokenizer recognises `#` comments only, alongside R operators (`<<-`,
`%in%`, `[[`). A Rust `//` becomes two division operators:

```
expected expression near "/"
```

[`cases/00-comment.rs`](cases/00-comment.rs) and
[`cases/00-comment-fails.rs`](cases/00-comment-fails.rs) differ by exactly one
comment line, and only the second fails. Every file in a real crate has
comments, `base16ct` included — doc comments alone account for much of why it
never reaches the later stages.

### 1. `'` is always a string opener, so lifetimes break the lexer

[`internal/matrixir/lexical.go:261`](https://github.com/tarekwasfy01/Code-Transpiler/blob/main/internal/matrixir/lexical.go#L261)

```go
if c == '"' || c == '\'' || c == '`' {
```

`<'a>` and `Formatter<'_>` are consumed as unterminated char literals. The
delimiter pairing in `AnalyzeTokenStructure` then desyncs, which is why the
reported error is a bracket complaint at an offset that always turns out to be
a lifetime. Case: [`cases/04-lifetime.rs`](cases/04-lifetime.rs).

This one looks genuinely small to fix: for `source == "rust"`, treat `'` as a
literal only when it matches `'\?.'`, and emit a lifetime token otherwise.

### 2. The Rust parser is the R parser

[`internal/backend/frontend_fact_parser.go:387`](https://github.com/tarekwasfy01/Code-Transpiler/blob/main/internal/backend/frontend_fact_parser.go#L387)

```go
func (p *factParser) parseIf() (ParsedNode, error) {
	p.next()
	if _, e := p.expect(tokLParen, ""); e != nil {   // requires  if ( ... )
```

This is a second, separate tokenizer from the one in defect 1 — the two stages
disagree — but both are R's. Parenthesised conditions are mandatory, so Rust's `if cond { }` can never
parse — it fails with `expected "" near "<first token of cond>"`. Case:
[`cases/02-if-block.rs`](cases/02-if-block.rs). Macro invocations are likewise
unhandled ([`cases/03-macro.rs`](cases/03-macro.rs)).

The emitted semantic contract is hardcoded R for a file declared as Rust:

```json
"language_profile": "rust",
"value_model": "tagged_dynamic_binary64",
"index_base": 1,
"type_contract": { "truth": "r_compatible", "ownership": "unknown" }
```

### 3. `fn` is never recognised as a declaration

This is the one that yields empty output. Node kinds produced for two
equivalent programs, via [`prober/main.go`](prober/main.go):

| input | function nodes |
|---|---|
| [`cases/05-equivalent.R`](cases/05-equivalent.R) — `f <- function(a) {…}` | `FUNCTION:1, CLOSURE:1, function:5, parameter:2` |
| [`cases/01-trivial.rs`](cases/01-trivial.rs) — `fn f(a: i32) -> i32 {…}` | **none** — `call:2, CALL:1` |

`fn f(a: i32)` is parsed as a *call* to `f`, with `fn` left as a stray
identifier. The tree holds no functions, so the Nim emitter correctly emits
nothing. Note that case 01 reports **no parse error** — it is the silent path.

## The silent-failure path

`-runtime` defaults to `true`. The fallback catches the parse failure, writes a
syntactically valid empty program, and exits `0`. The machinery to report this
already exists and is correct when asked:

```
-native      strict native frontend for "rust" is not implemented
-no-runtime  1/1 translations failed: [{nim DIRECT_NATIVE_UNAVAILABLE (stage=direct)}]
(default)    exit 0, empty file
```

Refusing to emit a program with zero functions when the input declared some
would turn a silent wrong answer into an honest error, without touching the
frontend at all.

## Scope of a real fix

Defects 0 and 1 are patches. Defects 2 and 3 are not: there is no Rust frontend to
repair. Supporting Rust means writing one — items, generics, lifetimes,
`impl`/traits, `match`, macros, `?` — and lowering it into a semantic model
that isn't R-shaped, since the type contract is R throughout. That is
comparable in size to the existing ~1,400-line R frontend, and probably larger.

## Contents

```
cases/     minimal inputs, one per defect, plus the R control
prober/    drop-in cmd/dbg that prints UAST node-kind counts
outputs/   the base16ct run: transcript and the one emitted .nim
repro.sh   clone, build, run everything
```

`prober/main.go` imports `internal/backend`, so it only compiles from inside
the upstream tree; `repro.sh` copies it to `cmd/dbg` for you.

Upstream is MIT. No upstream or crate source is vendored here — `repro.sh`
fetches what it needs.