nandi/code-transpiler-rust-frontend-findings · README.md

nandi/code-transpiler-rust-frontend-findingspublic⑂ Fork 0

⑂ 5375d17f26e5be6bd30310edeabaaa1211437b03

README.md · 151 lines · 5.8 KBmarkdown    Blame HistoryRaw
Reproducible bug report for the Rust frontend in Code-Transpiler   1# Rust → Nim in Code-Transpiler: what actually happens
3A reproducible bug report for [tarekwasfy01/Code-Transpiler](https://github.com/tarekwasfy01/Code-Transpiler),
4which lists `rust` as a supported source language.
6**Summary: Rust cannot be used as a source language, and the CLI does not say so.**
7Most real Rust fails in the lexer. The rest parses into a tree containing no
8functions, and the CLI writes an empty-but-valid target file and exits `0`.
10Nothing here is a criticism of the project's R-derived core, which works. The
11issue is specifically the Rust frontend, and the silent failure that hides it.
13## Reproduce
16
19Clones upstream, builds `cmd/r2many`, runs the minimal cases through the CLI,
20then re-runs them through a prober that reports which UAST node kinds the
21frontend actually produced. Needs Go 1.26+ and network access. Recorded output:
22[`EXPECTED.md`](EXPECTED.md).
24## What was tried first
26A random real crate: `base16ct` 1.0.0 — 613 lines, pure Rust, `no_std`, no
27dependencies. Five of its six source files failed to parse. The sixth, which
28contains three functions, transpiled to this complete Nim program:
35For the minimal cases in `cases/`, the emitted Nim file is even smaller: one
36byte, a newline. Exit code 0 in both situations.
38Full transcript: [`outputs/base16ct-1.0.0-transcript.txt`](outputs/base16ct-1.0.0-transcript.txt).
40## The four defects
42### 0. `//` line comments are not supported
44[`internal/backend/parser.go:157`](https://github.com/tarekwasfy01/Code-Transpiler/blob/main/internal/backend/parser.go#L157)
46This tokenizer recognises `#` comments only, alongside R operators (`<<-`,
47`%in%`, `[[`). A Rust `//` becomes two division operators:
50expected expression near "/"
53[`cases/00-comment.rs`](cases/00-comment.rs) and
54[`cases/00-comment-fails.rs`](cases/00-comment-fails.rs) differ by exactly one
55comment line, and only the second fails. Every file in a real crate has
56comments, `base16ct` included — doc comments alone account for much of why it
57never reaches the later stages.
59### 1. `'` is always a string opener, so lifetimes break the lexer
61[`internal/matrixir/lexical.go:261`](https://github.com/tarekwasfy01/Code-Transpiler/blob/main/internal/matrixir/lexical.go#L261)
64if c == '"' || c == '\'' || c == '`' {
67`<'a>` and `Formatter<'_>` are consumed as unterminated char literals. The
68delimiter pairing in `AnalyzeTokenStructure` then desyncs, which is why the
69reported error is a bracket complaint at an offset that always turns out to be
70a lifetime. Case: [`cases/04-lifetime.rs`](cases/04-lifetime.rs).
72This one looks genuinely small to fix: for `source == "rust"`, treat `'` as a
73literal only when it matches `'\?.'`, and emit a lifetime token otherwise.
75### 2. The Rust parser is the R parser
77[`internal/backend/frontend_fact_parser.go:387`](https://github.com/tarekwasfy01/Code-Transpiler/blob/main/internal/backend/frontend_fact_parser.go#L387)
80func (p *factParser) parseIf() (ParsedNode, error) {
81
82	if _, e := p.expect(tokLParen, ""); e != nil {   // requires  if ( ... )
85This is a second, separate tokenizer from the one in defect 1 — the two stages
86disagree — but both are R's. Parenthesised conditions are mandatory, so Rust's `if cond { }` can never
87parse — it fails with `expected "" near "<first token of cond>"`. Case:
88[`cases/02-if-block.rs`](cases/02-if-block.rs). Macro invocations are likewise
89unhandled ([`cases/03-macro.rs`](cases/03-macro.rs)).
91The emitted semantic contract is hardcoded R for a file declared as Rust:
94"language_profile": "rust",
95"value_model": "tagged_dynamic_binary64",
96"index_base": 1,
97"type_contract": { "truth": "r_compatible", "ownership": "unknown" }
100### 3. `fn` is never recognised as a declaration
102This is the one that yields empty output. Node kinds produced for two
103equivalent programs, via [`prober/main.go`](prober/main.go):
105| input | function nodes |
106
107| [`cases/05-equivalent.R`](cases/05-equivalent.R) — `f <- function(a) {…}` | `FUNCTION:1, CLOSURE:1, function:5, parameter:2` |
108| [`cases/01-trivial.rs`](cases/01-trivial.rs) — `fn f(a: i32) -> i32 {…}` | **none** — `call:2, CALL:1` |
110`fn f(a: i32)` is parsed as a *call* to `f`, with `fn` left as a stray
111identifier. The tree holds no functions, so the Nim emitter correctly emits
112nothing. Note that case 01 reports **no parse error** — it is the silent path.
114## The silent-failure path
116`-runtime` defaults to `true`. The fallback catches the parse failure, writes a
117syntactically valid empty program, and exits `0`. The machinery to report this
118already exists and is correct when asked:
121-native      strict native frontend for "rust" is not implemented
122-no-runtime  1/1 translations failed: [{nim DIRECT_NATIVE_UNAVAILABLE (stage=direct)}]
123(default)    exit 0, empty file
126Refusing to emit a program with zero functions when the input declared some
127would turn a silent wrong answer into an honest error, without touching the
128frontend at all.
130## Scope of a real fix
132Defects 0 and 1 are patches. Defects 2 and 3 are not: there is no Rust frontend to
133repair. Supporting Rust means writing one — items, generics, lifetimes,
134`impl`/traits, `match`, macros, `?` — and lowering it into a semantic model
135that isn't R-shaped, since the type contract is R throughout. That is
136comparable in size to the existing ~1,400-line R frontend, and probably larger.
138## Contents
141cases/     minimal inputs, one per defect, plus the R control
142prober/    drop-in cmd/dbg that prints UAST node-kind counts
143outputs/   the base16ct run: transcript and the one emitted .nim
144repro.sh   clone, build, run everything
147`prober/main.go` imports `internal/backend`, so it only compiles from inside
148the upstream tree; `repro.sh` copies it to `cmd/dbg` for you.
150Upstream is MIT. No upstream or crate source is vendored here — `repro.sh`
151fetches what it needs.

Reproducible bug report for the Rust frontend in Code-Transpiler 5375d17 nandi 19h ago	1	# Rust → Nim in Code-Transpiler: what actually happens
	2
	3	A reproducible bug report for [tarekwasfy01/Code-Transpiler](https://github.com/tarekwasfy01/Code-Transpiler),
	4	which lists `rust` as a supported source language.
	5
	6	Summary: Rust cannot be used as a source language, and the CLI does not say so.
	7	Most real Rust fails in the lexer. The rest parses into a tree containing no
	8	functions, and the CLI writes an empty-but-valid target file and exits `0`.
	9
	10	Nothing here is a criticism of the project's R-derived core, which works. The
	11	issue is specifically the Rust frontend, and the silent failure that hides it.
	12
	13	## Reproduce
	14
	15	```bash
	16	./repro.sh
	17	```
	18
	19	Clones upstream, builds `cmd/r2many`, runs the minimal cases through the CLI,
	20	then re-runs them through a prober that reports which UAST node kinds the
	21	frontend actually produced. Needs Go 1.26+ and network access. Recorded output:
	22	[`EXPECTED.md`](EXPECTED.md).
	23
	24	## What was tried first
	25
	26	A random real crate: `base16ct` 1.0.0 — 613 lines, pure Rust, `no_std`, no
	27	dependencies. Five of its six source files failed to parse. The sixth, which
	28	contains three functions, transpiled to this complete Nim program:
	29
	30	```nim
	31	{
	32	}
	33	```
	34
	35	For the minimal cases in `cases/`, the emitted Nim file is even smaller: one
	36	byte, a newline. Exit code 0 in both situations.
	37
	38	Full transcript: [`outputs/base16ct-1.0.0-transcript.txt`](outputs/base16ct-1.0.0-transcript.txt).
	39
	40	## The four defects
	41
	42	### 0. `//` line comments are not supported
	43
	44	[`internal/backend/parser.go:157`](https://github.com/tarekwasfy01/Code-Transpiler/blob/main/internal/backend/parser.go#L157)
	45
	46	This tokenizer recognises `#` comments only, alongside R operators (`<<-`,
	47	`%in%`, `[[`). A Rust `//` becomes two division operators:
	48
	49	```
	50	expected expression near "/"
	51	```
	52
	53	[`cases/00-comment.rs`](cases/00-comment.rs) and
	54	[`cases/00-comment-fails.rs`](cases/00-comment-fails.rs) differ by exactly one
	55	comment line, and only the second fails. Every file in a real crate has
	56	comments, `base16ct` included — doc comments alone account for much of why it
	57	never reaches the later stages.
	58
	59	### 1. `'` is always a string opener, so lifetimes break the lexer
	60
	61	[`internal/matrixir/lexical.go:261`](https://github.com/tarekwasfy01/Code-Transpiler/blob/main/internal/matrixir/lexical.go#L261)
	62
	63	```go
	64	if c == '"' \|\| c == '\'' \|\| c == '`' {
65	```
66
67	`<'a>` and `Formatter<'_>` are consumed as unterminated char literals. The
68	delimiter pairing in `AnalyzeTokenStructure` then desyncs, which is why the
69	reported error is a bracket complaint at an offset that always turns out to be
70	a lifetime. Case: [`cases/04-lifetime.rs`](cases/04-lifetime.rs).
71
72	This one looks genuinely small to fix: for `source == "rust"`, treat `'` as a
73	literal only when it matches `'\?.'`, and emit a lifetime token otherwise.
74
75	### 2. The Rust parser is the R parser
76
77	[`internal/backend/frontend_fact_parser.go:387`](https://github.com/tarekwasfy01/Code-Transpiler/blob/main/internal/backend/frontend_fact_parser.go#L387)
78
79	```go
80	func (p *factParser) parseIf() (ParsedNode, error) {
81	p.next()
82	if _, e := p.expect(tokLParen, ""); e != nil { // requires if ( ... )
83	```
84
85	This is a second, separate tokenizer from the one in defect 1 — the two stages
86	disagree — but both are R's. Parenthesised conditions are mandatory, so Rust's `if cond { }` can never
87	parse — it fails with `expected "" near "<first token of cond>"`. Case:
88	[`cases/02-if-block.rs`](cases/02-if-block.rs). Macro invocations are likewise
89	unhandled ([`cases/03-macro.rs`](cases/03-macro.rs)).
90
91	The emitted semantic contract is hardcoded R for a file declared as Rust:
92
93	```json
94	"language_profile": "rust",
95	"value_model": "tagged_dynamic_binary64",
96	"index_base": 1,
97	"type_contract": { "truth": "r_compatible", "ownership": "unknown" }
98	```
99
100	### 3. `fn` is never recognised as a declaration
101
102	This is the one that yields empty output. Node kinds produced for two
103	equivalent programs, via [`prober/main.go`](prober/main.go):
104
105	\| input \| function nodes \|
106	\|---\|---\|
107	\| [`cases/05-equivalent.R`](cases/05-equivalent.R) — `f <- function(a) {…}` \| `FUNCTION:1, CLOSURE:1, function:5, parameter:2` \|
108	\| [`cases/01-trivial.rs`](cases/01-trivial.rs) — `fn f(a: i32) -> i32 {…}` \| none — `call:2, CALL:1` \|
109
110	`fn f(a: i32)` is parsed as a call to `f`, with `fn` left as a stray
111	identifier. The tree holds no functions, so the Nim emitter correctly emits
112	nothing. Note that case 01 reports no parse error — it is the silent path.
113
114	## The silent-failure path
115
116	`-runtime` defaults to `true`. The fallback catches the parse failure, writes a
117	syntactically valid empty program, and exits `0`. The machinery to report this
118	already exists and is correct when asked:
119
120	```
121	-native strict native frontend for "rust" is not implemented
122	-no-runtime 1/1 translations failed: [{nim DIRECT_NATIVE_UNAVAILABLE (stage=direct)}]
123	(default) exit 0, empty file
124	```
125
126	Refusing to emit a program with zero functions when the input declared some
127	would turn a silent wrong answer into an honest error, without touching the
128	frontend at all.
129
130	## Scope of a real fix
131
132	Defects 0 and 1 are patches. Defects 2 and 3 are not: there is no Rust frontend to
133	repair. Supporting Rust means writing one — items, generics, lifetimes,
134	`impl`/traits, `match`, macros, `?` — and lowering it into a semantic model
135	that isn't R-shaped, since the type contract is R throughout. That is
136	comparable in size to the existing ~1,400-line R frontend, and probably larger.
137
138	## Contents
139
140	```
141	cases/ minimal inputs, one per defect, plus the R control
142	prober/ drop-in cmd/dbg that prints UAST node-kind counts
143	outputs/ the base16ct run: transcript and the one emitted .nim
144	repro.sh clone, build, run everything
145	```
146
147	`prober/main.go` imports `internal/backend`, so it only compiles from inside
148	the upstream tree; `repro.sh` copies it to `cmd/dbg` for you.
149
150	Upstream is MIT. No upstream or crate source is vendored here — `repro.sh`
151	fetches what it needs.