nandi/rustnim · 04eb29f

⑂ 04eb29f

Expand `macro_rules!` rather than translating it to a Nim template

Nim has template and macro, so a shape correspondence with macro_rules!
exists, and it is not small: of 1,833 definitions across a 400-crate sample,
36% are a single rule with no repetition -- exactly a template -- 12% are
multi-rule, 28% use repetition and would be varargs in a macro, and 22% munch
token trees and have no mechanical translation. About 76% has a translatable
shape.

We expand instead, and the reason is this project's approach rather than
Nim's expressiveness. A Nim template body is untyped: substituted first and
type-checked after. The lowering here is type-directed at nearly every step,
needing a type to choose div over /, to size a cast, to pick a literal's
width. Translating a body would mean lowering Rust with no type information,
which is the guessing the project exists to refuse. Expanding at the call
site gives ordinary Rust in a context where the types are known.

src/mrules.rs implements the single-rule case. A definition it cannot handle
is recorded with its reason, so a call site reports "`foo!` cannot be
expanded: `$(..)` repetition is not implemented yet" instead of "unknown
macro". Captured fragments are parenthesised on substitution, so square!(2+3)
is 25 and not 11.

This moved the survey more than every previous feature combined: 3 to 18 of
400 crates accepted. A macro_rules! used to be a hard stop at item level,
failing a whole crate on sight.

It also prompted a stricter metric, since "rustnim exits 0" is not "the
output is real". Of those 18, 12 produce Nim the Nim compiler accepts.
Compiling is still not behaving; only tests/cases/ is checked against rustc
for identical output. DESIGN.md now states all three numbers.

serde is assessed and not attempted, with the reasoning recorded: its derive
expands cleanly but is generic over a Serializer, so unlike bitflags! and log
it has no observable behaviour until a format crate supplies one. A shim
would have to pick a format and implement that. serde_json's exact output is
pinned in DESIGN.md for whenever that happens. The JSON prelude groundwork
written before that assessment is reverted rather than shipped unused.

43 differential cases, all green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

nandithebull committed 2026-09-18T23:27:23-07:00 Browse files

04eb29f parent: 7db7991

modified Cargo.lock +1 -0 Viewed

@@ -38,6 +38,7 @@ version = "0.1.0"
38	38	dependencies = [
39	39	"bitflags",
40	40	"log",
	41	+ "proc-macro2",
41	42	"syn",
42	43	]
43	44

  dependencies = [
   "bitflags",
   "log",
++ "proc-macro2",
   "syn",
  ]

modified Cargo.toml +1 -0 Viewed

@@ -4,6 +4,7 @@ version = "0.1.0"
4	4	edition = "2024"
5	5
6	6	[dependencies]
	7	+proc-macro2 = "1"
7	8	syn = { version = "3.0.5", features = ["full", "extra-traits", "visit"] }
8	9
9	10	# Used only by the test corpus: a case carrying `//@ extern: bitflags` is

  edition = "2024"
  [dependencies]
++proc-macro2 = "1"
  syn = { version = "3.0.5", features = ["full", "extra-traits", "visit"] }
  # Used only by the test corpus: a case carrying `//@ extern: bitflags` is

modified DESIGN.md +63 -0 Viewed

@@ -609,6 +609,69 @@ do not know is rejected rather than declared without the `const`.
609	609	`tests/cases/036-extern-c.rs` calls `abs`, `labs`, `strlen` and `atoi` through
610	610	this path, byte-identical to rustc.
611	611
	612	+## `macro_rules!`: expanded, not translated
	613	+
	614	+Nim has `template` and `macro`, so a shape-level correspondence with
	615	+`macro_rules!` exists. Across 1,833 definitions in a 400-crate sample:
	616	+
	617	+\| shape \| share \| Nim equivalent \|
	618	+\|---\|---\|---\|
	619	+\| single rule, no repetition \| 36% \| a `template` \|
	620	+\| multiple rules \| 12% \| a `macro` dispatching on shape \|
	621	+\| `$(..)` repetition \| 28% \| `varargs` in a `macro` \|
	622	+\| `:tt` token-tree munching \| 22% \| an interpreter; not mechanical \|
	623	+
	624	+So ~76% has a translatable shape. **We expand instead, and the reason is the
	625	+type-directed lowering.** A Nim template body is untyped: substituted first,
	626	+type-checked after. This lowering needs a type at nearly every step — to
	627	+choose `div` over `/`, to size a `cast`, to pick an integer literal's width.
	628	+Translating a macro body would mean lowering Rust with no type information,
	629	+which is exactly the guessing the project refuses. Expanding at the call site
	630	+yields ordinary Rust in a context where the types are known, so it lowers like
	631	+anything else. Same applicability, faithful output.
	632	+
	633	+`src/mrules.rs` implements the 36% case: one rule, no repetition, no `:tt`. A
	634	+definition it cannot handle is recorded with its reason, so a call site says
	635	+"`foo!` cannot be expanded: `$(..)` repetition is not implemented yet" rather
	636	+than "unknown macro". Captured fragments are parenthesised on substitution, so
	637	+`square!(2 + 3)` is 25 and not 11.
	638	+
	639	+**This moved the survey more than everything before it combined: 3 → 18 of
	640	+400 crates accepted.** A `macro_rules!` used to be a hard stop at item level,
	641	+failing a whole crate on sight.
	642	+
	643	+### A stricter number
	644	+
	645	+"rustnim exits 0" is not "the output is real". Of those 18, **12 produce Nim
	646	+that the Nim compiler accepts**:
	647	+
	648	+```
	649	+adler2 arrayref cfg_aliases ×3 cfg-if ×2 ctor-lite darling ×4
	650	+```
	651	+
	652	+Compiling is still not behaving: only the cases in `tests/cases/` are checked
	653	+against rustc for identical output. Three numbers, in increasing strength —
	654	+accepted 18, compiles 12, behaviourally verified only the corpus.
	655	+
	656	+## `serde`: assessed, not attempted
	657	+
	658	+`serde` is 49 dependents in `libcosmic`'s tree and the most generic crate
	659	+looked at here: 17,237 lines, 369 `impl<`, 922 `where` clauses, 849 uses of
	660	+`'de`, 324 associated types. `serde_derive` is another 8,975 lines of *proc
	661	+macro* — a program that runs at compile time, so it can be expanded (as
	662	+`bitflags!` was, with `RUSTC_BOOTSTRAP=1`) but never translated.
	663	+
	664	+The derive's expansion is small and clean — thirteen lines for a two-field
	665	+struct. But it is generic over a `Serializer`, so unlike `bitflags!`
	666	+(self-contained) and `log` (a facade with a defined no-op default), it has no
	667	+observable behaviour at all until a format crate supplies one. A shim would
	668	+therefore have to pick a format and implement that, which is a narrower and
	669	+much larger commitment than either previous shim.
	670	+
	671	+`serde_json`'s exact output was pinned for whenever that is attempted:
	672	+declaration order, no whitespace, `null` for `None`, and a float keeps the
	673	+`.0` that `Display` drops — `{"x":-3,"ratio":2.0,"maybe":null}`.
	674	+
612	675	## Proof of byte-identity for `base16ct`
613	676
614	677	[`PROOF.md`](PROOF.md) sets out what is actually established: exhaustive

  `tests/cases/036-extern-c.rs` calls `abs`, `labs`, `strlen` and `atoi` through
  this path, byte-identical to rustc.
++## `macro_rules!`: expanded, not translated
++
++Nim has `template` and `macro`, so a shape-level correspondence with
++`macro_rules!` exists. Across 1,833 definitions in a 400-crate sample:
++
++| shape | share | Nim equivalent |
++|---|---|---|
++| single rule, no repetition | 36% | a `template` |
++| multiple rules | 12% | a `macro` dispatching on shape |
++| `$(..)` repetition | 28% | `varargs` in a `macro` |
++| `:tt` token-tree munching | 22% | an interpreter; not mechanical |
++
++So ~76% has a translatable shape. **We expand instead, and the reason is the
++type-directed lowering.** A Nim template body is *untyped*: substituted first,
++type-checked after. This lowering needs a type at nearly every step — to
++choose `div` over `/`, to size a `cast`, to pick an integer literal's width.
++Translating a macro body would mean lowering Rust with no type information,
++which is exactly the guessing the project refuses. Expanding at the call site
++yields ordinary Rust in a context where the types are known, so it lowers like
++anything else. Same applicability, faithful output.
++
++`src/mrules.rs` implements the 36% case: one rule, no repetition, no `:tt`. A
++definition it cannot handle is recorded *with its reason*, so a call site says
++"`foo!` cannot be expanded: `$(..)` repetition is not implemented yet" rather
++than "unknown macro". Captured fragments are parenthesised on substitution, so
++`square!(2 + 3)` is 25 and not 11.
++
++**This moved the survey more than everything before it combined: 3 → 18 of
++400 crates accepted.** A `macro_rules!` used to be a hard stop at item level,
++failing a whole crate on sight.
++
++### A stricter number
++
++"rustnim exits 0" is not "the output is real". Of those 18, **12 produce Nim
++that the Nim compiler accepts**:
++
++```
++adler2  arrayref  cfg_aliases ×3  cfg-if ×2  ctor-lite  darling ×4
++```
++
++Compiling is still not behaving: only the cases in `tests/cases/` are checked
++against rustc for identical output. Three numbers, in increasing strength —
++accepted 18, compiles 12, behaviourally verified only the corpus.
++
++## `serde`: assessed, not attempted
++
++`serde` is 49 dependents in `libcosmic`'s tree and the most generic crate
++looked at here: 17,237 lines, 369 `impl<`, 922 `where` clauses, 849 uses of
++`'de`, 324 associated types. `serde_derive` is another 8,975 lines of *proc
++macro* — a program that runs at compile time, so it can be expanded (as
++`bitflags!` was, with `RUSTC_BOOTSTRAP=1`) but never translated.
++
++The derive's expansion is small and clean — thirteen lines for a two-field
++struct. But it is **generic over a `Serializer`**, so unlike `bitflags!`
++(self-contained) and `log` (a facade with a defined no-op default), it has no
++observable behaviour at all until a format crate supplies one. A shim would
++therefore have to pick a format and implement *that*, which is a narrower and
++much larger commitment than either previous shim.
++
++`serde_json`'s exact output was pinned for whenever that is attempted:
++declaration order, no whitespace, `null` for `None`, and a float keeps the
++`.0` that `Display` drops — `{"x":-3,"ratio":2.0,"maybe":null}`.
++
  ## Proof of byte-identity for `base16ct`
  [`PROOF.md`](PROOF.md) sets out what is actually established: exhaustive

modified README.md +13 -0 Viewed

@@ -103,6 +103,19 @@ That is the answer to `libc`, which is not transpiled: it is 129,594 lines
103	103	of which 54,544 are constants and 7,660 are declarations, against 121 function
104	104	bodies in the whole crate. Nim reaches those symbols natively.
105	105
	106	+## Macros
	107	+
	108	+A `macro_rules!` is expanded at the call site, not translated into a Nim
	109	+`template`. The shape-level correspondence is real — 36% of macros in a
	110	+400-crate sample are a single rule with no repetition, which is exactly a
	111	+template — but a template body is untyped, and this lowering is type-directed
	112	+throughout. Expanding gives ordinary Rust where the types are known.
	113	+
	114	+That change alone took the survey from 3 to 18 of 400 crates accepted, of
	115	+which 12 produce Nim the Nim compiler accepts. (Accepted, compiles, and
	116	+behaviourally verified are three different bars; only `tests/cases/` clears
	117	+the third.)
	118	+
106	119	## Does it generalise?
107	120
108	121	`base16ct` is the crate this was built toward, so a second one was tried.

  of which 54,544 are constants and 7,660 are declarations, against 121 function
  bodies in the whole crate. Nim reaches those symbols natively.
++## Macros
++
++A `macro_rules!` is **expanded at the call site**, not translated into a Nim
++`template`. The shape-level correspondence is real — 36% of macros in a
++400-crate sample are a single rule with no repetition, which is exactly a
++template — but a template body is untyped, and this lowering is type-directed
++throughout. Expanding gives ordinary Rust where the types are known.
++
++That change alone took the survey from 3 to 18 of 400 crates accepted, of
++which 12 produce Nim the Nim compiler accepts. (Accepted, compiles, and
++behaviourally verified are three different bars; only `tests/cases/` clears
++the third.)
++
  ## Does it generalise?
  `base16ct` is the crate this was built toward, so a second one was tried.

modified src/lower.rs +50 -4 Viewed

@@ -220,6 +220,10 @@ pub struct Lowerer {
220	220	foreign: std::collections::HashSet<String>,
221	221	/// Const-qualified C pointer aliases already declared.
222	222	const_ptrs: std::collections::HashSet<String>,
	223	+ /// `macro_rules!` definitions we can expand, and the reasons for those we
	224	+ /// cannot, so a call site can say why rather than "unknown macro".
	225	+ mrules: HashMap<String, crate::mrules::MacroDef>,
	226	+ mrules_bad: HashMap<String, String>,
223	227	/// Types declared by a `bitflags!` invocation.
224	228	bitflags: std::collections::HashSet<String>,
225	229	/// `(type, flag) -> nim const name`.
@@ -301,6 +305,8 @@ impl Lowerer {
301	305	assoc_consts: HashMap::new(),
302	306	foreign: std::collections::HashSet::new(),
303	307	const_ptrs: std::collections::HashSet::new(),
	308	+ mrules: HashMap::new(),
	309	+ mrules_bad: HashMap::new(),
304	310	bitflags: std::collections::HashSet::new(),
305	311	flag_consts: HashMap::new(),
306	312	use_map: HashMap::new(),
@@ -572,6 +578,25 @@ impl Lowerer {
572	578	Item::Macro(m) if path_name(&m.mac.path) == "bitflags" => {
573	579	self.collect_bitflags(&m.mac)?;
574	580	}
	581	+ // A `macro_rules!` definition is recorded, not emitted: it is
	582	+ // expanded at each call site, where the types its body needs are
	583	+ // known. See `src/mrules.rs` for why that beats translating it
	584	+ // into a Nim template.
	585	+ Item::Macro(m) if path_name(&m.mac.path) == "macro_rules" => {
	586	+ if let Some(name) = &m.ident {
	587	+ match crate::mrules::parse(m.mac.tokens.clone()) {
	588	+ Ok(def) => {
	589	+ self.mrules.insert(name.to_string(), def);
	590	+ }
	591	+ Err(why) => {
	592	+ // Recorded as unusable rather than silently absent:
	593	+ // a call site gets this reason instead of "unknown
	594	+ // macro".
	595	+ self.mrules_bad.insert(name.to_string(), why);
	596	+ }
	597	+ }
	598	+ }
	599	+ }
575	600	Item::ForeignMod(f) => {
576	601	for it in &f.items {
577	602	if let syn::ForeignItem::Fn(fi) = it {
@@ -1288,6 +1313,7 @@ impl Lowerer {
1288	1313	Ok(())
1289	1314	}
1290	1315	Item::Macro(m) if path_name(&m.mac.path) == "bitflags" => self.emit_bitflags(&m.mac),
	1316	+ Item::Macro(m) if path_name(&m.mac.path) == "macro_rules" => Ok(()),
1291	1317	Item::Type(_) => Ok(()), // expanded at every use site
1292	1318	Item::Trait(t) => {
1293	1319	// We do not model trait resolution, so a declaration generates
@@ -5010,10 +5036,30 @@ impl Lowerer {
5010	5036	}
5011	5037	Ok(format!("@[{}]", parts.join(", ")))
5012	5038	}
5013		- other => Err(format!(
5014		- "unsupported macro `{other}!`; a macro whose expansion is not \
5015		- known cannot be lowered faithfully"
5016		- )),
	5039	+ other => {
	5040	+ if let Some(why) = self.mrules_bad.get(other) {
	5041	+ return Err(format!("`{other}!` cannot be expanded: {why}"));
	5042	+ }
	5043	+ if let Some(def) = self.mrules.get(other).cloned() {
	5044	+ let expanded = def
	5045	+ .expand(mac.tokens.clone())
	5046	+ .map_err(\|e\| format!("expanding `{other}!`: {e}"))?;
	5047	+ // The expansion is ordinary Rust, lowered in a context
	5048	+ // where its types are known.
	5049	+ let e: Expr = syn::parse2(expanded.clone()).map_err(\|_\| {
	5050	+ format!(
	5051	+ "`{other}!` expands to something that is not an \
	5052	+ expression: `{}`",
	5053	+ expanded
	5054	+ )
	5055	+ })?;
	5056	+ return Ok(self.expr(&e)?.code);
	5057	+ }
	5058	+ Err(format!(
	5059	+ "unsupported macro `{other}!`; a macro whose expansion is not \
	5060	+ known cannot be lowered faithfully"
	5061	+ ))
	5062	+ }
5017	5063	}
5018	5064	}
5019	5065

      foreign: std::collections::HashSet<String>,
      /// Const-qualified C pointer aliases already declared.
      const_ptrs: std::collections::HashSet<String>,
++    /// `macro_rules!` definitions we can expand, and the reasons for those we
++    /// cannot, so a call site can say *why* rather than "unknown macro".
++    mrules: HashMap<String, crate::mrules::MacroDef>,
++    mrules_bad: HashMap<String, String>,
      /// Types declared by a `bitflags!` invocation.
      bitflags: std::collections::HashSet<String>,
      /// `(type, flag) -> nim const name`.
              assoc_consts: HashMap::new(),
              foreign: std::collections::HashSet::new(),
              const_ptrs: std::collections::HashSet::new(),
++            mrules: HashMap::new(),
++            mrules_bad: HashMap::new(),
              bitflags: std::collections::HashSet::new(),
              flag_consts: HashMap::new(),
              use_map: HashMap::new(),
              Item::Macro(m) if path_name(&m.mac.path) == "bitflags" => {
                  self.collect_bitflags(&m.mac)?;
              }
++            // A `macro_rules!` definition is recorded, not emitted: it is
++            // expanded at each call site, where the types its body needs are
++            // known. See `src/mrules.rs` for why that beats translating it
++            // into a Nim template.
++            Item::Macro(m) if path_name(&m.mac.path) == "macro_rules" => {
++                if let Some(name) = &m.ident {
++                    match crate::mrules::parse(m.mac.tokens.clone()) {
++                        Ok(def) => {
++                            self.mrules.insert(name.to_string(), def);
++                        }
++                        Err(why) => {
++                            // Recorded as unusable rather than silently absent:
++                            // a call site gets this reason instead of "unknown
++                            // macro".
++                            self.mrules_bad.insert(name.to_string(), why);
++                        }
++                    }
++                }
++            }
              Item::ForeignMod(f) => {
                  for it in &f.items {
                      if let syn::ForeignItem::Fn(fi) = it {
                  Ok(())
              }
              Item::Macro(m) if path_name(&m.mac.path) == "bitflags" => self.emit_bitflags(&m.mac),
++            Item::Macro(m) if path_name(&m.mac.path) == "macro_rules" => Ok(()),
              Item::Type(_) => Ok(()), // expanded at every use site
              Item::Trait(t) => {
                  // We do not model trait resolution, so a declaration generates
                  }
                  Ok(format!("@[{}]", parts.join(", ")))
              }
--            other => Err(format!(
++            other => {
--                "unsupported macro `{other}!`; a macro whose expansion is not \
++                if let Some(why) = self.mrules_bad.get(other) {
--                 known cannot be lowered faithfully"
++                    return Err(format!("`{other}!` cannot be expanded: {why}"));
--            )),
++                }
++                if let Some(def) = self.mrules.get(other).cloned() {
++                    let expanded = def
++                        .expand(mac.tokens.clone())
++                        .map_err(|e| format!("expanding `{other}!`: {e}"))?;
++                    // The expansion is ordinary Rust, lowered in a context
++                    // where its types are known.
++                    let e: Expr = syn::parse2(expanded.clone()).map_err(|_| {
++                        format!(
++                            "`{other}!` expands to something that is not an \
++                             expression: `{}`",
++                            expanded
++                        )
++                    })?;
++                    return Ok(self.expr(&e)?.code);
++                }
++                Err(format!(
++                    "unsupported macro `{other}!`; a macro whose expansion is not \
++                     known cannot be lowered faithfully"
++                ))
++            }
          }
      }

modified src/main.rs +1 -0 Viewed

@@ -4,6 +4,7 @@
4	4
5	5	mod fmt;
6	6	mod macros;
	7	+mod mrules;
7	8	mod lower;
8	9	mod ty;
9	10

  mod fmt;
  mod macros;
++mod mrules;
  mod lower;
  mod ty;

added src/mrules.rs +224 -0 Viewed

new file mode 100644
@@ -0,0 +1,224 @@
	1	+//! A `macro_rules!` expander.
	2	+//!
	3	+//! The alternative was translating each `macro_rules!` into a Nim `template`,
	4	+//! which is the shape-level correspondence: 36% of the 1,833 definitions in a
	5	+//! 400-crate sample are a single rule with no repetition, which is exactly a
	6	+//! template, and another 40% map to a Nim `macro` over `varargs`.
	7	+//!
	8	+//! That correspondence is real but it does not survive contact with this
	9	+//! project's approach. A Nim template body is untyped: it is substituted and
	10	+//! only then type-checked. Our lowering is type-directed throughout — it needs
	11	+//! a type to choose `div` over `/`, to size a `cast`, to pick an integer
	12	+//! literal's width. Translating a macro body would mean lowering Rust with no
	13	+//! type information, which is precisely the guessing the project refuses.
	14	+//!
	15	+//! Expanding at the call site does not have that problem: the expansion is
	16	+//! ordinary Rust in a context where types are known, so it lowers like any
	17	+//! other code. Same applicability, faithful output.
	18	+//!
	19	+//! Implemented here: a single rule, no repetition, no `:tt`. That is the 36%.
	20	+
	21	+use proc_macro2::{Delimiter, Group, Ident, TokenStream, TokenTree};
	22	+use std::collections::HashMap;
	23	+
	24	+/// One element of a matcher: a captured fragment, or a token to match exactly.
	25	+#[derive(Debug, Clone)]
	26	+enum Pat {
	27	+ Frag(String),
	28	+ Tok(String),
	29	+ /// A delimited group, matched recursively.
	30	+ Group(Delimiter, Vec<Pat>),
	31	+}
	32	+
	33	+#[derive(Debug, Clone)]
	34	+pub struct MacroDef {
	35	+ pattern: Vec<Pat>,
	36	+ body: TokenStream,
	37	+}
	38	+
	39	+/// Parse `macro_rules!`'s body: `( $matcher ) => { $transcriber };`.
	40	+pub fn parse(tokens: TokenStream) -> Result<MacroDef, String> {
	41	+ let t: Vec<TokenTree> = tokens.into_iter().collect();
	42	+ // matcher group, `=`, `>`, transcriber group, optional `;`
	43	+ let (matcher, rest) = match t.split_first() {
	44	+ Some((TokenTree::Group(g), rest)) => (g.clone(), rest),
	45	+ _ => return Err("expected a matcher group".into()),
	46	+ };
	47	+ let arrow: String = rest.iter().take(2).map(\|t\| t.to_string()).collect();
	48	+ if arrow != "=>" {
	49	+ return Err("expected `=>`".into());
	50	+ }
	51	+ let body = match rest.get(2) {
	52	+ Some(TokenTree::Group(g)) => g.stream(),
	53	+ _ => return Err("expected a transcriber group".into()),
	54	+ };
	55	+ if rest.len() > 4 \|\| (rest.len() == 4 && rest[3].to_string() != ";") {
	56	+ return Err("more than one rule is not implemented yet".into());
	57	+ }
	58	+ Ok(MacroDef { pattern: parse_pattern(matcher.stream())?, body })
	59	+}
	60	+
	61	+fn parse_pattern(ts: TokenStream) -> Result<Vec<Pat>, String> {
	62	+ let mut out = Vec::new();
	63	+ let mut it = ts.into_iter().peekable();
	64	+ while let Some(t) = it.next() {
	65	+ match t {
	66	+ TokenTree::Punct(p) if p.as_char() == '$' => {
	67	+ match it.next() {
	68	+ Some(TokenTree::Ident(name)) => {
	69	+ // `$name:fragment`
	70	+ match it.next() {
	71	+ Some(TokenTree::Punct(c)) if c.as_char() == ':' => {}
	72	+ _ => return Err("expected `:` after a fragment name".into()),
	73	+ }
	74	+ let kind = match it.next() {
	75	+ Some(TokenTree::Ident(k)) => k.to_string(),
	76	+ _ => return Err("expected a fragment specifier".into()),
	77	+ };
	78	+ if kind == "tt" {
	79	+ return Err(
	80	+ "`:tt` makes a macro a token-tree interpreter, which \
	81	+ has no mechanical translation"
	82	+ .into(),
	83	+ );
	84	+ }
	85	+ out.push(Pat::Frag(name.to_string()));
	86	+ }
	87	+ Some(TokenTree::Group(_)) => {
	88	+ return Err(
	89	+ "`$(..)` repetition is not implemented yet; it maps to \
	90	+ `varargs` in a Nim macro"
	91	+ .into(),
	92	+ )
	93	+ }
	94	+ _ => return Err("unexpected token after `$`".into()),
	95	+ }
	96	+ }
	97	+ TokenTree::Group(g) => {
	98	+ out.push(Pat::Group(g.delimiter(), parse_pattern(g.stream())?))
	99	+ }
	100	+ other => out.push(Pat::Tok(other.to_string())),
	101	+ }
	102	+ }
	103	+ Ok(out)
	104	+}
	105	+
	106	+impl MacroDef {
	107	+ /// Match an invocation's tokens and substitute them into the body.
	108	+ pub fn expand(&self, input: TokenStream) -> Result<TokenStream, String> {
	109	+ let mut binds = HashMap::new();
	110	+ let toks: Vec<TokenTree> = input.into_iter().collect();
	111	+ let used = match_seq(&self.pattern, &toks, &mut binds)?;
	112	+ if used != toks.len() {
	113	+ return Err("the invocation has tokens the matcher does not consume".into());
	114	+ }
	115	+ Ok(substitute(self.body.clone(), &binds))
	116	+ }
	117	+}
	118	+
	119	+/// Match `pats` against the front of `toks`, returning how many were consumed.
	120	+fn match_seq(
	121	+ pats: &[Pat],
	122	+ toks: &[TokenTree],
	123	+ binds: &mut HashMap<String, TokenStream>,
	124	+) -> Result<usize, String> {
	125	+ let mut i = 0;
	126	+ for (k, p) in pats.iter().enumerate() {
	127	+ match p {
	128	+ Pat::Tok(s) => {
	129	+ let t = toks.get(i).ok_or("the invocation ends before the matcher does")?;
	130	+ if t.to_string() != *s {
	131	+ return Err(format!("expected `{s}`, found `{t}`"));
	132	+ }
	133	+ i += 1;
	134	+ }
	135	+ Pat::Group(d, inner) => {
	136	+ let Some(TokenTree::Group(g)) = toks.get(i) else {
	137	+ return Err("expected a delimited group".into());
	138	+ };
	139	+ if g.delimiter() != *d {
	140	+ return Err("mismatched delimiter".into());
	141	+ }
	142	+ let sub: Vec<TokenTree> = g.stream().into_iter().collect();
	143	+ let n = match_seq(inner, &sub, binds)?;
	144	+ if n != sub.len() {
	145	+ return Err("group has tokens the matcher does not consume".into());
	146	+ }
	147	+ i += 1;
	148	+ }
	149	+ Pat::Frag(name) => {
	150	+ // A fragment runs to the next literal token in the matcher, or
	151	+ // to the end. That is what makes `$a:expr, $b:expr` split on
	152	+ // the comma rather than swallowing it.
	153	+ let stop = pats[k + 1..].iter().find_map(\|p\| match p {
	154	+ Pat::Tok(s) => Some(s.clone()),
	155	+ _ => None,
	156	+ });
	157	+ let start = i;
	158	+ let mut depth = 0i32;
	159	+ while i < toks.len() {
	160	+ let s = toks[i].to_string();
	161	+ if let Some(stop) = &stop {
	162	+ if depth == 0 && s == *stop {
	163	+ break;
	164	+ }
	165	+ }
	166	+ match &toks[i] {
	167	+ TokenTree::Punct(p) if p.as_char() == '<' => depth += 1,
	168	+ TokenTree::Punct(p) if p.as_char() == '>' => depth -= 1,
	169	+ _ => {}
	170	+ }
	171	+ i += 1;
	172	+ }
	173	+ if i == start {
	174	+ return Err(format!("nothing matched fragment `${name}`"));
	175	+ }
	176	+ binds.insert(
	177	+ name.clone(),
	178	+ toks[start..i].iter().cloned().collect::<TokenStream>(),
	179	+ );
	180	+ }
	181	+ }
	182	+ }
	183	+ Ok(i)
	184	+}
	185	+
	186	+/// Replace every `$name` in the body with what it captured.
	187	+fn substitute(ts: TokenStream, binds: &HashMap<String, TokenStream>) -> TokenStream {
	188	+ let mut out = Vec::new();
	189	+ let mut it = ts.into_iter().peekable();
	190	+ while let Some(t) = it.next() {
	191	+ match t {
	192	+ TokenTree::Punct(p) if p.as_char() == '$' => {
	193	+ match it.peek() {
	194	+ Some(TokenTree::Ident(name)) => {
	195	+ let n = name.to_string();
	196	+ it.next();
	197	+ match binds.get(&n) {
	198	+ // Parenthesised so that a captured expression keeps
	199	+ // its own precedence, as Rust's `expr` fragments do.
	200	+ Some(v) => out.push(TokenTree::Group(Group::new(
	201	+ Delimiter::Parenthesis,
	202	+ v.clone(),
	203	+ ))),
	204	+ None => {
	205	+ out.push(TokenTree::Punct(p));
	206	+ out.push(TokenTree::Ident(Ident::new(
	207	+ &n,
	208	+ proc_macro2::Span::call_site(),
	209	+ )));
	210	+ }
	211	+ }
	212	+ }
	213	+ _ => out.push(TokenTree::Punct(p)),
	214	+ }
	215	+ }
	216	+ TokenTree::Group(g) => {
	217	+ let inner = substitute(g.stream(), binds);
	218	+ out.push(TokenTree::Group(Group::new(g.delimiter(), inner)));
	219	+ }
	220	+ other => out.push(other),
	221	+ }
	222	+ }
	223	+ out.into_iter().collect()
	224	+}

++//! A `macro_rules!` expander.
++//!
++//! The alternative was translating each `macro_rules!` into a Nim `template`,
++//! which is the shape-level correspondence: 36% of the 1,833 definitions in a
++//! 400-crate sample are a single rule with no repetition, which is exactly a
++//! template, and another 40% map to a Nim `macro` over `varargs`.
++//!
++//! That correspondence is real but it does not survive contact with this
++//! project's approach. A Nim template body is *untyped*: it is substituted and
++//! only then type-checked. Our lowering is type-directed throughout — it needs
++//! a type to choose `div` over `/`, to size a `cast`, to pick an integer
++//! literal's width. Translating a macro body would mean lowering Rust with no
++//! type information, which is precisely the guessing the project refuses.
++//!
++//! Expanding at the call site does not have that problem: the expansion is
++//! ordinary Rust in a context where types are known, so it lowers like any
++//! other code. Same applicability, faithful output.
++//!
++//! Implemented here: a single rule, no repetition, no `:tt`. That is the 36%.
++
++use proc_macro2::{Delimiter, Group, Ident, TokenStream, TokenTree};
++use std::collections::HashMap;
++
++/// One element of a matcher: a captured fragment, or a token to match exactly.
++#[derive(Debug, Clone)]
++enum Pat {
++    Frag(String),
++    Tok(String),
++    /// A delimited group, matched recursively.
++    Group(Delimiter, Vec<Pat>),
++}
++
++#[derive(Debug, Clone)]
++pub struct MacroDef {
++    pattern: Vec<Pat>,
++    body: TokenStream,
++}
++
++/// Parse `macro_rules!`'s body: `( $matcher ) => { $transcriber };`.
++pub fn parse(tokens: TokenStream) -> Result<MacroDef, String> {
++    let t: Vec<TokenTree> = tokens.into_iter().collect();
++    // matcher group, `=`, `>`, transcriber group, optional `;`
++    let (matcher, rest) = match t.split_first() {
++        Some((TokenTree::Group(g), rest)) => (g.clone(), rest),
++        _ => return Err("expected a matcher group".into()),
++    };
++    let arrow: String = rest.iter().take(2).map(|t| t.to_string()).collect();
++    if arrow != "=>" {
++        return Err("expected `=>`".into());
++    }
++    let body = match rest.get(2) {
++        Some(TokenTree::Group(g)) => g.stream(),
++        _ => return Err("expected a transcriber group".into()),
++    };
++    if rest.len() > 4 || (rest.len() == 4 && rest[3].to_string() != ";") {
++        return Err("more than one rule is not implemented yet".into());
++    }
++    Ok(MacroDef { pattern: parse_pattern(matcher.stream())?, body })
++}
++
++fn parse_pattern(ts: TokenStream) -> Result<Vec<Pat>, String> {
++    let mut out = Vec::new();
++    let mut it = ts.into_iter().peekable();
++    while let Some(t) = it.next() {
++        match t {
++            TokenTree::Punct(p) if p.as_char() == '$' => {
++                match it.next() {
++                    Some(TokenTree::Ident(name)) => {
++                        // `$name:fragment`
++                        match it.next() {
++                            Some(TokenTree::Punct(c)) if c.as_char() == ':' => {}
++                            _ => return Err("expected `:` after a fragment name".into()),
++                        }
++                        let kind = match it.next() {
++                            Some(TokenTree::Ident(k)) => k.to_string(),
++                            _ => return Err("expected a fragment specifier".into()),
++                        };
++                        if kind == "tt" {
++                            return Err(
++                                "`:tt` makes a macro a token-tree interpreter, which \
++                                 has no mechanical translation"
++                                    .into(),
++                            );
++                        }
++                        out.push(Pat::Frag(name.to_string()));
++                    }
++                    Some(TokenTree::Group(_)) => {
++                        return Err(
++                            "`$(..)` repetition is not implemented yet; it maps to \
++                             `varargs` in a Nim macro"
++                                .into(),
++                        )
++                    }
++                    _ => return Err("unexpected token after `$`".into()),
++                }
++            }
++            TokenTree::Group(g) => {
++                out.push(Pat::Group(g.delimiter(), parse_pattern(g.stream())?))
++            }
++            other => out.push(Pat::Tok(other.to_string())),
++        }
++    }
++    Ok(out)
++}
++
++impl MacroDef {
++    /// Match an invocation's tokens and substitute them into the body.
++    pub fn expand(&self, input: TokenStream) -> Result<TokenStream, String> {
++        let mut binds = HashMap::new();
++        let toks: Vec<TokenTree> = input.into_iter().collect();
++        let used = match_seq(&self.pattern, &toks, &mut binds)?;
++        if used != toks.len() {
++            return Err("the invocation has tokens the matcher does not consume".into());
++        }
++        Ok(substitute(self.body.clone(), &binds))
++    }
++}
++
++/// Match `pats` against the front of `toks`, returning how many were consumed.
++fn match_seq(
++    pats: &[Pat],
++    toks: &[TokenTree],
++    binds: &mut HashMap<String, TokenStream>,
++) -> Result<usize, String> {
++    let mut i = 0;
++    for (k, p) in pats.iter().enumerate() {
++        match p {
++            Pat::Tok(s) => {
++                let t = toks.get(i).ok_or("the invocation ends before the matcher does")?;
++                if t.to_string() != *s {
++                    return Err(format!("expected `{s}`, found `{t}`"));
++                }
++                i += 1;
++            }
++            Pat::Group(d, inner) => {
++                let Some(TokenTree::Group(g)) = toks.get(i) else {
++                    return Err("expected a delimited group".into());
++                };
++                if g.delimiter() != *d {
++                    return Err("mismatched delimiter".into());
++                }
++                let sub: Vec<TokenTree> = g.stream().into_iter().collect();
++                let n = match_seq(inner, &sub, binds)?;
++                if n != sub.len() {
++                    return Err("group has tokens the matcher does not consume".into());
++                }
++                i += 1;
++            }
++            Pat::Frag(name) => {
++                // A fragment runs to the next literal token in the matcher, or
++                // to the end. That is what makes `$a:expr, $b:expr` split on
++                // the comma rather than swallowing it.
++                let stop = pats[k + 1..].iter().find_map(|p| match p {
++                    Pat::Tok(s) => Some(s.clone()),
++                    _ => None,
++                });
++                let start = i;
++                let mut depth = 0i32;
++                while i < toks.len() {
++                    let s = toks[i].to_string();
++                    if let Some(stop) = &stop {
++                        if depth == 0 && s == *stop {
++                            break;
++                        }
++                    }
++                    match &toks[i] {
++                        TokenTree::Punct(p) if p.as_char() == '<' => depth += 1,
++                        TokenTree::Punct(p) if p.as_char() == '>' => depth -= 1,
++                        _ => {}
++                    }
++                    i += 1;
++                }
++                if i == start {
++                    return Err(format!("nothing matched fragment `${name}`"));
++                }
++                binds.insert(
++                    name.clone(),
++                    toks[start..i].iter().cloned().collect::<TokenStream>(),
++                );
++            }
++        }
++    }
++    Ok(i)
++}
++
++/// Replace every `$name` in the body with what it captured.
++fn substitute(ts: TokenStream, binds: &HashMap<String, TokenStream>) -> TokenStream {
++    let mut out = Vec::new();
++    let mut it = ts.into_iter().peekable();
++    while let Some(t) = it.next() {
++        match t {
++            TokenTree::Punct(p) if p.as_char() == '$' => {
++                match it.peek() {
++                    Some(TokenTree::Ident(name)) => {
++                        let n = name.to_string();
++                        it.next();
++                        match binds.get(&n) {
++                            // Parenthesised so that a captured expression keeps
++                            // its own precedence, as Rust's `expr` fragments do.
++                            Some(v) => out.push(TokenTree::Group(Group::new(
++                                Delimiter::Parenthesis,
++                                v.clone(),
++                            ))),
++                            None => {
++                                out.push(TokenTree::Punct(p));
++                                out.push(TokenTree::Ident(Ident::new(
++                                    &n,
++                                    proc_macro2::Span::call_site(),
++                                )));
++                            }
++                        }
++                    }
++                    _ => out.push(TokenTree::Punct(p)),
++                }
++            }
++            TokenTree::Group(g) => {
++                let inner = substitute(g.stream(), binds);
++                out.push(TokenTree::Group(Group::new(g.delimiter(), inner)));
++            }
++            other => out.push(other),
++        }
++    }
++    out.into_iter().collect()
++}

added tests/cases/037-macro-rules.rs +40 -0 Viewed

new file mode 100644
@@ -0,0 +1,40 @@
	1	+// A `macro_rules!` is expanded at the call site, not translated into a Nim
	2	+// template. The shape-level correspondence is real -- a single-rule macro is
	3	+// a Nim template -- but a template body is untyped, and this lowering is
	4	+// type-directed throughout: it needs a type to choose `div` over `/`, to size
	5	+// a `cast`, to pick a literal's width. Expanding gives ordinary Rust in a
	6	+// context where those types are known.
	7	+
	8	+macro_rules! square {
	9	+ ($x:expr) => {
	10	+ $x * $x
	11	+ };
	12	+}
	13	+
	14	+macro_rules! clamp_to {
	15	+ ($v:expr, $lo:expr, $hi:expr) => {
	16	+ if $v < $lo { $lo } else if $v > $hi { $hi } else { $v }
	17	+ };
	18	+}
	19	+
	20	+macro_rules! first_of {
	21	+ ($a:expr, $b:expr) => {
	22	+ if $a != 0 { $a } else { $b }
	23	+ };
	24	+}
	25	+
	26	+fn main() {
	27	+ // Precedence is preserved: `square!(2 + 3)` is 25, not 11.
	28	+ println!("{} {}", square!(4), square!(2 + 3));
	29	+
	30	+ let a: i32 = 5;
	31	+ println!("{}", square!(a));
	32	+
	33	+ println!("{} {} {}", clamp_to!(15, 0, 10), clamp_to!(-4, 0, 10), clamp_to!(7, 0, 10));
	34	+ println!("{} {}", first_of!(0, 9), first_of!(3, 9));
	35	+
	36	+ // Nested, and with the macro's own type context.
	37	+ let w: u8 = 200;
	38	+ println!("{}", clamp_to!(w, 0u8, 100u8));
	39	+ println!("{}", square!(3i64) + square!(4i64));
	40	+}

++// A `macro_rules!` is expanded at the call site, not translated into a Nim
++// template. The shape-level correspondence is real -- a single-rule macro *is*
++// a Nim template -- but a template body is untyped, and this lowering is
++// type-directed throughout: it needs a type to choose `div` over `/`, to size
++// a `cast`, to pick a literal's width. Expanding gives ordinary Rust in a
++// context where those types are known.
++
++macro_rules! square {
++    ($x:expr) => {
++        $x * $x
++    };
++}
++
++macro_rules! clamp_to {
++    ($v:expr, $lo:expr, $hi:expr) => {
++        if $v < $lo { $lo } else if $v > $hi { $hi } else { $v }
++    };
++}
++
++macro_rules! first_of {
++    ($a:expr, $b:expr) => {
++        if $a != 0 { $a } else { $b }
++    };
++}
++
++fn main() {
++    // Precedence is preserved: `square!(2 + 3)` is 25, not 11.
++    println!("{} {}", square!(4), square!(2 + 3));
++
++    let a: i32 = 5;
++    println!("{}", square!(a));
++
++    println!("{} {} {}", clamp_to!(15, 0, 10), clamp_to!(-4, 0, 10), clamp_to!(7, 0, 10));
++    println!("{} {}", first_of!(0, 9), first_of!(3, 9));
++
++    // Nested, and with the macro's own type context.
++    let w: u8 = 200;
++    println!("{}", clamp_to!(w, 0u8, 100u8));
++    println!("{}", square!(3i64) + square!(4i64));
++}