rickub
Sign in Get started
nandi/rustnimpublic⑂ Fork 0
⑂ 3d4a7d2
Branches
main View all branches → View all tags →
Commits
⬇ Clone ▾
git clone https://git.rickub.com/nandi/rustnim.git
git clone ssh://git@rickub.com/nandi/rustnim.git

Host key fingerprint (ed25519): SHA256:iycHnxEyq0Q7uyVpB7JlznP0G7JrTPXLYRcAU5CSLhc — verify it before your first connect.

Expand `$(..)` repetition in macro_rules

Recounting the 1,833 definitions in the 400-crate sample by the expander's
actual boundary -- one rule, any repetition, no :tt -- gives 661 without
repetition and 271 with, so 932 of 1,833 are now expandable. The rest are
multi-rule (481) or token-tree munchers (420).

Two things here are easy to get wrong and are both pinned by
tests/cases/038. First, `>` closes `=>` and `->` as well as a generic
argument list, so the nesting depth used to find where a fragment ends must
not go negative; without that `$($a:expr => $b:expr),*` swallows the whole
invocation. Second, a repetition's separator may be inside the pattern rather
than between iterations -- `$first:expr $(, $rest:expr)*` has no separator,
its comma leads each iteration -- so a trailing fragment stops at the
separator when there is one and at whatever starts the next iteration when
there is not.

Repetition macros are usually written around a block expression with
statements, `{{ let mut m = ..; m }}`, so those now lower to Nim's `block:`
expression. Hoisting the statements into the enclosing scope was the obvious
thing and is wrong: the same macro expanded at two call sites then declares
the same binding twice in one scope. That is a general improvement, not a
macro-specific one -- Rust code uses block expressions directly too.

The survey did not move: 18 of 400 before, 18 after, 12 compilable either
way. Seventh feature running with that result, for the same reason each time
-- the crates it unblocks hit their next blocker. The capability is real and
tested; the crate count is gated by something else.

44 differential cases, all green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nandithebull committed 2026-09-18T23:38:10-07:00 Browse files
3d4a7d2 parent: 04eb29f
modified DESIGN.md +42 -9
@@ -630,15 +630,48 @@ which is exactly the guessing the project refuses. Expanding at the call site
630630 yields ordinary Rust in a context where the types are known, so it lowers like
631631 anything else. Same applicability, faithful output.
632632
633-`src/mrules.rs` implements the 36% case: one rule, no repetition, no `:tt`. A
634-definition it cannot handle is recorded *with its reason*, so a call site says
635-"`foo!` cannot be expanded: `$(..)` repetition is not implemented yet" rather
636-than "unknown macro". Captured fragments are parenthesised on substitution, so
637-`square!(2 + 3)` is 25 and not 11.
638-
639-**This moved the survey more than everything before it combined: 3 → 18 of
640-400 crates accepted.** A `macro_rules!` used to be a hard stop at item level,
641-failing a whole crate on sight.
633+`src/mrules.rs` implements single-rule macros, **with `$(..)` repetition**.
634+Recounting the 1,833 definitions by the expander's actual boundary — one rule,
635+any repetition, no `:tt`:
636+
637+| | count |
638+|---|---|
639+| single rule, no repetition | 661 |
640+| single rule, with repetition | 271 |
641+| **expandable** | **932 (51%)** |
642+| multiple rules | 481 |
643+| `:tt` | 420 |
644+
645+A definition it cannot handle is recorded *with its reason*, so a call site
646+says "`foo!` cannot be expanded: nested `$(..)` repetition is not implemented
647+yet" rather than "unknown macro". Captured fragments are parenthesised on
648+substitution, so `square!(2 + 3)` is 25 and not 11.
649+
650+Repetition needed two things that are easy to get wrong, both caught by
651+`tests/cases/038`:
652+
653+- `>` closes `=>` and `->` as well as a generic argument list, so the nesting
654+ depth used to find a fragment's end must not go negative. Without that,
655+ `$($a:expr => $b:expr),*` swallows the whole invocation.
656+- A repetition's separator may be *inside* the pattern rather than between
657+ iterations — `$first:expr $(, $rest:expr)*` has no separator, its comma
658+ leads each iteration. A trailing fragment therefore stops at the separator
659+ when there is one and at whatever starts the next iteration when there is
660+ not.
661+
662+A block expression with statements (`{{ let mut m = ..; m }}`, which is how
663+these macros are usually written) now lowers to Nim's `block:` expression
664+rather than being hoisted, so the same macro expanded at two call sites does
665+not collide in one scope.
666+
667+**Expansion moved the survey more than everything before it combined: 3 → 18
668+of 400 crates accepted.** A `macro_rules!` used to be a hard stop at item
669+level, failing a whole crate on sight.
670+
671+Repetition then moved it by **zero** — 18 before, 18 after — for the reason
672+every previous feature did: the crates it unblocked hit their next blocker.
673+That is now seven features running. The capability is real and tested; the
674+crate count is gated by something else.
642675
643676 ### A stricter number
644677
@@ -630,15 +630,48 @@ which is exactly the guessing the project refuses. Expanding at the call site
630 yields ordinary Rust in a context where the types are known, so it lowers like630 yields ordinary Rust in a context where the types are known, so it lowers like
631 anything else. Same applicability, faithful output.631 anything else. Same applicability, faithful output.
632 632
633-`src/mrules.rs` implements the 36% case: one rule, no repetition, no `:tt`. A633+`src/mrules.rs` implements single-rule macros, **with `$(..)` repetition**.
634-definition it cannot handle is recorded *with its reason*, so a call site says634+Recounting the 1,833 definitions by the expander's actual boundary — one rule,
635-"`foo!` cannot be expanded: `$(..)` repetition is not implemented yet" rather635+any repetition, no `:tt`:
636-than "unknown macro". Captured fragments are parenthesised on substitution, so636+
637-`square!(2 + 3)` is 25 and not 11.637+| | count |
638-638+|---|---|
639-**This moved the survey more than everything before it combined: 3 → 18 of639+| single rule, no repetition | 661 |
640-400 crates accepted.** A `macro_rules!` used to be a hard stop at item level,640+| single rule, with repetition | 271 |
641-failing a whole crate on sight.641+| **expandable** | **932 (51%)** |
642+| multiple rules | 481 |
643+| `:tt` | 420 |
644+
645+A definition it cannot handle is recorded *with its reason*, so a call site
646+says "`foo!` cannot be expanded: nested `$(..)` repetition is not implemented
647+yet" rather than "unknown macro". Captured fragments are parenthesised on
648+substitution, so `square!(2 + 3)` is 25 and not 11.
649+
650+Repetition needed two things that are easy to get wrong, both caught by
651+`tests/cases/038`:
652+
653+- `>` closes `=>` and `->` as well as a generic argument list, so the nesting
654+ depth used to find a fragment's end must not go negative. Without that,
655+ `$($a:expr => $b:expr),*` swallows the whole invocation.
656+- A repetition's separator may be *inside* the pattern rather than between
657+ iterations — `$first:expr $(, $rest:expr)*` has no separator, its comma
658+ leads each iteration. A trailing fragment therefore stops at the separator
659+ when there is one and at whatever starts the next iteration when there is
660+ not.
661+
662+A block expression with statements (`{{ let mut m = ..; m }}`, which is how
663+these macros are usually written) now lowers to Nim's `block:` expression
664+rather than being hoisted, so the same macro expanded at two call sites does
665+not collide in one scope.
666+
667+**Expansion moved the survey more than everything before it combined: 3 → 18
668+of 400 crates accepted.** A `macro_rules!` used to be a hard stop at item
669+level, failing a whole crate on sight.
670+
671+Repetition then moved it by **zero** — 18 before, 18 after — for the reason
672+every previous feature did: the crates it unblocked hit their next blocker.
673+That is now seven features running. The capability is real and tested; the
674+crate count is gated by something else.
642 675
643 ### A stricter number676 ### A stricter number
644 677
modified README.md +6 -2
@@ -111,8 +111,12 @@ A `macro_rules!` is **expanded at the call site**, not translated into a Nim
111111 template — but a template body is untyped, and this lowering is type-directed
112112 throughout. Expanding gives ordinary Rust where the types are known.
113113
114-That change alone took the survey from 3 to 18 of 400 crates accepted, of
115-which 12 produce Nim the Nim compiler accepts. (Accepted, compiles, and
114+Single-rule macros are supported, with `$(..)` repetition — 932 of the 1,833
115+definitions in that sample, 51%. The rest are multi-rule (481) or token-tree
116+munchers (420).
117+
118+Expansion took the survey from 3 to 18 of 400 crates accepted, of which 12
119+produce Nim the Nim compiler accepts. (Accepted, compiles, and
116120 behaviourally verified are three different bars; only `tests/cases/` clears
117121 the third.)
118122
@@ -111,8 +111,12 @@ A `macro_rules!` is **expanded at the call site**, not translated into a Nim
111 template — but a template body is untyped, and this lowering is type-directed111 template — but a template body is untyped, and this lowering is type-directed
112 throughout. Expanding gives ordinary Rust where the types are known.112 throughout. Expanding gives ordinary Rust where the types are known.
113 113
114-That change alone took the survey from 3 to 18 of 400 crates accepted, of114+Single-rule macros are supported, with `$(..)` repetition — 932 of the 1,833
115-which 12 produce Nim the Nim compiler accepts. (Accepted, compiles, and115+definitions in that sample, 51%. The rest are multi-rule (481) or token-tree
116+munchers (420).
117+
118+Expansion took the survey from 3 to 18 of 400 crates accepted, of which 12
119+produce Nim the Nim compiler accepts. (Accepted, compiles, and
116 behaviourally verified are three different bars; only `tests/cases/` clears120 behaviourally verified are three different bars; only `tests/cases/` clears
117 the third.)121 the third.)
118 122
modified src/lower.rs +49 -1
@@ -3610,6 +3610,51 @@ impl Lowerer {
36103610 // `if` and `match` are expressions in both languages, but only
36113611 // when every arm is itself a single expression.
36123612 Expr::If(i) => self.if_expr(i, expect),
3613+ // A block with statements used as a value: the statements are
3614+ // emitted ahead of the line being built and the block's trailing
3615+ // expression becomes the value. Every caller lowers its
3616+ // sub-expressions before emitting its own line, which is what
3617+ // makes that ordering hold.
3618+ Expr::Block(b)
3619+ if b.label.is_none()
3620+ && b.block.stmts.len() > 1
3621+ && matches!(b.block.stmts.last(), Some(Stmt::Expr(_, None))) =>
3622+ {
3623+ if self.in_loop_cond {
3624+ return Err("a block expression in a loop condition is not \
3625+ implemented yet: its statements would run once, \
3626+ before the loop"
3627+ .into());
3628+ }
3629+ // Nim's `block:` is an expression too, so the statements get
3630+ // their own scope rather than being hoisted into the enclosing
3631+ // one -- which would collide if the same block is written
3632+ // twice, as a macro expanded at two call sites is.
3633+ let tmp = self.fresh("Blk");
3634+ self.line(&format!("let {} = block:", tmp));
3635+ self.indent += 1;
3636+ self.push_scope();
3637+ let saved = self.target.take();
3638+ let before = self.out.len();
3639+ let tail = self.block_body_at(&b.block, expect)?;
3640+ let ty = tail.as_ref().and_then(|v| v.ty.clone());
3641+ match tail {
3642+ Some(v) => {
3643+ let code = v.code.clone();
3644+ self.line(&code);
3645+ }
3646+ None => {
3647+ return Err(
3648+ "a block used as a value needs a trailing expression".into()
3649+ )
3650+ }
3651+ }
3652+ let _ = before;
3653+ self.target = saved;
3654+ self.pop_scope();
3655+ self.indent -= 1;
3656+ Ok(Val::new(tmp, ty))
3657+ }
36133658 Expr::Block(b) if b.block.stmts.len() == 1 => {
36143659 if let Some(Stmt::Expr(e, None)) = b.block.stmts.first() {
36153660 self.expr_at(e, expect)
@@ -5183,7 +5228,10 @@ fn expressible(e: &Expr) -> bool {
51835228 // `unsafe { .. }` is transparent, so it is an expression exactly when
51845229 // its block is one.
51855230 Expr::Unsafe(u) => single_expr(&u.block).is_some_and(expressible),
5186- Expr::Match(_) | Expr::Block(_) | Expr::Loop(_) | Expr::While(_) | Expr::ForLoop(_) => false,
5231+ Expr::Block(b) => {
5232+ b.label.is_none() && matches!(b.block.stmts.last(), Some(Stmt::Expr(_, None)))
5233+ }
5234+ Expr::Match(_) | Expr::Loop(_) | Expr::While(_) | Expr::ForLoop(_) => false,
51875235 _ => true,
51885236 }
51895237 }
@@ -3610,6 +3610,51 @@ impl Lowerer {
3610 // `if` and `match` are expressions in both languages, but only3610 // `if` and `match` are expressions in both languages, but only
3611 // when every arm is itself a single expression.3611 // when every arm is itself a single expression.
3612 Expr::If(i) => self.if_expr(i, expect),3612 Expr::If(i) => self.if_expr(i, expect),
3613+ // A block with statements used as a value: the statements are
3614+ // emitted ahead of the line being built and the block's trailing
3615+ // expression becomes the value. Every caller lowers its
3616+ // sub-expressions before emitting its own line, which is what
3617+ // makes that ordering hold.
3618+ Expr::Block(b)
3619+ if b.label.is_none()
3620+ && b.block.stmts.len() > 1
3621+ && matches!(b.block.stmts.last(), Some(Stmt::Expr(_, None))) =>
3622+ {
3623+ if self.in_loop_cond {
3624+ return Err("a block expression in a loop condition is not \
3625+ implemented yet: its statements would run once, \
3626+ before the loop"
3627+ .into());
3628+ }
3629+ // Nim's `block:` is an expression too, so the statements get
3630+ // their own scope rather than being hoisted into the enclosing
3631+ // one -- which would collide if the same block is written
3632+ // twice, as a macro expanded at two call sites is.
3633+ let tmp = self.fresh("Blk");
3634+ self.line(&format!("let {} = block:", tmp));
3635+ self.indent += 1;
3636+ self.push_scope();
3637+ let saved = self.target.take();
3638+ let before = self.out.len();
3639+ let tail = self.block_body_at(&b.block, expect)?;
3640+ let ty = tail.as_ref().and_then(|v| v.ty.clone());
3641+ match tail {
3642+ Some(v) => {
3643+ let code = v.code.clone();
3644+ self.line(&code);
3645+ }
3646+ None => {
3647+ return Err(
3648+ "a block used as a value needs a trailing expression".into()
3649+ )
3650+ }
3651+ }
3652+ let _ = before;
3653+ self.target = saved;
3654+ self.pop_scope();
3655+ self.indent -= 1;
3656+ Ok(Val::new(tmp, ty))
3657+ }
3613 Expr::Block(b) if b.block.stmts.len() == 1 => {3658 Expr::Block(b) if b.block.stmts.len() == 1 => {
3614 if let Some(Stmt::Expr(e, None)) = b.block.stmts.first() {3659 if let Some(Stmt::Expr(e, None)) = b.block.stmts.first() {
3615 self.expr_at(e, expect)3660 self.expr_at(e, expect)
@@ -5183,7 +5228,10 @@ fn expressible(e: &Expr) -> bool {
5183 // `unsafe { .. }` is transparent, so it is an expression exactly when5228 // `unsafe { .. }` is transparent, so it is an expression exactly when
5184 // its block is one.5229 // its block is one.
5185 Expr::Unsafe(u) => single_expr(&u.block).is_some_and(expressible),5230 Expr::Unsafe(u) => single_expr(&u.block).is_some_and(expressible),
5186- Expr::Match(_) | Expr::Block(_) | Expr::Loop(_) | Expr::While(_) | Expr::ForLoop(_) => false,5231+ Expr::Block(b) => {
5232+ b.label.is_none() && matches!(b.block.stmts.last(), Some(Stmt::Expr(_, None)))
5233+ }
5234+ Expr::Match(_) | Expr::Loop(_) | Expr::While(_) | Expr::ForLoop(_) => false,
5187 _ => true,5235 _ => true,
5188 }5236 }
5189 }5237 }
modified src/mrules.rs +218 -19
@@ -28,8 +28,19 @@ enum Pat {
2828 Tok(String),
2929 /// A delimited group, matched recursively.
3030 Group(Delimiter, Vec<Pat>),
31+ /// `$( .. ) sep? op` — a repetition. `op` is `*`, `+` or `?`.
32+ Rep { inner: Vec<Pat>, sep: Option<String>, op: char },
3133 }
3234
35+/// What a fragment captured: once, or once per repetition.
36+#[derive(Debug, Clone)]
37+enum Cap {
38+ One(TokenStream),
39+ Seq(Vec<TokenStream>),
40+}
41+
42+type Binds = HashMap<String, Cap>;
43+
3344 #[derive(Debug, Clone)]
3445 pub struct MacroDef {
3546 pattern: Vec<Pat>,
@@ -84,12 +95,33 @@ fn parse_pattern(ts: TokenStream) -> Result<Vec<Pat>, String> {
8495 }
8596 out.push(Pat::Frag(name.to_string()));
8697 }
87- Some(TokenTree::Group(_)) => {
88- return Err(
89- "`$(..)` repetition is not implemented yet; it maps to \
90- `varargs` in a Nim macro"
91- .into(),
92- )
98+ Some(TokenTree::Group(g)) => {
99+ let inner = parse_pattern(g.stream())?;
100+ if inner.iter().any(|p| matches!(p, Pat::Rep { .. })) {
101+ return Err("nested `$(..)` repetition is not implemented yet".into());
102+ }
103+ // `$( .. ) sep? op`: an optional separator token, then
104+ // the operator.
105+ let mut sep = None;
106+ let op = loop {
107+ match it.next() {
108+ Some(TokenTree::Punct(p))
109+ if matches!(p.as_char(), '*' | '+' | '?') =>
110+ {
111+ break p.as_char()
112+ }
113+ Some(t) => {
114+ if sep.is_some() {
115+ return Err(
116+ "a repetition separator must be one token".into()
117+ );
118+ }
119+ sep = Some(t.to_string());
120+ }
121+ None => return Err("a repetition needs `*`, `+` or `?`".into()),
122+ }
123+ };
124+ out.push(Pat::Rep { inner, sep, op });
93125 }
94126 _ => return Err("unexpected token after `$`".into()),
95127 }
@@ -106,9 +138,9 @@ fn parse_pattern(ts: TokenStream) -> Result<Vec<Pat>, String> {
106138 impl MacroDef {
107139 /// Match an invocation's tokens and substitute them into the body.
108140 pub fn expand(&self, input: TokenStream) -> Result<TokenStream, String> {
109- let mut binds = HashMap::new();
141+ let mut binds = Binds::new();
110142 let toks: Vec<TokenTree> = input.into_iter().collect();
111- let used = match_seq(&self.pattern, &toks, &mut binds)?;
143+ let used = match_seq(&self.pattern, &toks, &mut binds, None)?;
112144 if used != toks.len() {
113145 return Err("the invocation has tokens the matcher does not consume".into());
114146 }
@@ -117,10 +149,14 @@ impl MacroDef {
117149 }
118150
119151 /// Match `pats` against the front of `toks`, returning how many were consumed.
152+/// `outer_stop` is where a trailing fragment must stop when the pattern
153+/// itself does not say: inside `$(, $rest:expr)*` that is the repetition's own
154+/// leading comma, which is where the next iteration begins.
120155 fn match_seq(
121156 pats: &[Pat],
122157 toks: &[TokenTree],
123- binds: &mut HashMap<String, TokenStream>,
158+ binds: &mut Binds,
159+ outer_stop: Option<&str>,
124160 ) -> Result<usize, String> {
125161 let mut i = 0;
126162 for (k, p) in pats.iter().enumerate() {
@@ -140,20 +176,67 @@ fn match_seq(
140176 return Err("mismatched delimiter".into());
141177 }
142178 let sub: Vec<TokenTree> = g.stream().into_iter().collect();
143- let n = match_seq(inner, &sub, binds)?;
179+ let n = match_seq(inner, &sub, binds, None)?;
144180 if n != sub.len() {
145181 return Err("group has tokens the matcher does not consume".into());
146182 }
147183 i += 1;
148184 }
185+ Pat::Rep { inner, sep, op } => {
186+ // Everything after the repetition that is a literal token
187+ // marks where it has to stop.
188+ let stop = next_literal(&pats[k + 1..]).or_else(|| outer_stop.map(str::to_string));
189+ // A trailing fragment inside the repetition stops at the
190+ // separator if there is one, or otherwise at whatever starts
191+ // the next iteration.
192+ let inner_stop = sep.clone().or_else(|| next_literal(inner));
193+ let names = frag_names(inner);
194+ let mut count = 0usize;
195+ for n in &names {
196+ binds.insert(n.clone(), Cap::Seq(Vec::new()));
197+ }
198+ while i < toks.len() {
199+ if let Some(stop) = &stop {
200+ if toks[i].to_string() == *stop {
201+ break;
202+ }
203+ }
204+ let mut one = Binds::new();
205+ let used = match_seq(inner, &toks[i..], &mut one, inner_stop.as_deref())?;
206+ if used == 0 {
207+ break;
208+ }
209+ for n in &names {
210+ let v = match one.remove(n) {
211+ Some(Cap::One(ts)) => ts,
212+ _ => return Err(format!("`${n}` did not capture in a repetition")),
213+ };
214+ match binds.get_mut(n) {
215+ Some(Cap::Seq(v0)) => v0.push(v),
216+ _ => unreachable!(),
217+ }
218+ }
219+ count += 1;
220+ i += used;
221+ match sep {
222+ Some(sp) if i < toks.len() && toks[i].to_string() == *sp => i += 1,
223+ Some(_) => break,
224+ None => {}
225+ }
226+ if *op == '?' {
227+ break;
228+ }
229+ }
230+ if *op == '+' && count == 0 {
231+ return Err("`$(..)+` needs at least one repetition".into());
232+ }
233+ }
149234 Pat::Frag(name) => {
150235 // A fragment runs to the next literal token in the matcher, or
151236 // to the end. That is what makes `$a:expr, $b:expr` split on
152237 // the comma rather than swallowing it.
153- let stop = pats[k + 1..].iter().find_map(|p| match p {
154- Pat::Tok(s) => Some(s.clone()),
155- _ => None,
156- });
238+ let stop = next_literal(&pats[k + 1..])
239+ .or_else(|| outer_stop.map(str::to_string));
157240 let start = i;
158241 let mut depth = 0i32;
159242 while i < toks.len() {
@@ -163,9 +246,12 @@ fn match_seq(
163246 break;
164247 }
165248 }
249+ // `<`/`>` are counted so a generic argument list is not
250+ // split, but `>` also closes `=>` and `->`, so the depth
251+ // never goes negative.
166252 match &toks[i] {
167253 TokenTree::Punct(p) if p.as_char() == '<' => depth += 1,
168- TokenTree::Punct(p) if p.as_char() == '>' => depth -= 1,
254+ TokenTree::Punct(p) if p.as_char() == '>' => depth = (depth - 1).max(0),
169255 _ => {}
170256 }
171257 i += 1;
@@ -175,7 +261,7 @@ fn match_seq(
175261 }
176262 binds.insert(
177263 name.clone(),
178- toks[start..i].iter().cloned().collect::<TokenStream>(),
264+ Cap::One(toks[start..i].iter().cloned().collect::<TokenStream>()),
179265 );
180266 }
181267 }
@@ -183,13 +269,99 @@ fn match_seq(
183269 Ok(i)
184270 }
185271
186-/// Replace every `$name` in the body with what it captured.
187-fn substitute(ts: TokenStream, binds: &HashMap<String, TokenStream>) -> TokenStream {
272+/// The first literal token that can follow, looking through a repetition:
273+/// after `$first:expr` in `($first:expr $(, $rest:expr)*)` the next literal is
274+/// the repetition's leading comma, which is where `$first` has to stop.
275+fn next_literal(pats: &[Pat]) -> Option<String> {
276+ for p in pats {
277+ match p {
278+ Pat::Tok(s) => return Some(s.clone()),
279+ Pat::Rep { inner, sep, .. } => {
280+ if let Some(s) = next_literal(inner) {
281+ return Some(s);
282+ }
283+ if let Some(s) = sep {
284+ return Some(s.clone());
285+ }
286+ }
287+ Pat::Group(..) | Pat::Frag(_) => return None,
288+ }
289+ }
290+ None
291+}
292+
293+/// The fragment names a pattern captures.
294+fn frag_names(pats: &[Pat]) -> Vec<String> {
295+ let mut out = Vec::new();
296+ for p in pats {
297+ match p {
298+ Pat::Frag(n) => out.push(n.clone()),
299+ Pat::Group(_, inner) => out.extend(frag_names(inner)),
300+ Pat::Rep { inner, .. } => out.extend(frag_names(inner)),
301+ Pat::Tok(_) => {}
302+ }
303+ }
304+ out
305+}
306+
307+/// Replace every `$name` in the body with what it captured, and expand every
308+/// `$( .. ) sep? op` once per repetition.
309+fn substitute(ts: TokenStream, binds: &Binds) -> TokenStream {
188310 let mut out = Vec::new();
189311 let mut it = ts.into_iter().peekable();
190312 while let Some(t) = it.next() {
191313 match t {
192314 TokenTree::Punct(p) if p.as_char() == '$' => {
315+ // `$( .. ) sep? op` in the transcriber.
316+ if let Some(TokenTree::Group(g)) = it.peek() {
317+ let g = g.clone();
318+ it.next();
319+ let mut sep = None;
320+ loop {
321+ match it.peek() {
322+ Some(TokenTree::Punct(p))
323+ if matches!(p.as_char(), '*' | '+' | '?') =>
324+ {
325+ it.next();
326+ break;
327+ }
328+ Some(t) => {
329+ sep = Some(t.clone());
330+ it.next();
331+ }
332+ None => break,
333+ }
334+ }
335+ let names: Vec<String> = fragments_in(g.stream())
336+ .into_iter()
337+ .filter(|n| matches!(binds.get(n), Some(Cap::Seq(_))))
338+ .collect();
339+ let n = names
340+ .iter()
341+ .filter_map(|n| match binds.get(n) {
342+ Some(Cap::Seq(v)) => Some(v.len()),
343+ _ => None,
344+ })
345+ .max()
346+ .unwrap_or(0);
347+ for idx in 0..n {
348+ let mut one: Binds = binds.clone();
349+ for nm in &names {
350+ if let Some(Cap::Seq(v)) = binds.get(nm) {
351+ if let Some(x) = v.get(idx) {
352+ one.insert(nm.clone(), Cap::One(x.clone()));
353+ }
354+ }
355+ }
356+ if idx > 0 {
357+ if let Some(s) = &sep {
358+ out.push(s.clone());
359+ }
360+ }
361+ out.extend(substitute(g.stream(), &one));
362+ }
363+ continue;
364+ }
193365 match it.peek() {
194366 Some(TokenTree::Ident(name)) => {
195367 let n = name.to_string();
@@ -197,10 +369,18 @@ fn substitute(ts: TokenStream, binds: &HashMap<String, TokenStream>) -> TokenStr
197369 match binds.get(&n) {
198370 // Parenthesised so that a captured expression keeps
199371 // its own precedence, as Rust's `expr` fragments do.
200- Some(v) => out.push(TokenTree::Group(Group::new(
372+ Some(Cap::One(v)) => out.push(TokenTree::Group(Group::new(
201373 Delimiter::Parenthesis,
202374 v.clone(),
203375 ))),
376+ Some(Cap::Seq(_)) => {
377+ // Used outside a `$(..)`; Rust rejects this too.
378+ out.push(TokenTree::Punct(p));
379+ out.push(TokenTree::Ident(Ident::new(
380+ &n,
381+ proc_macro2::Span::call_site(),
382+ )));
383+ }
204384 None => {
205385 out.push(TokenTree::Punct(p));
206386 out.push(TokenTree::Ident(Ident::new(
@@ -222,3 +402,22 @@ fn substitute(ts: TokenStream, binds: &HashMap<String, TokenStream>) -> TokenStr
222402 }
223403 out.into_iter().collect()
224404 }
405+
406+/// The `$name` references appearing in a token stream.
407+fn fragments_in(ts: TokenStream) -> Vec<String> {
408+ let mut out = Vec::new();
409+ let mut it = ts.into_iter().peekable();
410+ while let Some(t) = it.next() {
411+ match t {
412+ TokenTree::Punct(p) if p.as_char() == '$' => {
413+ if let Some(TokenTree::Ident(n)) = it.peek() {
414+ out.push(n.to_string());
415+ it.next();
416+ }
417+ }
418+ TokenTree::Group(g) => out.extend(fragments_in(g.stream())),
419+ _ => {}
420+ }
421+ }
422+ out
423+}
@@ -28,8 +28,19 @@ enum Pat {
28 Tok(String),28 Tok(String),
29 /// A delimited group, matched recursively.29 /// A delimited group, matched recursively.
30 Group(Delimiter, Vec<Pat>),30 Group(Delimiter, Vec<Pat>),
31+ /// `$( .. ) sep? op` — a repetition. `op` is `*`, `+` or `?`.
32+ Rep { inner: Vec<Pat>, sep: Option<String>, op: char },
31 }33 }
32 34
35+/// What a fragment captured: once, or once per repetition.
36+#[derive(Debug, Clone)]
37+enum Cap {
38+ One(TokenStream),
39+ Seq(Vec<TokenStream>),
40+}
41+
42+type Binds = HashMap<String, Cap>;
43+
33 #[derive(Debug, Clone)]44 #[derive(Debug, Clone)]
34 pub struct MacroDef {45 pub struct MacroDef {
35 pattern: Vec<Pat>,46 pattern: Vec<Pat>,
@@ -84,12 +95,33 @@ fn parse_pattern(ts: TokenStream) -> Result<Vec<Pat>, String> {
84 }95 }
85 out.push(Pat::Frag(name.to_string()));96 out.push(Pat::Frag(name.to_string()));
86 }97 }
87- Some(TokenTree::Group(_)) => {98+ Some(TokenTree::Group(g)) => {
88- return Err(99+ let inner = parse_pattern(g.stream())?;
89- "`$(..)` repetition is not implemented yet; it maps to \100+ if inner.iter().any(|p| matches!(p, Pat::Rep { .. })) {
90- `varargs` in a Nim macro"101+ return Err("nested `$(..)` repetition is not implemented yet".into());
91- .into(),102+ }
92- )103+ // `$( .. ) sep? op`: an optional separator token, then
104+ // the operator.
105+ let mut sep = None;
106+ let op = loop {
107+ match it.next() {
108+ Some(TokenTree::Punct(p))
109+ if matches!(p.as_char(), '*' | '+' | '?') =>
110+ {
111+ break p.as_char()
112+ }
113+ Some(t) => {
114+ if sep.is_some() {
115+ return Err(
116+ "a repetition separator must be one token".into()
117+ );
118+ }
119+ sep = Some(t.to_string());
120+ }
121+ None => return Err("a repetition needs `*`, `+` or `?`".into()),
122+ }
123+ };
124+ out.push(Pat::Rep { inner, sep, op });
93 }125 }
94 _ => return Err("unexpected token after `$`".into()),126 _ => return Err("unexpected token after `$`".into()),
95 }127 }
@@ -106,9 +138,9 @@ fn parse_pattern(ts: TokenStream) -> Result<Vec<Pat>, String> {
106 impl MacroDef {138 impl MacroDef {
107 /// Match an invocation's tokens and substitute them into the body.139 /// Match an invocation's tokens and substitute them into the body.
108 pub fn expand(&self, input: TokenStream) -> Result<TokenStream, String> {140 pub fn expand(&self, input: TokenStream) -> Result<TokenStream, String> {
109- let mut binds = HashMap::new();141+ let mut binds = Binds::new();
110 let toks: Vec<TokenTree> = input.into_iter().collect();142 let toks: Vec<TokenTree> = input.into_iter().collect();
111- let used = match_seq(&self.pattern, &toks, &mut binds)?;143+ let used = match_seq(&self.pattern, &toks, &mut binds, None)?;
112 if used != toks.len() {144 if used != toks.len() {
113 return Err("the invocation has tokens the matcher does not consume".into());145 return Err("the invocation has tokens the matcher does not consume".into());
114 }146 }
@@ -117,10 +149,14 @@ impl MacroDef {
117 }149 }
118 150
119 /// Match `pats` against the front of `toks`, returning how many were consumed.151 /// Match `pats` against the front of `toks`, returning how many were consumed.
152+/// `outer_stop` is where a trailing fragment must stop when the pattern
153+/// itself does not say: inside `$(, $rest:expr)*` that is the repetition's own
154+/// leading comma, which is where the next iteration begins.
120 fn match_seq(155 fn match_seq(
121 pats: &[Pat],156 pats: &[Pat],
122 toks: &[TokenTree],157 toks: &[TokenTree],
123- binds: &mut HashMap<String, TokenStream>,158+ binds: &mut Binds,
159+ outer_stop: Option<&str>,
124 ) -> Result<usize, String> {160 ) -> Result<usize, String> {
125 let mut i = 0;161 let mut i = 0;
126 for (k, p) in pats.iter().enumerate() {162 for (k, p) in pats.iter().enumerate() {
@@ -140,20 +176,67 @@ fn match_seq(
140 return Err("mismatched delimiter".into());176 return Err("mismatched delimiter".into());
141 }177 }
142 let sub: Vec<TokenTree> = g.stream().into_iter().collect();178 let sub: Vec<TokenTree> = g.stream().into_iter().collect();
143- let n = match_seq(inner, &sub, binds)?;179+ let n = match_seq(inner, &sub, binds, None)?;
144 if n != sub.len() {180 if n != sub.len() {
145 return Err("group has tokens the matcher does not consume".into());181 return Err("group has tokens the matcher does not consume".into());
146 }182 }
147 i += 1;183 i += 1;
148 }184 }
185+ Pat::Rep { inner, sep, op } => {
186+ // Everything after the repetition that is a literal token
187+ // marks where it has to stop.
188+ let stop = next_literal(&pats[k + 1..]).or_else(|| outer_stop.map(str::to_string));
189+ // A trailing fragment inside the repetition stops at the
190+ // separator if there is one, or otherwise at whatever starts
191+ // the next iteration.
192+ let inner_stop = sep.clone().or_else(|| next_literal(inner));
193+ let names = frag_names(inner);
194+ let mut count = 0usize;
195+ for n in &names {
196+ binds.insert(n.clone(), Cap::Seq(Vec::new()));
197+ }
198+ while i < toks.len() {
199+ if let Some(stop) = &stop {
200+ if toks[i].to_string() == *stop {
201+ break;
202+ }
203+ }
204+ let mut one = Binds::new();
205+ let used = match_seq(inner, &toks[i..], &mut one, inner_stop.as_deref())?;
206+ if used == 0 {
207+ break;
208+ }
209+ for n in &names {
210+ let v = match one.remove(n) {
211+ Some(Cap::One(ts)) => ts,
212+ _ => return Err(format!("`${n}` did not capture in a repetition")),
213+ };
214+ match binds.get_mut(n) {
215+ Some(Cap::Seq(v0)) => v0.push(v),
216+ _ => unreachable!(),
217+ }
218+ }
219+ count += 1;
220+ i += used;
221+ match sep {
222+ Some(sp) if i < toks.len() && toks[i].to_string() == *sp => i += 1,
223+ Some(_) => break,
224+ None => {}
225+ }
226+ if *op == '?' {
227+ break;
228+ }
229+ }
230+ if *op == '+' && count == 0 {
231+ return Err("`$(..)+` needs at least one repetition".into());
232+ }
233+ }
149 Pat::Frag(name) => {234 Pat::Frag(name) => {
150 // A fragment runs to the next literal token in the matcher, or235 // A fragment runs to the next literal token in the matcher, or
151 // to the end. That is what makes `$a:expr, $b:expr` split on236 // to the end. That is what makes `$a:expr, $b:expr` split on
152 // the comma rather than swallowing it.237 // the comma rather than swallowing it.
153- let stop = pats[k + 1..].iter().find_map(|p| match p {238+ let stop = next_literal(&pats[k + 1..])
154- Pat::Tok(s) => Some(s.clone()),239+ .or_else(|| outer_stop.map(str::to_string));
155- _ => None,
156- });
157 let start = i;240 let start = i;
158 let mut depth = 0i32;241 let mut depth = 0i32;
159 while i < toks.len() {242 while i < toks.len() {
@@ -163,9 +246,12 @@ fn match_seq(
163 break;246 break;
164 }247 }
165 }248 }
249+ // `<`/`>` are counted so a generic argument list is not
250+ // split, but `>` also closes `=>` and `->`, so the depth
251+ // never goes negative.
166 match &toks[i] {252 match &toks[i] {
167 TokenTree::Punct(p) if p.as_char() == '<' => depth += 1,253 TokenTree::Punct(p) if p.as_char() == '<' => depth += 1,
168- TokenTree::Punct(p) if p.as_char() == '>' => depth -= 1,254+ TokenTree::Punct(p) if p.as_char() == '>' => depth = (depth - 1).max(0),
169 _ => {}255 _ => {}
170 }256 }
171 i += 1;257 i += 1;
@@ -175,7 +261,7 @@ fn match_seq(
175 }261 }
176 binds.insert(262 binds.insert(
177 name.clone(),263 name.clone(),
178- toks[start..i].iter().cloned().collect::<TokenStream>(),264+ Cap::One(toks[start..i].iter().cloned().collect::<TokenStream>()),
179 );265 );
180 }266 }
181 }267 }
@@ -183,13 +269,99 @@ fn match_seq(
183 Ok(i)269 Ok(i)
184 }270 }
185 271
186-/// Replace every `$name` in the body with what it captured.272+/// The first literal token that can follow, looking through a repetition:
187-fn substitute(ts: TokenStream, binds: &HashMap<String, TokenStream>) -> TokenStream {273+/// after `$first:expr` in `($first:expr $(, $rest:expr)*)` the next literal is
274+/// the repetition's leading comma, which is where `$first` has to stop.
275+fn next_literal(pats: &[Pat]) -> Option<String> {
276+ for p in pats {
277+ match p {
278+ Pat::Tok(s) => return Some(s.clone()),
279+ Pat::Rep { inner, sep, .. } => {
280+ if let Some(s) = next_literal(inner) {
281+ return Some(s);
282+ }
283+ if let Some(s) = sep {
284+ return Some(s.clone());
285+ }
286+ }
287+ Pat::Group(..) | Pat::Frag(_) => return None,
288+ }
289+ }
290+ None
291+}
292+
293+/// The fragment names a pattern captures.
294+fn frag_names(pats: &[Pat]) -> Vec<String> {
295+ let mut out = Vec::new();
296+ for p in pats {
297+ match p {
298+ Pat::Frag(n) => out.push(n.clone()),
299+ Pat::Group(_, inner) => out.extend(frag_names(inner)),
300+ Pat::Rep { inner, .. } => out.extend(frag_names(inner)),
301+ Pat::Tok(_) => {}
302+ }
303+ }
304+ out
305+}
306+
307+/// Replace every `$name` in the body with what it captured, and expand every
308+/// `$( .. ) sep? op` once per repetition.
309+fn substitute(ts: TokenStream, binds: &Binds) -> TokenStream {
188 let mut out = Vec::new();310 let mut out = Vec::new();
189 let mut it = ts.into_iter().peekable();311 let mut it = ts.into_iter().peekable();
190 while let Some(t) = it.next() {312 while let Some(t) = it.next() {
191 match t {313 match t {
192 TokenTree::Punct(p) if p.as_char() == '$' => {314 TokenTree::Punct(p) if p.as_char() == '$' => {
315+ // `$( .. ) sep? op` in the transcriber.
316+ if let Some(TokenTree::Group(g)) = it.peek() {
317+ let g = g.clone();
318+ it.next();
319+ let mut sep = None;
320+ loop {
321+ match it.peek() {
322+ Some(TokenTree::Punct(p))
323+ if matches!(p.as_char(), '*' | '+' | '?') =>
324+ {
325+ it.next();
326+ break;
327+ }
328+ Some(t) => {
329+ sep = Some(t.clone());
330+ it.next();
331+ }
332+ None => break,
333+ }
334+ }
335+ let names: Vec<String> = fragments_in(g.stream())
336+ .into_iter()
337+ .filter(|n| matches!(binds.get(n), Some(Cap::Seq(_))))
338+ .collect();
339+ let n = names
340+ .iter()
341+ .filter_map(|n| match binds.get(n) {
342+ Some(Cap::Seq(v)) => Some(v.len()),
343+ _ => None,
344+ })
345+ .max()
346+ .unwrap_or(0);
347+ for idx in 0..n {
348+ let mut one: Binds = binds.clone();
349+ for nm in &names {
350+ if let Some(Cap::Seq(v)) = binds.get(nm) {
351+ if let Some(x) = v.get(idx) {
352+ one.insert(nm.clone(), Cap::One(x.clone()));
353+ }
354+ }
355+ }
356+ if idx > 0 {
357+ if let Some(s) = &sep {
358+ out.push(s.clone());
359+ }
360+ }
361+ out.extend(substitute(g.stream(), &one));
362+ }
363+ continue;
364+ }
193 match it.peek() {365 match it.peek() {
194 Some(TokenTree::Ident(name)) => {366 Some(TokenTree::Ident(name)) => {
195 let n = name.to_string();367 let n = name.to_string();
@@ -197,10 +369,18 @@ fn substitute(ts: TokenStream, binds: &HashMap<String, TokenStream>) -> TokenStr
197 match binds.get(&n) {369 match binds.get(&n) {
198 // Parenthesised so that a captured expression keeps370 // Parenthesised so that a captured expression keeps
199 // its own precedence, as Rust's `expr` fragments do.371 // its own precedence, as Rust's `expr` fragments do.
200- Some(v) => out.push(TokenTree::Group(Group::new(372+ Some(Cap::One(v)) => out.push(TokenTree::Group(Group::new(
201 Delimiter::Parenthesis,373 Delimiter::Parenthesis,
202 v.clone(),374 v.clone(),
203 ))),375 ))),
376+ Some(Cap::Seq(_)) => {
377+ // Used outside a `$(..)`; Rust rejects this too.
378+ out.push(TokenTree::Punct(p));
379+ out.push(TokenTree::Ident(Ident::new(
380+ &n,
381+ proc_macro2::Span::call_site(),
382+ )));
383+ }
204 None => {384 None => {
205 out.push(TokenTree::Punct(p));385 out.push(TokenTree::Punct(p));
206 out.push(TokenTree::Ident(Ident::new(386 out.push(TokenTree::Ident(Ident::new(
@@ -222,3 +402,22 @@ fn substitute(ts: TokenStream, binds: &HashMap<String, TokenStream>) -> TokenStr
222 }402 }
223 out.into_iter().collect()403 out.into_iter().collect()
224 }404 }
405+
406+/// The `$name` references appearing in a token stream.
407+fn fragments_in(ts: TokenStream) -> Vec<String> {
408+ let mut out = Vec::new();
409+ let mut it = ts.into_iter().peekable();
410+ while let Some(t) = it.next() {
411+ match t {
412+ TokenTree::Punct(p) if p.as_char() == '$' => {
413+ if let Some(TokenTree::Ident(n)) = it.peek() {
414+ out.push(n.to_string());
415+ it.next();
416+ }
417+ }
418+ TokenTree::Group(g) => out.extend(fragments_in(g.stream())),
419+ _ => {}
420+ }
421+ }
422+ out
423+}
added tests/cases/038-macro-repetition.rs +46 -0
new file mode 100644
@@ -0,0 +1,46 @@
1+// `$(..)` repetition — 28% of the `macro_rules!` in a 400-crate sample. The
2+// matcher binds each fragment once per iteration, and the transcriber emits
3+// its body once per iteration, joined by the separator.
4+
5+macro_rules! sum_all {
6+ ($($x:expr),*) => {
7+ 0 $(+ $x)*
8+ };
9+}
10+
11+macro_rules! max_of {
12+ ($first:expr $(, $rest:expr)*) => {{
13+ let mut m = $first;
14+ $( if $rest > m { m = $rest; } )*
15+ m
16+ }};
17+}
18+
19+macro_rules! count_args {
20+ ($($x:expr),*) => {
21+ 0 $(+ { let _ = $x; 1 })*
22+ };
23+}
24+
25+macro_rules! pairs_sum {
26+ ($($a:expr => $b:expr),*) => {
27+ 0 $(+ $a * $b)*
28+ };
29+}
30+
31+fn main() {
32+ println!("{}", sum_all!());
33+ println!("{}", sum_all!(1));
34+ println!("{}", sum_all!(1, 2, 3, 4));
35+ // Precedence survives: each capture is parenthesised.
36+ println!("{}", sum_all!(1 + 1, 2 * 3));
37+
38+ println!("{}", max_of!(3));
39+ println!("{}", max_of!(3, 9, 2));
40+ println!("{}", max_of!(-5, -9, -1));
41+
42+ println!("{}", count_args!(7, 8, 9));
43+ println!("{}", count_args!());
44+
45+ println!("{}", pairs_sum!(2 => 3, 4 => 5));
46+}
new file mode 100644
@@ -0,0 +1,46 @@
1+// `$(..)` repetition — 28% of the `macro_rules!` in a 400-crate sample. The
2+// matcher binds each fragment once per iteration, and the transcriber emits
3+// its body once per iteration, joined by the separator.
4+
5+macro_rules! sum_all {
6+ ($($x:expr),*) => {
7+ 0 $(+ $x)*
8+ };
9+}
10+
11+macro_rules! max_of {
12+ ($first:expr $(, $rest:expr)*) => {{
13+ let mut m = $first;
14+ $( if $rest > m { m = $rest; } )*
15+ m
16+ }};
17+}
18+
19+macro_rules! count_args {
20+ ($($x:expr),*) => {
21+ 0 $(+ { let _ = $x; 1 })*
22+ };
23+}
24+
25+macro_rules! pairs_sum {
26+ ($($a:expr => $b:expr),*) => {
27+ 0 $(+ $a * $b)*
28+ };
29+}
30+
31+fn main() {
32+ println!("{}", sum_all!());
33+ println!("{}", sum_all!(1));
34+ println!("{}", sum_all!(1, 2, 3, 4));
35+ // Precedence survives: each capture is parenthesised.
36+ println!("{}", sum_all!(1 + 1, 2 * 3));
37+
38+ println!("{}", max_of!(3));
39+ println!("{}", max_of!(3, 9, 2));
40+ println!("{}", max_of!(-5, -9, -1));
41+
42+ println!("{}", count_args!(7, 8, 9));
43+ println!("{}", count_args!());
44+
45+ println!("{}", pairs_sum!(2 => 3, 4 => 5));
46+}