Expand multi-rule and recursive macro_rules
Rules are tried top to bottom and the first whose matcher matches wins, which is Rust's rule and is also what makes a recursive macro terminate: a base case sits above the recursive rule. Recursion then falls out for free, since an expansion containing another macro call is lowered like any expression and so expands again. A 64-level limit turns a macro that does not terminate into an error rather than a hang. That takes the expander to 1,369 of the 1,833 macro_rules! in the sample, 74%. What is left is token-tree munching (420) and nested repetition (44). One bug found by the test and worth recording, because it was silent rather than loud: a fragment must not span a top-level `,` or `;`. Without that a greedy `$x:expr` swallows the whole invocation, so `describe!(3, 4)` matched the *one-argument* rule and took the wrong branch with no error anywhere. A comma inside brackets lives in a Group token and is not at that level at all, which is what makes the rule safe. The survey is unchanged at 18 of 400 accepted and 12 Nim-compilable, as it was for repetition. Eight features running with that result. The remaining blockers name the reason: unsupported types (86) and calls into dependencies (49), which is the unbounded tail rather than the finite feature pool. 45 differential cases, all green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
d459281 parent: 3d4a7d2 modified
DESIGN.md +28 -16 | @@ -630,25 +630,29 @@ which is exactly the guessing the project refuses. Expanding at the call site | ||
| 630 | 630 | yields ordinary Rust in a context where the types are known, so it lowers like |
| 631 | 631 | anything else. Same applicability, faithful output. |
| 632 | 632 | |
| 633 | -`src/mrules.rs` implements single-rule macros, **with `$(..)` repetition**. | |
| 634 | -Recounting the 1,833 definitions by the expander's actual boundary — one rule, | |
| 635 | -any repetition, no `:tt`: | |
| 633 | +`src/mrules.rs` handles any number of rules, with `$(..)` repetition. | |
| 634 | +Measured against the 1,833 definitions in the sample: | |
| 636 | 635 | |
| 637 | -| | count | | |
| 638 | -|---|---| | |
| 639 | -| single rule, no repetition | 661 | | |
| 640 | -| single rule, with repetition | 271 | | |
| 641 | -| **expandable** | **932 (51%)** | | |
| 642 | -| multiple rules | 481 | | |
| 643 | -| `:tt` | 420 | | |
| 636 | +| | count | | | |
| 637 | +|---|---|---| | |
| 638 | +| **expandable** | **1,369** | **74%** | | |
| 639 | +| nested `$(..)` inside `$(..)` | 44 | 2% | | |
| 640 | +| `:tt` token-tree munching | 420 | 22% | | |
| 641 | + | |
| 642 | +Rules are tried top to bottom and the first whose matcher matches wins, which | |
| 643 | +is Rust's rule and is also what makes a recursive macro terminate — a base | |
| 644 | +case sits above the recursive rule. Recursion falls out for free: an expansion | |
| 645 | +containing another macro call is lowered like any expression, so it expands | |
| 646 | +again, with a 64-level limit so a non-terminating macro is an error rather | |
| 647 | +than a hang. | |
| 644 | 648 | |
| 645 | 649 | A definition it cannot handle is recorded *with its reason*, so a call site |
| 646 | 650 | says "`foo!` cannot be expanded: nested `$(..)` repetition is not implemented |
| 647 | 651 | yet" rather than "unknown macro". Captured fragments are parenthesised on |
| 648 | 652 | substitution, so `square!(2 + 3)` is 25 and not 11. |
| 649 | 653 | |
| 650 | -Repetition needed two things that are easy to get wrong, both caught by | |
| 651 | -`tests/cases/038`: | |
| 654 | +Three things here are easy to get wrong, all caught by `tests/cases/038` and | |
| 655 | +`039` rather than by reasoning: | |
| 652 | 656 | |
| 653 | 657 | - `>` closes `=>` and `->` as well as a generic argument list, so the nesting |
| 654 | 658 | depth used to find a fragment's end must not go negative. Without that, |
| @@ -658,6 +662,11 @@ Repetition needed two things that are easy to get wrong, both caught by | ||
| 658 | 662 | leads each iteration. A trailing fragment therefore stops at the separator |
| 659 | 663 | when there is one and at whatever starts the next iteration when there is |
| 660 | 664 | not. |
| 665 | +- **A fragment never spans a top-level `,` or `;`.** Without that, a greedy | |
| 666 | + `$x:expr` swallows the whole invocation and `describe!(3, 4)` matches the | |
| 667 | + *one-argument* rule, silently taking the wrong branch. A comma inside | |
| 668 | + brackets is within a `Group` token and so is not at this level at all, | |
| 669 | + which is why the rule is safe. | |
| 661 | 670 | |
| 662 | 671 | A block expression with statements (`{{ let mut m = ..; m }}`, which is how |
| 663 | 672 | these macros are usually written) now lowers to Nim's `block:` expression |
| @@ -668,10 +677,13 @@ not collide in one scope. | ||
| 668 | 677 | of 400 crates accepted.** A `macro_rules!` used to be a hard stop at item |
| 669 | 678 | level, failing a whole crate on sight. |
| 670 | 679 | |
| 671 | -Repetition then moved it by **zero** — 18 before, 18 after — for the reason | |
| 672 | -every previous feature did: the crates it unblocked hit their next blocker. | |
| 673 | -That is now seven features running. The capability is real and tested; the | |
| 674 | -crate count is gated by something else. | |
| 680 | +Repetition and multi-rule then each moved it by **zero** — 18 before, 18 | |
| 681 | +after, 12 compilable throughout — for the reason every previous feature did: | |
| 682 | +the crates they unblocked hit their next blocker. That is now eight features | |
| 683 | +running. The capabilities are real and tested; the crate count is gated by | |
| 684 | +something else, and the remaining blockers say what: unsupported types (86) | |
| 685 | +and calls into dependencies (49), which is the unbounded tail rather than the | |
| 686 | +finite feature pool. | |
| 675 | 687 | |
| 676 | 688 | ### A stricter number |
| 677 | 689 | |
| @@ -630,25 +630,29 @@ which is exactly the guessing the project refuses. Expanding at the call site | |||
| 630 | yields ordinary Rust in a context where the types are known, so it lowers like | 630 | yields ordinary Rust in a context where the types are known, so it lowers like |
| 631 | anything else. Same applicability, faithful output. | 631 | anything else. Same applicability, faithful output. |
| 632 | 632 | ||
| 633 | -`src/mrules.rs` implements single-rule macros, **with `$(..)` repetition**. | 633 | +`src/mrules.rs` handles any number of rules, with `$(..)` repetition. |
| 634 | -Recounting the 1,833 definitions by the expander's actual boundary — one rule, | 634 | +Measured against the 1,833 definitions in the sample: |
| 635 | -any repetition, no `:tt`: | ||
| 636 | 635 | ||
| 637 | -| | count | | 636 | +| | count | | |
| 638 | -|---|---| | 637 | +|---|---|---| |
| 639 | -| single rule, no repetition | 661 | | 638 | +| **expandable** | **1,369** | **74%** | |
| 640 | -| single rule, with repetition | 271 | | 639 | +| nested `$(..)` inside `$(..)` | 44 | 2% | |
| 641 | -| **expandable** | **932 (51%)** | | 640 | +| `:tt` token-tree munching | 420 | 22% | |
| 642 | -| multiple rules | 481 | | 641 | + |
| 643 | -| `:tt` | 420 | | 642 | +Rules are tried top to bottom and the first whose matcher matches wins, which |
| 643 | +is Rust's rule and is also what makes a recursive macro terminate — a base | ||
| 644 | +case sits above the recursive rule. Recursion falls out for free: an expansion | ||
| 645 | +containing another macro call is lowered like any expression, so it expands | ||
| 646 | +again, with a 64-level limit so a non-terminating macro is an error rather | ||
| 647 | +than a hang. | ||
| 644 | 648 | ||
| 645 | A definition it cannot handle is recorded *with its reason*, so a call site | 649 | A definition it cannot handle is recorded *with its reason*, so a call site |
| 646 | says "`foo!` cannot be expanded: nested `$(..)` repetition is not implemented | 650 | says "`foo!` cannot be expanded: nested `$(..)` repetition is not implemented |
| 647 | yet" rather than "unknown macro". Captured fragments are parenthesised on | 651 | yet" rather than "unknown macro". Captured fragments are parenthesised on |
| 648 | substitution, so `square!(2 + 3)` is 25 and not 11. | 652 | substitution, so `square!(2 + 3)` is 25 and not 11. |
| 649 | 653 | ||
| 650 | -Repetition needed two things that are easy to get wrong, both caught by | 654 | +Three things here are easy to get wrong, all caught by `tests/cases/038` and |
| 651 | -`tests/cases/038`: | 655 | +`039` rather than by reasoning: |
| 652 | 656 | ||
| 653 | - `>` closes `=>` and `->` as well as a generic argument list, so the nesting | 657 | - `>` closes `=>` and `->` as well as a generic argument list, so the nesting |
| 654 | depth used to find a fragment's end must not go negative. Without that, | 658 | depth used to find a fragment's end must not go negative. Without that, |
| @@ -658,6 +662,11 @@ Repetition needed two things that are easy to get wrong, both caught by | |||
| 658 | leads each iteration. A trailing fragment therefore stops at the separator | 662 | leads each iteration. A trailing fragment therefore stops at the separator |
| 659 | when there is one and at whatever starts the next iteration when there is | 663 | when there is one and at whatever starts the next iteration when there is |
| 660 | not. | 664 | not. |
| 665 | +- **A fragment never spans a top-level `,` or `;`.** Without that, a greedy | ||
| 666 | + `$x:expr` swallows the whole invocation and `describe!(3, 4)` matches the | ||
| 667 | + *one-argument* rule, silently taking the wrong branch. A comma inside | ||
| 668 | + brackets is within a `Group` token and so is not at this level at all, | ||
| 669 | + which is why the rule is safe. | ||
| 661 | 670 | ||
| 662 | A block expression with statements (`{{ let mut m = ..; m }}`, which is how | 671 | A block expression with statements (`{{ let mut m = ..; m }}`, which is how |
| 663 | these macros are usually written) now lowers to Nim's `block:` expression | 672 | these macros are usually written) now lowers to Nim's `block:` expression |
| @@ -668,10 +677,13 @@ not collide in one scope. | |||
| 668 | of 400 crates accepted.** A `macro_rules!` used to be a hard stop at item | 677 | of 400 crates accepted.** A `macro_rules!` used to be a hard stop at item |
| 669 | level, failing a whole crate on sight. | 678 | level, failing a whole crate on sight. |
| 670 | 679 | ||
| 671 | -Repetition then moved it by **zero** — 18 before, 18 after — for the reason | 680 | +Repetition and multi-rule then each moved it by **zero** — 18 before, 18 |
| 672 | -every previous feature did: the crates it unblocked hit their next blocker. | 681 | +after, 12 compilable throughout — for the reason every previous feature did: |
| 673 | -That is now seven features running. The capability is real and tested; the | 682 | +the crates they unblocked hit their next blocker. That is now eight features |
| 674 | -crate count is gated by something else. | 683 | +running. The capabilities are real and tested; the crate count is gated by |
| 684 | +something else, and the remaining blockers say what: unsupported types (86) | ||
| 685 | +and calls into dependencies (49), which is the unbounded tail rather than the | ||
| 686 | +finite feature pool. | ||
| 675 | 687 | ||
| 676 | ### A stricter number | 688 | ### A stricter number |
| 677 | 689 | ||
modified
README.md +3 -3 | @@ -111,9 +111,9 @@ A `macro_rules!` is **expanded at the call site**, not translated into a Nim | ||
| 111 | 111 | template — but a template body is untyped, and this lowering is type-directed |
| 112 | 112 | throughout. Expanding gives ordinary Rust where the types are known. |
| 113 | 113 | |
| 114 | -Single-rule macros are supported, with `$(..)` repetition — 932 of the 1,833 | |
| 115 | -definitions in that sample, 51%. The rest are multi-rule (481) or token-tree | |
| 116 | -munchers (420). | |
| 114 | +Any number of rules, with `$(..)` repetition and recursion — 1,369 of the | |
| 115 | +1,833 definitions in that sample, 74%. What remains is token-tree munching | |
| 116 | +(420) and nested repetition (44). | |
| 117 | 117 | |
| 118 | 118 | Expansion took the survey from 3 to 18 of 400 crates accepted, of which 12 |
| 119 | 119 | produce Nim the Nim compiler accepts. (Accepted, compiles, and |
| @@ -111,9 +111,9 @@ A `macro_rules!` is **expanded at the call site**, not translated into a Nim | |||
| 111 | template — but a template body is untyped, and this lowering is type-directed | 111 | template — but a template body is untyped, and this lowering is type-directed |
| 112 | throughout. Expanding gives ordinary Rust where the types are known. | 112 | throughout. Expanding gives ordinary Rust where the types are known. |
| 113 | 113 | ||
| 114 | -Single-rule macros are supported, with `$(..)` repetition — 932 of the 1,833 | 114 | +Any number of rules, with `$(..)` repetition and recursion — 1,369 of the |
| 115 | -definitions in that sample, 51%. The rest are multi-rule (481) or token-tree | 115 | +1,833 definitions in that sample, 74%. What remains is token-tree munching |
| 116 | -munchers (420). | 116 | +(420) and nested repetition (44). |
| 117 | 117 | ||
| 118 | Expansion took the survey from 3 to 18 of 400 crates accepted, of which 12 | 118 | Expansion took the survey from 3 to 18 of 400 crates accepted, of which 12 |
| 119 | produce Nim the Nim compiler accepts. (Accepted, compiles, and | 119 | produce Nim the Nim compiler accepts. (Accepted, compiles, and |
modified
src/lower.rs +15 -1 | @@ -224,6 +224,7 @@ pub struct Lowerer { | ||
| 224 | 224 | /// cannot, so a call site can say *why* rather than "unknown macro". |
| 225 | 225 | mrules: HashMap<String, crate::mrules::MacroDef>, |
| 226 | 226 | mrules_bad: HashMap<String, String>, |
| 227 | + macro_depth: usize, | |
| 227 | 228 | /// Types declared by a `bitflags!` invocation. |
| 228 | 229 | bitflags: std::collections::HashSet<String>, |
| 229 | 230 | /// `(type, flag) -> nim const name`. |
| @@ -307,6 +308,7 @@ impl Lowerer { | ||
| 307 | 308 | const_ptrs: std::collections::HashSet::new(), |
| 308 | 309 | mrules: HashMap::new(), |
| 309 | 310 | mrules_bad: HashMap::new(), |
| 311 | + macro_depth: 0, | |
| 310 | 312 | bitflags: std::collections::HashSet::new(), |
| 311 | 313 | flag_consts: HashMap::new(), |
| 312 | 314 | use_map: HashMap::new(), |
| @@ -5086,6 +5088,15 @@ impl Lowerer { | ||
| 5086 | 5088 | return Err(format!("`{other}!` cannot be expanded: {why}")); |
| 5087 | 5089 | } |
| 5088 | 5090 | if let Some(def) = self.mrules.get(other).cloned() { |
| 5091 | + // A multi-rule macro commonly recurses, shedding one | |
| 5092 | + // argument per step. The depth limit turns a macro that | |
| 5093 | + // does not terminate into an error rather than a hang. | |
| 5094 | + if self.macro_depth > 64 { | |
| 5095 | + return Err(format!( | |
| 5096 | + "`{other}!` expanded more than 64 levels deep; it \ | |
| 5097 | + does not appear to terminate" | |
| 5098 | + )); | |
| 5099 | + } | |
| 5089 | 5100 | let expanded = def |
| 5090 | 5101 | .expand(mac.tokens.clone()) |
| 5091 | 5102 | .map_err(|e| format!("expanding `{other}!`: {e}"))?; |
| @@ -5098,7 +5109,10 @@ impl Lowerer { | ||
| 5098 | 5109 | expanded |
| 5099 | 5110 | ) |
| 5100 | 5111 | })?; |
| 5101 | - return Ok(self.expr(&e)?.code); | |
| 5112 | + self.macro_depth += 1; | |
| 5113 | + let r = self.expr(&e); | |
| 5114 | + self.macro_depth -= 1; | |
| 5115 | + return Ok(r?.code); | |
| 5102 | 5116 | } |
| 5103 | 5117 | Err(format!( |
| 5104 | 5118 | "unsupported macro `{other}!`; a macro whose expansion is not \ |
| @@ -224,6 +224,7 @@ pub struct Lowerer { | |||
| 224 | /// cannot, so a call site can say *why* rather than "unknown macro". | 224 | /// cannot, so a call site can say *why* rather than "unknown macro". |
| 225 | mrules: HashMap<String, crate::mrules::MacroDef>, | 225 | mrules: HashMap<String, crate::mrules::MacroDef>, |
| 226 | mrules_bad: HashMap<String, String>, | 226 | mrules_bad: HashMap<String, String>, |
| 227 | + macro_depth: usize, | ||
| 227 | /// Types declared by a `bitflags!` invocation. | 228 | /// Types declared by a `bitflags!` invocation. |
| 228 | bitflags: std::collections::HashSet<String>, | 229 | bitflags: std::collections::HashSet<String>, |
| 229 | /// `(type, flag) -> nim const name`. | 230 | /// `(type, flag) -> nim const name`. |
| @@ -307,6 +308,7 @@ impl Lowerer { | |||
| 307 | const_ptrs: std::collections::HashSet::new(), | 308 | const_ptrs: std::collections::HashSet::new(), |
| 308 | mrules: HashMap::new(), | 309 | mrules: HashMap::new(), |
| 309 | mrules_bad: HashMap::new(), | 310 | mrules_bad: HashMap::new(), |
| 311 | + macro_depth: 0, | ||
| 310 | bitflags: std::collections::HashSet::new(), | 312 | bitflags: std::collections::HashSet::new(), |
| 311 | flag_consts: HashMap::new(), | 313 | flag_consts: HashMap::new(), |
| 312 | use_map: HashMap::new(), | 314 | use_map: HashMap::new(), |
| @@ -5086,6 +5088,15 @@ impl Lowerer { | |||
| 5086 | return Err(format!("`{other}!` cannot be expanded: {why}")); | 5088 | return Err(format!("`{other}!` cannot be expanded: {why}")); |
| 5087 | } | 5089 | } |
| 5088 | if let Some(def) = self.mrules.get(other).cloned() { | 5090 | if let Some(def) = self.mrules.get(other).cloned() { |
| 5091 | + // A multi-rule macro commonly recurses, shedding one | ||
| 5092 | + // argument per step. The depth limit turns a macro that | ||
| 5093 | + // does not terminate into an error rather than a hang. | ||
| 5094 | + if self.macro_depth > 64 { | ||
| 5095 | + return Err(format!( | ||
| 5096 | + "`{other}!` expanded more than 64 levels deep; it \ | ||
| 5097 | + does not appear to terminate" | ||
| 5098 | + )); | ||
| 5099 | + } | ||
| 5089 | let expanded = def | 5100 | let expanded = def |
| 5090 | .expand(mac.tokens.clone()) | 5101 | .expand(mac.tokens.clone()) |
| 5091 | .map_err(|e| format!("expanding `{other}!`: {e}"))?; | 5102 | .map_err(|e| format!("expanding `{other}!`: {e}"))?; |
| @@ -5098,7 +5109,10 @@ impl Lowerer { | |||
| 5098 | expanded | 5109 | expanded |
| 5099 | ) | 5110 | ) |
| 5100 | })?; | 5111 | })?; |
| 5101 | - return Ok(self.expr(&e)?.code); | 5112 | + self.macro_depth += 1; |
| 5113 | + let r = self.expr(&e); | ||
| 5114 | + self.macro_depth -= 1; | ||
| 5115 | + return Ok(r?.code); | ||
| 5102 | } | 5116 | } |
| 5103 | Err(format!( | 5117 | Err(format!( |
| 5104 | "unsupported macro `{other}!`; a macro whose expansion is not \ | 5118 | "unsupported macro `{other}!`; a macro whose expansion is not \ |
modified
src/mrules.rs +62 -25 | @@ -42,31 +42,48 @@ enum Cap { | ||
| 42 | 42 | type Binds = HashMap<String, Cap>; |
| 43 | 43 | |
| 44 | 44 | #[derive(Debug, Clone)] |
| 45 | -pub struct MacroDef { | |
| 45 | +struct Rule { | |
| 46 | 46 | pattern: Vec<Pat>, |
| 47 | 47 | body: TokenStream, |
| 48 | 48 | } |
| 49 | 49 | |
| 50 | -/// Parse `macro_rules!`'s body: `( $matcher ) => { $transcriber };`. | |
| 50 | +#[derive(Debug, Clone)] | |
| 51 | +pub struct MacroDef { | |
| 52 | + /// Rules in source order. Rust tries them top to bottom and the first | |
| 53 | + /// whose matcher matches wins, so order is semantics, not style. | |
| 54 | + rules: Vec<Rule>, | |
| 55 | +} | |
| 56 | + | |
| 57 | +/// Parse `macro_rules!`'s body: a `;`-separated list of | |
| 58 | +/// `( $matcher ) => { $transcriber }`. | |
| 51 | 59 | pub fn parse(tokens: TokenStream) -> Result<MacroDef, String> { |
| 52 | 60 | let t: Vec<TokenTree> = tokens.into_iter().collect(); |
| 53 | - // matcher group, `=`, `>`, transcriber group, optional `;` | |
| 54 | - let (matcher, rest) = match t.split_first() { | |
| 55 | - Some((TokenTree::Group(g), rest)) => (g.clone(), rest), | |
| 56 | - _ => return Err("expected a matcher group".into()), | |
| 57 | - }; | |
| 58 | - let arrow: String = rest.iter().take(2).map(|t| t.to_string()).collect(); | |
| 59 | - if arrow != "=>" { | |
| 60 | - return Err("expected `=>`".into()); | |
| 61 | + let mut rules = Vec::new(); | |
| 62 | + let mut i = 0; | |
| 63 | + while i < t.len() { | |
| 64 | + let matcher = match &t[i] { | |
| 65 | + TokenTree::Group(g) => g.clone(), | |
| 66 | + other => return Err(format!("expected a matcher group, found `{other}`")), | |
| 67 | + }; | |
| 68 | + let arrow: String = t[i + 1..].iter().take(2).map(|t| t.to_string()).collect(); | |
| 69 | + if arrow != "=>" { | |
| 70 | + return Err("expected `=>` after a matcher".into()); | |
| 71 | + } | |
| 72 | + let body = match t.get(i + 3) { | |
| 73 | + Some(TokenTree::Group(g)) => g.stream(), | |
| 74 | + _ => return Err("expected a transcriber group".into()), | |
| 75 | + }; | |
| 76 | + rules.push(Rule { pattern: parse_pattern(matcher.stream())?, body }); | |
| 77 | + i += 4; | |
| 78 | + // The separator is optional after the last rule. | |
| 79 | + if i < t.len() && t[i].to_string() == ";" { | |
| 80 | + i += 1; | |
| 81 | + } | |
| 61 | 82 | } |
| 62 | - let body = match rest.get(2) { | |
| 63 | - Some(TokenTree::Group(g)) => g.stream(), | |
| 64 | - _ => return Err("expected a transcriber group".into()), | |
| 65 | - }; | |
| 66 | - if rest.len() > 4 || (rest.len() == 4 && rest[3].to_string() != ";") { | |
| 67 | - return Err("more than one rule is not implemented yet".into()); | |
| 83 | + if rules.is_empty() { | |
| 84 | + return Err("a macro with no rules".into()); | |
| 68 | 85 | } |
| 69 | - Ok(MacroDef { pattern: parse_pattern(matcher.stream())?, body }) | |
| 86 | + Ok(MacroDef { rules }) | |
| 70 | 87 | } |
| 71 | 88 | |
| 72 | 89 | fn parse_pattern(ts: TokenStream) -> Result<Vec<Pat>, String> { |
| @@ -136,15 +153,25 @@ fn parse_pattern(ts: TokenStream) -> Result<Vec<Pat>, String> { | ||
| 136 | 153 | } |
| 137 | 154 | |
| 138 | 155 | impl MacroDef { |
| 139 | - /// Match an invocation's tokens and substitute them into the body. | |
| 156 | + /// Try each rule in order and substitute using the first that matches, | |
| 157 | + /// which is what Rust does. | |
| 140 | 158 | pub fn expand(&self, input: TokenStream) -> Result<TokenStream, String> { |
| 141 | - let mut binds = Binds::new(); | |
| 142 | 159 | let toks: Vec<TokenTree> = input.into_iter().collect(); |
| 143 | - let used = match_seq(&self.pattern, &toks, &mut binds, None)?; | |
| 144 | - if used != toks.len() { | |
| 145 | - return Err("the invocation has tokens the matcher does not consume".into()); | |
| 160 | + let mut why = Vec::new(); | |
| 161 | + for (n, rule) in self.rules.iter().enumerate() { | |
| 162 | + let mut binds = Binds::new(); | |
| 163 | + match match_seq(&rule.pattern, &toks, &mut binds, None) { | |
| 164 | + Ok(used) if used == toks.len() => { | |
| 165 | + return Ok(substitute(rule.body.clone(), &binds)) | |
| 166 | + } | |
| 167 | + Ok(_) => why.push(format!( | |
| 168 | + "rule {}: matched, but left tokens over", | |
| 169 | + n + 1 | |
| 170 | + )), | |
| 171 | + Err(e) => why.push(format!("rule {}: {e}", n + 1)), | |
| 172 | + } | |
| 146 | 173 | } |
| 147 | - Ok(substitute(self.body.clone(), &binds)) | |
| 174 | + Err(format!("no rule matched ({})", why.join("; "))) | |
| 148 | 175 | } |
| 149 | 176 | } |
| 150 | 177 | |
| @@ -241,8 +268,18 @@ fn match_seq( | ||
| 241 | 268 | let mut depth = 0i32; |
| 242 | 269 | while i < toks.len() { |
| 243 | 270 | let s = toks[i].to_string(); |
| 244 | - if let Some(stop) = &stop { | |
| 245 | - if depth == 0 && s == *stop { | |
| 271 | + if depth == 0 { | |
| 272 | + if let Some(stop) = &stop { | |
| 273 | + if s == *stop { | |
| 274 | + break; | |
| 275 | + } | |
| 276 | + } | |
| 277 | + // A fragment never spans a top-level `,` or `;`: an | |
| 278 | + // `expr` is one expression, and a comma at this level | |
| 279 | + // separates arguments rather than belonging to one. | |
| 280 | + // A comma inside brackets is within a `Group` token, | |
| 281 | + // so it is not at this level at all. | |
| 282 | + if s == "," || s == ";" { | |
| 246 | 283 | break; |
| 247 | 284 | } |
| 248 | 285 | } |
| @@ -42,31 +42,48 @@ enum Cap { | |||
| 42 | type Binds = HashMap<String, Cap>; | 42 | type Binds = HashMap<String, Cap>; |
| 43 | 43 | ||
| 44 | #[derive(Debug, Clone)] | 44 | #[derive(Debug, Clone)] |
| 45 | -pub struct MacroDef { | 45 | +struct Rule { |
| 46 | pattern: Vec<Pat>, | 46 | pattern: Vec<Pat>, |
| 47 | body: TokenStream, | 47 | body: TokenStream, |
| 48 | } | 48 | } |
| 49 | 49 | ||
| 50 | -/// Parse `macro_rules!`'s body: `( $matcher ) => { $transcriber };`. | 50 | +#[derive(Debug, Clone)] |
| 51 | +pub struct MacroDef { | ||
| 52 | + /// Rules in source order. Rust tries them top to bottom and the first | ||
| 53 | + /// whose matcher matches wins, so order is semantics, not style. | ||
| 54 | + rules: Vec<Rule>, | ||
| 55 | +} | ||
| 56 | + | ||
| 57 | +/// Parse `macro_rules!`'s body: a `;`-separated list of | ||
| 58 | +/// `( $matcher ) => { $transcriber }`. | ||
| 51 | pub fn parse(tokens: TokenStream) -> Result<MacroDef, String> { | 59 | pub fn parse(tokens: TokenStream) -> Result<MacroDef, String> { |
| 52 | let t: Vec<TokenTree> = tokens.into_iter().collect(); | 60 | let t: Vec<TokenTree> = tokens.into_iter().collect(); |
| 53 | - // matcher group, `=`, `>`, transcriber group, optional `;` | 61 | + let mut rules = Vec::new(); |
| 54 | - let (matcher, rest) = match t.split_first() { | 62 | + let mut i = 0; |
| 55 | - Some((TokenTree::Group(g), rest)) => (g.clone(), rest), | 63 | + while i < t.len() { |
| 56 | - _ => return Err("expected a matcher group".into()), | 64 | + let matcher = match &t[i] { |
| 57 | - }; | 65 | + TokenTree::Group(g) => g.clone(), |
| 58 | - let arrow: String = rest.iter().take(2).map(|t| t.to_string()).collect(); | 66 | + other => return Err(format!("expected a matcher group, found `{other}`")), |
| 59 | - if arrow != "=>" { | 67 | + }; |
| 60 | - return Err("expected `=>`".into()); | 68 | + let arrow: String = t[i + 1..].iter().take(2).map(|t| t.to_string()).collect(); |
| 69 | + if arrow != "=>" { | ||
| 70 | + return Err("expected `=>` after a matcher".into()); | ||
| 71 | + } | ||
| 72 | + let body = match t.get(i + 3) { | ||
| 73 | + Some(TokenTree::Group(g)) => g.stream(), | ||
| 74 | + _ => return Err("expected a transcriber group".into()), | ||
| 75 | + }; | ||
| 76 | + rules.push(Rule { pattern: parse_pattern(matcher.stream())?, body }); | ||
| 77 | + i += 4; | ||
| 78 | + // The separator is optional after the last rule. | ||
| 79 | + if i < t.len() && t[i].to_string() == ";" { | ||
| 80 | + i += 1; | ||
| 81 | + } | ||
| 61 | } | 82 | } |
| 62 | - let body = match rest.get(2) { | 83 | + if rules.is_empty() { |
| 63 | - Some(TokenTree::Group(g)) => g.stream(), | 84 | + return Err("a macro with no rules".into()); |
| 64 | - _ => return Err("expected a transcriber group".into()), | ||
| 65 | - }; | ||
| 66 | - if rest.len() > 4 || (rest.len() == 4 && rest[3].to_string() != ";") { | ||
| 67 | - return Err("more than one rule is not implemented yet".into()); | ||
| 68 | } | 85 | } |
| 69 | - Ok(MacroDef { pattern: parse_pattern(matcher.stream())?, body }) | 86 | + Ok(MacroDef { rules }) |
| 70 | } | 87 | } |
| 71 | 88 | ||
| 72 | fn parse_pattern(ts: TokenStream) -> Result<Vec<Pat>, String> { | 89 | fn parse_pattern(ts: TokenStream) -> Result<Vec<Pat>, String> { |
| @@ -136,15 +153,25 @@ fn parse_pattern(ts: TokenStream) -> Result<Vec<Pat>, String> { | |||
| 136 | } | 153 | } |
| 137 | 154 | ||
| 138 | impl MacroDef { | 155 | impl MacroDef { |
| 139 | - /// Match an invocation's tokens and substitute them into the body. | 156 | + /// Try each rule in order and substitute using the first that matches, |
| 157 | + /// which is what Rust does. | ||
| 140 | pub fn expand(&self, input: TokenStream) -> Result<TokenStream, String> { | 158 | pub fn expand(&self, input: TokenStream) -> Result<TokenStream, String> { |
| 141 | - let mut binds = Binds::new(); | ||
| 142 | let toks: Vec<TokenTree> = input.into_iter().collect(); | 159 | let toks: Vec<TokenTree> = input.into_iter().collect(); |
| 143 | - let used = match_seq(&self.pattern, &toks, &mut binds, None)?; | 160 | + let mut why = Vec::new(); |
| 144 | - if used != toks.len() { | 161 | + for (n, rule) in self.rules.iter().enumerate() { |
| 145 | - return Err("the invocation has tokens the matcher does not consume".into()); | 162 | + let mut binds = Binds::new(); |
| 163 | + match match_seq(&rule.pattern, &toks, &mut binds, None) { | ||
| 164 | + Ok(used) if used == toks.len() => { | ||
| 165 | + return Ok(substitute(rule.body.clone(), &binds)) | ||
| 166 | + } | ||
| 167 | + Ok(_) => why.push(format!( | ||
| 168 | + "rule {}: matched, but left tokens over", | ||
| 169 | + n + 1 | ||
| 170 | + )), | ||
| 171 | + Err(e) => why.push(format!("rule {}: {e}", n + 1)), | ||
| 172 | + } | ||
| 146 | } | 173 | } |
| 147 | - Ok(substitute(self.body.clone(), &binds)) | 174 | + Err(format!("no rule matched ({})", why.join("; "))) |
| 148 | } | 175 | } |
| 149 | } | 176 | } |
| 150 | 177 | ||
| @@ -241,8 +268,18 @@ fn match_seq( | |||
| 241 | let mut depth = 0i32; | 268 | let mut depth = 0i32; |
| 242 | while i < toks.len() { | 269 | while i < toks.len() { |
| 243 | let s = toks[i].to_string(); | 270 | let s = toks[i].to_string(); |
| 244 | - if let Some(stop) = &stop { | 271 | + if depth == 0 { |
| 245 | - if depth == 0 && s == *stop { | 272 | + if let Some(stop) = &stop { |
| 273 | + if s == *stop { | ||
| 274 | + break; | ||
| 275 | + } | ||
| 276 | + } | ||
| 277 | + // A fragment never spans a top-level `,` or `;`: an | ||
| 278 | + // `expr` is one expression, and a comma at this level | ||
| 279 | + // separates arguments rather than belonging to one. | ||
| 280 | + // A comma inside brackets is within a `Group` token, | ||
| 281 | + // so it is not at this level at all. | ||
| 282 | + if s == "," || s == ";" { | ||
| 246 | break; | 283 | break; |
| 247 | } | 284 | } |
| 248 | } | 285 | } |
added
tests/cases/039-macro-multi-rule.rs +44 -0 | new file mode 100644 | ||
| @@ -0,0 +1,44 @@ | ||
| 1 | +// Rules are tried top to bottom and the first whose matcher matches wins, so | |
| 2 | +// their order is semantics rather than style. That is also what makes a | |
| 3 | +// recursive macro terminate: a base-case rule sits above the recursive one. | |
| 4 | + | |
| 5 | +macro_rules! describe { | |
| 6 | + () => { 0 }; | |
| 7 | + ($x:expr) => { $x }; | |
| 8 | + ($x:expr, $y:expr) => { $x * 100 + $y }; | |
| 9 | +} | |
| 10 | + | |
| 11 | +// Recursion, shedding one argument per step. | |
| 12 | +macro_rules! total { | |
| 13 | + () => { 0 }; | |
| 14 | + ($x:expr) => { $x }; | |
| 15 | + ($x:expr, $($rest:expr),+) => { $x + total!($($rest),+) }; | |
| 16 | +} | |
| 17 | + | |
| 18 | +macro_rules! depth { | |
| 19 | + ($x:expr) => { 1 }; | |
| 20 | + ($x:expr, $($rest:expr),+) => { 1 + depth!($($rest),+) }; | |
| 21 | +} | |
| 22 | + | |
| 23 | +// Different shapes, not just different arities. | |
| 24 | +macro_rules! pick { | |
| 25 | + (first $a:expr, $b:expr) => { $a }; | |
| 26 | + (second $a:expr, $b:expr) => { $b }; | |
| 27 | + (sum $a:expr, $b:expr) => { $a + $b }; | |
| 28 | +} | |
| 29 | + | |
| 30 | +fn main() { | |
| 31 | + println!("{} {} {}", describe!(), describe!(7), describe!(3, 4)); | |
| 32 | + | |
| 33 | + println!("{}", total!()); | |
| 34 | + println!("{}", total!(5)); | |
| 35 | + println!("{}", total!(1, 2, 3, 4, 5)); | |
| 36 | + println!("{}", total!(2 * 3, 4 + 1)); | |
| 37 | + | |
| 38 | + println!("{} {}", depth!(9), depth!(9, 9, 9, 9)); | |
| 39 | + | |
| 40 | + println!("{} {} {}", pick!(first 10, 20), pick!(second 10, 20), pick!(sum 10, 20)); | |
| 41 | + | |
| 42 | + let a: i64 = 1000; | |
| 43 | + println!("{}", total!(a, a, a)); | |
| 44 | +} | |
| new file mode 100644 | |||
| @@ -0,0 +1,44 @@ | |||
| 1 | +// Rules are tried top to bottom and the first whose matcher matches wins, so | ||
| 2 | +// their order is semantics rather than style. That is also what makes a | ||
| 3 | +// recursive macro terminate: a base-case rule sits above the recursive one. | ||
| 4 | + | ||
| 5 | +macro_rules! describe { | ||
| 6 | + () => { 0 }; | ||
| 7 | + ($x:expr) => { $x }; | ||
| 8 | + ($x:expr, $y:expr) => { $x * 100 + $y }; | ||
| 9 | +} | ||
| 10 | + | ||
| 11 | +// Recursion, shedding one argument per step. | ||
| 12 | +macro_rules! total { | ||
| 13 | + () => { 0 }; | ||
| 14 | + ($x:expr) => { $x }; | ||
| 15 | + ($x:expr, $($rest:expr),+) => { $x + total!($($rest),+) }; | ||
| 16 | +} | ||
| 17 | + | ||
| 18 | +macro_rules! depth { | ||
| 19 | + ($x:expr) => { 1 }; | ||
| 20 | + ($x:expr, $($rest:expr),+) => { 1 + depth!($($rest),+) }; | ||
| 21 | +} | ||
| 22 | + | ||
| 23 | +// Different shapes, not just different arities. | ||
| 24 | +macro_rules! pick { | ||
| 25 | + (first $a:expr, $b:expr) => { $a }; | ||
| 26 | + (second $a:expr, $b:expr) => { $b }; | ||
| 27 | + (sum $a:expr, $b:expr) => { $a + $b }; | ||
| 28 | +} | ||
| 29 | + | ||
| 30 | +fn main() { | ||
| 31 | + println!("{} {} {}", describe!(), describe!(7), describe!(3, 4)); | ||
| 32 | + | ||
| 33 | + println!("{}", total!()); | ||
| 34 | + println!("{}", total!(5)); | ||
| 35 | + println!("{}", total!(1, 2, 3, 4, 5)); | ||
| 36 | + println!("{}", total!(2 * 3, 4 + 1)); | ||
| 37 | + | ||
| 38 | + println!("{} {}", depth!(9), depth!(9, 9, 9, 9)); | ||
| 39 | + | ||
| 40 | + println!("{} {} {}", pick!(first 10, 20), pick!(second 10, 20), pick!(sum 10, 20)); | ||
| 41 | + | ||
| 42 | + let a: i64 = 1000; | ||
| 43 | + println!("{}", total!(a, a, a)); | ||
| 44 | +} | ||