[llvm][mustache] Refactor tokenizer for clarity

[ilovepi]

This patch refactors the Mustache tokenizer by breaking the logic up
with helper functions to improve clarity and simplify the code.

[ilovepi]

• [llvm][mustache] Avoid extra allocations in parseSection #159199
• [llvm][mustache] Optimize accessor splitting with a single pass #159198
• [llvm][mustache] Avoid redundant saves in accessor splitting #159197
• [llvm][mustache] Use single pass when tokenizing #159196
• [llvm][mustache] Avoid extra copy for json strings #159195
• [llvm][mustache] Use BumpPtrAllocator to save ASTNodes #159194
• [llvm][mustache] Simplify debug logging #159193
• [llvm][mustache] Fix failing StandaloneIndentation test #159192
• [llvm][mustache] Introduce MustacheContext to simplify mustache APIs #159191
• [llvm][mustache] Remove out parameters from processTags() #159190
• [llvm][mustache] Refactor template rendering #159189
• [llvm][mustache] Refactor tokenizer for clarity #159188 👈 (View in Graphite)
• [llvm][mustache] Support setting delimiters in templates #159187
• [llvm][mustache] Precommit test for Set Delimiter #159186
• [llvm][mustache] Align standalone partial indentation with spec #159185
• [llvm][mustache] Precommit test for StandaloneIndentation #159184
• [llvm][mustache] Add support for Triple Mustache #159183
• [llvm][mustache] Precommit tests for Triple Mustache #159182
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[llvmbot]

@llvm/pr-subscribers-llvm-support

Author: Paul Kirth (ilovepi)

Changes

This patch refactors the Mustache tokenizer by breaking the logic up
with helper functions to improve clarity and simplify the code.

---

Full diff: https://github.com/llvm/llvm-project/pull/159188.diff

1 Files Affected:

• (modified) llvm/lib/Support/Mustache.cpp (+27-22)

diff --git a/llvm/lib/Support/Mustache.cpp b/llvm/lib/Support/Mustache.cpp
index 43ce6adbba41a..08317a7b98902 100644
--- a/llvm/lib/Support/Mustache.cpp
+++ b/llvm/lib/Support/Mustache.cpp
@@ -550,11 +550,34 @@ class Parser {
                      llvm::StringMap<SectionLambda> &SectionLambdas,
                      EscapeMap &Escapes);
 
+  void parseSection(ASTNode *Parent, ASTNode::Type Ty, const Accessor &A,
+                    llvm::StringMap<AstPtr> &Partials,
+                    llvm::StringMap<Lambda> &Lambdas,
+                    llvm::StringMap<SectionLambda> &SectionLambdas,
+                    EscapeMap &Escapes);
+
   SmallVector<Token> Tokens;
   size_t CurrentPtr;
   StringRef TemplateStr;
 };
 
+void Parser::parseSection(ASTNode *Parent, ASTNode::Type Ty, const Accessor &A,
+                          llvm::StringMap<AstPtr> &Partials,
+                          llvm::StringMap<Lambda> &Lambdas,
+                          llvm::StringMap<SectionLambda> &SectionLambdas,
+                          EscapeMap &Escapes) {
+  AstPtr CurrentNode =
+      createNode(Ty, A, Parent, Partials, Lambdas, SectionLambdas, Escapes);
+  size_t Start = CurrentPtr;
+  parseMustache(CurrentNode.get(), Partials, Lambdas, SectionLambdas, Escapes);
+  const size_t End = CurrentPtr - 1;
+  std::string RawBody;
+  for (std::size_t I = Start; I < End; I++)
+    RawBody += Tokens[I].RawBody;
+  CurrentNode->setRawBody(std::move(RawBody));
+  Parent->addChild(std::move(CurrentNode));
+}
+
 AstPtr Parser::parse(llvm::StringMap<AstPtr> &Partials,
                      llvm::StringMap<Lambda> &Lambdas,
                      llvm::StringMap<SectionLambda> &SectionLambdas,
@@ -604,31 +627,13 @@ void Parser::parseMustache(ASTNode *Parent, llvm::StringMap<AstPtr> &Partials,
       break;
     }
     case Token::Type::SectionOpen: {
-      CurrentNode = createNode(ASTNode::Section, A, Parent, Partials, Lambdas,
-                               SectionLambdas, Escapes);
-      size_t Start = CurrentPtr;
-      parseMustache(CurrentNode.get(), Partials, Lambdas, SectionLambdas,
-                    Escapes);
-      const size_t End = CurrentPtr - 1;
-      std::string RawBody;
-      for (std::size_t I = Start; I < End; I++)
-        RawBody += Tokens[I].RawBody;
-      CurrentNode->setRawBody(std::move(RawBody));
-      Parent->addChild(std::move(CurrentNode));
+      parseSection(Parent, ASTNode::Section, A, Partials, Lambdas,
+                   SectionLambdas, Escapes);
       break;
     }
     case Token::Type::InvertSectionOpen: {
-      CurrentNode = createNode(ASTNode::InvertSection, A, Parent, Partials,
-                               Lambdas, SectionLambdas, Escapes);
-      size_t Start = CurrentPtr;
-      parseMustache(CurrentNode.get(), Partials, Lambdas, SectionLambdas,
-                    Escapes);
-      const size_t End = CurrentPtr - 1;
-      std::string RawBody;
-      for (size_t Idx = Start; Idx < End; Idx++)
-        RawBody += Tokens[Idx].RawBody;
-      CurrentNode->setRawBody(std::move(RawBody));
-      Parent->addChild(std::move(CurrentNode));
+      parseSection(Parent, ASTNode::InvertSection, A, Partials, Lambdas,
+                   SectionLambdas, Escapes);
       break;
     }
     case Token::Type::Comment:

[ilovepi]

Merge activity

• Sep 29, 11:30 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Sep 30, 12:00 AM UTC: Graphite rebased this pull request as part of a merge.
• Sep 30, 12:27 AM UTC: Graphite rebased this pull request as part of a merge.
• Sep 30, 12:55 AM UTC: @ilovepi merged this pull request with Graphite.