[clang-format] Bug `OneLineFormatOffRegex` broken if regex matches on first token

[sebastianmueller01]

@owenca

.clang_format:

OneLineFormatOffRegex: MACRO_TEST

test.cpp:

int main() {
        MACRO_TEST();
        MACRO_TEST();
        MACRO_TEST();
        MACRO_TEST();
        MACRO_TEST();
}

Running
clang-format-21 -i --style=file:.clang_format test.cpp

Results in the following incorrectly formatted output:

int main() {
        MACRO_TEST();
  MACRO_TEST();
        MACRO_TEST();
  MACRO_TEST();
        MACRO_TEST();
}

It seems like if we previously skipped formatting a line, then the first token of the next line is skipped from the regex match check. In the example, since the only matching token is the first in the line, we get this behavior. This doesn't occur if the token matching is at the EOL.

When transitioning from FO_CurrentLine (or FO_NextLine) to FO_None, the code breaks immediately, so the current token never gets evaluated against the regex in the default case.
https://github.com/llvm/llvm-project/pull/137577/files#diff-8e6ad02b15a65cc2ff3b2e622d25b2b45e8e900f54ce9ee79026d76115076a5eR93-R98

I think a possible solution to this would be

Also instead of evaluateCurrentToken you might also be able to look for !Tok.Finalized but I haven't read through enough of the code to confirm.

[llvmbot]

@llvm/issue-subscribers-clang-format

Author: None (sebastianmueller01)

@owenca

.clang_format:

OneLineFormatOffRegex: MACRO_TEST

test.cpp:

int main() {
        MACRO_TEST();
        MACRO_TEST();
        MACRO_TEST();
        MACRO_TEST();
        MACRO_TEST();
}

Running
clang-format-21 -i --style=file:.clang_format test.cpp

Results in the following incorrectly formatted output:

int main() {
        MACRO_TEST();
  MACRO_TEST();
        MACRO_TEST();
  MACRO_TEST();
        MACRO_TEST();
}

It seems like if we previously skipped formatting a line, then the first token of the next line is skipped from the regex match check. In the example, since the only matching token is the first in the line, we get this behavior. This doesn't occur if the token matching is at the EOL.

When transitioning from FO_CurrentLine (or FO_NextLine) to FO_None, the code breaks immediately, so the current token never gets evaluated against the regex in the default case.
https://github.com/llvm/llvm-project/pull/137577/files#diff-8e6ad02b15a65cc2ff3b2e622d25b2b45e8e900f54ce9ee79026d76115076a5eR93-R98

I think a possible solution to this would be
<img width="1360" height="1148" alt="Image" src="https://github.com/user-attachments/assets/396ee3ea-0bb3-4a15-9e6d-b86f6fb1d10a" />

Also instead of evaluateCurrentToken you might also be able to look for !Tok.Finalized but I haven't read through enough of the code to confirm.

[sebastianmueller01]

Using: Ubuntu clang-format version 21.1.3

[kars10]

Here's an alternate proposal for a potential fix:

diff --git a/clang/lib/Format/FormatTokenLexer.cpp b/clang/lib/Format/FormatTokenLexer.cpp
index 86a5185a92a5..cdcc7dab3f0c 100644
--- a/clang/lib/Format/FormatTokenLexer.cpp
+++ b/clang/lib/Format/FormatTokenLexer.cpp
@@ -87,32 +87,37 @@ FormatTokenLexer::FormatTokenLexer(
 ArrayRef<FormatToken *> FormatTokenLexer::lex() {
   assert(Tokens.empty());
   assert(FirstInLineIndex == 0);
   enum { FO_None, FO_CurrentLine, FO_NextLine } FormatOff = FO_None;
   do {
     Tokens.push_back(getNextToken());
     auto &Tok = *Tokens.back();
     const auto NewlinesBefore = Tok.NewlinesBefore;
     switch (FormatOff) {
     case FO_CurrentLine:
-      if (NewlinesBefore == 0)
-        Tok.Finalized = true;
-      else
+      if (NewlinesBefore > 0)
         FormatOff = FO_None;
       break;
     case FO_NextLine:
-      if (NewlinesBefore > 1) {
+      if (NewlinesBefore > 1)
         FormatOff = FO_None;
-      } else {
-        Tok.Finalized = true;
-        FormatOff = FO_CurrentLine;
-      }
+      break;
+    default:
+      break;
+    }
+    switch (FormatOff) {
+    case FO_CurrentLine:
+      Tok.Finalized = true;
+      break;
+    case FO_NextLine:
+      Tok.Finalized = true;
+      FormatOff = FO_CurrentLine;
       break;
     default:
       if (!FormattingDisabled && FormatOffRegex.match(Tok.TokenText)) {
         if (Tok.is(tok::comment) &&
             (NewlinesBefore > 0 || Tokens.size() == 1)) {
           Tok.Finalized = true;
           FormatOff = FO_NextLine;
         } else {
           for (auto *Token : reverse(Tokens)) {
             Token->Finalized = true;

[HazardyKnusperkeks]

@sebastianmueller01 did you post a diff as screenshot?

[owenca]

/cherry-pick 72d6d6e