Clang crash in preprocessor code

[eldiener]

Bugzilla Link	23469
Version	trunk
OS	Windows XP
Attachments	Crash report
CC	@DougGregor,@zygoloid

Extended Description

Testing my Boost.VMD library clang crashes with message:

clang++.exe: note: diagnostic msg: PLEASE submit a bug report to http://llvm.org/bugs/ and include the crash backtrace, preprocessed source, and associated run script.

Attached in a zip file is the information requested.

The test can be also run from Boost.VMD on the 'develop' branch to see the crash.

[llvmbot]

With an assert build I get:

Assertion `NextLocalOffset + TokLength + 1 > NextLocalOffset && NextLocalOffset + TokLength + 1 <= CurrentLoadedOffset && "Ran out of source locations!"' failed.

[eldiener]

I don't know what the assert build messages mean but it is clearly a clang bug and not anything in my original code. Shouldn't clang be looking to fix this bug in their preprocessor ?

[llvmbot]

Well the testcase just needs huge amounts of memory.
(gcc compiles it successfully but uses 20GB peak.)

It there a clang switch that increases the max. amount of source locations?

[zygoloid]

It there a clang switch that increases the max. amount of source locations?

No. (You could try switching it to a 64-bit type and recompiling, but there will likely be a lot of places you need to change, and this is obviously not a supported configuration.) ~2GB of preprocessed source per translation unit (including all intermediate levels of macro expansion) is one of clang's implementation limits. We are unlikely to increase this limit any time soon, as source locations are responsible for a significant proportion of our memory usage.

[eldiener]

It there a clang switch that increases the max. amount of source locations?

No. (You could try switching it to a 64-bit type and recompiling, but there
will likely be a lot of places you need to change, and this is obviously not
a supported configuration.) ~2GB of preprocessed source per translation unit
(including all intermediate levels of macro expansion) is one of clang's
implementation limits. We are unlikely to increase this limit any time soon,
as source locations are responsible for a significant proportion of our
memory usage.

You should still be putting out some sort of error indicating what has happened rather than have an internal compiler error.

[Endilll]

Still crashing as of post-16 trunk: https://godbolt.org/z/65K9fachf (it takes 8 seconds to crash on my machine, so I guess it exceeds CE timeouts)
Reduced by C-Reduce (it took 2 days, which is the longest I've seen up until now):

#define BOOST_PP_CAT(a, b) a##55
#define BOOST_PP_WHILE_55(p, o, s) BOOST_PP_WHILE_56(, o, o(, s))
#define BOOST_PP_WHILE_56(p, o, s) BOOST_PP_WHILE_57(, o, o(, s))
#define BOOST_PP_WHILE_57(p, o, s)                                           BOOST_PP_WHILE_58_C(,, o, s o (, s)s
#define BOOST_PP_WHILE_59(p, o, s)                                                                                         BOOST_PP_WHILE_60 (, o , o (, s)
#define BOOST_PP_WHILE_60(p, o, s) BOOST_PP_WHILE_61(, o, o(, s))
#define BOOST_PP_WHILE_61(p, o, s) BOOST_PP_WHILE_62(, o, o(, s))
#define BOOST_PP_WHILE_62(p, o, s) BOOST_PP_WHILE_62_C(, , o, s s)
#define BOOST_PP_WHILE_58_C(c, p, o, s) BOOST_PP_WHILE_59(, o, o(, s))
#define BOOST_PP_WHILE_62_C(c, p, o, s) BOOST_PP_IIF(, o(, s) s, s)
#define BOOST_VMD_DETAIL_SEQUENCE_OP(d, state)                                 \
  state BOOST_VMD_DETAIL_SEQUENCE_STATE_INPUT state                            \
  BOOST_VMD_DETAIL_SEQUENCE_STATE_ELEM(state)                                  \
      BOOST_VMD_DETAIL_SEQUENCE_STATE_OUTTYPE(state)                           \
          BOOST_VMD_DETAIL_SEQUENCE_STATE_FROM state state
#define BOOST_PP_IIF(bit, t, f) t t(t
              BOOST_PP_CAT(BOOST_PP_WHILE_ , )(, BOOST_VMD_DETAIL_SEQUENCE_OP, \    				BOOST_VMD_DETAIL_SEQUENCE_EMPTY_TYPE ))        )

Assertion:

clang: /home/tonik/llvm-project/clang/lib/Basic/SourceManager.cpp:678:
clang::SourceLocation clang::SourceManager::createExpansionLocImpl(const clang::SrcMgr::ExpansionInfo &, unsigned int, int, SourceLocation::UIntTy):
Assertion `NextLocalOffset + Length + 1 > NextLocalOffset && NextLocalOffset + Length + 1 <= CurrentLoadedOffset && "Ran out of source locations!"' failed.

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: /opt/compiler-explorer/clang-16.0.0-assert/bin/clang -cc1 -triple i686-pc-windows-gnu -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -main-file-name test_doc_modifiers_return_type.cpp -mrelocation-model static -mthread-model posix -fmath-errno -mconstructor-aliases -target-cpu i686 -fno-unique-section-names -D BOOST_ALL_NO_LIB=1 -D BOOST_VMD_TEST_GENERAL_HEADER -O0 -Wall -pedantic-errors -std=c++11 -fdeprecated-macro -ferror-limit 19 -mstackrealign -fno-use-cxa-atexit -fno-inline -fobjc-runtime=gcc -fcxx-exceptions -fexceptions -x c++ test_doc_modifiers_return_type-2c46f6.cpp
1.      test_doc_modifiers_return_type-2c46f6.cpp:17:15 <Spelling=test_doc_modifiers_return_type-2c46f6.cpp:10:41>: current parser token 'BOOST_PP_IIF'
 #0 0x000056283864c247 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x38b8247)
 #1 0x000056283864a0ce llvm::sys::RunSignalHandlers() (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x38b60ce)
 #2 0x000056283864c8ff SignalHandler(int) Signals.cpp:0:0
 #3 0x00007f2746a42520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
 #4 0x00007f2746a96a7c pthread_kill (/lib/x86_64-linux-gnu/libc.so.6+0x96a7c)
 #5 0x00007f2746a42476 gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x42476)
 #6 0x00007f2746a287f3 abort (/lib/x86_64-linux-gnu/libc.so.6+0x287f3)
 #7 0x00007f2746a2871b (/lib/x86_64-linux-gnu/libc.so.6+0x2871b)
 #8 0x00007f2746a39e96 (/lib/x86_64-linux-gnu/libc.so.6+0x39e96)
 #9 0x000056283895a91c clang::SourceManager::createExpansionLocImpl(clang::SrcMgr::ExpansionInfo const&, unsigned int, int, unsigned int) (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x3bc691c)
#10 0x000056283895a6a2 clang::SourceManager::createMacroArgExpansionLoc(clang::SourceLocation, clang::SourceLocation, unsigned int) (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x3bc66a2)
#11 0x000056283ba5f4d5 clang::TokenLexer::updateLocForMacroArgTokens(clang::SourceLocation, clang::Token*, clang::Token*) (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x6ccb4d5)
#12 0x000056283ba5d3de clang::TokenLexer::ExpandFunctionArguments() (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x6cc93de)
#13 0x000056283ba5c607 clang::TokenLexer::Init(clang::Token&, clang::SourceLocation, clang::MacroInfo*, clang::MacroArgs*) (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x6cc8607)
#14 0x000056283ba2eecb clang::Preprocessor::EnterMacro(clang::Token&, clang::SourceLocation, clang::MacroInfo*, clang::MacroArgs*) (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x6c9aecb)
#15 0x000056283ba36a26 clang::Preprocessor::HandleMacroExpandedIdentifier(clang::Token&, clang::MacroDefinition const&) (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x6ca2a26)
#16 0x000056283ba57bf2 clang::Preprocessor::HandleIdentifier(clang::Token&) (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x6cc3bf2)
#17 0x000056283ba580da clang::Preprocessor::Lex(clang::Token&) (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x6cc40da)
#18 0x000056283a4d0ec8 clang::Parser::Initialize() (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x573cec8)
#19 0x000056283a4cbf41 clang::ParseAST(clang::Sema&, bool, bool) (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x5737f41)
#20 0x00005628390e8f60 clang::FrontendAction::Execute() (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x4354f60)
#21 0x000056283905c89f clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x42c889f)
#22 0x00005628391ba793 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x4426793)
#23 0x000056283621524b cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x148124b)
#24 0x0000562836211a6b ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&) driver.cpp:0:0
#25 0x00005628362113b6 clang_main(int, char**) (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x147d3b6)
#26 0x00007f2746a29d90 (/lib/x86_64-linux-gnu/libc.so.6+0x29d90)
#27 0x00007f2746a29e40 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e40)
#28 0x000056283620db9e _start (/opt/compiler-explorer/clang-16.0.0-assert/bin/clang+0x1479b9e)
[1]    3467322 IOT instruction (core dumped)  /opt/compiler-explorer/clang-16.0.0-assert/bin/clang "-cc1" "-triple"

[llvmbot]

@llvm/issue-subscribers-clang-frontend

[Endilll]

Possibly related: #29894