provide a diagnostic for running out of source locations

[mclow]

Bugzilla Link	30546
Version	unspecified
OS	All
CC	@hfinkel,@zygoloid

[mclow]

Download Boost1.62.0 RC1 from here: https://sourceforge.net/projects/boost/files/boost/1.62.0.rc.1/boost_1_62_0_rc1.tar.gz/download

Expand the archive somewhere. I'm going to assume ~/boost.

$ cd ~/boost/status

$ clang++ -x c++ -std=c++14 -stdlib=libc++ -g -O0 -fno-inline -Wall -g -DBOOST_ALL_NO_LIB=1 -I".." -c -o "junk.o" "../libs/vmd/test/test_doc_modifiers_return_type.cpp"

Clang will run for hours on this.

if you kill the compilation, sometimes it will print out stuff about "too few arguments to macro-like invocation"

[mclow]

If I append '-E' to compile command, after a few minutes I get this output:

error: too few arguments provided to function-like macro invocation
note: macro 'BOOST_PP_VARIADIC_ELEM_3' defined here
In file included from ../libs/vmd/test/test_doc_modifiers_return_type.cpp:7:
In file included from ../libs/vmd/test/test_doc_modifiers_return_type.cxx:7:
In file included from ../boost/vmd/elem.hpp:15:
In file included from ../boost/vmd/detail/sequence_elem.hpp:11:
In file included from ../boost/preprocessor/comparison/greater.hpp:17:
In file included from ../boost/preprocessor/comparison/less.hpp:17:
In file included from ../boost/preprocessor/comparison/less_equal.hpp:17:
In file included from ../boost/preprocessor/arithmetic/sub.hpp:19:
In file included from ../boost/preprocessor/control/while.hpp:20:
In file included from ../boost/preprocessor/detail/auto_rec.hpp:21:
../boost/preprocessor/control/iif.hpp:25:64: error: too few arguments provided
      to function-like macro invocation
#    define BOOST_PP_IIF_I(bit, t, f) BOOST_PP_IIF_ ## bit(t, f)
                                                               ^
note: macro 'BOOST_PP_VARIADIC_ELEM_3' defined here
error: too few arguments provided to function-like macro invocation
note: macro 'BOOST_PP_VARIADIC_ELEM_3' defined here
../libs/vmd/test/test_doc_modifiers_return_type.cpp:2:3: error: too few
      arguments provided to function-like macro invocation
//  (C) Copyright Edward Diener 2011-2015
  ^
note: macro 'BOOST_PP_VARIADIC_ELEM_3' defined here

[mclow]

Note: In the previous comment, I provided some warning output. The compilation did not end after those warnings were output. It continued.

[zygoloid]

../libs/vmd/test/test_doc_modifiers_return_type.cpp:2:3: error: too few
arguments provided to function-like macro invocation
// (C) Copyright Edward Diener 2011-2015
^

Yikes. Either the source manager has got corrupted somehow or we ran out of source locations. Any chance this compilation is producing >1GB of preprocessed source? (How big is the -E output using GCC?)

[mclow]

$ g++ -x c++ -std=c++14 -g -O0 -fno-inline -Wall -g -DBOOST_ALL_NO_LIB=1 -I".." -E "../libs/vmd/test/test_doc_modifiers_return_type.cpp" > junk.pp

finishes in 2m 58sec, and produces an output file that is 718KB in size.
This is gcc 6.2, btw.

[mclow]

I have reproduced this with the Boost 1.62.0 release:
https://sourceforge.net/projects/boost/files/boost/1.62.0/boost_1_62_0.tar.gz

(Unsurprising, since the VMD code didn't change between RC and release, but that's a more long-lived link)

I get more error output with today's build, though:

In file included from ../libs/vmd/test/test_doc_modifiers_return_type.cpp:7:
In file included from ../libs/vmd/test/test_doc_modifiers_return_type.cxx:7:
In file included from ../boost/vmd/elem.hpp:15:
In file included from ../boost/vmd/detail/sequence_elem.hpp:11:
In file included from ../boost/preprocessor/comparison/greater.hpp:17:
In file included from ../boost/preprocessor/comparison/less.hpp:17:
In file included from ../boost/preprocessor/comparison/less_equal.hpp:17:
In file included from ../boost/preprocessor/arithmetic/sub.hpp:20:
In file included from ../boost/preprocessor/tuple/elem.hpp:23:
../boost/preprocessor/variadic/elem.hpp:26:101: error: too few arguments provided to function-like macro invocation
#        define BOOST_PP_VARIADIC_ELEM(n, ...) BOOST_PP_CAT(BOOST_PP_VARIADIC_ELEM_, n)(__VA_ARGS__,)
                                                                                                    ^
../boost/preprocessor/variadic/elem.hpp:31:13: note: macro 'BOOST_PP_VARIADIC_ELEM_3' defined here
#    define BOOST_PP_VARIADIC_ELEM_3(e0, e1, e2, e3, ...) e3
            ^
In file included from ../libs/vmd/test/test_doc_modifiers_return_type.cpp:7:
In file included from ../libs/vmd/test/test_doc_modifiers_return_type.cxx:7:
In file included from ../boost/vmd/elem.hpp:15:
In file included from ../boost/vmd/detail/sequence_elem.hpp:11:
In file included from ../boost/preprocessor/comparison/greater.hpp:17:
In file included from ../boost/preprocessor/comparison/less.hpp:17:
In file included from ../boost/preprocessor/comparison/less_equal.hpp:17:
In file included from ../boost/preprocessor/arithmetic/sub.hpp:19:
In file included from ../boost/preprocessor/control/while.hpp:20:
In file included from ../boost/preprocessor/detail/auto_rec.hpp:21:
../boost/preprocessor/control/iif.hpp:25:64: error: too few arguments provided to function-like macro invocation
#    define BOOST_PP_IIF_I(bit, t, f) BOOST_PP_IIF_ ## bit(t, f)
                                                               ^
../boost/preprocessor/variadic/elem.hpp:31:13: note: macro 'BOOST_PP_VARIADIC_ELEM_3' defined here
#    define BOOST_PP_VARIADIC_ELEM_3(e0, e1, e2, e3, ...) e3
            ^
../libs/vmd/test/test_doc_modifiers_return_type.cpp:2:10: error: too few arguments provided to function-like macro invocation
//  (C) Copyright Edward Diener 2011-2015
         ^
../boost/vmd/detail/sequence_elem.hpp:830:4: note: expanded from macro 'opyright'
        ) \
   ^
../boost/preprocessor/variadic/elem.hpp:31:13: note: macro 'BOOST_PP_VARIADIC_ELEM_3' defined here
#    define BOOST_PP_VARIADIC_ELEM_3(e0, e1, e2, e3, ...) e3
            ^
../boost/preprocessor/variadic/elem.hpp:26:101: error: too few arguments provided to function-like macro invocation
#        define BOOST_PP_VARIADIC_ELEM(n, ...) BOOST_PP_CAT(BOOST_PP_VARIADIC_ELEM_, n)(__VA_ARGS__,)
                                                                                                    ^
../boost/preprocessor/variadic/elem.hpp:30:13: note: macro 'BOOST_PP_VARIADIC_ELEM_2' defined here
#    define BOOST_PP_VARIADIC_ELEM_2(e0, e1, e2, ...) e2
            ^
../libs/vmd/test/test_doc_modifiers_return_type.cpp:2:10: error: too few arguments provided to function-like macro invocation
//  (C) Copyright Edward Diener 2011-2015
         ^
../boost/vmd/detail/sequence_elem.hpp:836:33: note: expanded from macro 'opyright'
        BOOST_VMD_DETAIL_EQUAL_TYPE \
                                ^
../boost/preprocessor/variadic/elem.hpp:31:13: note: macro 'BOOST_PP_VARIADIC_ELEM_3' defined here
#    define BOOST_PP_VARIADIC_ELEM_3(e0, e1, e2, e3, ...) e3
            ^
../boost/preprocessor/variadic/elem.hpp:26:101: error: too few arguments provided to function-like macro invocation
#        define BOOST_PP_VARIADIC_ELEM(n, ...) BOOST_PP_CAT(BOOST_PP_VARIADIC_ELEM_, n)(__VA_ARGS__,)
                                                                                                    ^
../boost/preprocessor/variadic/elem.hpp:31:13: note: macro 'BOOST_PP_VARIADIC_ELEM_3' defined here
#    define BOOST_PP_VARIADIC_ELEM_3(e0, e1, e2, e3, ...) e3
            ^

[zygoloid]

Looks like this is failing in the way I expected. Each portion of the test succeeds by itself, but the complete test fails with this assertion:

clang-3.5: tools/clang/lib/Basic/SourceManager.cpp:627: clang::SourceLocation clang::SourceManager::createExpansionLocImpl(const clang::SrcMgr::ExpansionInfo &, unsigned int, int, unsigned int): Assertion `NextLocalOffset + TokLength + 1 > NextLocalOffset && NextLocalOffset + TokLength + 1 <= CurrentLoadedOffset && "Ran out of source locations!"' failed.

It seems like this test is doing a truly vast amount of macro expansion -- including intermediate expansions in macro processing, it's generating a total of over 2G distinct source locations. That's more than Clang can represent -- but only just; the failure occurs while expanding the very last macro in the test file.

We ought to try to give a diagnostic for this, but ultimately you've hit UB due to violating an implementation limit. You can avoid this by splitting the file into multiple smaller files or doing less macro expansion per test.

[zygoloid]

I've dug into this a bit more, just to be sure.

If I comment out the last test in the file, -Xclang -print-stats reports

1032835/18149329/297 obj/fn/builtin macros expanded, 1146298 on the fast path.
4008291 token paste (##) operations performed, 498271 on the fast path.

That's about 19M macro expansions. We're producing 217M SLocEntries, which seems fairly high for that number of expansions, but perhaps there are a lot of arguments and token pastes going on here too.

A large proportion of the SLocEntry space is being spent on one 286-byte macro, which appears to be expanded at least half a million times by this test. There are no unusually-large SLocEntries or anything else that might indicate Clang misbehavior.

It still looks like you're just blowing past the implementation limit through too much macro expansion.