Use automated enforcement of includes

[eddyashton]

We have regular issues with missing includes. These are minor, and easily resolved when they come up, but can be a significant user frustration when they occur in our public headers.

It should be possible to add automated checks for this, along with the existing formatters/linters we already have. Include-what-you-use looks like a promising option. It has a strict concept of required includes (no transitive includes, even for .cpp!), but I think that's standard and fine. It can even add comments explaining why each header is present.

I've experimented with this locally, and it has some problems with our funky toolchains, and a fun deeper issue error that's probably from a clang version mismatch (Error running 'iwyu': /build/iwyu-QrxlvO/iwyu-8.0/iwyu.cc:3353: Assertion failed: llvm::isa<clang::TemplateDecl>(named_decl)TemplateSpecializationType has no decl of type TemplateDecl?). This also makes a huge number of changes (git diff | wc -l = 1071), so will require a big-fix initial commit.

[eddyashton]

eg: #4058, #3203

[wintersteiger]

+1 I'm in favour if we can find something that works reliably and quickly. If we want to check only for missing headers, we could try to simply compile each of them separately, which should work fine even with our special flavour toolchain.

[achamayou]

I'm a bit confused about why our funky toolchain matters for this. It seems like doing this on the virtual build ought to just work? Is it just a matter of making sure the tool uses the same clang we do? It seems that version 10 will use clang 10, and your error message suggests 8 (which uses... clang 8!).

[achamayou]

To be clear, I am massively in favour!

[achamayou]

Let's revisit that now that we've upgraded compilers.

[achamayou]

So this work well:

# Missing dependency
sudo apt install libclang-15-dev

# Build and install iwyu
git clone https://github.com/include-what-you-use/include-what-you-use.git
git checkout clang_15.0
cd .. && mkdir build && cd build
cmake -G "Unix Makefiles" -DCMAKE_PREFIX_PATH=/usr/lib/llvm-15 ../include-what-you-use
make && sudo make install

# Build CCF with IWYU
cmake -GNinja -DCMAKE_BUILD_TYPE=Debug -DCOMPILE_TARGET=virtual -DCMAKE_CXX_INCLUDE_WHAT_YOU_USE="include-what-you-use;-w;-Xiwyu;" ..
ninja

The next steps may be:

• do a pass and run fix_includes.py, but probably fairly disruptive for future backports
• add this to the CI image and run the check automatically, if fast enough?
• otherwise, turn into a script and run periodically, perhaps before LTS branching?

[achamayou]

The output is large (~20klines diff), and some of the includes are questionable, although this is perhaps a matter of using the right options:

../samples/apps/nobuiltins/nobuiltins.cpp should add these lines:
#include <__functional/function.h>       // for __base
#include <__memory/unique_ptr.h>         // for make_unique, unique_ptr
#include <__utility/move.h>              // for move