Crash from Assertion `!Packs.empty() && "Pack expansion without unexpanded packs?"'

[ilya-biryukov]

The following code crashes Clang with assertion at head, see https://gcc.godbolt.org/z/j6zY4as9h

template <class... T>
struct Types {};
template <int& field>
using Forget = int;
template <int&... fields>
using SeqKey = Types<Forget<fields>...>;

template <typename Key, typename Value>
struct HelperBase {
  using ResponseParser = Key();
  HelperBase(ResponseParser response_parser) {}
};
template <int&... fields>
SeqKey<fields...> Parser();

template <int&... fields>
struct Helper : public HelperBase<SeqKey<fields...>, double> {
  using Key = SeqKey<fields...>;
  using Value = double;
  using ParentClass = HelperBase<Key, Value>;
  Helper() : ParentClass(Parser<fields...>) {}
};
void test() { Helper<>(); }

This should be relatively recent as it was found by our release testing.

[ilya-biryukov]

@alexfh says it might be old, actually, we only found it because we started building the compiler with libc++ hardening.
He pointed out that the code was added in
4a8f351
and it crashes on this line when assertions are disabled:

llvm-project/clang/lib/Sema/SemaTemplateDeduction.cpp

Line 957 in 363e036

PackElements += *FixedNumExpansions;

[alexfh]

@zygoloid I wonder if you remember this code and could help resolving this UB and the assertion.

[llvmbot]

@llvm/issue-subscribers-clang-frontend

Author: Ilya Biryukov (ilya-biryukov)

The following code crashes Clang with assertion at head, see https://gcc.godbolt.org/z/j6zY4as9h ```cpp template <class... T> struct Types {}; template <int& field> using Forget = int; template <int&... fields> using SeqKey = Types<Forget<fields>...>;

template <typename Key, typename Value>
struct HelperBase {
using ResponseParser = Key();
HelperBase(ResponseParser response_parser) {}
};
template <int&... fields>
SeqKey<fields...> Parser();

template <int&... fields>
struct Helper : public HelperBase<SeqKey<fields...>, double> {
using Key = SeqKey<fields...>;
using Value = double;
using ParentClass = HelperBase<Key, Value>;
Helper() : ParentClass(Parser<fields...>) {}
};
void test() { Helper<>(); }

This should be relatively recent as it was found by our release testing.
</details>

[ilya-biryukov]

I got to the bottom of it, we are loosing any mention of fields from the type alias using SeqKey = Types<Forget<fields>...> when deducing it against Types<>. We end up deducing Types<int...> against Types<> and the code fails, because it (rightfully) assumes that PackExpansionType should have some unexpanded types in its pattern. We would probably want to deduce against Types<Forget<fields>...> instead, but that's the tricky part of template instantiations that always confused me (on one hand Forget<fields> has to be substituted when written, in the template itself; on the other hand, fields is an unexpanded parameter pack and if we loose it, the pack expansions type does not make sense).

See draft #100144 for (an invalid) fix that bring some other problems with it, I'll need to explore something along those lines for a proper fix.
@zygoloid if you have directions, that'd be helpful.

[cor3ntin]

Ping @mizvekov, our sugar expert :)

[mizvekov]

I think in particular anything that relied on type deduction preserving sugar would be extremely fragile.

The fix in #100144 is invalid. We can't use as-written template arguments where resolved arguments are expected, this will crash in a bunch of places.

To fix that particular sugar loss in type deduction, we need sugared resolved arguments, which is implemented by https://reviews.llvm.org/D134113
Though that patch in particular had a non-trivial impact on compile-time-tracker tests.

This might be fine as is, but It just so always happens that google will have some internal thing where the impact is enormous. I have some ideas to reduce impact in that case.

I intend to pick up that patch again soon after I restart my resugaring work, which is after my current thing is done.

[ilya-biryukov]

The fix in #100144 is invalid. We can't use as-written template arguments where resolved arguments are expected, this will crash in a bunch of places.

Yep, fully agree, the description of the PR says exactly that and that's why in draft state.

To fix that particular sugar loss in type deduction, we need sugared resolved arguments, which is implemented by https://reviews.llvm.org/D134113 Though that patch in particular had a non-trivial impact on compile-time-tracker tests.

Could you elaborate on this? Do we have other similar cases for deduction that work with sugar in Clang?
I am a little surprised that we need to rely on sugar over resolved templates for correctness, I have previously assumed that the sugar is there for improved diagnostics, tooling and non-standard static analysis (e.g. to track nullability and other attributes), but it would not affect correctness. E.g. we would have things like DecltypeType for sugar on non-dependent expressions and keep it as a canonical type for dependent expression.

This might be fine as is, but It just so always happens that google will have some internal thing where the impact is enormous. I have some ideas to reduce impact in that case.

I work at Google, and I am happy to check we're not seeing any compilation timeouts (i.e. enormous impact). Although our infrastructure does not allow to check how much slower the compilation will get until this actually gets released and we can stare at the graphs monitoring our build latencies, etc. That's still something, though, and given it's a correctness fix, we would be willing to take some regression and help reduce it.

[mizvekov]

Could you elaborate on this? Do we have other similar cases for deduction that work with sugar in Clang? I am a little surprised that we need to rely on sugar over resolved templates for correctness, I have previously assumed that the sugar is there for improved diagnostics, tooling and non-standard static analysis (e.g. to track nullability and other attributes), but it would not affect correctness. E.g. we would have things like DecltypeType for sugar on non-dependent expressions and keep it as a canonical type for dependent expression.

Until about a three years ago, type deduction in clang did not preserve any type sugar at all.
We started implementing it in the context of auto type deduction: https://reviews.llvm.org/D110216
And slowly improved it over time.

But we missed preserving type sugar in type deduction through template specializations, mainly because
it's a more massive change, with bigger compile time impact.

After that is taken care of, I think we would be preserving it for practically all cases in type deduction itself.
But there might be a few corner cases we missed, and this could have a long tail until we find everything.

But in general, we have other issues, for one aspect, how we represent some semantic adjustments.
We are quite trigger happy to eliminate type sugar when we need unqualified types, for one.
For another, some adjustments where we need to lose reference types also lose sugar.
We have a mechanism to express these adjustments without losing sugar, but it's a bit expensive and we don't use it everywhere.

So code in clang that relies on type sugar in order not to crash is highly suspect.
I mean if it would be obvious we handle all cases in that particular situation, that would be fine, but it's hardly ever obvious. If it has to go through type deduction, that would be a massive NO.

I work at Google, and I am happy to check we're not seeing any compilation timeouts (i.e. enormous impact). Although our infrastructure does not allow to check how much slower the compilation will get until this actually gets released and we can stare at the graphs monitoring our build latencies, etc. That's still something, though, and given it's a correctness fix, we would be willing to take some regression and help reduce it.

Thanks.
Though it shouldn't be about correctness in the sense of producing valid programs, but about producing better quality diagnostics and improving some kinds of analysis.

[mizvekov]

By the way, the change which introduced the crash, 4a8f351, is much older than when we started preserving sugar in type deduction. So it's unlikely the change intended to rely on it.