Characters that are not valid in identifiers should be allowed as preprocessing tokens

[Halalaluyafail3]

C allows using characters (to be clear, character here means a unicode scalar value) or universal character names that are not valid at the start of an identifier as preprocessing tokens:

The categories of preprocessing tokens are: header names, identifiers,
preprocessing numbers, character constants, string literals, punctuators, and
both single universal character names as well as single non-white-space
characters that do not lexically match the other preprocessing token
categories.

Section 6.4 "Lexical elements" Paragraph 3 ISO/IEC 9899:2024

However, Clang does not follow this specifically for characters e.g. the following is incorrectly rejected:

#define STR(X)#X
#define CAT(X,Y)X##Y
int main(){
    const char*CAT(z,̈)=STR(☺);
}

Clang does accept this when the combining diaresis character and smiling face emoji are replaced by \u0308 and \u263A, respectively.

[llvmbot]

@llvm/issue-subscribers-clang-frontend

Author: None (Halalaluyafail3)

C allows using characters (to be clear, character here means a unicode scalar value) or universal character names that are not valid at the start of an identifier as preprocessing tokens: > The categories of preprocessing tokens are: header names, identifiers, > preprocessing numbers, character constants, string literals, punctuators, and > both single universal character names as well as single non-white-space > characters that do not lexically match the other preprocessing token > categories.

Section 6.4 "Lexical elements" Paragraph 3 ISO/IEC 9899:2024

However, Clang does not follow this specifically for characters e.g. the following is incorrectly rejected:

#define STR(X)#X
#define CAT(X,Y)X##Y
int main(){
    const char*CAT(z,̈)=STR(☺);
}

Clang does accept this when the combining diaresis character and smiling face emoji are replaced by \u0308 and \u263A, respectively.

[shafik]

CC @cor3ntin @tahonermann

[cor3ntin]

duplicate of #97836