Skip to content

Commit 5a2c7aa

Browse files
committed
[lex.pptoken] Separate out keyword placeholders
The _preprocessing-token_ term is defined as one of another set of grammar productions, with the three placeholder tokens standing out as specific single tokens. Moving the placeholder tokens into their own production simplifies both the _preprocessing-token_ and _keyword_ grammar productions, plus it guarantees a consistency for the set of placeholder between both places. It also makes it easier for [cpp] to directly cross-reference where needed. Pedantically, the term placeholder token itself is problematic as tokens are a phase 7 concern, where they become keywords. Within the earlier phases where these placeholders are used the correct term is a preprocessing token, so strictly these should be placeholder preprocessing tokens, which is a mouthful. So renaming the term to placeholder keyword, which is hopefully less of a concern as there are no keywords in phase 3--6 to confuse with phase 7 keywords. Finally, clean up the informative list in text of the various kinds of preprocessing tokens, rendering as a list in the source (one kind per line) and adding a cross-reference to each such item.
1 parent 784cc65 commit 5a2c7aa

File tree

2 files changed

+44
-30
lines changed

2 files changed

+44
-30
lines changed

source/lex.tex

Lines changed: 42 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -532,17 +532,15 @@
532532
\indextext{token!preprocessing|(}%
533533
\begin{bnf}
534534
\nontermdef{preprocessing-token}\br
535+
keyword-placeholder\br
535536
header-name\br
536-
import-keyword\br
537-
module-keyword\br
538-
export-keyword\br
539-
identifier\br
540537
pp-number\br
538+
preprocessing-op-or-punc\br
539+
identifier\br
541540
character-literal\br
542541
user-defined-character-literal\br
543542
string-literal\br
544543
user-defined-string-literal\br
545-
preprocessing-op-or-punc\br
546544
\textnormal{each non-whitespace character that cannot be one of the above}
547545
\end{bnf}
548546

@@ -552,12 +550,15 @@
552550
In this document,
553551
glyphs are used to identify
554552
elements of the basic character set\iref{lex.charset}.
555-
The categories of preprocessing token are: header names,
556-
placeholder tokens produced by preprocessing \tcode{import} and \tcode{module} directives
557-
(\grammarterm{import-keyword}, \grammarterm{module-keyword}, and \grammarterm{export-keyword}),
558-
identifiers, preprocessing numbers, character literals (including user-defined character
559-
literals), string literals (including user-defined string literals), preprocessing
560-
operators and punctuators, and single non-whitespace characters that do not lexically
553+
The categories of preprocessing token are:
554+
keyword plaeholders\iref{lex.placeholder},
555+
header names\iref{lex.header},
556+
preprocessing numbers\iref{lex.ppnumber},
557+
preprocessing operators and punctuators\iref{lex.operators},
558+
identifiers\iref{lex.name},
559+
character literals\iref{lex.ccon} including user-defined character literals,
560+
string literals\iref{lex.string} including user-defined string literals,
561+
and single non-whitespace characters that do not lexically
561562
match the other preprocessing token categories.
562563
If a \unicode{0027}{apostrophe}, a \unicode{0022}{quotation mark},
563564
or any character not in the basic character set
@@ -584,20 +585,6 @@
584585
shall have the lexical form of a keyword, an identifier, a literal,
585586
or an operator or punctuator.
586587

587-
\pnum
588-
\indexgrammar{\idxgram{import-keyword}}%
589-
\indexgrammar{\idxgram{module-keyword}}%
590-
\indexgrammar{\idxgram{export-keyword}}%
591-
The \grammarterm{import-keyword} is produced
592-
by processing an \keyword{import} directive\iref{cpp.import},
593-
the \grammarterm{module-keyword} is produced
594-
by preprocessing a \keyword{module} directive\iref{cpp.module}, and
595-
the \grammarterm{export-keyword} is produced
596-
by preprocessing either of the previous two directives.
597-
\begin{note}
598-
None has any observable spelling.
599-
\end{note}
600-
601588
\pnum
602589
If the input stream has been parsed into preprocessing tokens up to a
603590
given character:
@@ -684,6 +671,35 @@
684671
\end{example}
685672
\indextext{token!preprocessing|)}
686673

674+
\rSec1[lex.placeholder]{Keyword placeholders}
675+
\indextext{placeholder!keyword|(}%
676+
\indexgrammar{\idxgram{import-keyword}}%
677+
\indexgrammar{\idxgram{module-keyword}}%
678+
\indexgrammar{\idxgram{export-keyword}}%
679+
680+
\begin{bnf}
681+
\nontermdef{keyword-placeholder}\br
682+
export-keyword\br
683+
import-keyword\br
684+
module-keyword
685+
\end{bnf}
686+
687+
\pnum
688+
Keyword placeholders are produced by preprocessing \keyword{import}
689+
and \keyword{module} directives.
690+
The \grammarterm{import-keyword} is produced
691+
by preprocessing an \keyword{import} directive\iref{cpp.import},
692+
the \grammarterm{module-keyword} is produced
693+
by preprocessing a \keyword{module} directive\iref{cpp.module}, and
694+
the \grammarterm{export-keyword} is produced
695+
by preprocessing either of the previous two directives.
696+
697+
\pnum
698+
\begin{note}
699+
Keyword placeholders do not have any observable spelling.
700+
\end{note}
701+
\indextext{placeholder!keyword|(}%
702+
687703
\rSec1[lex.header]{Header names}
688704

689705
\indextext{header!name|(}%
@@ -1020,9 +1036,7 @@
10201036
\begin{bnf}
10211037
\nontermdef{keyword}\br
10221038
\textnormal{any identifier listed in \tref{lex.key}}\br
1023-
\grammarterm{import-keyword}\br
1024-
\grammarterm{module-keyword}\br
1025-
\grammarterm{export-keyword}
1039+
\grammarterm{keyword-placeholder}
10261040
\end{bnf}
10271041

10281042
\pnum

source/preprocessor.tex

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1262,7 +1262,7 @@
12621262
\pnum
12631263
The \tcode{module} and \tcode{export} (if it exists) preprocessing tokens
12641264
are replaced by the \grammarterm{module-keyword} and
1265-
\grammarterm{export-keyword} preprocessing tokens respectively.
1265+
\grammarterm{export-keyword} keyword placeholders respectively\iref{lex.placeholder}.
12661266
\begin{note}
12671267
This makes the line no longer a directive
12681268
so it is not removed at the end of phase 4.
@@ -1320,7 +1320,7 @@
13201320
In all three forms of \grammarterm{pp-import},
13211321
the \tcode{import} and \tcode{export} (if it exists) preprocessing tokens
13221322
are replaced by the \grammarterm{import-keyword} and
1323-
\grammarterm{export-keyword} preprocessing tokens respectively.
1323+
\grammarterm{export-keyword} keyword placeholders respectively\iref{lex.placeholder}.
13241324
\begin{note}
13251325
This makes the line no longer a directive
13261326
so it is not removed at the end of phase 4.

0 commit comments

Comments
 (0)