Skip to content

Commit a8be5b3

Browse files
committed
[lex.pptoken] Separate out keyword placeholders
The _preprocessing-token_ term is defined as one of another set of grammar productions, with the three placeholder tokens standing out as specific single tokens. Moving the placeholder tokens into their own production simplifies both the _preprocessing-token_ and _keyword_ grammar productions, plus it guarantees a consistency for the set of placeholder between both places. It also makes it easier for [cpp] to directly cross-reference where needed. Pedantically, the term placeholder token itself is problematic as tokens are a phase 7 concern, where they become keywords. Within the earlier phases where these placeholders are used the correct term is a preprocessing token, so strictly these should be placeholder preprocessing tokens, which is a mouthful. So renaming the term to placeholder keyword, which is hopefully less of a concern as there are no keywords in phase 3--6 to confuse with phase 7 keywords. Finally, clean up the informative list in text of the various kinds of preprocessing tokens, rendering as a list in the source (one kind per line) and adding a cross-reference to each such item.
1 parent 13a33ac commit a8be5b3

File tree

2 files changed

+44
-30
lines changed

2 files changed

+44
-30
lines changed

source/lex.tex

Lines changed: 42 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -532,17 +532,15 @@
532532
\indextext{token!preprocessing|(}%
533533
\begin{bnf}
534534
\nontermdef{preprocessing-token}\br
535+
keyword-placeholder\br
535536
header-name\br
536-
import-keyword\br
537-
module-keyword\br
538-
export-keyword\br
539-
identifier\br
540537
pp-number\br
538+
preprocessing-op-or-punc\br
539+
identifier\br
541540
character-literal\br
542541
user-defined-character-literal\br
543542
string-literal\br
544543
user-defined-string-literal\br
545-
preprocessing-op-or-punc\br
546544
\textnormal{each non-whitespace character that cannot be one of the above}
547545
\end{bnf}
548546

@@ -552,12 +550,15 @@
552550
In this document,
553551
glyphs are used to identify
554552
elements of the basic character set\iref{lex.charset}.
555-
The categories of preprocessing token are: header names,
556-
placeholder tokens produced by preprocessing \tcode{import} and \tcode{module} directives
557-
(\grammarterm{import-keyword}, \grammarterm{module-keyword}, and \grammarterm{export-keyword}),
558-
identifiers, preprocessing numbers, character literals (including user-defined character
559-
literals), string literals (including user-defined string literals), preprocessing
560-
operators and punctuators, and single non-whitespace characters that do not lexically
553+
The categories of preprocessing token are:
554+
keyword plaeholders\iref{lex.placeholder},
555+
header names\iref{lex.header},
556+
preprocessing numbers\iref{lex.ppnumber},
557+
preprocessing operators and punctuators\iref{lex.operators},
558+
identifiers\iref{lex.name},
559+
character literals\iref{lex.ccon} including user-defined character literals,
560+
string literals\iref{lex.string} including user-defined string literals,
561+
and single non-whitespace characters that do not lexically
561562
match the other preprocessing token categories.
562563
If a \unicode{0027}{apostrophe} or a \unicode{0022}{quotation mark} character
563564
matches the last category, the program is ill-formed.
@@ -585,20 +586,6 @@
585586
shall have the lexical form of a keyword, an identifier, a literal,
586587
or an operator or punctuator.
587588

588-
\pnum
589-
\indexgrammar{\idxgram{import-keyword}}%
590-
\indexgrammar{\idxgram{module-keyword}}%
591-
\indexgrammar{\idxgram{export-keyword}}%
592-
The \grammarterm{import-keyword} is produced
593-
by processing an \keyword{import} directive\iref{cpp.import},
594-
the \grammarterm{module-keyword} is produced
595-
by preprocessing a \keyword{module} directive\iref{cpp.module}, and
596-
the \grammarterm{export-keyword} is produced
597-
by preprocessing either of the previous two directives.
598-
\begin{note}
599-
None has any observable spelling.
600-
\end{note}
601-
602589
\pnum
603590
If the input stream has been parsed into preprocessing tokens up to a
604591
given character:
@@ -685,6 +672,35 @@
685672
\end{example}
686673
\indextext{token!preprocessing|)}
687674

675+
\rSec1[lex.placeholder]{Keyword placeholders}
676+
\indextext{placeholder!keyword|(}%
677+
\indexgrammar{\idxgram{import-keyword}}%
678+
\indexgrammar{\idxgram{module-keyword}}%
679+
\indexgrammar{\idxgram{export-keyword}}%
680+
681+
\begin{bnf}
682+
\nontermdef{keyword-placeholder}\br
683+
export-keyword\br
684+
import-keyword\br
685+
module-keyword
686+
\end{bnf}
687+
688+
\pnum
689+
Keyword placeholders are produced by preprocessing \keyword{import}
690+
and \keyword{module} directives.
691+
The \grammarterm{import-keyword} is produced
692+
by preprocessing an \keyword{import} directive\iref{cpp.import},
693+
the \grammarterm{module-keyword} is produced
694+
by preprocessing a \keyword{module} directive\iref{cpp.module}, and
695+
the \grammarterm{export-keyword} is produced
696+
by preprocessing either of the previous two directives.
697+
698+
\pnum
699+
\begin{note}
700+
Keyword placeholders do not have any observable spelling.
701+
\end{note}
702+
\indextext{placeholder!keyword|(}%
703+
688704
\rSec1[lex.header]{Header names}
689705

690706
\indextext{header!name|(}%
@@ -1021,9 +1037,7 @@
10211037
\begin{bnf}
10221038
\nontermdef{keyword}\br
10231039
\textnormal{any identifier listed in \tref{lex.key}}\br
1024-
\grammarterm{import-keyword}\br
1025-
\grammarterm{module-keyword}\br
1026-
\grammarterm{export-keyword}
1040+
\grammarterm{keyword-placeholder}
10271041
\end{bnf}
10281042

10291043
\pnum

source/preprocessor.tex

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1262,7 +1262,7 @@
12621262
\pnum
12631263
The \tcode{module} and \tcode{export} (if it exists) preprocessing tokens
12641264
are replaced by the \grammarterm{module-keyword} and
1265-
\grammarterm{export-keyword} preprocessing tokens respectively.
1265+
\grammarterm{export-keyword} keyword placeholders respectively\iref{lex.placeholder}.
12661266
\begin{note}
12671267
This makes the line no longer a directive
12681268
so it is not removed at the end of phase 4.
@@ -1320,7 +1320,7 @@
13201320
In all three forms of \grammarterm{pp-import},
13211321
the \tcode{import} and \tcode{export} (if it exists) preprocessing tokens
13221322
are replaced by the \grammarterm{import-keyword} and
1323-
\grammarterm{export-keyword} preprocessing tokens respectively.
1323+
\grammarterm{export-keyword} keyword placeholders respectively\iref{lex.placeholder}.
13241324
\begin{note}
13251325
This makes the line no longer a directive
13261326
so it is not removed at the end of phase 4.

0 commit comments

Comments
 (0)