Note practical difference between syntactic & lexical definitions

Co-authored-by: Blaise Pabon <[email protected]>
Co-authored-by: Adam Turner <[email protected]>

Author: 3 people

Doc/reference/introduction.rst

@@ -183,13 +183,19 @@ This does *not* mean that there is an empty first alternative.
 
 .. index:: lexical definitions
 
-.. note::
-
-   There is some difference between *lexical* and *syntactic* analysis:
-   the :term:`lexical analyzer` operates on the individual characters of the
-   input source, while the *parser* (syntactic analyzer) operates on the stream
-   of :term:`tokens <token>` generated by the lexical analysis.
-   However, in some cases the exact boundary between the two phases is a
-   CPython implementation detail.
-
-   This documentation uses the same BNF grammar for both.
+There is some difference between *lexical* and *syntactic* analysis:
+the :term:`lexical analyzer` operates on the individual characters of the
+input source, while the *parser* (syntactic analyzer) operates on the stream
+of :term:`tokens <token>` generated by the lexical analysis.
+However, in some cases the exact boundary between the two phases is a
+CPython implementation detail.
+
+The practical difference between the two is that in *lexical* definitions,
+all whitespace is significant.
+The lexical analyzer :ref:`discards <whitespace>` all whitespace that is not
+converted to tokens like :data:`token.INDENT` or :data:`~token.NEWLINE`.
+*Syntactic* definitions then use these tokens, rather than source characters.
+
+This documentation uses the same BNF grammar for both styles of definitions.
+All uses of BNF in the next chapter (“Lexical Analysis”) are lexical definitions;
+uses in subsequent chapters are syntactic definitions.