Partial documentation update

Author: nikic

README.md

@@ -3,12 +3,12 @@ PHP Parser
 
 [![Coverage Status](https://coveralls.io/repos/github/nikic/PHP-Parser/badge.svg?branch=master)](https://coveralls.io/github/nikic/PHP-Parser?branch=master)
 
-This is a PHP 5.2 to PHP 8.1 parser written in PHP. Its purpose is to simplify static code analysis and
+This is a PHP parser written in PHP. Its purpose is to simplify static code analysis and
 manipulation.
 
 [Documentation for version 5.x][doc_master] (in development; for running on PHP >= 7.1; for parsing PHP 7.0 to PHP 8.2, with limited support for parsing PHP 5.x).
 
-[**Documentation for version 4.x**][doc_4_x] (stable; for running on PHP >= 7.0; for parsing PHP 5.2 to PHP 8.1).
+[**Documentation for version 4.x**][doc_4_x] (stable; for running on PHP >= 7.0; for parsing PHP 5.2 to PHP 8.2).
 
 [Documentation for version 3.x][doc_3_x] (unsupported; for running on PHP >= 5.5; for parsing PHP 5.2 to PHP 7.2).
 
@@ -17,12 +17,12 @@ Features
 
 The main features provided by this library are:
 
- * Parsing PHP 5, PHP 7, and PHP 8 code into an abstract syntax tree (AST).
+ * Parsing PHP 7, and PHP 8 code into an abstract syntax tree (AST).
    * Invalid code can be parsed into a partial AST.
    * The AST contains accurate location information.
  * Dumping the AST in human-readable form.
  * Converting an AST back to PHP code.
-   * Experimental: Formatting can be preserved for partially changed ASTs.
+   * Formatting can be preserved for partially changed ASTs.
  * Infrastructure to traverse and modify ASTs.
  * Resolution of namespaced names.
  * Evaluation of constant expressions.
@@ -53,7 +53,7 @@ function test($foo)
 }
 CODE;
 
-$parser = (new ParserFactory)->create(ParserFactory::PREFER_PHP7);
+$parser = (new ParserFactory())->createForNewestSupportedVersion();
 try {
     $ast = $parser->parse($code);
 } catch (Error $error) {

doc/0_Introduction.markdown

@@ -1,7 +1,7 @@
 Introduction
 ============
 
-This project is a PHP 5.2 to PHP 8.0 parser **written in PHP itself**.
+This project is a PHP parser **written in PHP itself**.
 
 What is this for?
 -----------------
@@ -26,16 +26,29 @@ programmatic PHP code analysis are incidentally PHP developers, not C developers
 What can it parse?
 ------------------
 
-The parser supports parsing PHP 5.2-8.0, with the following exceptions:
+The parser supports parsing PHP 7 and PHP 8 code, with the following exceptions:
 
  * Namespaced names containing whitespace (e.g. `Foo \ Bar` instead of `Foo\Bar`) are not supported.
    These are illegal in PHP 8, but are legal in earlier versions. However, PHP-Parser does not
    support them for any version.
 
+PHP-Parser 4.x had full support for parsing PHP 5. PHP-Parser 5.x has only limited support, with the
+following caveats:
+
+ * Some variable expressions like `$$foo[0]` are valid in both PHP 5 and PHP 7, but have different 
+   interpretation. In such cases, the PHP 7 AST will always be constructed (using `($$foo)[0]`
+   rather than `${$foo[0]}`).
+ * Declarations of the form `global $$var[0]` are not supported in PHP 7 and will cause a parse 
+   error. In error recovery mode, it is possible to continue parsing after such declarations.
+
 As the parser is based on the tokens returned by `token_get_all` (which is only able to lex the PHP
 version it runs on), additionally a wrapper for emulating tokens from newer versions is provided.
-This allows to parse PHP 7.4 source code running on PHP 7.0, for example. This emulation is somewhat
-hacky and not perfect, but it should work well on any sane code.
+This allows to parse PHP 8.0 source code running on PHP 7.1, for example. This emulation is not
+perfect, but works well in practice.
+
+Finally, it should be noted that the parser aims to accept all valid code, not reject all invalid
+code. It will generally accept code that is only valid in newer versions (even when targeting an
+older one), and accept code that is syntactically correct, but would result in a compiler error.
 
 What output does it produce?
 ----------------------------
@@ -63,7 +76,7 @@ This matches the structure of the code: An echo statement, which takes two strin
 with the values `Hi` and `World`.
 
 You can also see that the AST does not contain any whitespace information (but most comments are saved).
-So using it for formatting analysis is not possible.
+However, it does retain accurate position information, which can be used to inspect precise formatting.
 
 What else can it do?
 --------------------
@@ -74,7 +87,7 @@ Apart from the parser itself this package also bundles support for some other, r
    that "pretty printing" does not imply that the output is especially pretty. It's just how it's
    called ;)
  * Support for serializing and unserializing the node tree to JSON
- * Support for dumping the node tree in a human readable form (see the section above for an
+ * Support for dumping the node tree in a human-readable form (see the section above for an
    example of how the output looks like)
  * Infrastructure for traversing and changing the AST (node traverser and node visitors)
  * A node visitor for resolving namespaced names

doc/2_Usage_of_basic_components.markdown

@@ -12,7 +12,7 @@ To bootstrap the library, include the autoloader generated by composer:
 require 'path/to/vendor/autoload.php';
 ```
 
-Additionally you may want to set the `xdebug.max_nesting_level` ini option to a higher value:
+Additionally, you may want to set the `xdebug.max_nesting_level` ini option to a higher value:
 
 ```php
 ini_set('xdebug.max_nesting_level', 3000);
@@ -29,25 +29,29 @@ In order to parse code, you first have to create a parser instance:
 
 ```php
 use PhpParser\ParserFactory;
-$parser = (new ParserFactory)->create(ParserFactory::PREFER_PHP7);
-```
+use PhpParser\PhpVersion;
+
+// Parser for the version you are running on.
+$parser = (new ParserFactory())->createForHostVersion();
 
-The factory accepts a kind argument, that determines how different PHP versions are treated:
+// Parser for the newest PHP version supported by the PHP-Parser library.
+$parser = (new ParserFactory())->createForNewestSupportedVersion();
 
-Kind | Behavior
------|---------
-`ParserFactory::PREFER_PHP7` | Try to parse code as PHP 7. If this fails, try to parse it as PHP 5.
-`ParserFactory::PREFER_PHP5` | Try to parse code as PHP 5. If this fails, try to parse it as PHP 7.
-`ParserFactory::ONLY_PHP7` | Parse code as PHP 7.
-`ParserFactory::ONLY_PHP5` | Parse code as PHP 5.
+// Parser for a specific PHP version.
+$parser = (new ParserFactory())->createForVersion(PhpVersion::fromString('8.1'));
+```
 
-Unless you have a strong reason to use something else, `PREFER_PHP7` is a reasonable default.
+Which version you should target depends on your use case. In many cases you will want to use the
+host version, as people typically analyze code for the version they are running on. However, when
+analyzing arbitrary code you are usually best off using the newest supported version, which tends
+to accept the widest range of code (unless there are breaking changes in PHP).
 
-The `create()` method optionally accepts a `Lexer` instance as the second argument. Some use cases
-that require customized lexers are discussed in the [lexer documentation](component/Lexer.markdown).
+The `createXYZ()` methods optionally accept an array of lexer options. Some use cases that require
+customized lexer options are discussed in the [lexer documentation](component/Lexer.markdown).
 
-Subsequently you can pass PHP code (including the opening `<?php` tag) to the `parse` method in order to
-create a syntax tree. If a syntax error is encountered, an `PhpParser\Error` exception will be thrown:
+Subsequently, you can pass PHP code (including the opening `<?php` tag) to the `parse()` method in
+order to create a syntax tree. If a syntax error is encountered, an `PhpParser\Error` exception will
+be thrown by default:
 
 ```php
 <?php
@@ -62,7 +66,7 @@ function printLine($msg) {
 printLine('Hello World!!!');
 CODE;
 
-$parser = (new ParserFactory)->create(ParserFactory::PREFER_PHP7);
+$parser = (new ParserFactory())->createForHostVersion();
 
 try {
     $stmts = $parser->parse($code);
@@ -77,7 +81,7 @@ A parser instance can be reused to parse multiple files.
 Node dumping
 ------------
 
-To dump the abstract syntax tree in human readable form, a `NodeDumper` can be used:
+To dump the abstract syntax tree in human-readable form, a `NodeDumper` can be used:
 
 ```php
 <?php
@@ -171,7 +175,7 @@ with them easier they are grouped into three categories:
 
  * `PhpParser\Node\Stmt`s are statement nodes, i.e. language constructs that do not return
    a value and can not occur in an expression. For example a class definition is a statement.
-   It doesn't return a value and you can't write something like `func(class A {});`.
+   It doesn't return a value, and you can't write something like `func(class A {});`.
  * `PhpParser\Node\Expr`s are expression nodes, i.e. language constructs that return a value
    and thus can occur in other expressions. Examples of expressions are `$var`
    (`PhpParser\Node\Expr\Variable`) and `func()` (`PhpParser\Node\Expr\FuncCall`).
@@ -201,15 +205,14 @@ can then be retrieved using `hasAttribute()`, `getAttribute()` and `getAttribute
 By default the lexer adds the `startLine`, `endLine` and `comments` attributes. `comments` is an array
 of `PhpParser\Comment[\Doc]` instances.
 
-The start line can also be accessed using `getLine()`/`setLine()` (instead of `getAttribute('startLine')`).
+The start line can also be accessed using `getStartLine()` (instead of `getAttribute('startLine')`).
 The last doc comment from the `comments` attribute can be obtained using `getDocComment()`.
 
 Pretty printer
 --------------
 
-The pretty printer component compiles the AST back to PHP code. As the parser does not retain formatting
-information the formatting is done using a specified scheme. Currently there is only one scheme available,
-namely `PhpParser\PrettyPrinter\Standard`.
+The pretty printer component compiles the AST back to PHP code according to a specified scheme.
+Currently, there is only one scheme available, namely `PhpParser\PrettyPrinter\Standard`.
 
 ```php
 use PhpParser\Error;
@@ -218,8 +221,8 @@ use PhpParser\PrettyPrinter;
 
 $code = "<?php echo 'Hi ', hi\\getTarget();";
 
-$parser = (new ParserFactory)->create(ParserFactory::PREFER_PHP7);
-$prettyPrinter = new PrettyPrinter\Standard;
+$parser = (new ParserFactory())->createForHostVersion();
+$prettyPrinter = new PrettyPrinter\Standard();
 
 try {
     // parse
@@ -254,10 +257,13 @@ single expression using `prettyPrintExpr()`.
 The `prettyPrintFile()` method can be used to print an entire file. This will include the opening `<?php` tag
 and handle inline HTML as the first/last statement more gracefully.
 
+There is also a pretty-printing mode which retains formatting for parts of the AST that have not
+been changed, which requires additional setup.
+
 > Read more: [Pretty printing documentation](component/Pretty_printing.markdown)
 
-Node traversation
------------------
+Node traversal
+--------------
 
 The above pretty printing example used the fact that the source code was known and thus it was easy to
 write code that accesses a certain part of a node tree and changes it. Normally this is not the case.
@@ -272,7 +278,7 @@ use PhpParser\NodeTraverser;
 use PhpParser\ParserFactory;
 use PhpParser\PrettyPrinter;
 
-$parser        = (new ParserFactory)->create(ParserFactory::PREFER_PHP7);
+$parser        = (new ParserFactory())->createForHostVersion();
 $traverser     = new NodeTraverser;
 $prettyPrinter = new PrettyPrinter\Standard;
 
@@ -303,8 +309,7 @@ The corresponding node visitor might look like this:
 use PhpParser\Node;
 use PhpParser\NodeVisitorAbstract;
 
-class MyNodeVisitor extends NodeVisitorAbstract
-{
+class MyNodeVisitor extends NodeVisitorAbstract {
     public function leaveNode(Node $node) {
         if ($node instanceof Node\Scalar\String_) {
             $node->value = 'foo';
@@ -326,7 +331,7 @@ public function afterTraverse(array $nodes);
 ```
 
 The `beforeTraverse()` method is called once before the traversal begins and is passed the nodes the
-traverser was called with. This method can be used for resetting values before traversation or
+traverser was called with. This method can be used for resetting values before traversal or
 preparing the tree for traversal.
 
 The `afterTraverse()` method is similar to the `beforeTraverse()` method, with the only difference that
@@ -342,8 +347,8 @@ The `enterNode()` method can additionally return the value `NodeTraverser::DONT_
 which instructs the traverser to skip all children of the current node. To furthermore prevent subsequent
 visitors from visiting the current node, `NodeTraverser::DONT_TRAVERSE_CURRENT_AND_CHILDREN` can be used instead.
 
-The `leaveNode()` method can additionally return the value `NodeTraverser::REMOVE_NODE`, in which
-case the current node will be removed from the parent array. Furthermore it is possible to return
+Both methods can additionally return the value `NodeTraverser::REMOVE_NODE`, in which
+case the current node will be removed from the parent array. Furthermore, it is possible to return
 an array of nodes, which will be merged into the parent array at the offset of the current node.
 I.e. if in `array(A, B, C)` the node `B` should be replaced with `array(X, Y, Z)` the result will
 be `array(A, X, Y, Z, C)`.
@@ -372,8 +377,9 @@ unqualified function and constant names. These are resolved at runtime and thus
 know which function they are referring to. In most cases this is a non-issue as the global functions
 are meant.
 
-Also the `NameResolver` adds a `namespacedName` subnode to class, function and constant declarations
-that contains the namespaced name instead of only the shortname that is available via `name`.
+Additionally, the `NameResolver` adds a `namespacedName` subnode to class, function and constant
+declarations that contains the namespaced name instead of only the shortname that is available via
+`name`.
 
 > Read more: [Name resolution documentation](component/Name_resolution.markdown)
 
@@ -396,7 +402,7 @@ use PhpParser\NodeVisitor\NameResolver;
 $inDir  = '/some/path';
 $outDir = '/some/other/path';
 
-$parser        = (new ParserFactory)->create(ParserFactory::PREFER_PHP7);
+$parser        = (new ParserFactory())->createForNewestSupportedVersion();
 $traverser     = new NodeTraverser;
 $prettyPrinter = new PrettyPrinter\Standard;
 

lib/PhpParser/Lexer.php

@@ -15,7 +15,6 @@ class Lexer {
     protected $pos;
     protected $prevCloseTagHasNewline;
 
-    protected $tokenMap;
     protected $dropTokens;
 
     private $attributeStartLineUsed;