Merge branch 'pass_context' into copilot/fix-failing-tests

Author: danog

UPGRADING.md

@@ -1,3 +1,78 @@
+# Upgrading from Psalm 6 to Psalm 7
+
+## Changed
+
+- [BC] Property Psalm\Type\Atomic\TCallable#$is_pure was removed and replaced with an `$allowed_mutations` property
+- [BC] Property Psalm\Type\Atomic\TClosure#$is_pure was removed and replaced with an `$allowed_mutations` property
+- [BC] Property Psalm\Type\Atomic\CallableTrait#$is_pure was removed and replaced with an `$allowed_mutations` property
+- [BC] Method Psalm\Type\Atomic\TCallable#setIsPure() was removed and replaced with a `setAllowedMutations` method 
+- [BC] Method Psalm\Type\Atomic\TClosure#setIsPure() was removed and replaced with a `setAllowedMutations` method 
+- [BC] Method Psalm\Type\Atomic\CallableTrait#setIsPure() was removed and replaced with a `setAllowedMutations` method 
+- [BC] Property Psalm\Type\Atomic\TCallable#$value changed default value from NULL to 'callable'
+- [BC] The $value parameter of Psalm\Type\Atomic\TCallable#__construct() was removed
+- [BC] The $value parameter of Psalm\Type\Atomic\TClosure#__construct() was removed
+- [BC] Property $pure of `Psalm\Context`, `Psalm\Storage\FunctionLikeStorage` was removed and replaced with an `isPure()` method, reading from the new `$allowed_mutations` property.
+- [BC] Property $mutation_free of `Psalm\Context`, `Psalm\Storage\FunctionLikeStorage`, `Psalm\Storage\ClassLikeStorage` was removed and replaced with an `isMutationFree()` method, reading from the new `$allowed_mutations` property.
+- [BC] Property $external_mutation_free of of `Psalm\Context`, `Psalm\Storage\FunctionLikeStorage`, `Psalm\Storage\ClassLikeStorage` was removed and replaced with an `isExternalMutationFree()` method, reading from the new `$allowed_mutations` property.
+- [BC] Method signalMutationOnlyInferred() was added to interface Psalm\StatementsSource
+- [BC] Method signalMutation() was added to interface Psalm\StatementsSource
+
+- [BC] Taints are now *internally* represented by a bitmap (an integer), instead of an array of strings. Users can still use the usual string taint identifiers (including custom ones, which will be automatically registered by Psalm), but internally, the type of `Psalm\Type\TaintKind` taint types is now an integer.
+
+- [BC] The maximum number of usable taint *types* (including both native taints and custom taints) is now equal to 32 on 32-bit systems and 64 on 64-bit systems: this should be enough for the vast majority of usecases, if more taint types are needed, consider merging some taint types or using some native taint types.  
+
+- [BC] `Psalm\Plugin\EventHandler\AddTaintsInterface::addTaints` and `Psalm\Plugin\EventHandler\RemoveTaintsInterface::removeTaints` now must return an integer taint instead of an array of strings (see the new [taint documentation](https://psalm.dev/docs/security_analysis/custom_taint_sources/) for more info).  
+
+- [BC] The type of the `$taints` parameter of `Psalm\Codebase::addTaintSource` and  `Psalm\Codebase::addTaintSink` was changed to an integer
+
+- [BC] Type of property `Psalm\Storage\FunctionLikeParameter::$sinks` changed from `array|null` to `int`
+
+- [BC] Type of property `Psalm\Storage\FunctionLikeStorage::$taint_source_types` changed from `array` to `int`
+
+- [BC] Type of property `Psalm\Storage\FunctionLikeStorage::$added_taints` changed from `array` to `int`
+
+- [BC] Type of property `Psalm\Storage\FunctionLikeStorage::$removed_taints` changed from `array` to `int`
+
+- [BC] The `startScanningFiles`, `startAnalyzingFiles`, `startAlteringFiles` of `Psalm\Progress\Progress` and subclasses were removed and replaced with a new `startPhase` method, taking a `Psalm\Progress\Phase` enum case.
+
+- [BC] The `start` method was removed, use `expand`, instead; the progress is reset to 0 when changing the current phase.  
+
+- [BC] Method `doesTerminalSupportUtf8` of class `Psalm\Progress\Progress` became final
+
+- [BC] Method debug() of class Psalm\Progress\Progress changed from concrete to abstract
+
+- [BC] Method alterFileDone() of class Psalm\Progress\Progress changed from concrete to abstract
+
+- [BC] Method expand() of class Psalm\Progress\Progress changed from concrete to abstract
+
+- [BC] Method taskDone() of class Psalm\Progress\Progress changed from concrete to abstract
+
+- [BC] Method finish() of class Psalm\Progress\Progress changed from concrete to abstract
+
+- [BC] The return type of Psalm\Type::getListAtomic() changed from Psalm\Type\Atomic\TKeyedArray to the non-covariant Psalm\Type\Atomic\TKeyedArray|Psalm\Type\Atomic\TArray
+
+- [BC] The return type of Psalm\Type::getListAtomic() changed from Psalm\Type\Atomic\TKeyedArray to Psalm\Type\Atomic\TKeyedArray|Psalm\Type\Atomic\TArray
+
+- [BC] The return type of Psalm\Type::getNonEmptyListAtomic() changed from Psalm\Type\Atomic\TKeyedArray to the non-covariant Psalm\Type\Atomic\TKeyedArray|Psalm\Type\Atomic\TArray
+
+- [BC] The return type of Psalm\Type::getNonEmptyListAtomic() changed from Psalm\Type\Atomic\TKeyedArray to Psalm\Type\Atomic\TKeyedArray|Psalm\Type\Atomic\TArray
+
+- [BC] Class Psalm\Type\Atomic\TKeyedArray became final
+
+- [BC] Class Psalm\Type\Atomic\TKeyedArray can only be created using the new `make` or `makeCallable` factory methods, the constructor was rendered private.  
+
+- [BC] Class Psalm\Type\Atomic\TCallableKeyedArray has been deleted, and replaced with a new `is_callable` flag in Psalm\Type\Atomic\TKeyedArray
+
+- [BC] Class Psalm\Type\Atomic\TCallableInterface has been deleted, use `\Psalm\Type\Atomic::isCallableType()` instead
+
+## Removed
+
+- [BC] Constant Psalm\Type\Atomic\TKeyedArray::NAME_ARRAY was removed
+
+- [BC] Constant Psalm\Type\Atomic\TKeyedArray::NAME_LIST was removed
+
+- [BC] Psalm\Type\Atomic\TKeyedArray#__construct() was made private
+
 # Upgrading from Psalm 5 to Psalm 6
 ## Changed
 

bin/ci/test-with-real-projects.sh

@@ -6,10 +6,12 @@ set -x
 SCRIPT_DIR="$(dirname "${BASH_SOURCE[0]}")"
 SCRIPT_DIR="$(realpath "$SCRIPT_DIR")"
 PSALM="$(readlink -f "$SCRIPT_DIR/../../psalm")"
-PSALM_PHAR="$(readlink -f "$SCRIPT_DIR/../../build/psalm.phar")"
+PSALM_PHAR="$(readlink -f "$SCRIPT_DIR/../../build/psalm.phar" || echo "")"
 
 if [ ! -f "$PSALM_PHAR" ]; then PSALM_PHAR="$PSALM"; fi
 
+which gsed > /dev/null && sed=gsed || sed=sed
+
 rm -Rf /tmp/testing-with-real-projects
 mkdir -p /tmp/testing-with-real-projects
 cd /tmp/testing-with-real-projects
@@ -54,7 +56,7 @@ psl)
 	composer install --ignore-platform-reqs
 	# Avoid conflicts with old psalm when running phar tests
 	rm -rf vendor/vimeo/psalm
-	sed 's/ErrorOutputBehavior::Packed, ErrorOutputBehavior::Discard/ErrorOutputBehavior::Discard/g' -i src/Psl/Shell/execute.php
+	$sed 's/ErrorOutputBehavior::Packed, ErrorOutputBehavior::Discard/ErrorOutputBehavior::Discard/g' -i src/Psl/Shell/execute.php
 	"$PSALM_PHAR" --monochrome -c config/psalm.xml --set-baseline=psalm-baseline.xml || FAIL=$?
 	"$PSALM_PHAR" --monochrome -c config/psalm-static-analysis.xml tests/static-analysis --set-baseline=psalm-baseline-static-analysis.xml || FAIL=$?
 	;;

bin/docker/Dockerfile

@@ -57,7 +57,7 @@ RUN set -eux; \
 # Enable linker optimization (this sorts the hash buckets to improve cache locality, and is non-default)
 # https://github.com/docker-library/php/issues/272
 # -D_LARGEFILE_SOURCE and -D_FILE_OFFSET_BITS=64 (https://www.php.net/manual/en/intro.filesystem.php)
-ENV PHP_CFLAGS="-fstack-protector-strong -fpic -fpie -O3 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64"
+ENV PHP_CFLAGS="-fstack-protector-strong -fpic -fpie -O3 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g"
 ENV PHP_CPPFLAGS="$PHP_CFLAGS"
 ENV PHP_LDFLAGS="-Wl,-O1 -pie"
 
@@ -185,6 +185,7 @@ RUN set -eux; \
 	php --version
 
 COPY bin/docker/docker-php-* /usr/local/bin/
+ADD https://raw.githubusercontent.com/php/php-src/refs/heads/PHP-8.4/.gdbinit /root/.gdbinit
 
 # This line invalidates cache when master branch changes
 ADD https://github.com/vimeo/psalm/commits/master.atom /dev/null

composer.json

@@ -118,7 +118,7 @@
         "lint": "@php parallel-lint ./src ./tests",
         "phpunit": [
             "Composer\\Config::disableProcessTimeout",
-            "paratest -f --runner=WrapperRunner"
+            "@php paratest -f --runner=WrapperRunner"
         ],
         "phpunit-std": [
             "Composer\\Config::disableProcessTimeout",

config.xsd

@@ -81,7 +81,7 @@
         <xs:attribute name="skipChecksOnUnresolvableIncludes" type="xs:boolean" default="false" />
         <xs:attribute name="sealAllMethods" type="xs:boolean" default="true" />
         <xs:attribute name="sealAllProperties" type="xs:boolean" default="true" />
-        <xs:attribute name="runTaintAnalysis" type="xs:boolean" default="false" />
+        <xs:attribute name="runTaintAnalysis" type="xs:boolean" default="true" />
         <xs:attribute name="usePhpStormMetaPath" type="xs:boolean" default="true" />
         <xs:attribute name="allowInternalNamedArgumentCalls" type="xs:boolean" default="true" />
         <xs:attribute name="allowNamedArgumentCalls" type="xs:boolean" default="true" />
@@ -264,6 +264,7 @@
             <xs:element name="ImplicitToStringCast" type="IssueHandlerType" minOccurs="0" />
             <xs:element name="ImpureByReferenceAssignment" type="IssueHandlerType" minOccurs="0" />
             <xs:element name="ImpureFunctionCall" type="IssueHandlerType" minOccurs="0" />
+            <xs:element name="ImpureGlobalVariable" type="IssueHandlerType" minOccurs="0" />
             <xs:element name="ImpureMethodCall" type="IssueHandlerType" minOccurs="0" />
             <xs:element name="ImpurePropertyAssignment" type="IssueHandlerType" minOccurs="0" />
             <xs:element name="ImpurePropertyFetch" type="IssueHandlerType" minOccurs="0" />
@@ -342,10 +343,13 @@
             <xs:element name="MissingDependency" type="IssueHandlerType" minOccurs="0" />
             <xs:element name="MissingDocblockType" type="IssueHandlerType" minOccurs="0" />
             <xs:element name="MissingFile" type="IssueHandlerType" minOccurs="0" />
+            <xs:element name="MissingInterfaceImmutableAnnotation" type="IssueHandlerType" minOccurs="0" />
             <xs:element name="MissingImmutableAnnotation" type="IssueHandlerType" minOccurs="0" />
             <xs:element name="MissingOverrideAttribute" type="IssueHandlerType" minOccurs="0" />
             <xs:element name="MissingParamType" type="IssueHandlerType" minOccurs="0" />
             <xs:element name="MissingPropertyType" type="PropertyIssueHandlerType" minOccurs="0" />
+            <xs:element name="MissingAbstractPureAnnotation" type="IssueHandlerType" minOccurs="0" />
+            <xs:element name="MissingPureAnnotation" type="IssueHandlerType" minOccurs="0" />
             <xs:element name="MissingReturnType" type="IssueHandlerType" minOccurs="0" />
             <xs:element name="MissingTemplateParam" type="IssueHandlerType" minOccurs="0" />
             <xs:element name="MissingThrowsDocblock" type="ClassIssueHandlerType" minOccurs="0" />
@@ -368,6 +372,7 @@
             <xs:element name="MixedStringOffsetAssignment" type="IssueHandlerType" minOccurs="0" />
             <xs:element name="MoreSpecificImplementedParamType" type="IssueHandlerType" minOccurs="0" />
             <xs:element name="MoreSpecificReturnType" type="IssueHandlerType" minOccurs="0" />
+            <xs:element name="ImmutableDependency" type="PropertyIssueHandlerType" minOccurs="0" />
             <xs:element name="MutableDependency" type="PropertyIssueHandlerType" minOccurs="0" />
             <xs:element name="NamedArgumentNotAllowed" type="ArgumentIssueHandlerType" minOccurs="0" />
             <xs:element name="NoEnumProperties" type="ClassIssueHandlerType" minOccurs="0" />

dictionaries/InternalTaintSinkMap.php

@@ -5,59 +5,59 @@
 // This maps internal function names to sink types that we don’t want to end up there
 
 /**
- * @var non-empty-array<string, non-empty-list<list<TaintKind::*>>>
+ * @var non-empty-array<string, non-empty-list<int-mask-of<TaintKind::*>>>
  */
 return [
-'exec' => [['shell']],
-'create_function' => [[], ['eval']],
-'file_get_contents' => [['file']],
-'file_put_contents' => [['file']],
-'fopen' => [['file']],
-'unlink' => [['file']],
-'copy' => [['file'], ['file']],
-'file' => [['file']],
-'link' => [['file'], ['file']],
-'mkdir' => [['file']],
-'move_uploaded_file' => [['file'], ['file']],
-'parse_ini_file' => [['file']],
-'chown' => [['file']],
-'lchown' => [['file']],
-'readfile' => [['file']],
-'rename' => [['file'], ['file']],
-'rmdir' => [['file']],
-'header' => [['header']],
-'symlink' => [['file']],
-'tempnam' => [['file']],
-'igbinary_unserialize' => [['unserialize']],
-'ldap_search' => [[], ['ldap'], ['ldap']],
-'mysqli_query' => [[], ['sql']],
-'mysqli::query' => [['sql']],
-'mysqli_real_query' => [[], ['sql']],
-'mysqli::real_query' => [['sql']],
-'mysqli_multi_query' => [[], ['sql']],
-'mysqli::multi_query' => [['sql']],
-'mysqli_prepare' => [[], ['sql']],
-'mysqli::prepare' => [['sql']],
-'mysqli_stmt::__construct' => [[], ['sql']],
-'mysqli_stmt_prepare' => [[], ['sql']],
-'mysqli_stmt::prepare' => [['sql']],
-'passthru' => [['shell']],
-'pcntl_exec' => [['shell']],
-'pg_exec' => [[], ['sql']],
-'pg_prepare' => [[], [], ['sql']],
-'pg_put_line' => [[], ['sql']],
-'pg_query' => [[], ['sql']],
-'pg_query_params' => [[], ['sql']],
-'pg_send_prepare' => [[], [], ['sql']],
-'pg_send_query' => [[], ['sql']],
-'pg_send_query_params' => [[], ['sql'], []],
-'setcookie' => [['cookie'], ['cookie']],
-'shell_exec' => [['shell']],
-'system' => [['shell']],
-'unserialize' => [['unserialize']],
-'popen' => [['shell']],
-'proc_open' => [['shell']],
-'curl_init' => [['ssrf']],
-'curl_setopt' => [[], [], ['ssrf']],
-'getimagesize' => [['ssrf']],
+'exec' => [TaintKind::INPUT_SHELL],
+'create_function' => [0, TaintKind::INPUT_EVAL],
+'file_get_contents' => [TaintKind::INPUT_FILE|TaintKind::INPUT_SSRF],
+'file_put_contents' => [TaintKind::INPUT_FILE],
+'fopen' => [TaintKind::INPUT_FILE],
+'unlink' => [TaintKind::INPUT_FILE],
+'copy' => [TaintKind::INPUT_FILE|TaintKind::INPUT_SSRF, TaintKind::INPUT_FILE],
+'file' => [TaintKind::INPUT_FILE],
+'link' => [TaintKind::INPUT_FILE, TaintKind::INPUT_FILE],
+'mkdir' => [TaintKind::INPUT_FILE],
+'move_uploaded_file' => [TaintKind::INPUT_FILE, TaintKind::INPUT_FILE],
+'parse_ini_file' => [TaintKind::INPUT_FILE],
+'chown' => [TaintKind::INPUT_FILE],
+'lchown' => [TaintKind::INPUT_FILE],
+'readfile' => [TaintKind::INPUT_FILE],
+'rename' => [TaintKind::INPUT_FILE, TaintKind::INPUT_FILE],
+'rmdir' => [TaintKind::INPUT_FILE],
+'header' => [TaintKind::INPUT_HEADER],
+'symlink' => [TaintKind::INPUT_FILE],
+'tempnam' => [TaintKind::INPUT_FILE],
+'igbinary_unserialize' => [TaintKind::INPUT_UNSERIALIZE],
+'ldap_search' => [0, TaintKind::INPUT_LDAP, TaintKind::INPUT_LDAP],
+'mysqli_query' => [0, TaintKind::INPUT_SQL],
+'mysqli::query' => [TaintKind::INPUT_SQL],
+'mysqli_real_query' => [0, TaintKind::INPUT_SQL],
+'mysqli::real_query' => [TaintKind::INPUT_SQL],
+'mysqli_multi_query' => [0, TaintKind::INPUT_SQL],
+'mysqli::multi_query' => [TaintKind::INPUT_SQL],
+'mysqli_prepare' => [0, TaintKind::INPUT_SQL],
+'mysqli::prepare' => [TaintKind::INPUT_SQL],
+'mysqli_stmt::__construct' => [0, TaintKind::INPUT_SQL],
+'mysqli_stmt_prepare' => [0, TaintKind::INPUT_SQL],
+'mysqli_stmt::prepare' => [TaintKind::INPUT_SQL],
+'passthru' => [TaintKind::INPUT_SHELL],
+'pcntl_exec' => [TaintKind::INPUT_SHELL],
+'pg_exec' => [0, TaintKind::INPUT_SQL],
+'pg_prepare' => [0, 0, TaintKind::INPUT_SQL],
+'pg_put_line' => [0, TaintKind::INPUT_SQL],
+'pg_query' => [0, TaintKind::INPUT_SQL],
+'pg_query_params' => [0, TaintKind::INPUT_SQL],
+'pg_send_prepare' => [0, 0, TaintKind::INPUT_SQL],
+'pg_send_query' => [0, TaintKind::INPUT_SQL],
+'pg_send_query_params' => [0, TaintKind::INPUT_SQL, 0],
+'setcookie' => [TaintKind::INPUT_COOKIE, TaintKind::INPUT_COOKIE],
+'shell_exec' => [TaintKind::INPUT_SHELL],
+'system' => [TaintKind::INPUT_SHELL],
+'unserialize' => [TaintKind::INPUT_UNSERIALIZE],
+'popen' => [TaintKind::INPUT_SHELL],
+'proc_open' => [TaintKind::INPUT_SHELL],
+'curl_init' => [TaintKind::INPUT_SSRF],
+'curl_setopt' => [0, 0, TaintKind::INPUT_SSRF],
+'getimagesize' => [TaintKind::INPUT_SSRF],
 ];

docs/annotating_code/supported_annotations.md

@@ -370,7 +370,9 @@ class D {
 
 ### `@psalm-external-mutation-free`
 
-Used to annotate a class method that does not mutate state externally of the class's scope.
+Used to annotate a class method that does not mutate state externally of the class's scope.  
+
+Can also be used on classes to propagate the same annotation to all of its methods.
 
 ```php
 <?php
@@ -441,9 +443,15 @@ $anonymous = new /** @psalm-immutable */ class extends Foo
 };
 ```
 
+### `@psalm-mutable`
+
+Used to annotate a class where at least one property is mutable: this is the default behavior, but it can be explicitly marked for clarity.
+
 ### `@psalm-pure`
 
-Used to annotate a [pure function](https://en.wikipedia.org/wiki/Pure_function) - one whose output is just a function of its input.
+Used to annotate a [pure function](https://en.wikipedia.org/wiki/Pure_function) - one whose output is just a function of its input.  
+
+Can also be used on classes to auotmatically annotate all of its methods as pure and ban the usage of properties.  
 
 ```php
 <?php
@@ -484,6 +492,15 @@ foo(
 );
 ```
 
+### `@psalm-impure`
+
+Used to annotate a function that is not pure (nor mutation free, nor externally mutation free): this is the default, but Psalm always asks to explicitly annotate **abstract** methods with one of these four annotations:
+
+- `@psalm-pure`
+- `@psalm-mutation-free`
+- `@psalm-external-mutation-free`
+- `@psalm-impure`
+
 ### `@psalm-allow-private-mutation`
 
 Used to annotate readonly properties that can be mutated in a private context. With this, public properties can be read from another class but only be mutated within a method of its own class.

docs/annotating_code/type_syntax/callable_types.md

@@ -28,9 +28,22 @@ echo $adder(true);
 
 ## Pure callables
 
-For situations where the `callable` needs to be pure or immutable, the subtypes `pure-callable` and `pure-Closure` are also available.
+For situations where the `callable` needs to be pure, mutation-free or externally mutation-free, the following subtypes are available:
 
-This can be useful when the `callable` is used in a function marked with `@psalm-pure` or `@psalm-mutation-free`, for example:
+* Pure (no mutations or even read property accesses allowed), equivalent to marking functions or methods with `@psalm-pure` 
+  * `pure-callable`
+  * `pure-Closure`
+* Mutation-free (only internal property reads on `$this` are allowed for methods), equivalent to marking functions or methods with `@psalm-mutation-free`
+  * `self-accessing-callable`
+  * `self-accessing-Closure`
+* Externally mutation-free (internal property reads and writes on `$this` and `self` are allowed for methods), equivalent to marking functions or methods with `@psalm-external-mutation-free`
+  * `self-mutating-callable`
+  * `self-mutating-Closure`
+* Impure (the default behavior, all mutations allowed); functions or methods can also be explicitly marked as impure with `@psalm-impure`
+  * `impure-callable` (an alias to `callable`)
+  * `impure-Closure` (an alias to `Closure`)
+
+This can be useful when the `callable` is used in a function marked with `@psalm-pure` or `@psalm-mutation-free` or `@psalm-external-mutation-free`, for example:
 
 ```php
 <?php

docs/running_psalm/command_line_usage.md

@@ -33,7 +33,9 @@ Currently, Shepherd tracks type coverage (the percentage of types Psalm can infe
 
 ## Running Psalm faster
 
-Psalm has a couple of command-line options that will result in faster builds:
+To run Psalm up to 50% faster, use the [official docker image](https://psalm.dev/docs/running_psalm/installation/#docker-image).  
+
+Psalm also has a couple of command-line options that will result in faster builds:
 
 - `--threads=[n]` to run Psalm’s analysis in a number of threads
 - `--diff` which only checks files you’ve updated since the last run (and their dependents).