Merge branch 'master' into complex-type-hint-framework

Author: bdbaddog

CHANGES.txt

@@ -57,9 +57,44 @@ RELEASE  VERSION/DATE TO BE FILLED IN LATER
     - Fix of the --debug=sconscript option to return exist statements when using return
       statement with stop flag enabled
 
+  From Prabhu S. Khalsa:
+    - Fix typo in user documentation (issue #4458)
+
+  From Andrew Morrow:
+    - The NewParallel scheduler is now the default, the `tm_v2` flag is removed,
+      and the old scheduler is opt-in under `--experimental=legacy_sched`. Additionally,
+      the new scheduler is now used for -j1 builds as well.
+    - A python interpreter with support for the `threading` package is now required,
+      and this is enforced on startup. SCons currently sets its minimum supported
+      Python to 3.6, and it was not until Python 3.7 where `threading` became
+      default supported. In practice, we expect most real world Python 3.6 deployments
+      will have `threading` support enabled, so this will not be an issue.
+    - CacheDir writes no longer happen within the taskmaster critical section,
+      and therefore can run in parallel with both other CacheDir writes and the
+      taskmaster DAG walk.
+    - The NewParallel scheduler now only adds threads as new work requiring execution
+      is discovered, up to the limit set by -j. This should reduce resource utilization
+      when the achievable parallelism in the DAG is less than the -j limit.
+
   From Mats Wichmann:
     - Add support for Python 3.13 (as of alpha 2). So far only affects
       expected bytecodes in ActionTests.py.
+    - sconsign cleanup - remove some dead code, minor manpage tweaks.
+    - Be more cautious about encodings fetching command output on Windows.
+      Problem occurs in piped-spawn scenario, used by Configure tests.
+      Fixes #3529.
+    - Clarify/fix documentation of Scanners in User Guide and Manpage.
+      Fixes #4468.
+    - Fix bad typing in Action.py: process() and strfunction().
+    - Add Pseudo() to global functions, had been omitted. Fixes #4474.
+    - Improve handling of file data that SCons itself processes - try
+      harder to decode non-UTF-8 text. SCons.Util.to_Text now exists
+      to convert a byte stream, such as "raw" file data.  Fixes #3569, #4462.
+      The Pseudo manpage entry was updated to provide more clarity.
+    - The internal routine which implements the PyPackageDir function
+      would fail with an exception if called with a module which is
+      not found.  It will now return None.  Updated manpage entry and
+      docstring..
 
 
 RELEASE 4.6.0 -  Sun, 19 Nov 2023 17:22:20 -0700
@@ -7951,4 +7986,3 @@ A brief overview of important functionality available in release 0.01:
   - Linux packages available in RPM and Debian format.
 
   - Windows installer available.
-

CONTRIBUTING.rst

@@ -297,13 +297,13 @@ You may specifically list one or more tests to be run::
 
         $ python runtest.py SCons/BuilderTests.py
 
-        $ python runtest.py test/option-j.py test/Program.py
+        $ python runtest.py test/option/option-j.py test/Program.py
 
 You also use the ``-f`` option to execute just the tests listed in a specified
 text file::
 
         $ cat testlist.txt
-        test/option-j.py
+        test/option/option-j.py
         test/Program.py
         $ python runtest.py -f testlist.txt
 

RELEASE.txt

@@ -33,6 +33,7 @@ CHANGED/ENHANCED EXISTING FUNCTIONALITY
   that the generated function argument list matches the function's
   prototype when including a header file. Fixes GH Issue #4320
 - Now supports pre-release Python 3.13
+- Support for Python versions without support for the `threading` package has been removed
 
 FIXES
 -----
@@ -49,12 +50,32 @@ FIXES
 - MSVS: Fix the msvs project generation test scripts so that "false positive" tests
   results are not possible when the initial build is successful and the command-line
   build of the project file fails.
+- On Windows platform, when collecting command output (Configure checks),
+  make sure decoding of bytes doesn't fail.
+- Documentation indicated that both Pseudo() and env.Pseudo() were usable,
+  but Pseudo() did not work; is now enabled.
+- Improve handling of file data that SCons itself processes - as in
+  scanners - try harder to decode non-UTF-8 text.
+- PyPackageDir no longer fails if passed a module name which cannot be found,
+  now returns None.
+
 
 IMPROVEMENTS
 ------------
 
 - Use of NotImplemented instead of NotImplementedError for special methods
   of _ListVariable class
+- The NewParallel scheduler is now the default, the `tm_v2` flag is removed,
+  and the old scheduler is opt-in under `--experimental=legacy_sched`. Additionally,
+  the new scheduler is now used for -j1 builds as well.
+  NOTE: This should significantly improve SCons performance for larger parallel builds
+        (Larger -j values)
+- CacheDir writes no longer happen within the taskmaster critical section, and therefore
+  can run in parallel with both other CacheDir writes and the taskmaster DAG walk.
+- The NewParallel scheduler now only adds threads as new work requiring execution
+  is discovered, up to the limit set by -j. This should reduce resource utilization
+  when the achievable parallelism in the DAG is less than the -j limit.
+
 
 PACKAGING
 ---------
@@ -64,9 +85,10 @@ PACKAGING
 DOCUMENTATION
 -------------
 
-- List any significant changes to the documentation (not individual
-  typo fixes, even if they're mentioned in src/CHANGES.txt to give
-  the contributor credit)
+- Fixed the Scanner examples in the User Guide to be runnable and added
+  some more explantion. Clarified discussion of the scanner function in
+  the Scanner Objects section of the manpage.
+- The manpage entry for Pseudo was clarified.
 
 DEVELOPMENT
 -----------

SCons/Action.py

@@ -109,7 +109,7 @@
 from abc import ABC, abstractmethod
 from collections import OrderedDict
 from subprocess import DEVNULL, PIPE
-from typing import Optional
+from typing import List, Optional, Tuple
 
 import SCons.Debug
 import SCons.Errors
@@ -118,7 +118,7 @@
 
 # we use these a lot, so try to optimize them
 from SCons.Debug import logInstanceCreation
-from SCons.Subst import SUBST_SIG, SUBST_RAW
+from SCons.Subst import SUBST_CMD, SUBST_RAW, SUBST_SIG
 from SCons.Util import is_String, is_List
 from SCons.Util.sctyping import ExecutorType
 
@@ -129,13 +129,10 @@ class _null:
 execute_actions = True
 print_actions_presub = False
 
-# Use pickle protocol 1 when pickling functions for signature
-# otherwise python3 and python2 will yield different pickles
-# for the same object.
-# This is due to default being 1 for python 2.7, and 3 for 3.x
-# TODO: We can roll this forward to 2 (if it has value), but not
-# before a deprecation cycle as the sconsigns will change
-ACTION_SIGNATURE_PICKLE_PROTOCOL = 1
+# Use pickle protocol 4 when pickling functions for signature.
+# This is the common format since Python 3.4
+# TODO: use is commented out as not stable since 2017: e0bc3a04d5. Drop?
+# ACTION_SIGNATURE_PICKLE_PROTOCOL = 4
 
 
 def rfile(n):
@@ -450,7 +447,7 @@ def _do_create_action(act, kw):
         return act
 
     if is_String(act):
-        var=SCons.Util.get_environment_var(act)
+        var = SCons.Util.get_environment_var(act)
         if var:
             # This looks like a string that is purely an Environment
             # variable reference, like "$FOO" or "${FOO}".  We do
@@ -1012,18 +1009,19 @@ def __str__(self) -> str:
             return ' '.join(map(str, self.cmd_list))
         return str(self.cmd_list)
 
-    def process(self, target, source, env, executor: Optional[ExecutorType] = None, overrides: bool=False):
+
+    def process(self, target, source, env, executor=None, overrides: Optional[dict] = None) -> Tuple[List, bool, bool]:
         if executor:
-            result = env.subst_list(self.cmd_list, 0, executor=executor, overrides=overrides)
+            result = env.subst_list(self.cmd_list, SUBST_CMD, executor=executor, overrides=overrides)
         else:
-            result = env.subst_list(self.cmd_list, 0, target, source, overrides=overrides)
-        silent = None
-        ignore = None
+            result = env.subst_list(self.cmd_list, SUBST_CMD, target, source, overrides=overrides)
+        silent = False
+        ignore = False
         while True:
             try: c = result[0][0][0]
             except IndexError: c = None
-            if c == '@': silent = 1
-            elif c == '-': ignore = 1
+            if c == '@': silent = True
+            elif c == '-': ignore = True
             else: break
             result[0][0] = result[0][0][1:]
         try:
@@ -1033,7 +1031,7 @@ def process(self, target, source, env, executor: Optional[ExecutorType] = None,
             pass
         return result, ignore, silent
 
-    def strfunction(self, target, source, env, executor: Optional[ExecutorType] = None, overrides: bool=False):
+    def strfunction(self, target, source, env, executor: Optional[ExecutorType] = None, overrides: Optional[dict] = None) -> str:
         if self.cmdstr is None:
             return None
         if self.cmdstr is not _null:

SCons/Environment.py

@@ -2340,6 +2340,7 @@ def Local(self, *targets):
         return ret
 
     def Precious(self, *targets):
+        """Mark *targets* as precious: do not delete before building."""
         tlist = []
         for t in targets:
             tlist.extend(self.arg2nodes(t, self.fs.Entry))
@@ -2348,6 +2349,7 @@ def Precious(self, *targets):
         return tlist
 
     def Pseudo(self, *targets):
+        """Mark *targets* as pseudo: must not exist."""
         tlist = []
         for t in targets:
             tlist.extend(self.arg2nodes(t, self.fs.Entry))
@@ -2356,13 +2358,17 @@ def Pseudo(self, *targets):
         return tlist
 
     def Repository(self, *dirs, **kw) -> None:
+        """Specify Repository directories to search."""
         dirs = self.arg2nodes(list(dirs), self.fs.Dir)
         self.fs.Repository(*dirs, **kw)
 
     def Requires(self, target, prerequisite):
-        """Specify that 'prerequisite' must be built before 'target',
-        (but 'target' does not actually depend on 'prerequisite'
-        and need not be rebuilt if it changes)."""
+        """Specify that *prerequisite* must be built before *target*.
+
+        Creates an order-only relationship, not a full dependency.
+        *prerequisite* must exist before *target* can be built, but
+        a change to *prerequisite* does not trigger a rebuild of *target*.
+        """
         tlist = self.arg2nodes(target, self.fs.Entry)
         plist = self.arg2nodes(prerequisite, self.fs.Entry)
         for t in tlist:

SCons/Environment.xml

@@ -2880,20 +2880,41 @@ and &f-link-env-Prepend;.
 </arguments>
 <summary>
 <para>
-This returns a Directory Node similar to Dir.
-The python module / package is looked up and if located
-the directory is returned for the location.
-<parameter>modulename</parameter>
-Is a named python package / module to
-lookup the directory for it's location.
-</para>
-<para>
-If
-<parameter>modulename</parameter>
-is a list, SCons returns a list of Dir nodes.
+Finds the location of <parameter>modulename</parameter>,
+which can be a string or a sequence of strings,
+each representing the name of a &Python; module.
 Construction variables are expanded in
 <parameter>modulename</parameter>.
+Returns a Directory Node (see &f-link-Dir;),
+or a list of Directory Nodes if
+<parameter>modulename</parameter> is a sequence.
+<literal>None</literal> is returned for any module not found.
+</para>
+
+<para>
+When a Tool module which is installed as a
+&Python; module is used, you need
+to specify a <parameter>toolpath</parameter> argument to
+&f-link-Tool;,
+&f-link-Environment;
+or &f-link-Clone;,
+as tools outside the standard project locations
+(<filename>site_scons/site_tools</filename>)
+will not be found otherwise.
+Using &f-PyPackageDir; allows this path to be
+discovered at runtime instead of hardcoding the path.
 </para>
+
+<para>
+Example:
+</para>
+
+<example_commands>
+env = Environment(
+    tools=["default", "ExampleTool"],
+    toolpath=[PyPackageDir("example_tool")]
+)
+</example_commands>
 </summary>
 </scons_function>
 
@@ -2988,6 +3009,10 @@ but the target file(s) do not actually
 depend on the prerequisites
 and will not be rebuilt simply because
 the prerequisite file(s) change.
+<parameter>target</parameter> and
+<parameter>prerequisite</parameter> may each
+be a string or Node, or a list of strings or Nodes.
+Returns a list of the affected target nodes.
 </para>
 
 <para>