[FEATURE] Does it support compiling multiple Python files concurrently using multiple threads?

[17Reset]

I have a Python project containing thousands of files. Using PyArmor to obfuscate all Python code would be extremely time-consuming. Is it possible to support batch obfuscation of multiple files to improve speed?

[jondy]

Thread may not work, here it's Python doc

CPython implementation detail: In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing or concurrent.futures.ProcessPoolExecutor. However, threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.
https://docs.python.org/3.14/library/threading.html

I'll try to implement this experimental feature for pyarmor gen by ProcessPoolExecutor next week

[jondy]

I have uploaded pre-release v9.2.4 to enable this feature

https://pyarmor.dashingsoft.com/downloads/temp/pyarmor-9.2.4.tar.gz

Before pyarmor gen, enable it by this way

pyarmor cfg max_workers 1

The usage of this option
;; Enable ProcessPoolExecuter to generate the obfuscated scripts
;;
;; 0: normal build
;; 1: use ProcessPoolExecutor with max_workers=None
;; N > 1: use ProcessPoolExecutor with max_workers=N
max_workers = 0

[17Reset]

D:\Downloads\pyarmor-9.2.4>pip install .
Processing d:\downloads\pyarmor-9.2.4
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pyarmor.cli.core~=8.1.0 (from pyarmor==9.2.4)
  Downloading pyarmor_cli_core-8.1.0-cp313-cp313-win_amd64.whl.metadata (3.0 kB)
Downloading pyarmor_cli_core-8.1.0-cp313-cp313-win_amd64.whl (421 kB)
Building wheels for collected packages: pyarmor
  Building wheel for pyarmor (pyproject.toml) ... done
  Created wheel for pyarmor: filename=pyarmor-9.2.4-py3-none-any.whl size=3056565 sha256=126b92d39a332f636010b129da16f688a322bdff8f0fa9ce5285ee15651cb4b3
  Stored in directory: c:\users\xaccel\appdata\local\pip\cache\wheels\95\51\85\a7459291da2d2b6a361afde7fb60fe855163b9f356971ec9b6
Successfully built pyarmor
Installing collected packages: pyarmor.cli.core, pyarmor
  Attempting uninstall: pyarmor
    Found existing installation: pyarmor 9.2.4
    Can't uninstall 'pyarmor'. No files were found to uninstall.
Successfully installed pyarmor-9.2.4 pyarmor.cli.core-8.1.0

D:\Downloads\pyarmor-9.2.4>pyarmor --version
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "D:\App\Python\Scripts\pyarmor.exe\__main__.py", line 2, in <module>
    from pyarmor.cli.__main__ import main
  File "D:\Downloads\pyarmor-9.2.4\pyarmor\cli\__main__.py", line 33, in <module>
    from .generate import Builder
  File "D:\Downloads\pyarmor-9.2.4\pyarmor\cli\generate.py", line 26, in <module>
    from .core import Pytransform3
ModuleNotFoundError: No module named 'pyarmor.cli.core'

[jondy]

how about install it directly?

pip install https://pyarmor.dashingsoft.com/downloads/temp/pyarmor-9.2.4.tar.gz             

[17Reset]

OK, installation completed successfully, but the following error occurred during execution:

D:\Downloads>pip install https://pyarmor.dashingsoft.com/downloads/temp/pyarmor-9.2.4.tar.gz
Collecting https://pyarmor.dashingsoft.com/downloads/temp/pyarmor-9.2.4.tar.gz
  Downloading https://pyarmor.dashingsoft.com/downloads/temp/pyarmor-9.2.4.tar.gz (3.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 139.9 kB/s  0:00:20
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pyarmor.cli.core~=8.1.0 (from pyarmor==9.2.4)
  Using cached pyarmor_cli_core-8.1.0-cp313-cp313-win_amd64.whl.metadata (3.0 kB)
Using cached pyarmor_cli_core-8.1.0-cp313-cp313-win_amd64.whl (421 kB)
Building wheels for collected packages: pyarmor
  Building wheel for pyarmor (pyproject.toml) ... done
  Created wheel for pyarmor: filename=pyarmor-9.2.4-py3-none-any.whl size=3056593 sha256=b806b54cb91907f5451a277fc017f39a4c2f039bc9f2051199acc3ce33900289
  Stored in directory: c:\users\xaccel\appdata\local\pip\cache\wheels\a8\c9\eb\c0f5f6bc648479b8464774a0ca765c1556d1e83575a85435ad
Successfully built pyarmor
Installing collected packages: pyarmor.cli.core, pyarmor
Successfully installed pyarmor-9.2.4 pyarmor.cli.core-8.1.0

D:\Downloads>pyarmor --version
Pyarmor 9.2.4 (group), 006040, non-profits

License Type    : pyarmor-group
License No.     : pyarmor-vax-006040
License To      : xAccel
License Product : non-profits

BCC Mode        : Yes
RFT Mode        : Yes
CI/CD Mode      : No

Notes
* Offline obfuscation
447.22


Process SpawnProcess-7:
Process SpawnProcess-1:
Traceback (most recent call last):
Traceback (most recent call last):
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
Process SpawnProcess-3:
Traceback (most recent call last):
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
Process SpawnProcess-2:
Traceback (most recent call last):
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
Process SpawnProcess-6:
Traceback (most recent call last):
Process SpawnProcess-4:
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
Traceback (most recent call last):
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
Process SpawnProcess-5:
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
Traceback (most recent call last):
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
Process SpawnProcess-9:
Traceback (most recent call last):
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
Process SpawnProcess-13:
Process SpawnProcess-8:
Traceback (most recent call last):
Traceback (most recent call last):
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
Process SpawnProcess-10:
Process SpawnProcess-12:
Traceback (most recent call last):
Traceback (most recent call last):
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
Process SpawnProcess-11:
Traceback (most recent call last):
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
Process SpawnProcess-14:
Traceback (most recent call last):
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
Process SpawnProcess-15:
Traceback (most recent call last):
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
Process SpawnProcess-16:
Process SpawnProcess-18:
Traceback (most recent call last):
Traceback (most recent call last):
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
Process SpawnProcess-17:
Traceback (most recent call last):
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
Process SpawnProcess-20:
Traceback (most recent call last):
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
Process SpawnProcess-19:
Traceback (most recent call last):
  File "D:\App\Python\Lib\multiprocessing\process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "D:\App\Python\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\App\Python\Lib\concurrent\futures\process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "D:\App\Python\Lib\multiprocessing\queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
ModuleNotFoundError: No module named 'pyarmor.cli'; 'pyarmor' is not a package
MESSAGE_ERROR: A process in the process pool was terminated abruptly while the future was running or pending.

[jondy]

Comment this line in the source pyarmor/__init__.py

# sys.path.insert(0, abspath(dirname(__file__)))

The __init__.py of package pyarmor changes sys.path results in this issue.

The final solution is only installing package pyarmor.cli, because package pyarmor is only for compatible with old Pyarmor 7, and I don't want to touch it.