Merge branch 'JuliaPy:main' into hh-timedelta64

Author: hhaensel

.github/workflows/tests-nightly.yml

@@ -38,6 +38,7 @@ jobs:
       - uses: julia-actions/julia-runtest@v1
         env:
           JULIA_DEBUG: PythonCall
+          JULIA_NUM_THREADS: '2'
       - uses: julia-actions/julia-processcoverage@v1
       - uses: codecov/codecov-action@v1
         with:

.github/workflows/tests.yml

@@ -43,6 +43,7 @@ jobs:
         uses: julia-actions/julia-runtest@v1
         env:
           JULIA_DEBUG: PythonCall
+          JULIA_NUM_THREADS: '2'
       - name: Process coverage
         uses: julia-actions/julia-processcoverage@v1
       - name: Upload coverage to Codecov
@@ -82,6 +83,8 @@ jobs:
       - name: Run tests
         run: |
           pytest -s --nbval --cov=pysrc ./pytest/
+        env:
+          PYTHON_JULIACALL_THREADS: '2'
       - name: Upload coverage to Codecov
         uses: codecov/codecov-action@v2
         env:

Project.toml

@@ -1,7 +1,7 @@
 name = "PythonCall"
 uuid = "6099a3de-0909-46bc-b1f4-468b9a2dfc0d"
 authors = ["Christopher Doris <github.com/cjdoris>"]
-version = "0.9.21"
+version = "0.9.23"
 
 [deps]
 CondaPkg = "992eb4ea-22a4-4c89-a5bb-47a3300528ab"

README.md

@@ -40,9 +40,8 @@ In this example we use the Python module JuliaCall from an IPython notebook to t
 
 ## What about PyCall?
 
-The existing package [PyCall](https://github.com/JuliaPy/PyCall.jl) is another similar interface to Python. Here we note some key differences, but a more detailed comparison is in the documentation.
+The existing package [PyCall](https://github.com/JuliaPy/PyCall.jl) is another similar interface to Python. Here we note some key differences:.
 - PythonCall supports a wider range of conversions between Julia and Python, and the conversion mechanism is extensible.
 - PythonCall by default never copies mutable objects when converting, but instead directly wraps the mutable object. This means that modifying the converted object modifies the original, and conversion is faster.
 - PythonCall does not usually automatically convert results to Julia values, but leaves them as Python objects. This makes it easier to do Pythonic things with these objects (e.g. accessing methods) and is type-stable.
-- PythonCall installs dependencies into a separate Conda environment for each Julia project. This means each Julia project can have an isolated set of Python dependencies.
-- PythonCall supports Julia 1.6.1+ and Python 3.8+ whereas PyCall supports Julia 0.7+ and Python 2.7+.
+- PythonCall installs dependencies into a separate Conda environment for each Julia project using [CondaPkg](https://github.com/JuliaPy/CondaPkg.jl). This means each Julia project can have an isolated set of Python dependencies.

bump.jl

@@ -22,7 +22,7 @@ function bumpver(file, pattern, oldver, newver)
 end
 
 bumpver("Project.toml", "version = \"{}\"\n", oldver, newver)
-bumpver("setup.cfg", "version = {}\n", oldver, newver)
+bumpver("pyproject.toml", "version = \"{}\"\n", oldver, newver)
 bumpver("pysrc/juliacall/__init__.py", "__version__ = '{}'\n", oldver, newver)
 bumpver("pysrc/juliacall/juliapkg.json", "\"version\": \"={}\"", oldver, newver)
 bumpver("pysrc/juliacall/juliapkg-dev.json", "\"version\": \"={}\"", oldver, newver)

docs/make.jl

@@ -19,7 +19,6 @@ makedocs(
         ],
         "compat.md",
         "faq.md",
-        "pycall.md",
         "releasenotes.md",
     ],
 )

docs/src/faq.md

@@ -1,22 +1,27 @@
 # FAQ & Troubleshooting
 
+## Can I use PythonCall and PyCall together?
+
+Yes, you can use both PyCall and PythonCall in the same Julia session. This is platform-dependent:
+- On most systems the Python interpreter used by PythonCall and PyCall must be the same (see below).
+- On Windows it appears to be possible for PythonCall and PyCall to use different interpreters.
+
+To force PythonCall to use the same Python interpreter as PyCall, set the environment variable [`JULIA_PYTHONCALL_EXE`](@ref pythoncall-config) to `"@PyCall"`. Note that this will opt out of automatic dependency management using CondaPkg.
+
+Alternatively, to force PyCall to use the same interpreter as PythonCall, set the environment variable `PYTHON` to [`PythonCall.python_executable_path()`](@ref) and then `Pkg.build("PyCall")`. You will need to do this each time you change project, because PythonCall by default uses a different Python for each project.
+
 ## Is PythonCall/JuliaCall thread safe?
 
-No.
-
-Some rules if you are writing multithreaded code:
-- Only call Python functions from the first thread.
-- You probably also need to call `PythonCall.GC.disable()` on the main thread before any
-  threaded block of code. Remember to call `PythonCall.GC.enable()` again afterwards.
-  (This is because Julia finalizers can be called from any thread.)
-- Julia intentionally causes segmentation faults as part of the GC safepoint mechanism.
-  If unhandled, these segfaults will result in termination of the process. To enable signal handling,
-  set `PYTHON_JULIACALL_HANDLE_SIGNALS=yes` before any calls to import juliacall. This is equivalent
-  to starting julia with `julia --handle-signals=yes`, the default behavior in Julia. 
-  See discussion [here](https://github.com/JuliaPy/PythonCall.jl/issues/219#issuecomment-1605087024) for more information.
-- You may still encounter problems.
-
-Related issues: [#201](https://github.com/JuliaPy/PythonCall.jl/issues/201), [#202](https://github.com/JuliaPy/PythonCall.jl/issues/202)
+Yes, as of v0.9.22, provided you handle the GIL correctly. See the guides for
+[PythonCall](@ref jl-multi-threading) and [JuliaCall](@ref py-multi-threading).
+
+Before, tricks such as disabling the garbage collector were required. See the
+[old docs](https://juliapy.github.io/PythonCall.jl/v0.9.21/faq/#Is-PythonCall/JuliaCall-thread-safe?).
+
+Related issues:
+[#201](https://github.com/JuliaPy/PythonCall.jl/issues/201),
+[#202](https://github.com/JuliaPy/PythonCall.jl/issues/202),
+[#529](https://github.com/JuliaPy/PythonCall.jl/pull/529)
 
 ## Issues when Numpy arrays are expected
 

docs/src/juliacall-reference.md

@@ -1,4 +1,4 @@
-# JuliaCall API Reference
+# [JuliaCall API Reference](@id jl-reference)
 
 ## Constants
 
@@ -93,8 +93,9 @@ replaced with `!!`.
 
 ###### Members
 - `_jl_raw()`: Convert to a [`RawValue`](#juliacall.RawValue). (See also [`pyjlraw`](@ref).)
-- `_jl_display()`: Display the object using Julia's display mechanism.
-- `_jl_help()`: Display help for the object.
+- `_jl_display(mime=None)`: Display the object using Julia's display mechanism.
+- `_jl_help(mime=None)`: Display help for the object.
+- `_jl_call_nogil(*args, **kwargs)`: Call this with the GIL disabled.
 `````
 
 `````@customdoc
@@ -217,4 +218,5 @@ single tuple, it will need to be wrapped in another tuple.
 ###### Members
 - `_jl_any()`: Convert to a [`AnyValue`](#juliacall.AnyValue) (or subclass). (See also
   [`pyjl`](@ref).)
+- `_jl_call_nogil(*args, **kwargs)`: Call this with the GIL disabled.
 `````

docs/src/juliacall.md

@@ -124,3 +124,79 @@ be configured in two ways:
 | `-X juliacall-threads=<N\|auto>` | `PYTHON_JULIACALL_THREADS=<N\|auto>` | Launch N threads. |
 | `-X juliacall-warn-overwrite=<yes\|no>` | `PYTHON_JULIACALL_WARN_OVERWRITE=<yes\|no>` | Enable or disable method overwrite warnings. |
 | `-X juliacall-autoload-ipython-extension=<yes\|no>` | `PYTHON_JULIACALL_AUTOLOAD_IPYTHON_EXTENSION=<yes\|no>` | Enable or disable IPython extension autoloading. |
+
+## [Multi-threading](@id py-multi-threading)
+
+From v0.9.22, JuliaCall supports multi-threading in Julia and/or Python, with some
+caveats.
+
+Most importantly, you can only call Python code while Python's
+[Global Interpreter Lock (GIL)](https://docs.python.org/3/glossary.html#term-global-interpreter-lock)
+is locked by the current thread. You can use JuliaCall from any Python thread, and the GIL
+will be locked whenever any JuliaCall function is used. However, to leverage the benefits
+of multi-threading, you can unlock the GIL while executing any Julia code that does not
+interact with Python.
+
+The simplest way to do this is using the `_jl_call_nogil` method on Julia functions to
+call the function with the GIL unlocked.
+
+```python
+from concurrent.futures import ThreadPoolExecutor, wait
+from juliacall import Main as jl
+pool = ThreadPoolExecutor(4)
+fs = [pool.submit(jl.Libc.systemsleep._jl_call_nogil, 5) for _ in range(4)]
+wait(fs)
+```
+
+In the above example, we call `Libc.systemsleep(5)` on four threads. Because we
+called it with `_jl_call_nogil`, the GIL was unlocked, allowing the threads to run in
+parallel, taking about 5 seconds in total.
+
+If we did not use `_jl_call_nogil` (i.e. if we did `pool.submit(jl.Libc.systemsleep, 5)`)
+then the above code will take 20 seconds because the sleeps run one after another.
+
+It is very important that any function called with `_jl_call_nogil` does not interact
+with Python at all unless it re-locks the GIL first, such as by using
+[PythonCall.GIL.@lock](@ref).
+
+You can also use [multi-threading from Julia](@ref jl-multi-threading).
+
+### Caveat: Julia's task scheduler
+
+If you try the above example with a Julia function that yields to the task scheduler,
+such as `sleep` instead of `Libc.systemsleep`, then you will likely experience a hang.
+
+In this case, you need to yield back to Julia's scheduler periodically to allow the task
+to continue. You can use the following pattern instead of `wait(fs)`:
+```python
+jl_yield = getattr(jl, "yield")
+while True:
+  # yield to Julia's task scheduler
+  jl_yield()
+  # wait for up to 0.1 seconds for the threads to finish
+  state = wait(fs, timeout=0.1)
+  # if they finished then stop otherwise try again
+  if not state.not_done:
+    break
+```
+
+Set the `timeout` parameter smaller to let Julia's scheduler cycle more frequently.
+
+Future versions of JuliaCall may provide tooling to make this simpler.
+
+### [Caveat: Signal handling](@id py-multi-threading-signal-handling)
+
+We recommend setting [`PYTHON_JULIACALL_HANDLE_SIGNALS=yes`](@ref julia-config)
+before importing JuliaCall with multiple threads.
+
+This is because Julia intentionally causes segmentation faults as part of the GC
+safepoint mechanism. If unhandled, these segfaults will result in termination of the
+process. See discussion
+[here](https://github.com/JuliaPy/PythonCall.jl/issues/219#issuecomment-1605087024)
+for more information.
+
+Note however that this interferes with Python's own signal handling, so for example
+Ctrl-C will not raise `KeyboardInterrupt`.
+
+Future versions of JuliaCall may make this the default behaviour when using multiple
+threads.