Skip to content

Conversation

@jonatanklosko
Copy link
Member

This is actually a bug upstream in pybind11, which is a popular library for implementing C extension modules in Python. The bug concerns the GIL, it has been reported a long time ago (pybind/pybind11#2888) and still exists. I found another viable way for us to manage the GIL, such that we don't run into that issue. More details in the code comment.

For the record, below is the code that reliably reproduced the issue for me.

script.exs
IO.gets("PID: #{:os.getpid()} Press enter...")

Mix.install([
  {:pythonx, path: "~/git/pythonx"},
])

Pythonx.uv_init("""
[project]
name = "project"
version = "0.0.0"
requires-python = "==3.13.*"
dependencies = [
  "matplotlib"
]
""")

globals = %{}

# Start a long-running evaluation that releases the GIL, it likely
# runs on the first dirty scheduler thread. This way we can start
# another evaluation and it will run on another dirty scheduler
# thread.
spawn(fn ->
  {_, _globals} =
    Pythonx.eval(
      """
      import time
      time.sleep(20)
      """,
      %{}
    )
end)

Process.sleep(100)

{_, globals} =
  Pythonx.eval(
    """
    import matplotlib
    matplotlib.use("Agg")

    import matplotlib.pyplot as plt
    plt.plot([1, 2], [1, 2])
    """,
    globals
  )

{_, _globals} =
  Pythonx.eval(
    """
    import io
    buffer = io.BytesIO()
    plt.gcf().savefig(buffer, format="png", bbox_inches="tight")
    """,
    globals
  )

(I also removed the interpreter finalization NIF, because we don't use and it's unlikely we ever will, so it's not worth keeping it in sync).

@jonatanklosko
Copy link
Member Author

Looks like it doesn't work on Windows Python 3.10, I will keep investigating.

@jonatanklosko jonatanklosko merged commit 1dd28d0 into main Feb 25, 2025
9 checks passed
@jonatanklosko jonatanklosko deleted the jk-gil branch February 25, 2025 06:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants