Merge remote-tracking branch 'origin/main' into pr/JamesWrigley/667

Author: Christopher Doris

docs/src/conversion-to-julia.md

@@ -31,6 +31,8 @@ From Python, the arguments to a Julia function will be converted according to th
 | `datetime.date`/`datetime.time`/`datetime.datetime`                                                          | `Date`/`Time`/`DateTime`                                    |
 | `datetime.timedelta`                                                                                         | `Microsecond` (or `Millisecond` or `Second` on overflow)    |
 | `numpy.intXX`/`numpy.uintXX`/`numpy.floatXX`                                                                 | `IntXX`/`UIntXX`/`FloatXX`                                  |
+| `numpy.datetime64`                                                                                           | `NumpyDates.DateTime64`                                     |
+| `numpy.timedelta64`                                                                                          | `NumpyDates.TimeDelta64`                                    |
 | **Standard priority (other reasonable conversions).**                                                        |                                                             |
 | `None`                                                                                                       | `Missing`                                                   |
 | `bytes`                                                                                                      | `Vector{UInt8}`, `Vector{Int8}`, `String`                   |
@@ -48,6 +50,8 @@ From Python, the arguments to a Julia function will be converted according to th
 | `ctypes.c_char_p`                                                                                            | `Cstring`, `Ptr{Cchar}`, `Ptr`                              |
 | `ctypes.c_wchar_p`                                                                                           | `Cwstring`, `Ptr{Cwchar}`, `Ptr`                            |
 | `numpy.bool_`/`numpy.intXX`/`numpy.uintXX`/`numpy.floatXX`                                                   | `Bool`, `Integer`, `Rational`, `Real`, `Number`             |
+| `numpy.datetime64`                                                                                           | `NumpyDates.InlineDateTime64`, `Dates.DateTime`             |
+| `numpy.timedelta64`                                                                                          | `NumpyDates.InlineTimeDelta64`, `Dates.Period`              |
 | Objects satisfying the buffer or array interface                                                             | `Array`, `AbstractArray`                                    |
 | **Low priority (fallback to `Py`).**                                                                         |                                                             |
 | Anything                                                                                                     | `Py`                                                        |

docs/src/faq.md

@@ -1,6 +1,6 @@
 # FAQ & Troubleshooting
 
-## Can I use PythonCall and PyCall together?
+## [Can I use PythonCall and PyCall together?](@id faq-pycall)
 
 Yes, you can use both PyCall and PythonCall in the same Julia session. This is platform-dependent:
 - On most systems the Python interpreter used by PythonCall and PyCall must be the same (see below).

docs/src/pythoncall-reference.md

@@ -255,3 +255,18 @@ PythonCall.getptr
 PythonCall.pydel!
 PythonCall.unsafe_pynext
 ```
+
+## NumpyDates
+
+The submodule `PythonCall.NumpyDates` provides types corresponding to Numpy's `datetime64` and `timedelta64` types. Enables conversion of these Numpy types (either as scalars or in arrays) to native Julia types.
+
+```@docs
+PythonCall.NumpyDates
+PythonCall.NumpyDates.AbstractDateTime64
+PythonCall.NumpyDates.InlineDateTime64
+PythonCall.NumpyDates.DateTime64
+PythonCall.NumpyDates.AbstractTimeDelta64
+PythonCall.NumpyDates.InlineTimeDelta64
+PythonCall.NumpyDates.TimeDelta64
+PythonCall.NumpyDates.Unit
+```

docs/src/pythoncall.md

@@ -240,17 +240,21 @@ to your current Julia project containing Python and any required Python packages
 ENV["JULIA_CONDAPKG_BACKEND"] = "Null"
 ENV["JULIA_PYTHONCALL_EXE"] = "/path/to/python"  # optional
 ENV["JULIA_PYTHONCALL_EXE"] = "@PyCall"  # optional
+ENV["JULIA_PYTHONCALL_EXE"] = "@venv" # optional
 ```
 
 By setting the CondaPkg backend to Null, it will never install any Conda packages. In this
 case, PythonCall will use whichever Python is currently installed and in your `PATH`. You
 must have already installed any Python packages that you need.
 
 If `python` is not in your `PATH`, you will also need to set `JULIA_PYTHONCALL_EXE` to its
-path.
+path. Relative paths are resolved relative to the current active project.
 
 If you also use PyCall, you can set `JULIA_PYTHONCALL_EXE=@PyCall` to use the same Python
-interpreter.
+interpreter. [See here](@ref faq-pycall).
+
+If you have a Python virtual environment at `.venv` in your current active project, you
+can set `JULIA_PYTHONCALL_EXE=@venv` to use it.
 
 #### If you already have a Conda environment
 

docs/src/releasenotes.md

@@ -1,6 +1,12 @@
 # Release Notes
 
 ## Unreleased
+* Added `NumpyDates`: NumPy-compatible DateTime64/TimeDelta64 types and units.
+* Added `pyconvert` rules for NumpyDates types.
+* Added `PyArray` support for NumPy arrays of `datetime64` and `timedelta64`.
+* Added `juliacall.ArrayValue` support for Julia arrays of `InlineDateTime64` and `InlineTimeDelta64`.
+* If `JULIA_PYTHONCALL_EXE` is a relative path, it is now considered relative to the active project.
+* Added option `JULIA_PYTHONCALL_EXE=@venv` to use a Python virtual environment relative to the active project.
 * Added `PYTHON_JULIACALL_EXE` and `PYTHON_JULIACALL_PROJECT` for specifying the Julia binary and project to override JuliaPkg.
 * Bug fixes.
 * Internal: switch from Requires.jl to package extensions.

src/C/context.jl

@@ -75,14 +75,38 @@ function init_context()
             exe_path = PyCall.python::String
             CTX.lib_path = PyCall.libpython::String
             CTX.which = :PyCall
+        elseif exe_path == "@venv"
+            # load from a .venv in the active project
+            exe_path = abspath(dirname(Base.active_project()), ".venv")
+            if Sys.iswindows()
+                exe_path = abspath(exe_path, "Scripts", "python.exe")::String
+            else
+                exe_path = abspath(exe_path, "bin", "python")::String
+            end
         elseif startswith(exe_path, "@")
             error("invalid JULIA_PYTHONCALL_EXE=$exe_path")
         else
             # Otherwise we use the Python specified
             CTX.which = :unknown
+            if isabspath(exe_path)
+                # nothing to do
+            elseif '/' in exe_path || '\\' in exe_path
+                # it's a relative path, interpret it as relative to the current project
+                exe_path = abspath(dirname(Base.active_project()), exe_path)::String
+            else
+                # it's a command, find it in the PATH
+                given_exe_path = exe_path
+                exe_path = Sys.which(exe_path)
+                exe_path === nothing &&
+                    error("Python executable $(repr(given_exe_path)) not found.")
+                exe_path::String
+            end
         end
 
         # Ensure Python is runnable
+        if !ispath(exe_path)
+            error("Python executable $(repr(exe_path)) does not exist.")
+        end
         try
             run(pipeline(`$exe_path --version`, stdout = devnull, stderr = devnull))
         catch

src/Convert/Convert.jl

@@ -9,6 +9,7 @@ using ..PythonCall
 using ..Utils
 using ..C
 using ..Core
+using ..NumpyDates
 
 using Dates: Date, Time, DateTime, Second, Millisecond, Microsecond, Nanosecond
 
@@ -20,8 +21,7 @@ import ..PythonCall:
     pyconvert,
     PyConvertPriority
 
-export
-    pyconvert_isunconverted,
+export pyconvert_isunconverted,
     pyconvert_result,
     pyconvert_result,
     pyconvert_tryconvert,

src/Convert/numpy.jl

@@ -9,6 +9,76 @@ function (::pyconvert_rule_numpysimplevalue{R,SAFE})(::Type{T}, x::Py) where {R,
     end
 end
 
+function pyconvert_rule_datetime64(::Type{DateTime64}, x::Py)
+    pyconvert_return(C.PySimpleObject_GetValue(DateTime64, x))
+end
+
+function pyconvert_rule_datetime64(::Type{T}, x::Py) where {T<:InlineDateTime64}
+    pyconvert_tryconvert(T, C.PySimpleObject_GetValue(DateTime64, x))
+end
+
+function pyconvert_rule_datetime64(::Type{T}, x::Py) where {T<:NumpyDates.DatesInstant}
+    d = C.PySimpleObject_GetValue(DateTime64, x)
+    if isnan(d)
+        pyconvert_unconverted()
+    else
+        pyconvert_tryconvert(T, d)
+    end
+end
+
+function pyconvert_rule_datetime64(::Type{Missing}, x::Py)
+    d = C.PySimpleObject_GetValue(DateTime64, x)
+    if isnan(d)
+        pyconvert_return(missing)
+    else
+        pyconvert_unconverted()
+    end
+end
+
+function pyconvert_rule_datetime64(::Type{Nothing}, x::Py)
+    d = C.PySimpleObject_GetValue(DateTime64, x)
+    if isnan(d)
+        pyconvert_return(nothing)
+    else
+        pyconvert_unconverted()
+    end
+end
+
+function pyconvert_rule_timedelta64(::Type{TimeDelta64}, x::Py)
+    pyconvert_return(C.PySimpleObject_GetValue(TimeDelta64, x))
+end
+
+function pyconvert_rule_timedelta64(::Type{T}, x::Py) where {T<:InlineTimeDelta64}
+    pyconvert_tryconvert(T, C.PySimpleObject_GetValue(TimeDelta64, x))
+end
+
+function pyconvert_rule_timedelta64(::Type{T}, x::Py) where {T<:NumpyDates.DatesPeriod}
+    d = C.PySimpleObject_GetValue(TimeDelta64, x)
+    if isnan(d)
+        pyconvert_unconverted()
+    else
+        pyconvert_tryconvert(T, d)
+    end
+end
+
+function pyconvert_rule_timedelta64(::Type{Missing}, x::Py)
+    d = C.PySimpleObject_GetValue(TimeDelta64, x)
+    if isnan(d)
+        pyconvert_return(missing)
+    else
+        pyconvert_unconverted()
+    end
+end
+
+function pyconvert_rule_timedelta64(::Type{Nothing}, x::Py)
+    d = C.PySimpleObject_GetValue(TimeDelta64, x)
+    if isnan(d)
+        pyconvert_return(missing)
+    else
+        pyconvert_unconverted()
+    end
+end
+
 const NUMPY_SIMPLE_TYPES = [
     ("bool_", Bool),
     ("int8", Int8),
@@ -28,6 +98,7 @@ const NUMPY_SIMPLE_TYPES = [
 ]
 
 function init_numpy()
+    # simple numeric scalar types
     for (t, T) in NUMPY_SIMPLE_TYPES
         isbool = occursin("bool", t)
         isint = occursin("int", t) || isbool
@@ -54,4 +125,36 @@ function init_numpy()
         iscomplex && pyconvert_add_rule(name, Complex, rule)
         isnumber && pyconvert_add_rule(name, Number, rule)
     end
+
+    # datetime64
+    pyconvert_add_rule(
+        "numpy:datetime64",
+        DateTime64,
+        pyconvert_rule_datetime64,
+        PYCONVERT_PRIORITY_ARRAY,
+    )
+    pyconvert_add_rule("numpy:datetime64", InlineDateTime64, pyconvert_rule_datetime64)
+    pyconvert_add_rule(
+        "numpy:datetime64",
+        NumpyDates.DatesInstant,
+        pyconvert_rule_datetime64,
+    )
+    pyconvert_add_rule("numpy:datetime64", Missing, pyconvert_rule_datetime64)
+    pyconvert_add_rule("numpy:datetime64", Nothing, pyconvert_rule_datetime64)
+
+    # timedelta64
+    pyconvert_add_rule(
+        "numpy:timedelta64",
+        TimeDelta64,
+        pyconvert_rule_timedelta64,
+        PYCONVERT_PRIORITY_ARRAY,
+    )
+    pyconvert_add_rule("numpy:timedelta64", InlineTimeDelta64, pyconvert_rule_timedelta64)
+    pyconvert_add_rule(
+        "numpy:timedelta64",
+        NumpyDates.DatesPeriod,
+        pyconvert_rule_timedelta64,
+    )
+    pyconvert_add_rule("numpy:timedelta64", Missing, pyconvert_rule_timedelta64)
+    pyconvert_add_rule("numpy:timedelta64", Nothing, pyconvert_rule_timedelta64)
 end

src/JlWrap/JlWrap.jl

@@ -7,6 +7,7 @@ module JlWrap
 
 using ..PythonCall
 using ..Utils
+using ..NumpyDates: NumpyDates
 using ..C
 using ..Core
 using ..Convert

src/JlWrap/array.jl

@@ -223,6 +223,8 @@ pyjlarray_isarrayabletype(::Type{T}) where {T} = T in (
     Complex{Float32},
     Complex{Float64},
 )
+pyjlarray_isarrayabletype(::Type{NumpyDates.InlineDateTime64{U}}) where {U} = true
+pyjlarray_isarrayabletype(::Type{NumpyDates.InlineTimeDelta64{U}}) where {U} = true
 pyjlarray_isarrayabletype(::Type{T}) where {T<:Tuple} =
     isconcretetype(T) &&
     Base.allocatedinline(T) &&
@@ -235,22 +237,45 @@ const PYTYPESTRDESCR = IdDict{Type,Tuple{String,Py}}()
 pytypestrdescr(::Type{T}) where {T} =
     get!(PYTYPESTRDESCR, T) do
         c = Utils.islittleendian() ? '<' : '>'
-        T == Bool ? ("$(c)b$(sizeof(Bool))", PyNULL) :
-        T == Int8 ? ("$(c)i1", PyNULL) :
-        T == UInt8 ? ("$(c)u1", PyNULL) :
-        T == Int16 ? ("$(c)i2", PyNULL) :
-        T == UInt16 ? ("$(c)u2", PyNULL) :
-        T == Int32 ? ("$(c)i4", PyNULL) :
-        T == UInt32 ? ("$(c)u4", PyNULL) :
-        T == Int64 ? ("$(c)i8", PyNULL) :
-        T == UInt64 ? ("$(c)u8", PyNULL) :
-        T == Float16 ? ("$(c)f2", PyNULL) :
-        T == Float32 ? ("$(c)f4", PyNULL) :
-        T == Float64 ? ("$(c)f8", PyNULL) :
-        T == Complex{Float16} ? ("$(c)c4", PyNULL) :
-        T == Complex{Float32} ? ("$(c)c8", PyNULL) :
-        T == Complex{Float64} ? ("$(c)c16", PyNULL) :
-        if isstructtype(T) && isconcretetype(T) && Base.allocatedinline(T)
+        if T == Bool
+            ("$(c)b$(sizeof(Bool))", PyNULL)
+        elseif T == Int8
+            ("$(c)i1", PyNULL)
+        elseif T == UInt8
+            ("$(c)u1", PyNULL)
+        elseif T == Int16
+            ("$(c)i2", PyNULL)
+        elseif T == UInt16
+            ("$(c)u2", PyNULL)
+        elseif T == Int32
+            ("$(c)i4", PyNULL)
+        elseif T == UInt32
+            ("$(c)u4", PyNULL)
+        elseif T == Int64
+            ("$(c)i8", PyNULL)
+        elseif T == UInt64
+            ("$(c)u8", PyNULL)
+        elseif T == Float16
+            ("$(c)f2", PyNULL)
+        elseif T == Float32
+            ("$(c)f4", PyNULL)
+        elseif T == Float64
+            ("$(c)f8", PyNULL)
+        elseif T == Complex{Float16}
+            ("$(c)c4", PyNULL)
+        elseif T == Complex{Float32}
+            ("$(c)c8", PyNULL)
+        elseif T == Complex{Float64}
+            ("$(c)c16", PyNULL)
+        elseif isconcretetype(T) &&
+               T <: Union{NumpyDates.InlineDateTime64,NumpyDates.InlineTimeDelta64}
+            u, m = NumpyDates.unitpair(T)
+            tc = T <: NumpyDates.InlineDateTime64 ? 'M' : 'm'
+            us =
+                u == NumpyDates.UNBOUND_UNITS ? "" :
+                m == 1 ? "[$(Symbol(u))]" : "[$(m)$(Symbol(u))]"
+            ("$(c)$(tc)8$(us)", PyNULL)
+        elseif isstructtype(T) && isconcretetype(T) && Base.allocatedinline(T)
             n = fieldcount(T)
             flds = []
             for i = 1:n