Skip to content

Commit d02f723

Browse files
antonwolfyantonwolfy
authored
Merge branch 'master' into impl-binary_repr
2 parents 356408f + 050f9b3 commit d02f723

File tree

8 files changed

+237
-99
lines changed

8 files changed

+237
-99
lines changed

‎README.md‎

Lines changed: 44 additions & 85 deletions
Original file line numberDiff line numberDiff line change
@@ -6,117 +6,76 @@
66
[![Build Sphinx](https://github.com/IntelPython/dpnp/workflows/Build%20Sphinx/badge.svg)](https://intelpython.github.io/dpnp)
77
[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/IntelPython/dpnp/badge)](https://securityscorecards.dev/viewer/?uri=github.com/IntelPython/dpnp)
88

9+
<img align="left" src="https://spec.oneapi.io/oneapi-logo-white-scaled.jpg" alt="oneAPI logo" width="75"/>
10+
911
# DPNP - Data Parallel Extension for NumPy*
12+
13+
Data Parallel Extension for NumPy* or `dpnp` is a Python library that
14+
implements a subset of NumPy* that can be executed on any data parallel device.
15+
The subset is a drop-in replacement of core NumPy* functions and numerical data types.
16+
1017
[API coverage summary](https://intelpython.github.io/dpnp/reference/comparison.html#summary)
1118

1219
[Full documentation](https://intelpython.github.io/dpnp/)
1320

14-
[DPNP C++ backend documentation](https://intelpython.github.io/dpnp/backend_doc/)
21+
`Dpnp` is the core part of a larger family of [data-parallel Python libraries and tools](https://www.intel.com/content/www/us/en/developer/tools/oneapi/distribution-for-python.html)
22+
to program on XPUs.
1523

16-
## Build from source:
17-
Ensure you have the following prerequisite packages installed:
1824

19-
- `cython`
20-
- `cmake >=3.21`
21-
- `dpcpp_linux-64` or `dpcpp_win-64` (depending on your OS)
22-
- `dpctl`
23-
- `mkl-devel-dpcpp`
24-
- `onedpl-devel`
25-
- `ninja`
26-
- `numpy >=1.19,<1.25a0`
27-
- `python`
28-
- `scikit-build`
29-
- `setuptools`
30-
- `sysroot_linux-64 >=2.28` (only on Linux OS)
31-
- `tbb-devel`
25+
# Installing
3226

33-
After these steps, `dpnp` can be built in debug mode as follows:
27+
You can install the library using `conda`, `mamba` or [pip](https://pypi.org/project/dpnp/)
28+
package managers. It is also available as part of the [Intel(R) Distribution for Python](https://www.intel.com/content/www/us/en/developer/tools/oneapi/distribution-for-python.html)
29+
(IDP).
3430

35-
```bash
36-
git clone https://github.com/IntelPython/dpnp
37-
cd dpnp
38-
python scripts/build_locally.py
39-
```
31+
## Intel(R) Distribution for Python
4032

41-
## Install Wheel Package via pip
42-
Install DPNP
43-
```cmd
44-
python -m pip install --index-url https://software.repos.intel.com/python/pypi dpnp
45-
```
33+
You can find the most recent release of `dpnp` every quarter as part of the IDP
34+
releases.
4635

47-
Set path to Performance Libraries in case of using venv or system Python:
48-
```cmd
49-
export LD_LIBRARY_PATH=<path_to_your_env>/lib
50-
```
36+
To get the library from the latest release, follow the instructions from
37+
[Get Started With Intel® Distribution for Python](https://www.intel.com/content/www/us/en/developer/articles/technical/get-started-with-intel-distribution-for-python.html).
5138

52-
It is also required to set following environment variables:
53-
```cmd
54-
export OCL_ICD_FILENAMES_RESET=1
55-
export OCL_ICD_FILENAMES=libintelocl.so
56-
```
39+
## Conda
5740

58-
## Run test
59-
```bash
60-
pytest
61-
# or
62-
pytest tests/test_matmul.py -s -v
63-
# or
64-
python -m unittest tests/test_mixins.py
65-
```
41+
To install `dpnp` from the Intel(R) conda channel, use the following command:
6642

67-
## Run numpy external test
6843
```bash
69-
. ./0.env.sh
70-
python -m tests.third_party.numpy_ext
71-
# or
72-
python -m tests.third_party.numpy_ext core/tests/test_umath.py
73-
# or
74-
python -m tests.third_party.numpy_ext core/tests/test_umath.py::TestHypot::test_simple
44+
conda install dpnp -c https://software.repos.intel.com/python/conda/ -c conda-forge
7545
```
7646

77-
### Building documentation:
47+
## Pip
48+
49+
The `dpnp` can be installed using `pip` obtaining wheel packages either from
50+
PyPi or from Intel(R) channel. To install `dpnp` wheel package from Intel(R)
51+
channel, run the following command:
52+
7853
```bash
79-
Prerequisites:
80-
$ conda install sphinx sphinx_rtd_theme
81-
Building:
82-
1. Install dpnp into your python environment
83-
2. $ cd doc && make html
84-
3. The documentation will be in doc/_build/html
54+
python -m pip install --index-url https://software.repos.intel.com/python/pypi dpnp
8555
```
8656

87-
## Packaging:
57+
## Installing the bleeding edge
58+
59+
To try out the latest features, install `dpnp` using our development channel on
60+
Anaconda cloud:
61+
8862
```bash
89-
. ./0.env.sh
90-
conda-build conda-recipe/
63+
conda install dpnp -c dppy/label/dev -c https://software.repos.intel.com/python/conda/ -c conda-forge
9164
```
9265

93-
## Run benchmark:
94-
```bash
95-
cd benchmarks/
9666

97-
asv run --python=python --bench <filename without .py>
98-
# example:
99-
asv run --python=python --bench bench_elementwise
67+
# Building
10068

101-
# or
69+
Refer to our [Documentation](https://intelpython.github.io/dpnp/quick_start_guide.html)
70+
for more information on setting up a development environment and building `dpnp`
71+
from the source.
10272

103-
asv run --python=python --bench <class>.<bench>
104-
# example:
105-
asv run --python=python --bench Elementwise.time_square
10673

107-
# add --quick option to run every case once but looks like first execution has additional overheads and takes a lot of time (need to be investigated)
108-
```
74+
# Running Tests
10975

76+
Tests are located in folder [dpnp/tests](dpnp/tests).
11077

111-
## Tests matrix:
112-
| # |Name |OS |distributive|interpreter|python used from|SYCL queue manager|build commands set |forced environment |
113-
|---|------------------------------------|-----|------------|-----------|:--------------:|:----------------:|------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------|
114-
|1 |Ubuntu 20.04 Python37 |Linux|Ubuntu 20.04|Python 3.7 | IntelOneAPI | local |export DPNP_DEBUG=1 python setup.py clean python setup.py build_clib python setup.py build_ext --inplace pytest |cmake-3.19.2, valgrind, pytest-valgrind, conda-build, pytest, hypothesis |
115-
|2 |Ubuntu 20.04 Python38 |Linux|Ubuntu 20.04|Python 3.8 | IntelOneAPI | local |export DPNP_DEBUG=1 python setup.py clean python setup.py build_clib python setup.py build_ext --inplace pytest |cmake-3.19.2, valgrind, pytest-valgrind, conda-build, pytest, hypothesis |
116-
|3 |Ubuntu 20.04 Python39 |Linux|Ubuntu 20.04|Python 3.9 | IntelOneAPI | local |export DPNP_DEBUG=1 python setup.py clean python setup.py build_clib python setup.py build_ext --inplace pytest |cmake-3.19.2, valgrind, pytest-valgrind, conda-build, pytest, hypothesis |
117-
|4 |Ubuntu 20.04 External Tests Python37|Linux|Ubuntu 20.04|Python 3.7 | IntelOneAPI | local |export DPNP_DEBUG=1 python setup.py clean python setup.py build_clib python setup.py build_ext --inplace python -m tests_external.numpy.runtests|cmake-3.19.2, valgrind, pytest-valgrind, conda-build, pytest, hypothesis |
118-
|5 |Ubuntu 20.04 External Tests Python38|Linux|Ubuntu 20.04|Python 3.8 | IntelOneAPI | local |export DPNP_DEBUG=1 python setup.py clean python setup.py build_clib python setup.py build_ext --inplace python -m tests_external.numpy.runtests|cmake-3.19.2, valgrind, pytest-valgrind, conda-build, pytest, hypothesis |
119-
|6 |Ubuntu 20.04 External Tests Python39|Linux|Ubuntu 20.04|Python 3.9 | IntelOneAPI | local |export DPNP_DEBUG=1 python setup.py clean python setup.py build_clib python setup.py build_ext --inplace python -m tests_external.numpy.runtests|cmake-3.19.2, valgrind, pytest-valgrind, conda-build, pytest, hypothesis |
120-
|7 |Code style |Linux|Ubuntu 20.04|Python 3.8 | IntelOneAPI | local |python ./setup.py style |cmake-3.19.2, valgrind, pytest-valgrind, conda-build, pytest, hypothesis, conda-verify, pycodestyle, autopep8, black |
121-
|8 |Valgrind |Linux|Ubuntu 20.04| | IntelOneAPI | local |export DPNP_DEBUG=1 python setup.py clean python setup.py build_clib python setup.py build_ext --inplace |cmake-3.19.2, valgrind, pytest-valgrind, conda-build, pytest, hypothesis |
122-
|9 |Code coverage |Linux|Ubuntu 20.04|Python 3.8 | IntelOneAPI | local |export DPNP_DEBUG=1 python setup.py clean python setup.py build_clib python setup.py build_ext --inplace |cmake-3.19.2, valgrind, pytest-valgrind, conda-build, pytest, hypothesis, conda-verify, pycodestyle, autopep8, pytest-cov|
78+
To run the tests, use:
79+
```bash
80+
python -m pytest --pyargs dpnp
81+
```

‎dpnp/dpnp_iface_statistics.py‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@
6565
"amin",
6666
"average",
6767
"bincount",
68+
"corrcoef",
6869
"correlate",
6970
"cov",
7071
"max",
@@ -360,6 +361,127 @@ def bincount(x1, weights=None, minlength=0):
360361
return call_origin(numpy.bincount, x1, weights=weights, minlength=minlength)
361362

362363

364+
def corrcoef(x, y=None, rowvar=True, *, dtype=None):
365+
"""
366+
Return Pearson product-moment correlation coefficients.
367+
368+
For full documentation refer to :obj:`numpy.corrcoef`.
369+
370+
Parameters
371+
----------
372+
x : {dpnp.ndarray, usm_ndarray}
373+
A 1-D or 2-D array containing multiple variables and observations.
374+
Each row of `x` represents a variable, and each column a single
375+
observation of all those variables. Also see `rowvar` below.
376+
y : {None, dpnp.ndarray, usm_ndarray}, optional
377+
An additional set of variables and observations. `y` has the same
378+
shape as `x`.
379+
Default: ``None``.
380+
rowvar : {bool}, optional
381+
If `rowvar` is ``True``, then each row represents a variable,
382+
with observations in the columns. Otherwise, the relationship
383+
is transposed: each column represents a variable, while the rows
384+
contain observations.
385+
Default: ``True``.
386+
dtype : {None, dtype}, optional
387+
Data-type of the result.
388+
Default: ``None``.
389+
390+
Returns
391+
-------
392+
R : {dpnp.ndarray}
393+
The correlation coefficient matrix of the variables.
394+
395+
See Also
396+
--------
397+
:obj:`dpnp.cov` : Covariance matrix.
398+
399+
Examples
400+
--------
401+
In this example we generate two random arrays, ``xarr`` and ``yarr``, and
402+
compute the row-wise and column-wise Pearson correlation coefficients,
403+
``R``. Since `rowvar` is true by default, we first find the row-wise
404+
Pearson correlation coefficients between the variables of ``xarr``.
405+
406+
>>> import dpnp as np
407+
>>> np.random.seed(123)
408+
>>> xarr = np.random.rand(3, 3).astype(np.float32)
409+
>>> xarr
410+
array([[7.2858386e-17, 2.2066992e-02, 3.9520904e-01],
411+
[4.8012391e-01, 5.9377134e-01, 4.5147297e-01],
412+
[9.0728188e-01, 9.9387854e-01, 5.8399546e-01]], dtype=float32)
413+
>>> R1 = np.corrcoef(xarr)
414+
>>> R1
415+
array([[ 0.99999994, -0.6173796 , -0.9685411 ],
416+
[-0.6173796 , 1. , 0.7937219 ],
417+
[-0.9685411 , 0.7937219 , 0.9999999 ]], dtype=float32)
418+
419+
If we add another set of variables and observations ``yarr``, we can
420+
compute the row-wise Pearson correlation coefficients between the
421+
variables in ``xarr`` and ``yarr``.
422+
423+
>>> yarr = np.random.rand(3, 3).astype(np.float32)
424+
>>> yarr
425+
array([[0.17615308, 0.65354985, 0.15716429],
426+
[0.09373496, 0.2123185 , 0.84086883],
427+
[0.9011005 , 0.45206687, 0.00225109]], dtype=float32)
428+
>>> R2 = np.corrcoef(xarr, yarr)
429+
>>> R2
430+
array([[ 0.99999994, -0.6173796 , -0.968541 , -0.48613155, 0.9951523 ,
431+
-0.8900264 ],
432+
[-0.6173796 , 1. , 0.7937219 , 0.9875833 , -0.53702235,
433+
0.19083664],
434+
[-0.968541 , 0.7937219 , 0.9999999 , 0.6883078 , -0.9393724 ,
435+
0.74857277],
436+
[-0.48613152, 0.9875833 , 0.6883078 , 0.9999999 , -0.39783284,
437+
0.0342579 ],
438+
[ 0.9951523 , -0.53702235, -0.9393725 , -0.39783284, 0.99999994,
439+
-0.9305482 ],
440+
[-0.89002645, 0.19083665, 0.7485727 , 0.0342579 , -0.9305482 ,
441+
1. ]], dtype=float32)
442+
443+
Finally if we use the option ``rowvar=False``, the columns are now
444+
being treated as the variables and we will find the column-wise Pearson
445+
correlation coefficients between variables in ``xarr`` and ``yarr``.
446+
447+
>>> R3 = np.corrcoef(xarr, yarr, rowvar=False)
448+
>>> R3
449+
array([[ 1. , 0.9724453 , -0.9909503 , 0.8104691 , -0.46436927,
450+
-0.1643624 ],
451+
[ 0.9724453 , 1. , -0.9949381 , 0.6515728 , -0.6580445 ,
452+
0.07012729],
453+
[-0.99095035, -0.994938 , 1. , -0.72450536, 0.5790461 ,
454+
0.03047091],
455+
[ 0.8104691 , 0.65157276, -0.72450536, 1. , 0.14243561,
456+
-0.71102554],
457+
[-0.4643693 , -0.6580445 , 0.57904613, 0.1424356 , 0.99999994,
458+
-0.79727215],
459+
[-0.1643624 , 0.07012729, 0.03047091, -0.7110255 , -0.7972722 ,
460+
0.99999994]], dtype=float32)
461+
"""
462+
463+
out = dpnp.cov(x, y, rowvar, dtype=dtype)
464+
if out.ndim == 0:
465+
# scalar covariance
466+
# nan if incorrect value (nan, inf, 0), 1 otherwise
467+
return out / out
468+
469+
d = dpnp.diag(out)
470+
471+
stddev = dpnp.sqrt(d.real)
472+
out /= stddev[:, None]
473+
out /= stddev[None, :]
474+
475+
# Clip real and imaginary parts to [-1, 1]. This does not guarantee
476+
# abs(a[i,j]) <= 1 for complex arrays, but is the best we can do without
477+
# excessive work.
478+
dpnp.clip(out.real, -1, 1, out=out.real)
479+
if dpnp.iscomplexobj(out):
480+
dpnp.clip(out.imag, -1, 1, out=out.imag)
481+
482+
return out
483+
484+
363485
def correlate(x1, x2, mode="valid"):
364486
"""
365487
Cross-correlation of two 1-dimensional sequences.

‎tests/skipped_tests.tbl‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -310,11 +310,6 @@ tests/third_party/cupy/random_tests/test_sample.py::TestRandomIntegers2::test_bo
310310
tests/third_party/cupy/random_tests/test_sample.py::TestRandomIntegers2::test_goodness_of_fit
311311
tests/third_party/cupy/random_tests/test_sample.py::TestRandomIntegers2::test_goodness_of_fit_2
312312

313-
tests/third_party/cupy/statistics_tests/test_correlation.py::TestCorrcoef::test_corrcoef
314-
tests/third_party/cupy/statistics_tests/test_correlation.py::TestCorrcoef::test_corrcoef_diag_exception
315-
tests/third_party/cupy/statistics_tests/test_correlation.py::TestCorrcoef::test_corrcoef_rowvar
316-
tests/third_party/cupy/statistics_tests/test_correlation.py::TestCorrcoef::test_corrcoef_y
317-
318313
tests/third_party/cupy/statistics_tests/test_order.py::TestOrder::test_percentile_defaults[linear]
319314
tests/third_party/cupy/statistics_tests/test_order.py::TestOrder::test_percentile_defaults[lower]
320315
tests/third_party/cupy/statistics_tests/test_order.py::TestOrder::test_percentile_defaults[higher]

‎tests/skipped_tests_gpu.tbl‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -320,11 +320,6 @@ tests/third_party/cupy/random_tests/test_sample.py::TestRandomIntegers2::test_bo
320320
tests/third_party/cupy/random_tests/test_sample.py::TestRandomIntegers2::test_goodness_of_fit
321321
tests/third_party/cupy/random_tests/test_sample.py::TestRandomIntegers2::test_goodness_of_fit_2
322322

323-
tests/third_party/cupy/statistics_tests/test_correlation.py::TestCorrcoef::test_corrcoef
324-
tests/third_party/cupy/statistics_tests/test_correlation.py::TestCorrcoef::test_corrcoef_diag_exception
325-
tests/third_party/cupy/statistics_tests/test_correlation.py::TestCorrcoef::test_corrcoef_rowvar
326-
tests/third_party/cupy/statistics_tests/test_correlation.py::TestCorrcoef::test_corrcoef_y
327-
328323
tests/third_party/cupy/statistics_tests/test_order.py::TestOrder::test_percentile_defaults[linear]
329324
tests/third_party/cupy/statistics_tests/test_order.py::TestOrder::test_percentile_defaults[lower]
330325
tests/third_party/cupy/statistics_tests/test_order.py::TestOrder::test_percentile_defaults[higher]

‎tests/test_statistics.py‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -573,6 +573,61 @@ def test_std_error(self):
573573
dpnp.std(ia, ddof="1")
574574

575575

576+
class TestCorrcoef:
577+
@pytest.mark.usefixtures(
578+
"suppress_divide_invalid_numpy_warnings",
579+
"suppress_dof_numpy_warnings",
580+
)
581+
@pytest.mark.parametrize("dtype", get_all_dtypes())
582+
@pytest.mark.parametrize("rowvar", [True, False])
583+
def test_corrcoef(self, dtype, rowvar):
584+
dp_array = dpnp.array([[0, 1, 2], [3, 4, 0]], dtype=dtype)
585+
np_array = dpnp.asnumpy(dp_array)
586+
587+
expected = numpy.corrcoef(np_array, rowvar=rowvar)
588+
result = dpnp.corrcoef(dp_array, rowvar=rowvar)
589+
590+
assert_dtype_allclose(result, expected)
591+
592+
@pytest.mark.usefixtures(
593+
"suppress_divide_invalid_numpy_warnings",
594+
"suppress_dof_numpy_warnings",
595+
"suppress_mean_empty_slice_numpy_warnings",
596+
)
597+
@pytest.mark.parametrize("shape", [(2, 0), (0, 2)])
598+
def test_corrcoef_empty(self, shape):
599+
dp_array = dpnp.empty(shape, dtype=dpnp.int64)
600+
np_array = dpnp.asnumpy(dp_array)
601+
602+
result = dpnp.corrcoef(dp_array)
603+
expected = numpy.corrcoef(np_array)
604+
assert_dtype_allclose(result, expected)
605+
606+
@pytest.mark.usefixtures("suppress_complex_warning")
607+
@pytest.mark.parametrize("dt_in", get_all_dtypes(no_bool=True))
608+
@pytest.mark.parametrize("dt_out", get_float_complex_dtypes())
609+
def test_corrcoef_dtype(self, dt_in, dt_out):
610+
dp_array = dpnp.array([[0, 1, 2], [3, 4, 0]], dtype=dt_in)
611+
np_array = dpnp.asnumpy(dp_array)
612+
613+
expected = numpy.corrcoef(np_array, dtype=dt_out)
614+
result = dpnp.corrcoef(dp_array, dtype=dt_out)
615+
assert expected.dtype == result.dtype
616+
assert_allclose(result, expected, rtol=1e-6)
617+
618+
@pytest.mark.usefixtures(
619+
"suppress_divide_invalid_numpy_warnings",
620+
"suppress_dof_numpy_warnings",
621+
)
622+
def test_corrcoef_scalar(self):
623+
dp_array = dpnp.array(5)
624+
np_array = dpnp.asnumpy(dp_array)
625+
626+
result = dpnp.corrcoef(dp_array)
627+
expected = numpy.corrcoef(np_array)
628+
assert_dtype_allclose(result, expected)
629+
630+
576631
@pytest.mark.usefixtures("allow_fall_back_on_numpy")
577632
class TestBincount:
578633
@pytest.mark.parametrize(

0 commit comments

Comments
 (0)