TST: Add cygwin build to CI (numpy#18330)

* TST: Try to set up a test environment on Cygwin.

Edit: Fix spelling of "pull request"

TST: Follow workflow for other OSs using commands available on Cygwin.

Other GitHub actions set fetch-depth to zero and use ./.github/actions
for the actual testing.  Let's see if this works.

Edit: Drop shell from the last step in the action.

Apparently "shell" isn't valid for steps.  I hope the action knows to
use bash rather than Power Shell or cmd.exe, or this is going to get
very confused.

TST: Adjust for cygwin

The action assumes `sudo` exists, which is a bit of a problem.

TST: Fetch with Cygwin git

Make sure Cygwin git can access the repository.  Versioneer depends on
git, so this is kind of important.

Edit: Include closing quote.

Edit: Install Cygwin git.

Versioneer needs this in order to function.

Edit: FIXUP: build_src option is not spelled --verbose-config

It's --verbose-cfg.

TST: Check which version of Python is being used.

I want to be sure it's a Cygwin one, not a Windows one.

TST: Build a wheel for Cygwin.

I may need to install in a virtualenv, and installing from a wheel is
much faster than installing from a directory.

TST: Use runtests.py to build the extensions.

It couldn't find them during the tests, which is not a problem I've
run into locally.  Hopefully this fixes that.

TST: Drop separate build step.

TST: Cygwin CI: Install numpy and run tests from that.

Running with runtests.py didn't work, but this should.

TST: Add cygwin bin dir to path, not root dir.

TST: Make sure Cygwin build fails when tests fail

The tests managed to pass despite NumPy not importing.  This is not a good thing.

TST: Avoid steps after tests in Cygwin CI.

I really need this to fail if NumPy doesn't import.

TST: Ensure Cygwin CI is running in Cygwin.

Kinda defeats the point if it's not.

TST: Check that pip installed the C extensions.

This really shouldn't need to be checked, but the CI runs have been
failing due to a failed import of a C extension.  I also printed the
version, which will make me feel better about the right version being
found and used.

TST: Include closing quote in test command

TST: Report loaded modules

Still trying to find why the C extension module import fails.

TST: Check permissions on NumPy C extensions

TST: Ask python which files it's trying to load.

Hopefully this gives me a general direction for where to look for
"Cannot find file or directory"

TST: Work around powershell syntax weirdness

Drop a level of quoting and just write the command to a file, then run that.  It should still show DLL permissions.

TST: Fix line endings in script.

bash expects \n
powershell creates \r\n

TST: Check for DLLs required by C extensions in Cygwin CI.

It's still not working; hopefully this shows why.

TST: Add import checks to the dll testing.

I have a project where `python -c 'import module'` worked but
`pytest --pyargs module` did not.  Lets see if that happens here.

Allow for global installs

The runners have global install privileges, so pip will install NumPy there.  I need to account for that in my script.

Fix sed regex.

Lots of leaning-toothpick syndrome, but it actually does what it's supposed to locally.  Hopefully the CI agrees.

Stop trying to import NumPy modules from sourcedir

This doesn't work and hasn't for a while.

TST: Simplify PATH and make sure lapack is on it.

Most recent CI run said it couldn't find lapack.
Reduce PATH to just Cygwin directories, make sure /usr/lib/lapack is included (after /usr/bin), and try again.

Shortening PATH occasionally fixes some other problems.  Let's see if that works here.

TST: Change Cygwin CI python from 3.8 to 3.7

Let's see if this solves the problems.  There was a change in DLL load path handling in 3.8 that might be causing the "cannot load numpy.linalg.lapack_lite" errors.

TST: Stop running CI on PR branch.

STY: Wrap long lines in Cygwin workflow file.

TST: Update name of "main" branch used to trigger workflow.

I forgot this changed a while back.

TST: Specify full paths for commands.

Also use dash in more places.

TST: Change the newline-escape mechanism in Cygwin workflow.

Backslash is apparently difficult.  This reduces that step to a single
command and tells YAML the string is to be interpreted as a single
line.

TST: Move the test command onto one line so GitHub can read it.

Apparently the GitHub Actions YAML parser is incomplete.  It should be
reading what was there as a single-line string with no linebreak at
the end and all linebreaks in the middle replaced by spaces.  It seems
to have parsed this as two lines.

Does GitHub have a document saying which subset of YAML they actually
recognize?

TST: Add dependency on importlib-metadata.

Pytest didn't declare it.

TST: Move importlib-metadata earlier in install list.

I hope this means it actually gets installed.  The manager is ignoring
it right now.

TST: Add zipp to the package install list.

Also tell pip to install the test_requirements, so I don't keep
running into this problem.

TST: Specify full path to python for testing.

I apparently missed this earlier.

TST: Add CFFI and pytz to Cygwin test environment.

TST: Set global path to include /usr/lib/lapack.

I'd forgotten there was a line for this already.

TST: Shorten and unpin the Cygwin test requirements.

I don't want to build more modules than I have to.  It works fine on
my machine with the system setuptools, pytz, and CFFI as well as
system-ish Cython, so it should work fine on the CI runners.

TST: Make sure test requirements are actually installed.

Apparently
pytest->importlib_metadata->"typing_extensions; python_version <= 3.7"
isn't a declared dependency chain.  Mentioning the last two
explicitly should get them installed anyway.

TST: Fix quoting in command line.

* TST: Clean up the output from the extension module check.

Also serves to document the script a bit.

* BLD: Tell NumPy about functions that cause test failures.

Several of the test failures still happen.  I'm not entirely sure why.

BLD: Add more functions to the Cygwin replace list.

These functions are mentioned in test failures, so I mark them for
replacement, along with the more obvious functions that might get
called by functions that continue to fail.

TST: List more functions to be replaced on Cygwin.

casin and casinf don't pass the branch cut tests.  Let's see if
replacing them lets the test pass on CI.

TST: Mark more functions to be replaced on Cygwin.

I tried to note the tests each function fails by group, but I don't
remember all of them anymore.

Let's see if casin{,f,l} gets replaced on the CI runner.

BLD: Mark more functions for replacement on Cygwin.

I probably need to run `git clean` to see improvements locally.

BLD:List more functions to be replaced on Cygwin.

This is nearly the last of them.

There are still a few failures I don't know how to deal with.
The cabsl/hypotl overflows will be gone next Cygwin update.
I may have an idea for cpowl (if cpowl(x, n) == cexpl(n*clogl(x)))
I don't understand timezones, CFFI, or LAPACK,
and that's the rest of the failures.

BLD: Change list of functions marked for replacement on Cygwin.

I hoped this would convince `cpowl` to flag its overflows, but that
doesn't appear to have happened.

* TST: Return to requirements in test_requirements.txt

There will probably be compilation involved for coverage (from
pytest-cov), maybe also cython, but the rest look like pure python and
should be fast.

* BLD: Export random distribution functions on Cygwin.

CFFI should be able to find them now.  The Cygwin runtime DLL loader
is the Windows one, and the linker also shares most of the same
semantics.

* BUG: Set default hypotl to use npy_longdouble arithmetic.

The implementation is already there, and the tests require
npy_longdouble arithmetic, so I set up the boilerplate to make it so.

It seems to fix only np.abs(npy_clongdouble), not
abs(npy_clongdouble), for reasons I don't understand.

* Go back to old npy_hypotl and mark the failing test case.

* Undo the remaining changes from "SIMD: Force inlining all functions that accept AVX registers"

These changes are not present in `main`.  I see no commits likely to
have specifically changed whether these SIMD functions are inlined.
Adding these back to `main` is left for another PR.  The symptoms I
saw were segfaults, basically because function calls do not preserve
alignment information.

* Revert "Undo the remaining changes from "SIMD: Force inlining all functions that accept AVX registers""

This reverts commit bbd571a.

That commit was original to this branch, but is now in main, so
undoing it is a bad idea.

* TST: Move the DLL checker to a separate script

Instead of trying to write it every run, just add it to the repository
and use that.  I'm not sure how Windows git handles line endings; I
suspect it changes things to \r\n, which is not what I want.  I'm on
Cygwin, not MSYS, so everything expects \n.

* TST: Prettify the cabsl test xfail declaration.

Marking this in the parametrize call gets really big.  Moving this
inside the function is much shorter, even if it won't tell me when a
new Cygwin release makes this obsolete.  Since this only needs to wait
for one Cygwin release, I suppose that's not much of a stretch.

* Wrap long line and change condition order.

I want the `sys.platform == "cygwin"` check before I try to access
`np.clongdouble`, so the check doesn't crash the test on platforms
where np.clongdouble doesn't exist.

Author: DWesl

.github/workflows/cygwin.yml

@@ -0,0 +1,67 @@
+name: Test on Cygwin
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+jobs:
+  cygwin_build_test:
+    runs-on: windows-latest
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: recursive
+          fetch-depth: 0
+      - name: Install Cygwin
+        uses: egor-tensin/setup-cygwin@v3
+        with:
+          platform: x64
+          install-dir: 'C:\tools\cygwin'
+          packages: >
+            python37-devel python37-zipp python37-importlib-metadata
+            python37-cython python37-pip python37-wheel python37-cffi
+            python37-pytz python37-setuptools python37-pytest
+            python37-hypothesis liblapack-devel libopenblas
+            gcc-fortran git dash
+      - name: Set Windows PATH
+        uses: egor-tensin/cleanup-path@v1
+        with:
+          dirs: 'C:\tools\cygwin\bin;C:\tools\cygwin\lib\lapack'
+      - name: Verify that bash is Cygwin bash
+        run: |
+          command bash
+          bash -c "uname -svrmo"
+      - name: Update with Cygwin git
+        # fetch-depth=0 above should make this short.
+        run: |
+          dash -c "which git; /usr/bin/git fetch --all -p"
+      - name: Verify python version
+        # Make sure it's the Cygwin one, not a Windows one
+        run: |
+          dash -c "which python3.7; /usr/bin/python3.7 --version -V"
+      - name: Build NumPy wheel
+        run: |
+          dash -c "/usr/bin/python3.7 -m pip install 'setuptools<49.2.0' pytest pytz cffi pickle5 importlib_metadata typing_extensions"
+          dash -c "/usr/bin/python3.7 -m pip install -r test_requirements.txt"
+          dash -c "/usr/bin/python3.7 setup.py bdist_wheel"
+      - name: Install new NumPy
+        run: |
+          bash -c "/usr/bin/python3.7 -m pip install dist/numpy-*cp37*.whl"
+      - name: Run NumPy test suite
+        run: >-
+          dash -c "/usr/bin/python3.7 runtests.py -n -vv"
+      - name: Upload wheel if tests fail
+        uses: actions/upload-artifact@v2
+        if: failure()
+        with:
+          name: numpy-cygwin-wheel
+          path: dist/numpy-*cp37*.whl
+      - name: On failure check the extension modules
+        if: failure()
+        run: |
+          dash -c "/usr/bin/python3.7 -m pip show numpy"
+          dash -c "/usr/bin/python3.7 -m pip show -f numpy | grep .dll"
+          dash -c "/bin/tr -d '\r' <tools/list_installed_dll_dependencies_cygwin.sh >list_dlls_unix.sh"
+          dash "list_dlls_unix.sh"

numpy/core/include/numpy/random/distributions.h

@@ -28,7 +28,7 @@ extern "C" {
 #define RAND_INT_MAX INT64_MAX
 #endif
 
-#ifdef _MSC_VER
+#if defined(_MSC_VER) || defined(__CYGWIN__)
 #define DECLDIR __declspec(dllexport)
 #else
 #define DECLDIR extern

numpy/core/src/common/npy_config.h

@@ -96,6 +96,51 @@
 #undef HAVE_POWL
 #endif
 
+#ifdef __CYGWIN__
+/* Loss of precision */
+#undef HAVE_CASINHL
+#undef HAVE_CASINH
+#undef HAVE_CASINHF
+
+/* Loss of precision */
+#undef HAVE_CATANHL
+#undef HAVE_CATANH
+#undef HAVE_CATANHF
+
+/* Loss of precision and branch cuts */
+#undef HAVE_CATANL
+#undef HAVE_CATAN
+#undef HAVE_CATANF
+
+/* Branch cuts */
+#undef HAVE_CACOSHF
+#undef HAVE_CACOSH
+
+/* Branch cuts */
+#undef HAVE_CSQRTF
+#undef HAVE_CSQRT
+
+/* Branch cuts and loss of precision */
+#undef HAVE_CASINF
+#undef HAVE_CASIN
+#undef HAVE_CASINL
+
+/* Branch cuts */
+#undef HAVE_CACOSF
+#undef HAVE_CACOS
+
+/* log2(exp2(i)) off by a few eps */
+#undef HAVE_LOG2
+
+/* np.power(..., dtype=np.complex256) doesn't report overflow */
+#undef HAVE_CPOWL
+#undef HAVE_CEXPL
+
+/* Builtin abs reports overflow */
+#undef HAVE_CABSL
+#undef HAVE_HYPOTL
+#endif
+
 /* Disable broken gnu trig functions */
 #if defined(HAVE_FEATURES_H)
 #include <features.h>

numpy/core/tests/test_scalarmath.py

@@ -680,10 +680,18 @@ def _test_abs_func(self, absfunc, test_dtype):
 
     @pytest.mark.parametrize("dtype", floating_types + complex_floating_types)
     def test_builtin_abs(self, dtype):
+        if sys.platform == "cygwin" and dtype == np.clongdouble:
+            pytest.xfail(
+                reason="absl is computed in double precision on cygwin"
+            )
         self._test_abs_func(abs, dtype)
 
     @pytest.mark.parametrize("dtype", floating_types + complex_floating_types)
     def test_numpy_abs(self, dtype):
+        if sys.platform == "cygwin" and dtype == np.clongdouble:
+            pytest.xfail(
+                reason="absl is computed in double precision on cygwin"
+            )
         self._test_abs_func(np.abs, dtype)
 
 class TestBitShifts:

tools/list_installed_dll_dependencies_cygwin.sh

@@ -0,0 +1,38 @@
+#!/bin/dash
+# Check permissions and dependencies on installed DLLs
+# DLLs need execute permissions to be used
+# DLLs must be able to find their dependencies
+# This checks both of those, then does a direct test
+# The best way of checking whether a C extension module is importable
+# is trying to import it.  The rest is trying to give reasons why it
+# isn't importing.
+#
+# One of the tools and the extension for shared libraries are
+# Cygwin-specific, but the rest should work on most platforms with
+# /bin/sh
+
+py_ver=3.7
+site_packages=$(python${py_ver} -m pip show numpy | \
+		    grep Location | cut -d " " -f 2 -);
+dll_list=$(for name in $(python${py_ver} -m pip show -f numpy | \
+			     grep -F .dll); do echo ${site_packages}/${name}; done)
+echo "Checks for existence, permissions and file type"
+ls -l ${dll_list}
+file ${dll_list}
+echo "Dependency checks"
+ldd ${dll_list} | grep -F -e " => not found" && exit 1
+cygcheck ${dll_list} >cygcheck_dll_list 2>cygcheck_missing_deps
+grep -F -e "cygcheck: track_down: could not find " cygcheck_missing_deps && exit 1
+echo "Import tests"
+mkdir -p dist/
+cd dist/
+for name in ${dll_list};
+do
+    echo ${name}
+    ext_module=$(echo ${name} | \
+                     sed -E \
+			 -e "s/^\/+(home|usr).*?site-packages\/+//" \
+			 -e "s/.cpython-3.m?-x86(_64)?-cygwin.dll$//" \
+			 -e "s/\//./g")
+    python${py_ver} -c "import ${ext_module}"
+done