feat: support Deequ 2.0.0 (Spark 3.1) and 2.0.1 (Spark 3.2) (#100)

* adding support for Deequ 2.0/Spark 3.2

* adding support for deequ 2.0.1, but keeping support to previous versions

* keep pyspark as optional dependency

* black formatting

* testing on github workflow

* try and fix workflow trigger

* install poetry

* update lock

* downgrading python to 3.7

* adding java installation

* installing different python versions for different pyspark versions

* temporarily removing spark 2.* from tests

* set SPARK_VERSION for running tests

* testing spark 2.2 and 2.4

* downgrading Java version for pyspark 2.x

* remove 2.2 from tests

* naming steps

* drop support for Spark 2.2

* using pyspark api to retrive spark version

* get spark version from subprocess

* change _get_spark_version strategy to avoid deadlocks

* debugging spark version

* fix parsing of spark stdout

* better deequ coord configuration

* improve suggestions unit tests

* improve tests

* support spark 3.1

* fix pyspark version on github workflow

* better check of deequ v2

* fix missing argument

* update numpy and refresh poetry.lock

* improve unsupported spark version error message

* handling deequ V1 as an exception not a rule

* update matrix strategy to not fail-fast

* dropping pyspark 2.4 from CI

Author: lecardozo

.github/workflows/base.yml

@@ -0,0 +1,36 @@
+on:
+  push:
+    branches:
+     - "*"
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        PYSPARK_VERSION: ["3.0", "3.1.3", "3.2"]
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - uses: actions/setup-python@v2
+        name: Install Python 3.8
+        with:
+          python-version: 3.8
+
+      - uses: actions/setup-java@v1
+        name: Setup Java 11
+        if: startsWith(matrix.PYSPARK_VERSION, '3')
+        with:
+          java-version: '11'
+
+      - name: Running tests with pyspark==${{matrix.PYSPARK_VERSION}}
+        env:
+          SPARK_VERSION: ${{matrix.PYSPARK_VERSION}}
+        run: |
+          pip install --upgrade pip
+          pip install poetry
+          poetry install
+          poetry add pyspark==$SPARK_VERSION
+          poetry run python -m pytest -s tests