Bump dependencies and defaults; drop support for Python 3.7 (#367)

* dump all deps
* drop support for python 3.7 and enable ci for python 3.11 and 3.12
* update defaults: spark -> 3.5.0 hadoop -> 3.3.6
* clean up pip-tools section
* add test for normalize_keys and fix lint problems
* update changelog

Author: nchammas

.github/workflows/flintrock.yaml

@@ -17,10 +17,11 @@ jobs:
           - ubuntu-20.04
           - macos-11
         python-version:
-          - "3.7"
           - "3.8"
           - "3.9"
           - "3.10"
+          - "3.11"
+          - "3.12"
     name: ${{ matrix.os }} / Python ${{ matrix.python-version }}
     steps:
       - uses: actions/checkout@v3

CHANGES.md

@@ -6,15 +6,16 @@
 
 ### Changed
 
-* [#348]: Bumped default Spark to 3.2; dropped support for Python 3.6; added CI build for Python 3.10.
+* [#348], [#367]: Bumped default Spark to 3.5.0 and default Hadoop to 3.3.6; dropped support for Python 3.6 and 3.7; added CI builds for Python 3.10, 3.11, and 3.12.
 * [#361]: Migrated from AdoptOpenJDK, which is deprecated, to Adoptium OpenJDK.
-* [#362][#366]: Improved Flintrock's ability to cleanup after launch failures.
+* [#362], [#366]: Improved Flintrock's ability to cleanup after launch failures.
 * [#366]: Deprecated `--ec2-spot-request-duration`, which is not needed for one-time spot instances launched using the RunInstances API.
 
 [#348]: https://github.com/nchammas/flintrock/pull/348
 [#361]: https://github.com/nchammas/flintrock/pull/361
 [#362]: https://github.com/nchammas/flintrock/pull/362
 [#366]: https://github.com/nchammas/flintrock/pull/366
+[#367]: https://github.com/nchammas/flintrock/pull/367
 
 ## [2.0.0] - 2021-06-10
 

CONTRIBUTING.md

@@ -80,9 +80,11 @@ There are a few things you should do before diving in to write a new feature or
 
 If you are changing anything about Flintrock's dependencies, be sure to update the compiled requirements using [pip-tools]:
 
+[pip-tools]: https://github.com/jazzband/pip-tools
+
 ```shell
 function update-deps() {
-    pip install -U "pip-tools==6.8.0"
+    pip install -U "pip-tools==7.3.0"
 
     pip-compile -U requirements/user.in -o requirements/user.pip
     pip-compile -U requirements/developer.in -o requirements/developer.pip
@@ -91,19 +93,13 @@ function update-deps() {
     # Uncomment whichever set of requirements makes sense for you.
     # pip-sync requirements/user.pip
     # pip-sync requirements/developer.pip
-    pip-sync requirements/maintainer.pip
+    # pip-sync requirements/maintainer.pip
 }
 
 update-deps
 ```
 
-After doing that, make sure your environment matches what's in the compiled requirements by running `pip-sync` against the appropriate requirements file:
-
-```
-pip-sync requirements/[user|developer|maintainer].pip
-```
-
-[pip-tools]: https://github.com/jazzband/pip-tools
+`pip-compile` takes the provided set of input requirements, like `user.in` and compiles them into a full list of pinned transitive dependencies, like `user.pip`. This is similar to a lock file. `pip-sync` ensures that the current active virtual environment has exactly the dependencies listed in the provided pip file, no more and no less.
 
 #### Coordinate first
 

README.md

@@ -30,7 +30,7 @@ Here's a quick way to launch a cluster on EC2, assuming you already have an [AWS
 ```sh
 flintrock launch test-cluster \
     --num-slaves 1 \
-    --spark-version 3.3.0 \
+    --spark-version 3.5.0 \
     --ec2-key-name key_name \
     --ec2-identity-file /path/to/key.pem \
     --ec2-ami ami-0aeeebd8d2ab47354 \
@@ -87,12 +87,12 @@ these steps:
    better performance.
 3. Make sure Flintrock is configured to use Hadoop/HDFS 2.7+. Earlier
    versions of Hadoop do not have solid implementations of `s3a://`.
-   Flintrock's default is Hadoop 3.3.4, so you don't need to do anything
+   Flintrock's default is Hadoop 3.3.6, so you don't need to do anything
    here if you're using a vanilla configuration.
 4. Call Spark with the hadoop-aws package to enable `s3a://`. For example:
    ```sh
-   spark-submit --packages org.apache.hadoop:hadoop-aws:3.3.4 my-app.py
-   pyspark --packages org.apache.hadoop:hadoop-aws:3.3.4
+   spark-submit --packages org.apache.hadoop:hadoop-aws:3.3.6 my-app.py
+   pyspark --packages org.apache.hadoop:hadoop-aws:3.3.6
    ```
    If you have issues using the package, consult the [hadoop-aws troubleshooting
    guide](http://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html)
@@ -114,7 +114,7 @@ Before using Flintrock, take a quick look at the
 notice and [license](https://github.com/nchammas/flintrock/blob/master/LICENSE)
 and make sure you're OK with their terms.
 
-**Flintrock requires Python 3.7 or newer**, unless you are using one
+**Flintrock requires Python 3.8 or newer**, unless you are using one
 of our **standalone packages**. Flintrock has been thoroughly tested
 only on OS X, but it should run on all POSIX systems.
 A motivated contributor should be able to add
@@ -252,7 +252,7 @@ provider: ec2
 
 services:
   spark:
-    version: 3.3.0
+    version: 3.5.0
 
 launch:
   num-slaves: 1

flintrock/config.yaml.template

@@ -1,6 +1,6 @@
 services:
   spark:
-    version: 3.3.0
+    version: 3.5.0
     # git-commit: latest  # if not 'latest', provide a full commit SHA; e.g. d6dc12ef0146ae409834c78737c116050961f350
     # git-repository:  # optional; defaults to https://github.com/apache/spark
     # optional; defaults to download from a dynamically selected Apache mirror
@@ -12,7 +12,7 @@ services:
     # download-source: "s3://some-bucket/spark/{v}/"
     # executor-instances: 1
   hdfs:
-    version: 3.3.4
+    version: 3.3.6
     # optional; defaults to download from a dynamically selected Apache mirror
     #   - can be http, https, or s3 URL
     #   - must contain a {v} template corresponding to the version

flintrock/flintrock.py

@@ -292,7 +292,7 @@ def cli(cli_context, config, provider, debug):
 @click.option('--num-slaves', type=click.IntRange(min=1), required=True)
 @click.option('--java-version', type=click.IntRange(min=8), default=11)
 @click.option('--install-hdfs/--no-install-hdfs', default=False)
-@click.option('--hdfs-version', default='3.3.4')
+@click.option('--hdfs-version', default='3.3.6')
 @click.option('--hdfs-download-source',
               help=(
                   "URL to download Hadoop from. If an S3 URL, Flintrock will use the "
@@ -1121,10 +1121,13 @@ def normalize_keys(obj):
     """
     Used to map keys from config files to Python parameter names.
     """
-    if type(obj) != dict:
+    if not isinstance(obj, dict):
         return obj
     else:
-        return {k.replace('-', '_'): normalize_keys(v) for k, v in obj.items()}
+        return {
+            k.replace('-', '_'): normalize_keys(v)
+            for k, v in obj.items()
+        }
 
 
 def config_to_click(config: dict) -> dict:

requirements/developer.in

@@ -1,6 +1,5 @@
 -r user.pip
 pytest >= 3.5.0
 pytest-cov >= 2.5.1
-flake8 == 3.8.4
-freezegun == 1.1.0
+flake8 == 6.1.0
 # PyYAML  # requirement already covered by setup.py

requirements/developer.pip

@@ -1,119 +1,103 @@
 #
-# This file is autogenerated by pip-compile with python 3.7
-# To update, run:
+# This file is autogenerated by pip-compile with Python 3.8
+# by the following command:
 #
 #    pip-compile --output-file=requirements/developer.pip requirements/developer.in
 #
 -e file:.#egg=Flintrock
     # via -r requirements/user.pip
-attrs==22.1.0
-    # via pytest
-bcrypt==3.2.2
+bcrypt==4.0.1
     # via
     #   -r requirements/user.pip
     #   paramiko
-boto3==1.21.44
+boto3==1.29.4
     # via
     #   -r requirements/user.pip
     #   flintrock
-botocore==1.24.44
+botocore==1.32.4
     # via
     #   -r requirements/user.pip
     #   boto3
     #   flintrock
     #   s3transfer
-cffi==1.15.1
+cffi==1.16.0
     # via
     #   -r requirements/user.pip
-    #   bcrypt
     #   cryptography
     #   pynacl
-click==7.1.2
+click==8.1.7
     # via
     #   -r requirements/user.pip
     #   flintrock
-coverage[toml]==6.4.3
-    # via pytest-cov
-cryptography==37.0.4
+coverage[toml]==7.3.2
+    # via
+    #   coverage
+    #   pytest-cov
+cryptography==41.0.5
     # via
     #   -r requirements/user.pip
     #   flintrock
     #   paramiko
-flake8==3.8.4
-    # via -r requirements/developer.in
-freezegun==1.1.0
+exceptiongroup==1.2.0
+    # via pytest
+flake8==6.1.0
     # via -r requirements/developer.in
-importlib-metadata==4.12.0
-    # via
-    #   flake8
-    #   pluggy
-    #   pytest
-iniconfig==1.1.1
+iniconfig==2.0.0
     # via pytest
 jmespath==1.0.1
     # via
     #   -r requirements/user.pip
     #   boto3
     #   botocore
-mccabe==0.6.1
+mccabe==0.7.0
     # via flake8
-packaging==21.3
+packaging==23.2
     # via pytest
-paramiko==2.11.0
+paramiko==3.3.1
     # via
     #   -r requirements/user.pip
     #   flintrock
-pluggy==1.0.0
+pluggy==1.3.0
     # via pytest
-py==1.11.0
-    # via pytest
-pycodestyle==2.6.0
+pycodestyle==2.11.1
     # via flake8
 pycparser==2.21
     # via
     #   -r requirements/user.pip
     #   cffi
-pyflakes==2.2.0
+pyflakes==3.1.0
     # via flake8
 pynacl==1.5.0
     # via
     #   -r requirements/user.pip
     #   paramiko
-pyparsing==3.0.9
-    # via packaging
-pytest==7.1.2
+pytest==7.4.3
     # via
     #   -r requirements/developer.in
     #   pytest-cov
-pytest-cov==3.0.0
+pytest-cov==4.1.0
     # via -r requirements/developer.in
 python-dateutil==2.8.2
     # via
     #   -r requirements/user.pip
     #   botocore
-    #   freezegun
-pyyaml==6.0
+pyyaml==6.0.1
     # via
     #   -r requirements/user.pip
     #   flintrock
-s3transfer==0.5.2
+s3transfer==0.7.0
     # via
     #   -r requirements/user.pip
     #   boto3
 six==1.16.0
     # via
     #   -r requirements/user.pip
-    #   paramiko
     #   python-dateutil
 tomli==2.0.1
     # via
     #   coverage
     #   pytest
-typing-extensions==4.3.0
-    # via importlib-metadata
-urllib3==1.26.11
+urllib3==1.26.18
     # via
     #   -r requirements/user.pip
     #   botocore
-zipp==3.8.1
-    # via importlib-metadata

requirements/maintainer.in

@@ -1,4 +1,4 @@
 -r developer.pip
 wheel >= 0.31.0
-twine == 3.8.0
-PyInstaller == 5.3
+twine == 4.0.2
+PyInstaller == 6.2.0