Migrate to uv and commit hook

Author: BenGalewsky

.gitignore

@@ -170,11 +170,10 @@ cython_debug/
 #  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
-#.idea/
+.idea/
 
 # Ruff stuff:
 .ruff_cache/
 
 # PyPI configuration file
 .pypirc
-

.pre-commit-config.yaml

@@ -0,0 +1,80 @@
+ci:
+  autoupdate_commit_msg: "chore: update pre-commit hooks"
+  autofix_commit_msg: "style: pre-commit fixes"
+
+exclude: ^.cruft.json|.copier-answers.yml$
+
+repos:
+  - repo: https://github.com/adamchainz/blacken-docs
+    rev: "1.19.1"
+    hooks:
+      - id: blacken-docs
+        additional_dependencies: [black==24.*]
+
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: "v5.0.0"
+    hooks:
+      - id: check-added-large-files
+      - id: check-case-conflict
+      - id: check-merge-conflict
+      - id: check-symlinks
+      - id: check-yaml
+      - id: debug-statements
+      - id: end-of-file-fixer
+      - id: mixed-line-ending
+      - id: name-tests-test
+        args: ["--pytest-test-first"]
+      - id: requirements-txt-fixer
+      - id: trailing-whitespace
+
+  - repo: https://github.com/pre-commit/pygrep-hooks
+    rev: "v1.10.0"
+    hooks:
+      - id: rst-backticks
+      - id: rst-directive-colons
+      - id: rst-inline-touching-normal
+
+  - repo: https://github.com/rbubley/mirrors-prettier
+    rev: "v3.5.2"
+    hooks:
+      - id: prettier
+        types_or: [yaml, markdown, html, css, scss, javascript, json]
+        args: [--prose-wrap=always]
+
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: "v0.9.9"
+    hooks:
+      - id: ruff
+        args: ["--fix", "--show-fixes"]
+      - id: ruff-format
+
+  - repo: https://github.com/codespell-project/codespell
+    rev: "v2.4.1"
+    hooks:
+      - id: codespell
+
+  - repo: https://github.com/shellcheck-py/shellcheck-py
+    rev: "v0.10.0.1"
+    hooks:
+      - id: shellcheck
+
+  - repo: local
+    hooks:
+      - id: disallow-caps
+        name: Disallow improper capitalization
+        language: pygrep
+        entry: PyBind|Numpy|Cmake|CCache|Github|PyTest
+        exclude: .pre-commit-config.yaml
+
+  - repo: https://github.com/abravalheri/validate-pyproject
+    rev: "v0.23"
+    hooks:
+      - id: validate-pyproject
+        additional_dependencies: ["validate-pyproject-schema-store[all]"]
+
+  - repo: https://github.com/python-jsonschema/check-jsonschema
+    rev: "0.31.2"
+    hooks:
+      - id: check-dependabot
+      - id: check-github-workflows
+      - id: check-readthedocs

Dockerfile

@@ -0,0 +1,4 @@
+FROM python:3.11.6
+RUN python -m pip install --upgrade pip && \
+    python -m pip install dask dask-gateway distributed numpy pandas xarray zarr netcdf4 \
+    intake bokeh

README.md

@@ -1,12 +1,38 @@
-# HTCdaskGateway
+# NCSA HTCdaskGateway
 
-* A Dask Gateway client extension for heterogeneous cluster mode combining the Kubernetes backend for pain-free scheduler networking, with COFFEA-powered HTCondor workers and/or OKD [coming soon].
-* Latest [![PyPI version](https://badge.fury.io/py/htcdaskgateway.svg)](https://badge.fury.io/py/htcdaskgateway)
- is installed by default and deployed to the COFFEA-DASK notebook on EAF (https://analytics-hub.fnal.gov). A few lines will get you going!
-* The current image for workers/schedulers is: coffeateam/coffea-dask-cc7-gateway:0.7.12-fastjet-3.3.4.0rc9-g8a990fa
+Subclasses the Dask Gateway client to launch dask clusters in Kubernetes, but
+with HTCondor workers. This is a fork of the ingenious original idea by Maria
+Acosta at Fermilab as part of their Elastic Analysis Facility project.
+
+## How it Works
+
+This is a drop-in replacement for the official Dask Gateway client. It keeps the
+same authentication and interaction with the gateway server (which is assumed to
+be running in a Kubernetes cluster). When the user requests a new cluster, this
+client communicates with the gateway server and instructs it to launch a
+cluster. We are running a modified docker image in the cluster which only
+launches the scheduler, and assumes that HTC workers will evetually join.
+
+The client then uses the user's credentials to build an HTC Job file and submits
+it to the cluster. These jobs run the dask worker and have the necessary certs
+to present themselves to the scheduler.
+
+The scheduler then accepts them into the cluster and we are ready to `compute`
+
+- A Dask Gateway client extension for heterogeneous cluster mode combining the
+  Kubernetes backend for pain-free scheduler networking, with COFFEA-powered
+  HTCondor workers and/or OKD [coming soon].
+- Latest
+  [![PyPI version](https://badge.fury.io/py/htcdaskgateway.svg)](https://badge.fury.io/py/htcdaskgateway)
+  is installed by default and deployed to the COFFEA-DASK notebook on EAF
+  (https://analytics-hub.fnal.gov). A few lines will get you going!
+- The current image for workers/schedulers is:
+  coffeateam/coffea-dask-cc7-gateway:0.7.12-fastjet-3.3.4.0rc9-g8a990fa
 
 ## Basic usage @ Fermilab [EAF](https://analytics-hub.fnal.gov)
-* Make sure the notebook launched supports this functionality (COFFEA-DASK notebook)
+
+- Make sure the notebook launched supports this functionality (COFFEA-DASK
+  notebook)
 
 ```
 from htcdaskgateway import HTCGateway
@@ -26,25 +52,38 @@ client
 # When computations are finished, shutdown the cluster
 cluster.shutdown()
 ```
+
 ## Other functions worth checking out
-* This is a multi-tenant environment, and you are authenticated via JupyterHub Oauth which means that you can create as many* clusters as you wish
-* To list your clusters:
+
+- This is a multi-tenant environment, and you are authenticated via JupyterHub
+  Oauth which means that you can create as many\* clusters as you wish
+- To list your clusters:
+
 ```
 # Verify that the gateway is responding to requests by asking to list all its clusters
 clusters = gateway.list_clusters()
 clusters
 ```
-* To connect to a specific cluster from the list:
+
+- To connect to a specific cluster from the list:
+
 ```
 cluster = gateway.connect(cluster_name)
 cluster
 cluster.shutdown()
 ```
-* To gracefully close the cluster and remove HTCondor worker jobs associated to it:
+
+- To gracefully close the cluster and remove HTCondor worker jobs associated to
+  it:
+
 ```
 cluster.shutdown()
 ```
-* There are widgets implemented by Dask Gateway. Make sure to give them a try from your EAF COFFEA notebook, just execute the the `client` and `cluster` commands (after properly initializing them) in a cell like: 
+
+- There are widgets implemented by Dask Gateway. Make sure to give them a try
+  from your EAF COFFEA notebook, just execute the `client` and `cluster`
+  commands (after properly initializing them) in a cell like:
+
 ```
 -------------
 cluster = gateway.new_cluster()
@@ -53,7 +92,7 @@ cluster
 -------------
 client = cluster.get_client()
 client
-< Widget will apear after this step >
+< Widget will appear after this step >
 -------------
 cluster
 < Widget will appear after this step >

pyproject.toml

@@ -0,0 +1,97 @@
+[build-system]
+requires = ["hatchling", "hatch-vcs"]
+build-backend = "hatchling.build"
+
+
+[project]
+name = "ncsa-htcdaskgateway"
+authors = [
+  { name = "Ben Galewsky", email = "[email protected]" },
+    { name = "Maria P. Acosta F./Fermilab EAF project", email="[email protected]"}
+]
+description = "Launches a Dask Gateway cluster in K8s and joins HTCondor workers to it"
+readme = "README.md"
+requires-python = ">=3.9"
+classifiers = [
+  "Development Status :: 1 - Planning",
+  "Intended Audience :: Science/Research",
+  "Intended Audience :: Developers",
+  "License :: OSI Approved :: BSD License",
+  "Operating System :: OS Independent",
+  "Programming Language :: Python",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3 :: Only",
+  "Programming Language :: Python :: 3.9",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Topic :: Scientific/Engineering",
+  "Typing :: Typed",
+]
+dynamic = ["version"]
+dependencies = [
+  "dask-gateway==2024.1.0"
+]
+
+[project.urls]
+Homepage = "https://github.com/ncsa/dagster-ncsa"
+"Bug Tracker" = "https://github.com/ncsa/dagster-ncsa/issues"
+Discussions = "https://github.com/ncsa/dagster-ncsa/discussions"
+Changelog = "https://github.com/ncsa/dagster-ncsa/releases"
+
+
+[tool.hatch]
+version.source = "vcs"
+build.hooks.vcs.version-file = "src/htcdaskgateway/_version.py"
+
+[tool.hatch.envs.default]
+features = ["test"]
+scripts.test = "pytest {args}"
+
+
+[tool.uv]
+dev-dependencies = [
+  "dagster_ncsa[test]",
+]
+
+
+[tool.ruff]
+exclude = ["_version.py"]
+
+[tool.ruff.lint]
+extend-select = [
+  "ARG",      # flake8-unused-arguments
+  "B",        # flake8-bugbear
+  "C4",       # flake8-comprehensions
+  "EM",       # flake8-errmsg
+  "EXE",      # flake8-executable
+  "G",        # flake8-logging-format
+  "I",        # isort
+  "ICN",      # flake8-import-conventions
+  "NPY",      # NumPy specific rules
+  "PD",       # pandas-vet
+  "PGH",      # pygrep-hooks
+  "PIE",      # flake8-pie
+  "PL",       # pylint
+  "PT",       # flake8-pytest-style
+  "PTH",      # flake8-use-pathlib
+  "RET",      # flake8-return
+  "RUF",      # Ruff-specific
+  "SIM",      # flake8-simplify
+  "T20",      # flake8-print
+  "UP",       # pyupgrade
+  "YTT",      # flake8-2020
+]
+ignore = [
+  "PLR09",    # Too many <...>
+  "PLR2004",  # Magic value used in comparison
+]
+isort.required-imports = ["from __future__ import annotations"]
+# Uncomment if using a _compat.typing backport
+# typing-modules = ["dagster_ncsa._compat.typing"]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**" = ["T20"]
+"noxfile.py" = ["T20"]
+"_version.py" = ["UP035", "UP006"]

setup.py

@@ -1,6 +1,8 @@
-from .cluster import HTCGatewayCluster 
-from .gateway import HTCGateway 
+from __future__ import annotations
+
+from .cluster import HTCGatewayCluster
+from .gateway import HTCGateway
 
 # @author Maria A. - mapsacosta
 
-__all__ = ["HTCGatewayCluster", "HTCGateway"]
+__all__ = ["HTCGateway", "HTCGatewayCluster"]

src/htcdaskgateway/__init__.py

@@ -0,0 +1,21 @@
+# file generated by setuptools-scm
+# don't change, don't track in version control
+
+__all__ = ["__version__", "__version_tuple__", "version", "version_tuple"]
+
+TYPE_CHECKING = False
+if TYPE_CHECKING:
+    from typing import Tuple
+    from typing import Union
+
+    VERSION_TUPLE = Tuple[Union[int, str], ...]
+else:
+    VERSION_TUPLE = object
+
+version: str
+__version__: str
+__version_tuple__: VERSION_TUPLE
+version_tuple: VERSION_TUPLE
+
+__version__ = version = '0.1.23.dev12+g72f1d19.d20250317'
+__version_tuple__ = version_tuple = (0, 1, 23, 'dev12', 'g72f1d19.d20250317')