Major upgrade: Python 3.14, Pydantic v2, Ruff linter, and library updates (#572)

Co-authored-by: Jash Parekh <jparekh1@wayfair.com>

Author: jashparekh

.bandit

@@ -8,26 +8,10 @@ on:
       branches: ["main"]
 
 env:
-  PYTHON_VERSION: "3.12"
+  PYTHON_VERSION: "3.13"
 
 jobs:
-  bandit:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Check out code
-        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
-      - uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5.0.0
-        with:
-          python-version: ${{ env.PYTHON_VERSION }}
-      - name: Install dependencies
-        uses: ./.github/actions/install-dependencies
-        with:
-          test-requirements: "true"
-
-      - name: Run bandit
-        run: bandit --ini .bandit -r gbq
-
-  black:
+  ruff:
     runs-on: ubuntu-latest
     steps:
       - name: Check out code
@@ -41,26 +25,13 @@ jobs:
         with:
           test-requirements: "true"
 
-      - name: Run black
-        run: black --check gbq tests
-
-  flake8:
-    runs-on: ubuntu-latest
-    steps:
-        - name: Check out code
-          uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
-        - uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5.0.0
-          with:
-            python-version: ${{ env.PYTHON_VERSION }}
-        - name: Install dependencies
-          uses: ./.github/actions/install-dependencies
-          with:
-            test-requirements: "true"
+      - name: Run ruff linter
+        run: ruff check gbq tests
 
-        - name: Run flake8
-          run: flake8 gbq tests
+      - name: Run ruff formatter
+        run: ruff format --check gbq tests
 
-  isort:
+  bandit:
     runs-on: ubuntu-latest
     steps:
       - name: Check out code
@@ -73,8 +44,8 @@ jobs:
         with:
           test-requirements: "true"
 
-      - name: Run isort
-        run: isort --check-only gbq tests
+      - name: Run bandit
+        run: bandit -c pyproject.toml -r gbq
 
   mypy:
     runs-on: ubuntu-latest
@@ -90,7 +61,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [ "3.8", "3.9", "3.10", "3.11", "3.12" ]
+        python-version: [ "3.10", "3.11", "3.12", "3.13", "3.14" ]
     steps:
         - name: Check out code
           uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
@@ -133,7 +104,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [ "3.8", "3.9", "3.10", "3.11", "3.12" ]
+        python-version: [ "3.10", "3.11", "3.12", "3.13", "3.14" ]
     steps:
       - name: Check out code
         uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
@@ -153,7 +124,7 @@ jobs:
       - name: Check out code
         uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
       - name: Build devbox image
-        run: "docker-compose build devbox"
+        run: "docker compose build devbox"
 
   build-docs:
     runs-on: ubuntu-latest

.flake8

@@ -4,10 +4,50 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+## [1.1.0] - 2025-11-02
+
+### Added
+- Support for Python 3.10, 3.11, 3.12, 3.13, and 3.14
+- Replaced `pip` with `uv` package manager for faster dependency resolution
+- Replaced `black`, `isort`, `flake8`, and `autoflake` with `ruff` for unified linting and formatting
+
 ### Changed
-- Fixed get_bq_schema_from_json_schema() SchemaField parameters
+- Upgraded Pydantic from v1 to v2.12.3
+- Upgraded all core dependencies to latest versions:
+  - `google-cloud-bigquery` to 3.38.0
+  - `google-api-core` to 2.28.1
+  - `google-auth` to 2.42.1
+  - `google-cloud-core` to 2.5.0
+  - `google-crc32c` to 1.7.1
+  - `google-resumable-media` to 2.7.2
+  - `googleapis-common-protos` to 1.71.0
+- Updated documentation dependencies for Python 3.14 compatibility:
+  - `mkdocs` to 1.6.1
+  - `mkdocs-material` to 9.5.49
+  - `mkdocstrings[python]` to 0.27.0
+  - `mike` to 2.1.1
+- Updated CI workflow to use Docker Compose V2 (`docker compose` instead of `docker-compose`)
+- Moved bandit configuration from `.bandit` file to `pyproject.toml`
+- Updated `pyproject.toml` with ruff configuration for linting and formatting
+
+### Removed
+- Support for Python 3.8 and 3.9
+- Individual linter configurations (`.flake8`, `[tool.black]`, `[tool.isort]`)
 
-## [Unreleased]
+### Fixed
+- Fixed `SchemaField` API compatibility with google-cloud-bigquery 3.x
+- Fixed `QueryJob` instantiation in tests for google-cloud-bigquery 3.x compatibility
+- Fixed REPEATED mode detection for list-based schema fields in `get_bq_schema_from_record()`
+- Fixed Pydantic v2 `model_validator` compatibility issues
+- Fixed Docker Compose V2 compatibility in CI workflows
+
+### Internal
+- Updated Dockerfile to use Python 3.14
+- Installed `uv` in Docker images for package management
+- Updated test fixtures and helpers for new API compatibility
+- Regenerated `requirements.lock` with updated dependencies
 
 ## [1.0.5] - 2024-08-01
 

.github/workflows/main.yml

@@ -3,7 +3,7 @@
 [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/gbq)][pypi]
 [![codecov](https://codecov.io/gh/wayfair-incubator/gbq/branch/main/graph/badge.svg)][codecov]
 [![Checked with mypy](https://img.shields.io/badge/mypy-checked-blue)][mypy-home]
-[![Code style: black](https://img.shields.io/badge/code%20style-black-black.svg)][black-home]
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)][ruff-home]
 
 
 # GBQ
@@ -46,16 +46,14 @@ Check out the [project documentation](https://wayfair-incubator.github.io/gbq/)
 [ci]: https://github.com/wayfair-incubator/gbq/actions
 [codecov]: https://codecov.io/gh/wayfair-incubator/gbq
 [mypy-home]: http://mypy-lang.org/
-[black-home]: https://github.com/psf/black
+[ruff-home]: https://github.com/astral-sh/ruff
 [install-docker]: https://docs.docker.com/install/
 [pdbpp-home]: https://github.com/pdbpp/pdbpp
 [pdb-docs]: https://docs.python.org/3/library/pdb.html
 [pdbpp-docs]: https://github.com/pdbpp/pdbpp#usage
 [pytest-docs]: https://docs.pytest.org/en/latest/
 [mypy-docs]: https://mypy.readthedocs.io/en/stable/
-[black-docs]: https://black.readthedocs.io/en/stable/
-[isort-docs]: https://pycqa.github.io/isort/
-[flake8-docs]: http://flake8.pycqa.org/en/stable/
+[ruff-docs]: https://docs.astral.sh/ruff/
 [bandit-docs]: https://bandit.readthedocs.io/en/stable/
 [sem-ver]: https://semver.org/
 [pypi]: https://semver.org/

CHANGELOG.md

@@ -1,4 +1,4 @@
-version: "3.4"
+version: "3.9"
 
 x-mount-app-and-user-git-config: &mount-app-and-user-git-config
   volumes:

README.md

@@ -1,26 +1,42 @@
-FROM python:3.12-bookworm
+FROM python:3.14-bookworm
 
 ARG _USER="gbq"
 ARG _UID="1000"
 ARG _GID="100"
 ARG _SHELL="/bin/bash"
 
+# Install uv as root before creating the user
+ENV UV_NO_CACHE="true"
+RUN curl -LsSf https://astral.sh/uv/install.sh | sh && \
+    cp /root/.local/bin/uv /usr/local/bin/uv && \
+    cp /root/.local/bin/uvx /usr/local/bin/uvx
+
 RUN useradd -m -s "${_SHELL}" -N -u "${_UID}" "${_USER}"
 
-ENV USER ${_USER}
-ENV UID ${_UID}
-ENV GID ${_GID}
-ENV HOME /home/${_USER}
-ENV PATH "${HOME}/.local/bin/:${PATH}"
-ENV PIP_NO_CACHE_DIR "true"
+ENV USER=${_USER}
+ENV UID=${_UID}
+ENV GID=${_GID}
+ENV HOME=/home/${_USER}
+ENV PATH="${HOME}/.local/bin/:${PATH}"
 
 RUN mkdir /app && chown ${UID}:${GID} /app
 
-USER ${_USER}
-
-COPY --chown=${UID}:${GID} ./requirements* /app/
+# Copy requirements files as root since we'll install as root
+COPY ./requirements* /app/
 WORKDIR /app
 
-RUN pip install -r requirements.lock -r requirements-test.txt -r requirements-docs.txt
+# Install from requirements.txt first, then add test and docs requirements
+# Skip lock file if it's empty or just a comment
+# Install as root since --system requires write access to system site-packages
+RUN if [ -s requirements.lock ] && [ "$(grep -v '^#' requirements.lock | wc -l)" -gt 0 ]; then \
+        uv pip install --system -r requirements.lock -r requirements-test.txt -r requirements-docs.txt; \
+    else \
+        uv pip install --system -r requirements.txt -r requirements-test.txt -r requirements-docs.txt; \
+    fi
+
+# Change ownership of app directory to user
+RUN chown -R ${UID}:${GID} /app
+
+USER ${_USER}
 
 CMD bash

docker-compose.yml

@@ -1,8 +1,8 @@
 #!/usr/bin/env bash
 
-pip uninstall -y -r <(pip freeze)
-pip install -r requirements.txt
+uv pip uninstall --system -r <(uv pip freeze --system)
+uv pip install --system -r requirements.txt
 printf "# THIS IS AN AUTOGENERATED LOCKFILE. DO NOT EDIT MANUALLY.\n" > requirements.lock
-pip freeze --disable-pip-version-check --all >> requirements.lock
+uv pip freeze --system >> requirements.lock
 
-echo "Rebuild containers to verify there are no conflicts."
+echo "Rebuild containers to verify there are no conflicts."

docker/devbox.dockerfile

@@ -2,8 +2,7 @@
 
 set -eo pipefail
 
-BLACK_ACTION="--check"
-ISORT_ACTION="--check-only"
+RUFF_FIX=""
 
 function usage
 {
@@ -17,8 +16,7 @@ while [[ $# -gt 0 ]]; do
     arg="$1"
     case $arg in
         --format-code)
-        BLACK_ACTION="--quiet"
-        ISORT_ACTION=""
+        RUFF_FIX="--fix"
         ;;
         -h|--help)
         usage
@@ -37,20 +35,18 @@ done
 # only generate html locally
 pytest tests --cov-report html
 
-echo "Running autoflake..."
-autoflake -irv --remove-all-unused-import gbq tests
+echo "Running ruff linter..."
+ruff check ${RUFF_FIX} gbq tests
 
-echo "Running isort..."
-isort ${ISORT_ACTION} gbq tests
-
-echo "Running black..."
-black ${BLACK_ACTION} gbq tests
-
-echo "Running flake8..."
-flake8 gbq tests
+echo "Running ruff formatter..."
+if [ -z "${RUFF_FIX}" ]; then
+    ruff format --check gbq tests
+else
+    ruff format gbq tests
+fi
 
 echo "Running mypy..."
 mypy gbq
 
 echo "Running bandit..."
-bandit --ini .bandit --quiet -r gbq
+bandit -c pyproject.toml --quiet -r gbq

docker/lock_requirements.sh

@@ -16,13 +16,13 @@ details on how to [install docker][install-docker] on your computer.
 Once that is configured, the test suite can be run locally:
 
 ```bash
-docker-compose run --rm test
+docker compose run --rm test
 ```
 
 If you want to be able to execute code in the container:
 
 ```bash
-docker-compose run --rm devbox
+docker compose run --rm devbox
 (your code here)
 ```
 
@@ -50,11 +50,11 @@ See the documentation on [pdb][pdb-docs] and [pdb++][pdbpp-docs] for more inform
 You'll be unable to merge code unless the linting and tests pass. You can run these in your container via:
 
 ```bash
-docker-compose run --rm test
+docker compose run --rm test
 ```
 
 This will run the same tests, linting, and code coverage that are run by the CI pipeline. The only difference is that,
-when run locally, `black` and `isort` are configured to automatically correct issues they detect.
+when run locally, `ruff` is configured to automatically fix issues it detects (with the `--format-code` flag).
 
 Generally we should endeavor to write tests for every feature. Every new feature branch should increase the test
 coverage rather than decreasing it.
@@ -67,10 +67,8 @@ To customize / override a specific testing stage, please read the documentation
 
 1. [PyTest][pytest-docs]
 2. [MyPy][mypy-docs]
-3. [Black][black-docs]
-4. [Isort][isort-docs]
-5. [Flake8][flake8-docs]
-6. [Bandit][bandit-docs]
+3. [Ruff][ruff-docs]
+4. [Bandit][bandit-docs]
 
 ### Building the Library
 
@@ -119,8 +117,8 @@ This same pipeline also runs on the default branch when a maintainer merges a pu
 
 ### Lints
 
-The first set of jobs that run as part of the CI pipline are linters that perform static analysis on the code. This
-includes: [MyPy][mypy-docs], [Black][black-docs], [Isort][isort-docs], [Flake8][flake8-docs], and [Bandit][bandit-docs].
+The first set of jobs that run as part of the CI pipeline are linters that perform static analysis on the code. This
+includes: [MyPy][mypy-docs], [Ruff][ruff-docs], and [Bandit][bandit-docs].
 
 ### Tests
 
@@ -145,9 +143,7 @@ The pipeline runs the tests cases across each supported version of Python to ens
 [pdbpp-docs]: https://github.com/pdbpp/pdbpp#usage
 [pytest-docs]: https://docs.pytest.org/en/latest/
 [mypy-docs]: https://mypy.readthedocs.io/en/stable/
-[black-docs]: https://black.readthedocs.io/en/stable/
-[isort-docs]: https://pycqa.github.io/isort/
-[flake8-docs]: http://flake8.pycqa.org/en/stable/
+[ruff-docs]: https://docs.astral.sh/ruff/
 [bandit-docs]: https://bandit.readthedocs.io/en/stable/
 [sem-ver]: https://semver.org/
 [pep-517]: https://www.python.org/dev/peps/pep-0517