Release 0.31.0 (#126)

* Documentation improvement (#80)

* Documentation improvement

* GitPython is not required anymore (#83)

Installation docs upgrade

* Documentation improvement (#80)

* Documentation improvement

* GitPython is not required anymore (#83)

Installation docs upgrade

* feat: looker integration for compile and deploy, new command: bi

* tests fix, looker git with ssh

* chore: organizing imports, removed not used variables

* chore: dbt2looker as extra dependency, mypy errors fix, bi-git-key-path option instead of key-path

* chore: formatting, types-requests dependency for mypy

* refactor: refactoring bi, removing bi command, fix tests, moving bi flags to bi.yml instead of call option

* Fix after rebase

* Fix after rebase

* Fix after rebase

* Fix after rebase

* Airbyte integration (#86)

* [DATA-366] Update datahub package and add dbt extra (#79)

Co-authored-by: Piotr Sierkin <[email protected]>

* Documentation improvement (#80)

* Added IAP Airbyte authorization with service account file

Co-authored-by: Piotr Sierkin <[email protected]>
Co-authored-by: Piotr Pękala <[email protected]>
Co-authored-by: Pawel Pinkos <[email protected]>
Co-authored-by: Pawel Jurkiewicz <[email protected]>
Co-authored-by: pawelpinkos <[email protected]>

* Service account OIDC ID token in deploy command (#91)

* Added possibility to pass platform-agnostic service account OIDC ID token to deploy command

* Added changelog information

* added 'source freshness' command to dbt compile

* Bump copier (#93)

* [DATA-366] Update datahub package and add dbt extra (#79)

* feat: update datahub package and add dbt extra

* fix: replace datahub requirements

* fix: pre-commit coma

Co-authored-by: Piotr Sierkin <[email protected]>

* Documentation improvement (#80)

* Documentation improvement

* dp compile default environment has been set to local

* GitPython is not required anymore (#83)

Installation docs upgrade

* chore: bump datahub (#84)

Co-authored-by: Piotr Sierkin <[email protected]>

* [DATA-366] Update datahub package and add dbt extra (#79)

* feat: update datahub package and add dbt extra

* fix: replace datahub requirements

* fix: pre-commit coma

Co-authored-by: Piotr Sierkin <[email protected]>

* Documentation improvement (#80)

* Documentation improvement

* GitPython is not required anymore (#83)

Installation docs upgrade

* chore: bump packages except copier

* fix: package name

* chore: add specific version for pyyaml

* fix: misspelling

* chore: test docs

* chore: test

* test

* test

* test

* test

* test

* test

* chore: bump packages

* chore: add renovate

* chore: bump copier

* chore: update exception

* fix: update by removing not existing flag

* chore: update tests

* feat: update schema on manifest to v7

* chore: bump packages

* chore: delete renovate

* chore: bump python version

* chore: remove junk

* chore: bring back old exception

* chore: add matrix to tox

* chore: add new require and classifier

* trigger

* chore(docs): update badge and changelog

* chore(docs): replace old extension

* chore: add description

* chore(docs): add old suffix and envops

* chore(docs): update changelog

* chore(docs): update changelog

* chore(docs): update copier

* chore: pinup version for dbt2looker

* chore(docs): update changelog

* chore: replace var for more meaningful

Co-authored-by: Piotr Sierkin <[email protected]>
Co-authored-by: Piotr Pękala <[email protected]>

* chore: downgrade version to properly prepare release

* chore(docs): fix date in changelog

* BI deployment is disabled when config is missing (#95)

* DATA-629: Added Airbyte configuration documentation (#97)

* feat: disable sync bucket (#102)

* chore: add flag to turn off bucket syncing with artefacts

* chore: replace default value

* chore(docs): updat changelog

* fix: add readthedocs config file

* chore: repair failing tests

* feat: add test for sync_bucket

* chore(docs): cosmetic changes in changelog

* chore: rename `sync-bucket` to `disable-bucket-sync`

* chore(docs): update with flag

* chore: replace `sync_bucket` with `disable_bucket_sync`

* chore: update name for `_bucket_sync`

---------

Co-authored-by: Piotr Sierkin <[email protected]>

* Fix failing readthedocs builds (#104)

* Removed sphinx action from prepare-release pipeline (#105)

* Bump werkzeug from 2.2.2 to 2.2.3 (#106)

* Bump werkzeug from 2.2.2 to 2.2.3

Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3.
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](pallets/werkzeug@2.2.2...2.2.3)

---
updated-dependencies:
- dependency-name: werkzeug
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

* bump codeclimate-action to 4.0.0

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Pawel Jurkiewicz <[email protected]>

* Airbyte fix creating multiple connections (#109)

* Fixing tests.

* Fix failing test.

* Remove redundant line.

* Add info about fix to CHANGELOG.md.

* Add workspace_id entry to config.

* Get rid of walrus operator & add test for missing workspace id.

* Fix wrong connection id in Airflow ingestion task.

* Fix typo in test name.

* Add fetching default workspace if not available in config.

* Increasign test coverage.

* Update workspace_id configuration description.

* Bumping Python version to 3.8 in prepare-release workflow.

* Pin pydantic version (#112)

* Pin Pydantic version until Copier fixes their dependency.

* Add info to changelog.

* Upgrade dbt to 1.5.4 (#114)

* Upgrade dbt to 1.5.4

* Fixing imports for publish.py

* Update of dbt_schema_version

* Updating manifest.json for tests

* Fix manifest for testing

* Fixing manifest cntd.

* Fixing manifest cntd.

* Replace example-project with exampleproject

* Adding tutorial for upgrading manifest in tests cntd.

* Fixing typos in the docs

* Version fixes before release

---------

Co-authored-by: Michal Rudko <[email protected]>

* Databricks generate job (#116)

* new feature

* small fixes

* dbt databricks factory requirement

* python 3.9 onwards

* testing

* linting

* annotations fix

* linting

* no docs

* linting

* update docs

* testing

* tests with databricks

* testenv

* Upgrade pyyaml

* Dbt glue (#118)

* DBT glue

* DBT glue

* DBT glue

* DBT glue

* Dbt glue with spark (#120)

* Dbt glue with spark

* Fixing dp cli

* Dbt upgtade (#122)

* Dbt upgrade to 1.7

* Dbt upgrade to 1.7

* Fixing dp cli

* Upgrade versions

* Upgrade versions

* feat: Add support for python 3.11 and 3.12 (#125)

# Add Python 3.11 and 3.12 Support

  ## Summary
  Adds full support for Python 3.11 and 3.12 while maintaining Python 3.9/3.10 compatibility. Python 3.9 is deprecated (EOL Oct 2025) with support ending April 30, 2026.

  ## Changes

  **Dependencies**
  - Removed `pydantic<2` (unused, blocked Python 3.12)
  - Updated `packaging` to `>=23.0` (setuptools compatibility)
  - Updated `fsspec`, `gcsfs`, `s3fs` to `>=2024.6.0,<2025.0.0`
  - Added `setuptools<75.0.0` constraint in tox/requirements-dev

  **Testing**
  - Configured tox for `py39`, `py310`, `py311`, `py312`
  - All 112 tests pass on Python 3.10-3.12 (96% coverage)
  - Updated pre-commit: flake8 6.1.0, mypy 1.7.1

  **Code Quality**
  - Fixed exception handling: `ClientError` instead of bare `Exception`
  - Removed unnecessary `type: ignore` comments
  - Fixed line length violations

  **Documentation**
  - Updated `CONTRIBUTING.md` with dev setup and tox usage
  - Updated `.pre-commit-config.yaml` tool versions

  ## Testing
  ```bash
  tox              # All versions
  tox -e py312     # Specific version

  Results: 112 tests passed, 96% coverage on Python 3.9-3.12

  Release

  Version: 0.31.0 Type: Minor (non-breaking, adds Python 3.11/3.12 support) Target Branch: develop → main

  Breaking Changes

  None. Fully backward compatible.

* fix: rollback manual version bump and changelog edit

* fix: rollback another manual change in dp version

* ci: rollback missing default CHANGELOG configuration that was manually edited

* ci: fix release pipeline, add permissions to GH token

* ci: fix release pipeline, add permissions to GH token

* FIX #126  - Bump version and CHANGELOG for release 0.31.0

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Piotr Pękala <[email protected]>
Co-authored-by: Pawel Pinkos <[email protected]>
Co-authored-by: Piotr Sierkin <[email protected]>
Co-authored-by: Piotr Sierkin <[email protected]>
Co-authored-by: Pawel Jurkiewicz <[email protected]>
Co-authored-by: pawelpinkos <[email protected]>
Co-authored-by: github-actions <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jakub Szafran <[email protected]>
Co-authored-by: jszafran <[email protected]>
Co-authored-by: michalrudko <[email protected]>
Co-authored-by: Michal Rudko <[email protected]>
Co-authored-by: ptutak-getindata <[email protected]>
Co-authored-by: Dominik Gniewek-Węgrzyn <[email protected]>
Co-authored-by: Dominik Gniewek-Węgrzyn <[email protected]>

Author: 13 people

.github/workflows/prepare-release.yml

@@ -6,11 +6,14 @@ on:
       version_part:
         description: The part of the version to update (patch, minor or major)
         required: true
-        default: 'minor'
+        default: "minor"
 
 jobs:
   prepare-release:
     runs-on: ubuntu-latest
+    permissions:
+      contents: write # to create branch, tag and push the changes
+      pull-requests: write # to create pull request
     strategy:
       matrix:
         python-version: [3.9]
@@ -26,7 +29,7 @@ jobs:
         run: |
           echo "INPUT_VERSION_PART: ${{ github.event.inputs.version_part }}"
           python -c "if '${{ github.event.inputs.version_part }}' not in ['patch', 'minor', 'major']:   raise ValueError(\"'${{ github.event.inputs.version_part }}' must be one of ['patch', 'minor', 'major'])\")"
-      - name: Bump the version number  # bump2version is a maintained fork of original bumpversion
+      - name: Bump the version number # bump2version is a maintained fork of original bumpversion
         id: bump_version
         run: |
           pip install bump2version

.github/workflows/publish.yml

@@ -8,9 +8,11 @@ on:
 jobs:
   deploy:
     runs-on: ubuntu-latest
+    permissions:
+      contents: write # to push to develop, create releases and tags
     strategy:
       matrix:
-        python-version: [3.7]
+        python-version: ["3.9"]
     env:
       PYTHON_PACKAGE: data_pipelines_cli
     steps:

.github/workflows/python-package.yml

@@ -12,7 +12,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.9", "3.10"]
+        python-version: ["3.9", "3.10", "3.11", "3.12"]
 
     steps:
       - uses: actions/checkout@v3
@@ -40,12 +40,3 @@ jobs:
       - name: Test with tox
         run: |
           tox
-
-      - name: Report coverage
-        uses: paambaati/[email protected]
-        env:
-          CC_TEST_REPORTER_ID: ${{ secrets.CODE_CLIMATE }}
-        with:
-          coverageCommand: coverage xml
-          debug: true
-          coverageLocations: coverage.xml:coverage.py

.gitignore

@@ -4,6 +4,9 @@
 out/
 .idea_modules/
 
+# VSCode
+.vscode/
+
 ### macOS
 *.DS_Store
 .AppleDouble
@@ -125,4 +128,4 @@ venv.bak/
 
 docs/_build
 
-dp-testing
+dp-testing

.pre-commit-config.yaml

@@ -23,7 +23,7 @@ repos:
     - id: check-yaml
 
 - repo: https://github.com/pycqa/flake8
-  rev: 4.0.1
+  rev: 6.1.0
   hooks:
     - id: flake8
       additional_dependencies: [
@@ -33,7 +33,7 @@ repos:
       ]
 
 - repo: https://github.com/pre-commit/mirrors-mypy
-  rev: v0.961
+  rev: v1.7.1
   hooks:
     - id: mypy
       additional_dependencies:

.tool-versions

@@ -0,0 +1 @@
+python 3.12.12 3.9.24 3.10.19 3.11.14