Release 0.26.0 (#115)

* Documentation improvement (#80)

* Documentation improvement

* GitPython is not required anymore (#83)

Installation docs upgrade

* Documentation improvement (#80)

* Documentation improvement

* GitPython is not required anymore (#83)

Installation docs upgrade

* feat: looker integration for compile and deploy, new command: bi

* tests fix, looker git with ssh

* chore: organizing imports, removed not used variables

* chore: dbt2looker as extra dependency, mypy errors fix, bi-git-key-path option instead of key-path

* chore: formatting, types-requests dependency for mypy

* refactor: refactoring bi, removing bi command, fix tests, moving bi flags to bi.yml instead of call option

* Fix after rebase

* Fix after rebase

* Fix after rebase

* Fix after rebase

* Airbyte integration (#86)

* [DATA-366] Update datahub package and add dbt extra (#79)

Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>

* Documentation improvement (#80)

* Added IAP Airbyte authorization with service account file

Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>
Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com>
Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com>
Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com>
Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com>

* Service account OIDC ID token in deploy command (#91)

* Added possibility to pass platform-agnostic service account OIDC ID token to deploy command

* Added changelog information

* added 'source freshness' command to dbt compile

* Bump copier (#93)

* [DATA-366] Update datahub package and add dbt extra (#79)

* feat: update datahub package and add dbt extra

* fix: replace datahub requirements

* fix: pre-commit coma

Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>

* Documentation improvement (#80)

* Documentation improvement

* dp compile default environment has been set to local

* GitPython is not required anymore (#83)

Installation docs upgrade

* chore: bump datahub (#84)

Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>

* [DATA-366] Update datahub package and add dbt extra (#79)

* feat: update datahub package and add dbt extra

* fix: replace datahub requirements

* fix: pre-commit coma

Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>

* Documentation improvement (#80)

* Documentation improvement

* GitPython is not required anymore (#83)

Installation docs upgrade

* chore: bump packages except copier

* fix: package name

* chore: add specific version for pyyaml

* fix: misspelling

* chore: test docs

* chore: test

* test

* test

* test

* test

* test

* test

* chore: bump packages

* chore: add renovate

* chore: bump copier

* chore: update exception

* fix: update by removing not existing flag

* chore: update tests

* feat: update schema on manifest to v7

* chore: bump packages

* chore: delete renovate

* chore: bump python version

* chore: remove junk

* chore: bring back old exception

* chore: add matrix to tox

* chore: add new require and classifier

* trigger

* chore(docs): update badge and changelog

* chore(docs): replace old extension

* chore: add description

* chore(docs): add old suffix and envops

* chore(docs): update changelog

* chore(docs): update changelog

* chore(docs): update copier

* chore: pinup version for dbt2looker

* chore(docs): update changelog

* chore: replace var for more meaningful

Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>
Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com>

* chore: downgrade version to properly prepare release

* chore(docs): fix date in changelog

* BI deployment is disabled when config is missing (#95)

* DATA-629: Added Airbyte configuration documentation (#97)

* feat: disable sync bucket (#102)

* chore: add flag to turn off bucket syncing with artefacts

* chore: replace default value

* chore(docs): updat changelog

* fix: add readthedocs config file

* chore: repair failing tests

* feat: add test for sync_bucket

* chore(docs): cosmetic changes in changelog

* chore: rename `sync-bucket` to `disable-bucket-sync`

* chore(docs): update with flag

* chore: replace `sync_bucket` with `disable_bucket_sync`

* chore: update name for `_bucket_sync`

---------

Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>

* Fix failing readthedocs builds (#104)

* Removed sphinx action from prepare-release pipeline (#105)

* Bump werkzeug from 2.2.2 to 2.2.3 (#106)

* Bump werkzeug from 2.2.2 to 2.2.3

Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3.
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](pallets/werkzeug@2.2.2...2.2.3)

---
updated-dependencies:
- dependency-name: werkzeug
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* bump codeclimate-action to 4.0.0

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com>

* Airbyte fix creating multiple connections (#109)

* Fixing tests.

* Fix failing test.

* Remove redundant line.

* Add info about fix to CHANGELOG.md.

* Add workspace_id entry to config.

* Get rid of walrus operator & add test for missing workspace id.

* Fix wrong connection id in Airflow ingestion task.

* Fix typo in test name.

* Add fetching default workspace if not available in config.

* Increasign test coverage.

* Update workspace_id configuration description.

* Bumping Python version to 3.8 in prepare-release workflow.

* Pin pydantic version (#112)

* Pin Pydantic version until Copier fixes their dependency.

* Add info to changelog.

* Upgrade dbt to 1.5.4 (#114)

* Upgrade dbt to 1.5.4

* Fixing imports for publish.py

* Update of dbt_schema_version

* Updating manifest.json for tests

* Fix manifest for testing

* Fixing manifest cntd.

* Fixing manifest cntd.

* Replace example-project with exampleproject

* Adding tutorial for upgrading manifest in tests cntd.

* Fixing typos in the docs

* Version fixes before release

---------

Co-authored-by: Michal Rudko <michal.rudko@getindata.com>

* FIX #115  - Bump version and CHANGELOG for release 0.26.0

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Piotr Pękala <piotr.pekala@getindata.com>
Co-authored-by: Pawel Pinkos <pawel.pinkos@getindata.com>
Co-authored-by: Piotr Sierkin <psierkin@gmail.com>
Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>
Co-authored-by: Pawel Jurkiewicz <pawel.jurkiewicz94@gmail.com>
Co-authored-by: pawelpinkos <113952441+pawelpinkos@users.noreply.github.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jakub Szafran <33091690+jszafran@users.noreply.github.com>
Co-authored-by: jszafran <jszafran.pv@gmail.com>
Co-authored-by: michalrudko <michal.rudko@gmail.com>
Co-authored-by: Michal Rudko <michal.rudko@getindata.com>

Author: 11 people

CHANGELOG.md

@@ -2,6 +2,11 @@
 
 ## [Unreleased]
 
+## [0.26.0] - 2023-08-25
+
+-   Upgrade dbt to 1.5.4
+-   Upgrade acryl-datahub to 0.10.4
+
 ## [0.25.3] - 2023-07-04
 
 -   Pinned Pydantic's version (transient dependency from Copier) as new major version breaks our CLI.
@@ -265,7 +270,9 @@
 -   Draft of `dp init`, `dp create`, `dp template new`, `dp template list` and `dp dbt`
 -   Draft of `dp compile` and `dp deploy`
 
-[Unreleased]: https://github.com/getindata/data-pipelines-cli/compare/0.25.3...HEAD
+[Unreleased]: https://github.com/getindata/data-pipelines-cli/compare/0.26.0...HEAD
+
+[0.26.0]: https://github.com/getindata/data-pipelines-cli/compare/0.25.3...0.26.0
 
 [0.25.3]: https://github.com/getindata/data-pipelines-cli/compare/0.25.2...0.25.3
 

data_pipelines_cli/__init__.py

@@ -5,4 +5,4 @@
 pipelines.
 """
 
-version = "0.25.3"
+version = "0.26.0"

data_pipelines_cli/cli_commands/publish.py

@@ -5,9 +5,8 @@
 
 import click
 import yaml
-from dbt.contracts.graph.compiled import ManifestNode
 from dbt.contracts.graph.manifest import Manifest
-from dbt.contracts.graph.parsed import ColumnInfo
+from dbt.contracts.graph.nodes import ColumnInfo, ManifestNode
 
 from ..cli_constants import BUILD_DIR
 from ..cli_utils import echo_info, echo_warning

setup.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.25.3
+current_version = 0.26.0
 
 [bumpversion:file:setup.py]
 

setup.py

@@ -16,7 +16,7 @@
     "fsspec==2022.11.0",
     "packaging==21.3",
     "colorama==0.4.5",
-    "dbt-core==1.3.1",
+    "dbt-core==1.5.4",
     "pydantic<2",
 ]
 
@@ -27,19 +27,19 @@
 
 EXTRA_REQUIRE = {
     # DBT adapters
-    "bigquery": ["dbt-bigquery==1.3.0"],
-    "postgres": ["dbt-postgres==1.3.1"],
-    "snowflake": ["dbt-snowflake==1.3.0"],
-    "redshift": ["dbt-redshift==1.3.0"],
+    "bigquery": ["dbt-bigquery==1.5.5"],
+    "postgres": ["dbt-postgres==1.5.4"],
+    "snowflake": ["dbt-snowflake==1.5.2"],
+    "redshift": ["dbt-redshift==1.5.9"],
     "dbt-all": [
-        "dbt-bigquery==1.3.0",
-        "dbt-postgres==1.3.1",
-        "dbt-snowflake==1.3.0",
-        "dbt-redshift==1.3.0",
+        "dbt-bigquery==1.5.5",
+        "dbt-postgres==1.5.4",
+        "dbt-snowflake==1.5.2",
+        "dbt-redshift==1.5.9",
     ],
     # ---
     "docker": ["docker==6.0.1"],
-    "datahub": ["acryl-datahub[dbt]==0.9.3.2"],
+    "datahub": ["acryl-datahub[dbt]==0.10.4"],
     "git": ["GitPython==3.1.29"],
     "looker": ["dbt2looker==0.11.0"],
     "tests": [
@@ -69,7 +69,7 @@
 
 setup(
     name="data_pipelines_cli",
-    version="0.25.3",
+    version="0.26.0",
     description="CLI for data platform",
     long_description=README,
     long_description_content_type="text/markdown",

tests/cli_commands/test_prepare_env.py

@@ -19,7 +19,7 @@ class GenHomeProfilesCommandTestCase(unittest.TestCase):
             "outputs": {
                 "env_execution": {
                     "method": "service-account",
-                    "project": "example-project",
+                    "project": "exampleproject",
                     "dataset": "var21-dataset",
                     "keyfile": "/tmp/a/b/c/d.json",
                     "timeout_seconds": 150,

tests/cli_commands/test_publish.py

@@ -23,7 +23,7 @@ class PublishCommandTestCase(unittest.TestCase):
         "sources": [
             {
                 "name": "my_test_project_1337",
-                "database": "example-project",
+                "database": "exampleproject",
                 "schema": "username_private_working_dataset",
                 "tags": ["project:my_test_project_1337"],
                 "meta": {"dag": "experimental-dag"},

tests/goldens/config/base/bigquery.yml

@@ -1,5 +1,5 @@
 method: service-account
-project: example-project
+project: exampleproject
 dataset: example-dataset
 threads: 1
 keyfile: /var/keyfile.json

tests/goldens/config/staging/bigquery.yml

@@ -1,5 +1,5 @@
 method: service-account
-project: example-project
+project: exampleproject
 dataset: "{{ var('variable_2') }}-dataset"
 threads: "{{ var('variable_1') }}"
 keyfile: "{{ env_var('BIGQUERY_KEYFILE') }}"

tests/goldens/example_profiles/dev_bigquery.yml

@@ -2,7 +2,7 @@ bigquery:
   outputs:
     env_execution:
       method: service-account
-      project: example-project
+      project: exampleproject
       dataset: example-dataset
       keyfile: /var/keyfile.json
       timeout_seconds: 150