dbx support for pipelines (delta live tables)

[benbauer89]

Hey, I was wondering whether it's already possible to use dbx to deploy pipelines (delta live tables).
If not it would be a great feature :)

[renardeinside]

hi @benbauer89 ,
we have plans to support DLT, but currently are constrained from multiple sides.

[jiri-vinarek]

I agree it would be a great feature :) @renardeinside what are the main roadblocks? Just curious.

[guillesd]

yes, I am also really interested in this feature. Many of my clients are interested in using dbx but also DLTs... it feels like a natural thing that dbx supports pipelines! Can we help somewhere @renardeinside?

[renardeinside]

Hi everyone on this thread!

We're in progress of implementation, most probably DLT support will be delivered until end of first week of October (7-8 approx.)

[renardeinside]

implemented in #531 will be soonish released in 0.8.0

[ctaylor-carvana]

Amazing! Question, does this support permission setting as well? Just appending these to the pipeline list doesn’t seem to work. I can’t find any example of properly setting the permission. https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-api-guide.html access_control_list: - user_name: ***@***.*** permission_level: IS_OWNER - group_name: team_ngc permission_level: CAN_MANAGE Best, Christina Taylor ***@***.***

…

On Nov 2, 2022, at 9:17 AM, Ivan Trusov ***@***.***> wrote: implemented in #531 <https://www.google.com/url?q=https://github.com/databrickslabs/dbx/pull/531&source=gmail-imap&ust=1668007065000000&usg=AOvVaw0F2Q40KNTcQxGxqH3gxQvV> will be soonish released in 0.8.0 — Reply to this email directly, view it on GitHub <https://www.google.com/url?q=https://github.com/databrickslabs/dbx/discussions/250%23discussioncomment-4038072&source=gmail-imap&ust=1668007065000000&usg=AOvVaw2w5do9d-KgYpwYRI1i77IR>, or unsubscribe <https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AYPPEBBMSLKQFJPZBFMCZ5LWGKAZPANCNFSM5WLEQHCA&source=gmail-imap&ust=1668007065000000&usg=AOvVaw1EHWKsrhkVdlS0ydlrn1th>. You are receiving this because you are subscribed to this thread.

[renardeinside]

yes, it definitely does apply the permissions. Please note that they shall be provided on the level of the pipeline definition, e.g.:

environments:
  default:
    workflows:
      - name: "[dev][dbx-dlt-test]"
        workflow_type: "pipeline"
        development: true
        libraries:
          - notebook:
              path: "/Repos/some/path"
        clusters:
          - label: "default"
            autoscale:
              min_workers: 1
              max_workers: 3
              mode: "ENHANCED"
            driver_instance_pool_id: "instance-pool://itrusov-pool"
            instance_pool_id: "instance-pool://itrusov-pool"
        access_control_list:
          - user_name: "[email protected]"
            permission_level: "IS_OWNER"
          - group_name: "some-group"
            permission_level: "CAN_VIEW"

[ctaylor-carvana]

Good to know; thanks! I had trouble deploying both the pipeline and the workflow that has a pipeline task at once - when using [“some-pipeline”, “some-workflow"] the workflow cannot find the pipeline. Ended up creating two deployment files. Is that expected? Christina Taylor ***@***.***

…

On Nov 2, 2022, at 12:50 PM, Ivan Trusov ***@***.***> wrote: yes, it definitely does apply the permissions. Please note that they shall be provided on the level of the pipeline definition, e.g.: environments: default: workflows: - name: "[dev][dbx-dlt-test]" workflow_type: "pipeline" development: true libraries: - notebook: path: "/Repos/some/path" clusters: - label: "default" autoscale: min_workers: 1 max_workers: 3 mode: "ENHANCED" driver_instance_pool_id: "instance-pool://itrusov-pool" instance_pool_id: "instance-pool://itrusov-pool" access_control_list: - user_name: ***@***.***" permission_level: "IS_OWNER" - group_name: "some-group" permission_level: "CAN_VIEW" — Reply to this email directly, view it on GitHub <https://www.google.com/url?q=https://github.com/databrickslabs/dbx/discussions/250%23discussioncomment-4040964&source=gmail-imap&ust=1668019848000000&usg=AOvVaw2JTBaw3hum1Pp54DjtuOJe>, or unsubscribe <https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AYPPEBEESPZRSINOK7XNVYLWGKZYNANCNFSM5WLEQHCA&source=gmail-imap&ust=1668019848000000&usg=AOvVaw3l7sPJXLsqsLdYX4ZwGCSL>. You are receiving this because you are subscribed to this thread.

[renardeinside]

nope, that's not - I've created #539 to fix that one

[renardeinside]

Implemented and released in 0.8.0! 🔥