Skip to content

Data engineering template #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
Dec 2, 2024
Merged

Data engineering template #45

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
fbd8a92
Add initial data-engineering template
lennartkats-db Nov 17, 2024
54762a7
Fix init calls
lennartkats-db Nov 18, 2024
596ece0
Add --branch parameters for now
lennartkats-db Nov 18, 2024
448c4f3
Update contrib/templates/data-engineering/base/databricks_template_sc…
lennartkats-db Nov 18, 2024
5493ac4
Use vars for catalogs/schemas
lennartkats-db Nov 21, 2024
e15ceb1
Update based on feedback
lennartkats-db Nov 22, 2024
8e01c61
Merge remote-tracking branch 'databricks/data-engineering' into data-…
lennartkats-db Nov 22, 2024
9b87093
Fix newline
lennartkats-db Nov 22, 2024
3141102
Move template one folder up
lennartkats-db Nov 24, 2024
eca8f28
Add workaround for databricks-dlt conflict
lennartkats-db Nov 25, 2024
38a385d
Merge remote-tracking branch 'databricks/main' into data-engineering
lennartkats-db Nov 25, 2024
80a2e02
Update based on reviewer feedback
lennartkats-db Nov 28, 2024
d2ea026
Remove workaround for older databricks-dlt
lennartkats-db Nov 28, 2024
56cb317
Extend description
lennartkats-db Nov 28, 2024
93cc57a
Update .gitignore
lennartkats-db Nov 28, 2024
6a615bb
Rework package structure
lennartkats-db Nov 30, 2024
c447107
Remove settings.json from the present PR
lennartkats-db Nov 30, 2024
bed1101
Fix template name
lennartkats-db Dec 1, 2024
56093bd
Remove settings.json from this PR
lennartkats-db Dec 2, 2024
aab5cc5
Revert "Add --branch parameters for now"
lennartkats-db Dec 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion contrib/templates/data-engineering/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@ assets in DABs.
Install it using

```
databricks bundle init https://github.com/databricks/bundle-examples --template-dir contrib/templates/data-engineering/base
databricks bundle init https://github.com/databricks/bundle-examples --template-dir contrib/templates/data-engineering/base --branch data-engineering
```
2 changes: 1 addition & 1 deletion contrib/templates/data-engineering/base/databricks_template_schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,5 +42,5 @@
"order": 4
}
},
"success_message": "\n\nYour new project has been created in the '{{.project_name}}' directory!\n\nTo add an example asset to your project, use\n\n $ cd ${{.project_name}}\n $ databricks bundle init https://github.com/databricks/bundle-examples --template-dir contrib/templates/data-engineering/assets/etl-pipeline\n\nRefer to the README.md file for full \"getting started\" instructions!"
"success_message": "\n\nYour new project has been created in the '{{.project_name}}' directory!\n\nTo add an example asset to your project, use\n\n $ cd ${{.project_name}}\n $ databricks bundle init https://github.com/databricks/bundle-examples --template-dir contrib/templates/data-engineering/assets/etl-pipeline --branch data-engineering\n\nRefer to the README.md file for full \"getting started\" instructions!"
}
2 changes: 1 addition & 1 deletion contrib/templates/data-engineering/base/template/{{.project_name}}/scripts/add_asset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@


def init_bundle(asset_type: AssetType) -> None:
cmd = f"databricks bundle init https://github.com/databricks/bundle-examples --template-dir contrib/templates/data-engineering/assets/{asset_type}"
cmd = f"databricks bundle init https://github.com/databricks/bundle-examples --template-dir contrib/templates/data-engineering/assets/{asset_type} --branch data-engineering"
subprocess.run(cmd, shell=True)


Expand Down