Data engineering template #45
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lennartkats-db
commented
Nov 18, 2024
pietern
reviewed
Nov 18, 2024
contrib/templates/data-engineering/base/databricks_template_schema.json
Outdated
Show resolved
Hide resolved
contrib/templates/data-engineering/base/template/{{.project_name}}/README.md.tmpl
Show resolved
Hide resolved
contrib/templates/data-engineering/base/template/{{.project_name}}/scripts/test.py
Outdated
Show resolved
Hide resolved
contrib/templates/data-engineering/base/template/{{.project_name}}/conftest.py
Show resolved
Hide resolved
contrib/templates/data-engineering/base/template/{{.project_name}}/conftest.py
Outdated
Show resolved
Hide resolved
contrib/templates/data-engineering/assets/etl-pipeline/databricks_template_schema.json
Outdated
Show resolved
Hide resolved
...ta-engineering/assets/etl-pipeline/template/assets/{{.pipeline_name}}/explorations/README.md
Outdated
Show resolved
Hide resolved
...ta-engineering/assets/etl-pipeline/template/assets/{{.pipeline_name}}/explorations/README.md
Outdated
Show resolved
Hide resolved
fjakobs
reviewed
Nov 18, 2024
...ering/assets/etl-pipeline/template/assets/{{.pipeline_name}}/{{.pipeline_name}}.job.yml.tmpl
Show resolved
Hide resolved
fjakobs
reviewed
Nov 18, 2024
.../assets/etl-pipeline/template/assets/{{.pipeline_name}}/{{.pipeline_name}}.pipeline.yml.tmpl
Outdated
Show resolved
Hide resolved
fjakobs
reviewed
Nov 18, 2024
contrib/templates/data-engineering/base/template/{{.project_name}}/databricks.yml.tmpl
Show resolved
Hide resolved
fjakobs
reviewed
Nov 18, 2024
contrib/templates/data-engineering/base/template/{{.project_name}}/databricks.yml.tmpl
Show resolved
Hide resolved
pietern
reviewed
Nov 18, 2024
...ib/templates/data-engineering/assets/etl-pipeline/template/assets/{{.pipeline_name}}/main.py
Outdated
Show resolved
Hide resolved
…hema.json Co-authored-by: Pieter Noordhuis <[email protected]>
Merged
pietern
reviewed
Nov 26, 2024
...g/assets/etl-pipeline/template/assets/{{.pipeline_name}}/explorations/exploration.ipynb.tmpl
Show resolved
Hide resolved
...ib/templates/data-engineering/assets/etl-pipeline/template/assets/{{.pipeline_name}}/main.py
Outdated
Show resolved
Hide resolved
contrib/templates/data-engineering/databricks_template_schema.json
Outdated
Show resolved
Hide resolved
contrib/templates/data-engineering/template/{{.project_name}}/conftest.py
Show resolved
Hide resolved
contrib/templates/data-engineering/template/{{.project_name}}/conftest.py
Outdated
Show resolved
Hide resolved
contrib/templates/data-engineering/template/{{.project_name}}/pyproject.toml
Outdated
Show resolved
Hide resolved
contrib/templates/data-engineering/template/{{.project_name}}/scripts/test.py
Outdated
Show resolved
Hide resolved
...ta-engineering/assets/etl-pipeline/template/assets/{{.pipeline_name}}/explorations/README.md
Show resolved
Hide resolved
contrib/templates/data-engineering/assets/job/databricks_template_schema.json
Outdated
Show resolved
Hide resolved
pietern
reviewed
Nov 28, 2024
...ta-engineering/assets/etl-pipeline/template/assets/{{.pipeline_name}}/explorations/README.md
Show resolved
Hide resolved
...data-engineering/assets/etl-pipeline/template/assets/{{.pipeline_name}}/sources/dev/taxis.py
Outdated
Show resolved
Hide resolved
| # conftest.py is used to configure pytest. | ||
| # This file is in the root since it affects all tests through this bundle. | ||
| # It makes sure all 'assets/*' directories are added to `sys.path` so that | ||
| # tests can import them. |
Contributor
There was a problem hiding this comment.
What happens if you have 2 entries in this directory, both with a transformations directory?
I think this is contrary to what one would expect.
Contributor
Author
There was a problem hiding this comment.
👍 will look into an alternative approach
Contributor
Author
There was a problem hiding this comment.
For the latest incarnation, the pipeline name is included in the package path. We'll need to discuss this a bit further in the LakeFlow group and solicit feedback.
contrib/templates/data-engineering/template/{{.project_name}}/conftest.py
Show resolved
Hide resolved
This reverts commit 596ece0.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This adds a data engineering template.
uv run add-assetcan be used to add a pipeline to a projectuv run testcan be used to run all tests on serverless computeuv run pytestcan be used to run all tests on any type of compute as configured in the current IDE or .databrickscfg settingsHow to try this
You can give the template a try using
Note that each pipeline has a separate template. New pipelines can be added by using the short-hand
uv run add-assetor by manually instantiating the pipeline template with