Skip to content

Commit cfb8158

Browse files
authored
cli-pipelines template: remove project directory (#3294)
## Changes Remove project_pipelines directory and bubble up subdirectories. Consolidated 2 READMEs to have only 1 in the directory. ## Why For one pipeline, the transformations, explorations, and utilities folders will be in the root of the my_project directory. ## Tests Updated acceptance test output.
1 parent d1bc0af commit cfb8158

File tree

34 files changed

+87
-122
lines changed

34 files changed

+87
-122
lines changed

acceptance/pipelines/init/error-cases/output/my_project/README.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,23 @@ The 'my_project' project was generated by using the CLI Pipelines template.
2020
https://docs.databricks.com/dev-tools/vscode-ext.html. Or the PyCharm plugin from
2121
https://www.databricks.com/blog/announcing-pycharm-integration-databricks.
2222
23+
## Pipeline Structure
24+
25+
This folder defines all source code for the my_project_pipeline pipeline:
26+
27+
- `explorations`: Ad-hoc notebooks used to explore the data processed by this pipeline.
28+
- `transformations`: All dataset definitions and transformations.
29+
- `utilities` (optional): Utility functions and Python modules used in this pipeline.
30+
31+
## Getting Started
32+
33+
To get started, go to the `transformations` folder -- most of the relevant source code lives there:
34+
35+
* By convention, every dataset under `transformations` is in a separate file.
36+
* Take a look at the sample under "sample_trips_my_project.py" to get familiar with the syntax.
37+
Read more about the syntax at https://docs.databricks.com/dlt/python-ref.html.
38+
39+
For more tutorials and reference material, see https://docs.databricks.com/dlt.
2340
2441
## Deploying pipelines
2542

acceptance/pipelines/init/error-cases/output/my_project/databricks.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ bundle:
77
include:
88
- resources/*.yml
99
- resources/*/*.yml
10-
- my_project_pipeline/*.yml
10+
- ./*.yml
1111

1212
# Variable declarations. These variables are assigned in the dev/prod targets below.
1313
variables:

acceptance/pipelines/init/error-cases/output/my_project/my_project_pipeline/explorations/sample_exploration.ipynb renamed to acceptance/pipelines/init/error-cases/output/my_project/explorations/sample_exploration.ipynb

File renamed without changes.

acceptance/pipelines/init/error-cases/output/my_project/my_project_pipeline/my_project.pipeline.yml renamed to acceptance/pipelines/init/error-cases/output/my_project/my_project.pipeline.yml

File renamed without changes.

acceptance/pipelines/init/error-cases/output/my_project/my_project_pipeline/README.md

Lines changed: 0 additions & 22 deletions
This file was deleted.

acceptance/pipelines/init/error-cases/output/my_project/my_project_pipeline/transformations/sample_trips_my_project.py renamed to acceptance/pipelines/init/error-cases/output/my_project/transformations/sample_trips_my_project.py

File renamed without changes.

acceptance/pipelines/init/error-cases/output/my_project/my_project_pipeline/transformations/sample_zones_my_project.py renamed to acceptance/pipelines/init/error-cases/output/my_project/transformations/sample_zones_my_project.py

File renamed without changes.

acceptance/pipelines/init/error-cases/output/my_project/my_project_pipeline/utilities/utils.py renamed to acceptance/pipelines/init/error-cases/output/my_project/utilities/utils.py

File renamed without changes.

acceptance/pipelines/init/python/output/my_python_project/README.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,23 @@ The 'my_python_project' project was generated by using the CLI Pipelines templat
2020
https://docs.databricks.com/dev-tools/vscode-ext.html. Or the PyCharm plugin from
2121
https://www.databricks.com/blog/announcing-pycharm-integration-databricks.
2222
23+
## Pipeline Structure
24+
25+
This folder defines all source code for the my_python_project_pipeline pipeline:
26+
27+
- `explorations`: Ad-hoc notebooks used to explore the data processed by this pipeline.
28+
- `transformations`: All dataset definitions and transformations.
29+
- `utilities` (optional): Utility functions and Python modules used in this pipeline.
30+
31+
## Getting Started
32+
33+
To get started, go to the `transformations` folder -- most of the relevant source code lives there:
34+
35+
* By convention, every dataset under `transformations` is in a separate file.
36+
* Take a look at the sample under "sample_trips_my_python_project.py" to get familiar with the syntax.
37+
Read more about the syntax at https://docs.databricks.com/dlt/python-ref.html.
38+
39+
For more tutorials and reference material, see https://docs.databricks.com/dlt.
2340
2441
## Deploying pipelines
2542

acceptance/pipelines/init/python/output/my_python_project/databricks.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ bundle:
77
include:
88
- resources/*.yml
99
- resources/*/*.yml
10-
- my_python_project_pipeline/*.yml
10+
- ./*.yml
1111

1212
# Variable declarations. These variables are assigned in the dev/prod targets below.
1313
variables:

0 commit comments

Comments
 (0)