Skip to content

Commit 146dce6

Browse files
alyssa-dbalyssa-db
authored
acc: pipelines run enrich events output (#3355)
## Changes Acceptance tests, following #3337 and #3332 Test output of `pipelines run` based on test.toml API endpoints
1 parent 49f8b13 commit 146dce6

File tree

14 files changed

+171
-3
lines changed

14 files changed

+171
-3
lines changed

‎acceptance/pipelines/e2e/output.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]
2525
Update ID: [UUID]
2626

2727
Update for pipeline completed successfully.
28+
2829
Pipeline configurations for this update:
2930
• All tables are refreshed
3031

@@ -69,6 +70,7 @@ Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]
6970
Update ID: [UUID]
7071

7172
Update for pipeline completed successfully.
73+
7274
Pipeline configurations for this update:
7375
• All tables are refreshed
7476

‎acceptance/pipelines/run/no-wait/output.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,5 +13,6 @@ Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]
1313
Update ID: [UUID]
1414

1515
Update for pipeline completed successfully.
16+
1617
Pipeline configurations for this update:
1718
• All tables are refreshed

‎acceptance/pipelines/run/refresh-flags/output.txt‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]
1313
Update ID: [UUID]
1414

1515
Update for pipeline completed successfully.
16+
1617
Pipeline configurations for this update:
1718
• All tables are refreshed
1819

@@ -35,6 +36,7 @@ Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]
3536
Update ID: [UUID]
3637

3738
Update for pipeline completed successfully.
39+
3840
Pipeline configurations for this update:
3941
• All tables are refreshed
4042

@@ -54,6 +56,7 @@ Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]
5456
Update ID: [UUID]
5557

5658
Update for pipeline completed successfully.
59+
5760
Pipeline configurations for this update:
5861
• All tables are refreshed
5962

@@ -76,6 +79,7 @@ Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]
7679
Update ID: [UUID]
7780

7881
Update for pipeline completed successfully.
82+
7983
Pipeline configurations for this update:
8084
• All tables are refreshed
8185

‎acceptance/pipelines/run/restart/output.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/[UUID]
1313
Update ID: [UUID]
1414

1515
Update for pipeline completed successfully.
16+
1617
Pipeline configurations for this update:
1718
• All tables are refreshed
1819

‎acceptance/pipelines/run/run-info/databricks.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
bundle:
2+
name: test-pipeline-run
3+
4+
resources:
5+
pipelines:
6+
my_pipeline:
7+
name: test-pipeline
8+
libraries:
9+
- file:
10+
path: pipeline_file.py

‎acceptance/pipelines/run/run-info/out.test.toml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
Local = true
2+
Cloud = false
3+
4+
[EnvMatrix]
5+
DATABRICKS_CLI_DEPLOYMENT = ["terraform", "direct-exp"]

‎acceptance/pipelines/run/run-info/output.txt‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
2+
>>> [PIPELINES] deploy
3+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-run/default/files...
4+
Deploying resources...
5+
Updating deployment state...
6+
Deployment complete!
7+
View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?o=[NUMID]
8+
9+
=== Run pipeline and display pipeline configuration summary and events durations
10+
>>> [PIPELINES] run
11+
Update URL: [DATABRICKS_URL]/#joblist/pipelines/[UUID]/updates/test-update-123
12+
13+
[TIMESTAMP] update_progress "Update test-update-123 is COMPLETED."
14+
[TIMESTAMP] update_progress "Update test-update-123 is RUNNING."
15+
[TIMESTAMP] update_progress "Update test-update-123 is SETTING_UP_TABLES."
16+
[TIMESTAMP] update_progress "Update test-update-123 is INITIALIZING."
17+
[TIMESTAMP] update_progress "Update test-update-123 is WAITING_FOR_RESOURCES."
18+
[TIMESTAMP] update_progress "Update test-update-123 is CREATED."
19+
Update ID: test-update-123
20+
21+
Update for pipeline test-pipeline completed successfully.
22+
23+
Pipeline ID: test-pipeline-789
24+
Update start time: [TIMESTAMP]Z
25+
Update end time: [TIMESTAMP]Z
26+
27+
Pipeline configurations for this update:
28+
• All tables are fully refreshed
29+
• Update cause: Manual trigger
30+
• Serverless compute
31+
• Channel: CURRENT
32+
• Continuous pipeline
33+
• Development mode
34+
• Catalog: test_catalog
35+
• Schema: test_schema
36+
37+
Run Phase Duration
38+
--------- --------
39+
CREATED 1.0s
40+
WAITING_FOR_RESOURCES 1.0s
41+
INITIALIZING 1.0s
42+
SETTING_UP_TABLES 1.0s
43+
RUNNING 1m 25s

‎acceptance/pipelines/run/run-info/pipeline_file.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Simple pipeline file for testing
2+
import dlt
3+
4+
5+
@dlt.table
6+
def my_table():
7+
return spark.range(10)

‎acceptance/pipelines/run/run-info/script‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
trace $PIPELINES deploy
2+
title "Run pipeline and display pipeline configuration summary and events durations"
3+
# Test note: The mock API handler for "GET /api/2.0/pipelines/{pipeline_id}/events" doesn't support
4+
# the OrderBy query parameter. In production, the real API returns events in descending order (newest first), and
5+
# bundle/run/pipeline.go reverses them again to display chronologically.
6+
# So, test output appears "backwards" compared to production behavior.
7+
trace $PIPELINES run

‎acceptance/pipelines/run/run-info/test.toml‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
Local = true
2+
Cloud = false
3+
4+
[[Server]]
5+
Pattern = "GET /api/2.0/pipelines/{pipeline_id}/updates/{update_id}"
6+
Response.Body = '''
7+
{
8+
"update": {
9+
"update_id": "test-update-123",
10+
"state": "COMPLETED",
11+
"full_refresh": true,
12+
"cause": "Manual trigger",
13+
"creation_time": 1640995200000,
14+
"cluster_id": "test-cluster-456",
15+
"config": {
16+
"name": "test-pipeline",
17+
"id": "test-pipeline-789",
18+
"serverless": true,
19+
"channel": "CURRENT",
20+
"continuous": true,
21+
"development": true,
22+
"catalog": "test_catalog",
23+
"schema": "test_schema"
24+
}
25+
}
26+
}
27+
'''
28+
29+
[[Server]]
30+
Pattern = "GET /api/2.0/pipelines/{pipeline_id}/events"
31+
Response.Body = '''
32+
{
33+
"events": [
34+
{
35+
"timestamp": "2022-01-01T00:00:01.000Z",
36+
"event_type": "update_progress",
37+
"message": "Update test-update-123 is CREATED."
38+
},
39+
{
40+
"timestamp": "2022-01-01T00:00:02.000Z",
41+
"event_type": "update_progress",
42+
"message": "Update test-update-123 is WAITING_FOR_RESOURCES."
43+
},
44+
{
45+
"timestamp": "2022-01-01T00:00:03.000Z",
46+
"event_type": "update_progress",
47+
"message": "Update test-update-123 is INITIALIZING."
48+
},
49+
{
50+
"timestamp": "2022-01-01T00:00:04.000Z",
51+
"event_type": "update_progress",
52+
"message": "Update test-update-123 is SETTING_UP_TABLES."
53+
},
54+
{
55+
"timestamp": "2022-01-01T00:00:05.000Z",
56+
"event_type": "update_progress",
57+
"message": "Update test-update-123 is RUNNING."
58+
},
59+
{
60+
"timestamp": "2022-01-01T00:01:30.000Z",
61+
"event_type": "update_progress",
62+
"message": "Update test-update-123 is COMPLETED."
63+
}
64+
]
65+
}
66+
'''
67+
68+
[[Server]]
69+
Pattern = "POST /api/2.0/pipelines/{pipeline_id}/updates"
70+
Response.Body = '''
71+
{
72+
"update_id": "test-update-123"
73+
}
74+
'''

0 commit comments

Comments
 (0)