Skip to content

Commit b1cbf80

Browse files
cmcarthurmikealfarecolin-rogers-dbt
authored
move protos into external dependency (#1087)
Co-authored-by: Mike Alfare <[email protected]> Co-authored-by: Colin Rogers <[email protected]>
1 parent d8ebe13 commit b1cbf80

File tree

7 files changed

+13
-773
lines changed

7 files changed

+13
-773
lines changed

.pre-commit-config.yaml

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,14 +26,13 @@ repos:
2626
- --target-version=py310
2727
- --target-version=py311
2828
- --target-version=py312
29-
- --force-exclude=dbt-adapters/src/dbt/adapters/events/adapter_types_pb2.py
3029
additional_dependencies: [flaky]
3130

3231
- repo: https://github.com/pycqa/flake8
3332
rev: 7.1.1
3433
hooks:
3534
- id: flake8
36-
exclude: dbt-adapters/src/dbt/adapters/events/adapter_types_pb2.py|dbt-adapters/tests/|dbt-spark/tests/|dbt-bigquery/tests/|dbt-redshift/tests|dbt-postgres/tests|dbt-snowflake/tests
35+
exclude: dbt-adapters/tests/|dbt-spark/tests/|dbt-bigquery/tests/|dbt-redshift/tests|dbt-postgres/tests|dbt-snowflake/tests
3736
args:
3837
- --max-line-length=99
3938
- --select=E,F,W

dbt-adapters/pyproject.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ classifiers = [
2828
]
2929
dependencies = [
3030
"dbt-common>=1.13,<2.0",
31+
"dbt-protos>=1.0.291,<2.0",
3132
"pytz>=2015.7",
3233
# installed via dbt-common but used directly
3334
"agate>=1.0,<2.0",

dbt-adapters/src/dbt/adapters/events/README.md

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,7 @@ The event module provides types that represent what is happening in dbt in `even
88
When events are processed via `fire_event`, nearly everything is logged. Whether or not the user has enabled the debug flag, all debug messages are still logged to the file. However, some events are particularly time consuming to construct because they return a huge amount of data. Today, the only messages in this category are cache events and are only logged if the `--log-cache-events` flag is on. This is important because these messages should not be created unless they are going to be logged, because they cause a noticable performance degredation. These events use a "fire_event_if" functions.
99

1010
# Adding a New Event
11-
* Add a new message in types.proto, and a second message with the same name + "Msg". The "Msg" message should have two fields, an "info" field of EventInfo, and a "data" field referring to the message name without "Msg"
12-
* run the protoc compiler to update adapter_types_pb2.py: make adapter_proto_types
13-
* Add a wrapping class in core/dbt/adapters/event/types.py with a Level superclass plus code and message methods
14-
15-
We have switched from using betterproto to using google protobuf, because of a lack of support for Struct fields in betterproto.
16-
17-
The google protobuf interface is janky and very much non-Pythonic. The "generated" classes in types_pb2.py do not resemble regular Python classes. They do not have normal constructors; they can only be constructed empty. They can be "filled" by setting fields individually or using a json_format method like ParseDict. We have wrapped the logging events with a class (in types.py) which allows using a constructor -- keywords only, no positional parameters.
11+
All protos have been moved into the central protos repository. To edit an event proto, edit https://github.com/dbt-labs/proto-python-public or open an issue on that repository.
1812

1913
## Required for Every Event
2014

0 commit comments

Comments
 (0)