Skip to content

Commit 5feed5b

Browse files
DeltaMichaelDilyan Marinov
DeltaMichael
and
Dilyan Marinov
authored
vdk-structlog: put vdk init logs config behind flag (#3107)
## Why? Configuring the logs in the vdk_initialize hook causes commands like vdk server and vdk info to have the same verbose logging format as data job logs. ## What? - Put the log configuration for vdk_initialize behind a config option and disable it by default for local runs and enable it by default for cloud runs. Configuring the logging format in vdk_initialize might still be useful for cloud runs and we don't care about other vdk commands output in cloud environments - Move config classes and functions into separate file ## How was this tested ```sh (venv) 11:01:14 ~ $ export VDK_STRUCTLOG_FORMAT_INIT_LOGS=True (venv) 11:01:19 ~ $ vdk create 2024-02-15 11:01:21,922 [VDK] [INFO ] vdk.internal.builtin_plugins.b builtin_hook_impl.py :98 vdk_initialize - Setting: OP_ID: 14b52d54-af7d-4eab-a0a5-05b50d534aed-1707987681, ATTEMPT_ID: 14b52d54-af7d-4eab-a0a5-05b50d534aed-1707987681-aa061, EXECUTION_ID: 14b52d54-af7d-4eab-a0a5-05b50d534aed-1707987681 2024-02-15 11:01:21,923 [VDK] [INFO ] vdk.internal.cli_entry cli_entry.py :135 vdk_main - Start CLI vdk with args ['create'] Job Name: (venv) 11:01:52 ~ $ export VDK_STRUCTLOG_FORMAT_INIT_LOGS=False (venv) 11:01:58 ~ $ vdk create Job Name: ``` ```sh (venv) 11:03:25 ~ $ export VDK_STRUCTLOG_FORMAT_INIT_LOGS=True (venv) 11:03:27 ~ $ vdk version 2024-02-15 11:03:30,267 [VDK] [INFO ] vdk.internal.builtin_plugins.b builtin_hook_impl.py :98 vdk_initialize - Setting: OP_ID: 907e2fef-40c7-45c1-a492-f1cda0717748-1707987810, ATTEMPT_ID: 907e2fef-40c7-45c1-a492-f1cda0717748-1707987810-3568d, EXECUTION_ID: 907e2fef-40c7-45c1-a492-f1cda0717748-1707987810 2024-02-15 11:03:30,267 [VDK] [INFO ] vdk.internal.cli_entry cli_entry.py :135 vdk_main - Start CLI vdk with args ['version'] 2024-02-15 11:03:30,292 [VDK] [INFO ] vdk.internal.builtin_plugins.v version.py :85 version - Versatile Data Kit (VDK) Version: 0.3.1177416838 Build details: RELEASE_VERSION=0.3.1177416838, BUILD_DATE=Thu Feb 15 07:57:26 UTC 2024, BUILD_MACHINE_INFO=Linux runner-trzkizxz-project-28359933-concurrent-0rvcnn 5.4.235-144.344.amzn2.x86_64 #1 SMP Sun Mar 12 12:50:22 UTC 2023 x86_64 GNU/Linux, GITLAB_CI_JOB_ID=6177494231, GIT_COMMIT_SHA=a4c5a23b69ed2bd135b7f14658abd18c143e8759, GIT_BRANCH=main Python version: 3.11.7 64bit (/Users/mdilyan/Projects/versatile-data-kit/projects/vdk-plugins/vdk-structlog/venv/bin/python) Installed plugins: vdk-ingest-http (from package vdk-ingest-http, version 0.2.1156222304) vdk-sqlite (from package vdk-sqlite, version 0.1.1156222304) vdk-server (from package vdk-server, version 0.1.1174992336) vdk-structlog (from package vdk-structlog, version 0.1.0) vdk-control-service-properties (from package vdk-plugin-control-cli, version 0.1.948436673) vdk-execution-skip (from package vdk-plugin-control-cli, version 0.1.948436673) vdk-plugin-control-cli (from package vdk-plugin-control-cli, version 0.1.948436673) vdk-ingest-file (from package vdk-ingest-file, version 0.1.948436673) -------------------------------------------------------------------------------- (venv) 11:04:04 ~ $ export VDK_STRUCTLOG_FORMAT_INIT_LOGS=False (venv) 11:04:09 ~ $ vdk version Versatile Data Kit (VDK) Version: 0.3.1177416838 Build details: RELEASE_VERSION=0.3.1177416838, BUILD_DATE=Thu Feb 15 07:57:26 UTC 2024, BUILD_MACHINE_INFO=Linux runner-trzkizxz-project-28359933-concurrent-0rvcnn 5.4.235-144.344.amzn2.x86_64 #1 SMP Sun Mar 12 12:50:22 UTC 2023 x86_64 GNU/Linux, GITLAB_CI_JOB_ID=6177494231, GIT_COMMIT_SHA=a4c5a23b69ed2bd135b7f14658abd18c143e8759, GIT_BRANCH=main Python version: 3.11.7 64bit (/Users/mdilyan/Projects/versatile-data-kit/projects/vdk-plugins/vdk-structlog/venv/bin/python) Installed plugins: vdk-ingest-http (from package vdk-ingest-http, version 0.2.1156222304) vdk-sqlite (from package vdk-sqlite, version 0.1.1156222304) vdk-server (from package vdk-server, version 0.1.1174992336) vdk-structlog (from package vdk-structlog, version 0.1.0) vdk-control-service-properties (from package vdk-plugin-control-cli, version 0.1.948436673) vdk-execution-skip (from package vdk-plugin-control-cli, version 0.1.948436673) vdk-plugin-control-cli (from package vdk-plugin-control-cli, version 0.1.948436673) vdk-ingest-file (from package vdk-ingest-file, version 0.1.948436673) -------------------------------------------------------------------------------- ``` CI/CD ## What kind of change is this? Bugfix Signed-off-by: Dilyan Marinov <[email protected]> Co-authored-by: Dilyan Marinov <[email protected]>
1 parent 395beed commit 5feed5b

File tree

5 files changed

+274
-224
lines changed

5 files changed

+274
-224
lines changed

‎projects/vdk-plugins/vdk-structlog/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,11 +23,12 @@ pip install vdk-structlog
2323

2424
| Name | Description | Example Value | Possible Values |
2525
|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
26-
| use_structlog | Use the structlog logging config instead of using the one in vdk-core | "True" | "True", "False" |
26+
| use_structlog | Use the structlog logging config instead of using the one in vdk-core (True by default). | "True" | "True", "False" |
2727
| structlog_metadata | Configure the metadata that will be output along with the log message | "timestamp, level, logger_name, file_name, vdk_job_name | Any combination of the following: "timestamp, level, logger_name, file_name, line_number, function_name, vdk_job_name, vdk_step_name, vdk_step_type". Can be expanded by extra params and bound key-value pairs. See the bound logger examples for more information |
2828
| structlog_format | Configure the logging output format. Available formats: json, console, ltsv | "console" | "console", "json", "ltsv" |
2929
| structlog_console_log_pattern | Custom format string for console logging, applied only when`logging_format` is 'console'. Overrides `logging_metadata`. Note: For config.ini, %-signs should be escaped by doubling, e.g. %(asctime)s should become %%(asctime)s | "%(asctime)s %(name)-12s %(levelname)-8s %(message)s" | Any valid Python logging format string |
3030
| structlog_config_preset | Choose a configuration preset. Any config options set together with the preset will override the preset options. Available presets: LOCAL, CLOUD. | "CLOUD" | "console", "json", "ltsv" |
31+
| structlog_format_init_logs | Set to True to apply structlog formatting options to vdk initialization logs | "True" | "True", "False" |
3132
| log_level_module | Configure the log level of different Python modules separately | "a.b.c=INFO;foo.bar=ERROR" | Semicolon-separated list of pairs of Python module paths and log level labels |
3233
| syslog_host | Syslog host to which logs are emitted | "syslog.vmware.com" | Any valid host name |
3334
| syslog_port | Syslog port used to emit logs | 514 | Any valid port number |

‎projects/vdk-plugins/vdk-structlog/src/vdk/plugin/structlog/constants.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
STRUCTLOG_LOGGING_FORMAT_KEY = "structlog_format"
99
STRUCTLOG_CONSOLE_LOG_PATTERN = "structlog_console_custom_format"
1010
STRUCTLOG_CONFIG_PRESET = "structlog_config_preset"
11+
STRUCTLOG_FORMAT_INIT_LOGS = "structlog_format_init_logs"
1112

1213
STRUCTLOG_LOGGING_FORMAT_POSSIBLE_VALUES = ["console", "json", "ltsv"]
1314
STRUCTLOG_LOGGING_FORMAT_DEFAULT = "console"

‎projects/vdk-plugins/vdk-structlog/src/vdk/plugin/structlog/structlog_config.py‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
# Copyright 2021-2024 VMware, Inc.
2+
# SPDX-License-Identifier: Apache-2.0
3+
from vdk.internal.builtin_plugins.config import vdk_config
4+
from vdk.internal.core.config import Configuration
5+
from vdk.internal.core.config import ConfigurationBuilder
6+
from vdk.plugin.structlog.constants import DEFAULT_SYSLOG_ENABLED
7+
from vdk.plugin.structlog.constants import DEFAULT_SYSLOG_HOST
8+
from vdk.plugin.structlog.constants import DEFAULT_SYSLOG_PORT
9+
from vdk.plugin.structlog.constants import DEFAULT_SYSLOG_PROTOCOL
10+
from vdk.plugin.structlog.constants import DETAILED_LOGGING_FORMAT
11+
from vdk.plugin.structlog.constants import JSON_STRUCTLOG_LOGGING_METADATA_DEFAULT
12+
from vdk.plugin.structlog.constants import STRUCTLOG_CONFIG_PRESET
13+
from vdk.plugin.structlog.constants import STRUCTLOG_CONSOLE_LOG_PATTERN
14+
from vdk.plugin.structlog.constants import STRUCTLOG_FORMAT_INIT_LOGS
15+
from vdk.plugin.structlog.constants import STRUCTLOG_LOGGING_FORMAT_DEFAULT
16+
from vdk.plugin.structlog.constants import STRUCTLOG_LOGGING_FORMAT_KEY
17+
from vdk.plugin.structlog.constants import STRUCTLOG_LOGGING_FORMAT_POSSIBLE_VALUES
18+
from vdk.plugin.structlog.constants import STRUCTLOG_LOGGING_METADATA_ALL_KEYS
19+
from vdk.plugin.structlog.constants import STRUCTLOG_LOGGING_METADATA_KEY
20+
from vdk.plugin.structlog.constants import STRUCTLOG_USE_STRUCTLOG
21+
from vdk.plugin.structlog.constants import SYSLOG_ENABLED_KEY
22+
from vdk.plugin.structlog.constants import SYSLOG_HOST_KEY
23+
from vdk.plugin.structlog.constants import SYSLOG_PORT_KEY
24+
from vdk.plugin.structlog.constants import SYSLOG_PROTOCOL_KEY
25+
26+
27+
class StructlogConfig:
28+
def __init__(self, configuration: Configuration):
29+
presets = {
30+
"LOCAL": {
31+
STRUCTLOG_USE_STRUCTLOG: configuration.get_value(
32+
STRUCTLOG_USE_STRUCTLOG
33+
),
34+
STRUCTLOG_LOGGING_METADATA_KEY: ",".join(
35+
list(JSON_STRUCTLOG_LOGGING_METADATA_DEFAULT.keys())
36+
),
37+
STRUCTLOG_LOGGING_FORMAT_KEY: STRUCTLOG_LOGGING_FORMAT_DEFAULT,
38+
STRUCTLOG_CONSOLE_LOG_PATTERN: "",
39+
STRUCTLOG_CONFIG_PRESET: configuration.get_value(
40+
STRUCTLOG_CONFIG_PRESET
41+
),
42+
SYSLOG_HOST_KEY: DEFAULT_SYSLOG_HOST,
43+
SYSLOG_PORT_KEY: DEFAULT_SYSLOG_PORT,
44+
SYSLOG_PROTOCOL_KEY: DEFAULT_SYSLOG_PROTOCOL,
45+
SYSLOG_ENABLED_KEY: DEFAULT_SYSLOG_ENABLED,
46+
vdk_config.LOG_LEVEL_VDK.lower(): configuration.get_value(
47+
vdk_config.LOG_LEVEL_VDK.lower()
48+
),
49+
vdk_config.LOG_LEVEL_MODULE.lower(): configuration.get_value(
50+
vdk_config.LOG_LEVEL_MODULE.lower()
51+
),
52+
STRUCTLOG_FORMAT_INIT_LOGS: False,
53+
},
54+
"CLOUD": {
55+
STRUCTLOG_USE_STRUCTLOG: configuration.get_value(
56+
STRUCTLOG_USE_STRUCTLOG
57+
),
58+
STRUCTLOG_LOGGING_METADATA_KEY: "",
59+
STRUCTLOG_LOGGING_FORMAT_KEY: STRUCTLOG_LOGGING_FORMAT_DEFAULT,
60+
STRUCTLOG_CONSOLE_LOG_PATTERN: DETAILED_LOGGING_FORMAT,
61+
STRUCTLOG_CONFIG_PRESET: configuration.get_value(
62+
STRUCTLOG_CONFIG_PRESET
63+
),
64+
SYSLOG_HOST_KEY: DEFAULT_SYSLOG_HOST,
65+
SYSLOG_PORT_KEY: DEFAULT_SYSLOG_PORT,
66+
SYSLOG_PROTOCOL_KEY: DEFAULT_SYSLOG_PROTOCOL,
67+
SYSLOG_ENABLED_KEY: DEFAULT_SYSLOG_ENABLED,
68+
vdk_config.LOG_LEVEL_VDK.lower(): configuration.get_value(
69+
vdk_config.LOG_LEVEL_VDK.lower()
70+
),
71+
vdk_config.LOG_LEVEL_MODULE.lower(): configuration.get_value(
72+
vdk_config.LOG_LEVEL_MODULE.lower()
73+
),
74+
STRUCTLOG_FORMAT_INIT_LOGS: True,
75+
},
76+
}
77+
78+
self._config = presets[configuration.get_value(STRUCTLOG_CONFIG_PRESET)]
79+
80+
for key in configuration.list_config_keys():
81+
if not configuration.is_default(key):
82+
self._config[key] = configuration.get_value(key)
83+
84+
def get_use_structlog(self) -> bool:
85+
return self._config[STRUCTLOG_USE_STRUCTLOG]
86+
87+
def get_structlog_logging_metadata(self) -> str:
88+
return self._config[STRUCTLOG_LOGGING_METADATA_KEY]
89+
90+
def get_structlog_logging_format(self) -> str:
91+
return self._config[STRUCTLOG_LOGGING_FORMAT_KEY]
92+
93+
def get_structlog_console_log_pattern(self) -> str:
94+
return self._config[STRUCTLOG_CONSOLE_LOG_PATTERN]
95+
96+
def get_structlog_config_preset(self) -> str:
97+
return self._config[STRUCTLOG_CONFIG_PRESET]
98+
99+
def get_syslog_host(self) -> str:
100+
return self._config[SYSLOG_HOST_KEY]
101+
102+
def get_syslog_port(self) -> int:
103+
return self._config[SYSLOG_PORT_KEY]
104+
105+
def get_syslog_protocol(self) -> str:
106+
return self._config[SYSLOG_PROTOCOL_KEY]
107+
108+
def get_syslog_enabled(self) -> bool:
109+
return self._config[SYSLOG_ENABLED_KEY]
110+
111+
def get_log_level_vdk(self) -> str:
112+
return self._config[vdk_config.LOG_LEVEL_VDK.lower()]
113+
114+
def get_log_level_module(self) -> str:
115+
return self._config[vdk_config.LOG_LEVEL_MODULE.lower()]
116+
117+
def get_format_init_logs(self) -> str:
118+
return self._config[STRUCTLOG_FORMAT_INIT_LOGS]
119+
120+
121+
def add_definitions(config_builder: ConfigurationBuilder):
122+
config_builder.add(
123+
key=STRUCTLOG_LOGGING_METADATA_KEY,
124+
default_value=",".join(list(JSON_STRUCTLOG_LOGGING_METADATA_DEFAULT.keys())),
125+
description=(
126+
f"Possible values: {STRUCTLOG_LOGGING_METADATA_ALL_KEYS}"
127+
"User-defined key-value pairs added to the logger's context will be displayed after the metadata, "
128+
"but before the message"
129+
"Keys for user-defined key-value pairs have to be added in this config option for the values to be "
130+
"displayed in the metadata"
131+
),
132+
)
133+
134+
config_builder.add(
135+
key=STRUCTLOG_CONSOLE_LOG_PATTERN,
136+
default_value="",
137+
description="Custom format string for console logging. Leave empty for default format.",
138+
)
139+
140+
config_builder.add(
141+
key=STRUCTLOG_LOGGING_FORMAT_KEY,
142+
default_value=STRUCTLOG_LOGGING_FORMAT_DEFAULT,
143+
description=(
144+
f"Controls the logging output format. Possible values: {STRUCTLOG_LOGGING_FORMAT_POSSIBLE_VALUES}"
145+
),
146+
)
147+
148+
config_builder.add(
149+
key=SYSLOG_HOST_KEY,
150+
default_value=DEFAULT_SYSLOG_HOST,
151+
description="Hostname of the Syslog server.",
152+
)
153+
154+
config_builder.add(
155+
key=SYSLOG_PORT_KEY,
156+
default_value=DEFAULT_SYSLOG_PORT,
157+
description="Port of the Syslog server.",
158+
)
159+
160+
config_builder.add(
161+
key=SYSLOG_PROTOCOL_KEY,
162+
default_value=DEFAULT_SYSLOG_PROTOCOL,
163+
description="Syslog protocol (UDP or TCP).",
164+
)
165+
166+
config_builder.add(
167+
key=SYSLOG_ENABLED_KEY,
168+
default_value=DEFAULT_SYSLOG_ENABLED,
169+
description="Enable Syslog logging (True or False).",
170+
)
171+
172+
config_builder.add(
173+
key=STRUCTLOG_CONFIG_PRESET,
174+
default_value="LOCAL",
175+
description="Choose configuration preset. Any config options set together with the preset will override "
176+
"the preset options. Available presets: LOCAL, CLOUD",
177+
)
178+
179+
config_builder.add(
180+
key=STRUCTLOG_USE_STRUCTLOG,
181+
default_value=True,
182+
description="Use the structlog logging config instead of using the one in vdk-core",
183+
)
184+
185+
config_builder.add(
186+
key=STRUCTLOG_FORMAT_INIT_LOGS,
187+
default_value=False,
188+
description="Set to True to apply structlog formatting options to the vdk initialization logs",
189+
)

0 commit comments

Comments
 (0)