feat(integration-tests): system-wide overhaul of logging and error reporting. #1802

quinntaylormitchell · 2025-12-18T20:22:55Z

Description

Checklist

The PR satisfies the contribution guidelines.
This is a breaking change and that has been indicated in the PR title, OR this isn't a
breaking change.
Necessary docs have been updated, OR no docs need to be updated.

Validation performed

To see the specific tasks where the Asana app for GitHub is being used, see below:
- https://app.asana.com/0/0/1212861692182928

coderabbitai · 2025-12-18T20:23:00Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

junhaoliao

with the understanding that this is a draft, i just took a quick look

junhaoliao · 2026-01-05T07:25:52Z

.gitignore

 .task/
 build/
 dist/
+__*


what are we trying to exclude here?

The __pytest_logs directory that stores the log file for each test run.

That being said, right now the __pytest_logs directory gets created/stored under the clp/integration-tests directory. Do you think it might be better to have it under the build/integration-tests directory instead (used for other integration test stuff as well)? Then we wouldn't have to modify this file.

junhaoliao · 2026-01-05T07:26:27Z

integration-tests/.pytest.ini

+    --show-capture=no
+    --tb=short
    --strict-config
    --strict-markers


nit

Suggested change

--show-capture=no

--tb=short

--strict-config

--strict-markers

--show-capture=no

--strict-config

--strict-markers

--tb=short

junhaoliao · 2026-01-05T07:42:46Z

integration-tests/.pytest.ini

-log_cli_date_format = %Y-%m-%d %H:%M:%S,%f
-log_cli_format = %(name)s %(asctime)s [%(levelname)s] %(message)s
 log_cli_level = INFO
+log_cli_format = %(levelname)-8s %(asctime)-25s %(message)s


any reason to change this? i thought we were trying to match references like

clp/components/clp-py-utils/clp_py_utils/initialize-spider-db.py

Line 24 in 5de8bc9

logging_formatter = logging.Formatter("%(asctime)s [%(levelname)s] %(message)s")

My only reason was aesthetics, cause this is the format of the logs that will be reported to the user via the command line. If there is some reason we want to match references like you describe above, I can change it; I just wasn't aware we were doing that

junhaoliao · 2026-01-05T07:43:24Z

integration-tests/.pytest.ini

+
+log_file_mode = a
+log_file_level = DEBUG
+log_file_format = %(levelname)-8s %(asctime)-30s [%(name)s:%(filename)s:%(lineno)d]: %(message)s


any reason to use a different format string between the cli and the log file?

Just to provide more information to the dev; I don't want to crowd the cli output, especially for a successful (no failure) test run. I think it's safe to assume that the main reason a dev would be looking at the test log would be if something fails, and in the event of failure, they'll need more information.

junhaoliao · 2026-01-05T07:52:53Z

integration-tests/.pytest.ini

+log_cli_format = %(levelname)-8s %(asctime)-25s %(message)s
+log_cli_date_format = %Y-%m-%d %H:%M:%S
+
+log_file_mode = a


instead of appending to the same file, can isolate the logs from different test runs instead? e.g., use set_log_path with the default log_file_mode = w. see the test code at pytest-dev/pytest#4752 for example

If this isn't here, the test output file will contain all of the pytest logging messages followed by all of the output from the various subprocesses called during the test run, like this:

INFO 2025-01-06 event1 INFO 2025-01-06 event2 INFO 2025-01-06 event3 <output following event 1> <output following event 2> <output following event 3>

It seems better to me to have everything written to the log file in order, so that it looks like this:

INFO 2025-01-06 event1 <output following event 1> INFO 2025-01-06 event2 <output following event 2> INFO 2025-01-06 event3 <output following event 3>

The logs from different test runs are already isolated by the code in conftest.py.

junhaoliao · 2026-01-05T07:57:31Z

integration-tests/tests/utils/asserting_utils.py


-def run_and_assert(cmd: list[str], **kwargs: Any) -> subprocess.CompletedProcess[Any]:
+def run_and_assert(
+    request: pytest.FixtureRequest, cmd: list[str], **kwargs: Any


update below docstring for the param

do you mean the request? if so, right on, thank you

junhaoliao · 2026-01-05T08:00:14Z

integration-tests/tests/utils/package_utils.py



-def stop_clp_package(package_config: PackageConfig) -> None:
+def stop_clp_package(request: pytest.FixtureRequest, package_config: PackageConfig) -> None:


param docstring for request

right on, thanks

junhaoliao · 2026-01-05T08:02:30Z

integration-tests/tests/utils/package_utils.py

+
+    try:
+        run_and_assert(request, stop_cmd, timeout=DEFAULT_CMD_TIMEOUT_SECONDS)
+    except SubprocessError:


in what case would this be thrown here? i thought we would have caught this in run_and_assert? like also if there are other SubprocessError than CalledProcessError that might be thrown but uncaught in run_and_assert, why don't we modify run_and_assert to handle such in the first place?

Yeah that's a good point. See my message that I sent you offline.

junhaoliao · 2026-01-05T08:02:59Z

integration-tests/tests/utils/package_utils.py

+
+    try:
+        run_and_assert(request, start_cmd, timeout=DEFAULT_CMD_TIMEOUT_SECONDS)
+    except SubprocessError:


ditto -

in what case would this be thrown here? i thought we would have caught this in run_and_assert? like also if there are other SubprocessError than CalledProcessError that might be thrown but uncaught in run_and_assert, why don't we modify run_and_assert to handle such in the first place?

ditto my other comment -

junhaoliao · 2026-01-05T08:03:43Z

integration-tests/tests/utils/package_utils.py



-def start_clp_package(package_config: PackageConfig) -> None:
+def start_clp_package(request: pytest.FixtureRequest, package_config: PackageConfig) -> None:


param docstring for request

right on, thanks

Preliminary logging and error reporting overhaul.

1269402

junhaoliao self-requested a review December 18, 2025 20:25

jackluo923 and others added 5 commits December 19, 2025 12:19

Merge branch 'main' into integration-test-logging-overhaul

9dbf299

Merge branch 'main' into integration-test-logging-overhaul

e985fdf

Merge branch 'main' into integration-test-logging-overhaul

7b4a68b

Merge branch 'main' into feature branch

6fe0796

Clean up implementation.

0562f48

junhaoliao requested changes Jan 5, 2026

View reviewed changes

quinntaylormitchell added 2 commits January 5, 2026 21:58

Merge branch 'main' into feature branch

2080fd5

Address Junhao's comments.

6da4264

quinntaylormitchell requested a review from junhaoliao January 6, 2026 16:05

Remove erroneous line.

f5ec1b5

junhaoliao self-assigned this Jan 19, 2026

junhaoliao modified the milestones: Backlog, February 2026 Jan 19, 2026



		def stop_clp_package(package_config: PackageConfig) -> None:
		def stop_clp_package(request: pytest.FixtureRequest, package_config: PackageConfig) -> None:



		def start_clp_package(package_config: PackageConfig) -> None:
		def start_clp_package(request: pytest.FixtureRequest, package_config: PackageConfig) -> None:

feat(integration-tests): system-wide overhaul of logging and error reporting. #1802

Are you sure you want to change the base?

feat(integration-tests): system-wide overhaul of logging and error reporting. #1802

Uh oh!

Conversation

quinntaylormitchell commented Dec 18, 2025 • edited by junhaoliao Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Validation performed

Uh oh!

coderabbitai bot commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

junhaoliao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

quinntaylormitchell commented Dec 18, 2025 •

edited by junhaoliao

Loading

coderabbitai bot commented Dec 18, 2025 •

edited

Loading