Add Windows Test Runners for All Core Test Suites #1439

TimPansino · 2025-07-31T18:26:43Z

Overview

Add Windows test runners to Github Actions.
- This requires manually installs of Python using the setup-python action, which is a bit of a regression from our nice CI image. Hopefully this isn't a big deal for just the core suites, if it is we may have to explore a Windows docker image for the CI.
Add Windows test environments for all core test suites in tox.
Initial Windows support limited to Python 3.13

github-actions · 2025-07-31T18:28:27Z

🦙 MegaLinter status: ❌ ERROR

Descriptor	Linter	Files	Fixed	Errors	Elapsed time
❌ ACTION	actionlint	6		1	0.8s
✅ MARKDOWN	markdownlint	7	0	0	1.24s
✅ MARKDOWN	markdown-link-check	7		0	18.01s
✅ PYTHON	ruff	922	0	0	0.91s
✅ PYTHON	ruff-format	922	0	0	0.35s
✅ YAML	prettier	13	1	0	1.36s
✅ YAML	v8r	13		0	6.01s
✅ YAML	yamllint	13		0	0.67s

See detailed report in MegaLinter reports

codecov-commenter · 2025-07-31T18:41:11Z

Codecov Report

❌ Patch coverage is 82.60870% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.28%. Comparing base (02d0071) to head (357154a).

Files with missing lines	Patch %	Lines
newrelic/core/agent_control_health.py	82.22%	6 Missing and 2 partials ⚠️

Additional details and impacted files

@@                 Coverage Diff                 @@
##           develop-windows    #1439      +/-   ##
===================================================
- Coverage            81.37%   81.28%   -0.10%     
===================================================
  Files                  208      208              
  Lines                23566    23586      +20     
  Branches              3717     3723       +6     
===================================================
- Hits                 19177    19172       -5     
- Misses                3132     3170      +38     
+ Partials              1257     1244      -13

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

hmstepanek

Posting this now so I can get this feedback to you-I'd like to take a closer look at the healthcheck section changes when I have some more time.

hmstepanek · 2025-08-15T19:53:17Z

tox.ini

@@ -44,6 +44,19 @@ setupdir = {toxinidir}
 ; Fail tests when interpreters are missing.
 skip_missing_interpreters = false
 envlist =
+    # Core Features Tests run on Linux, MacOS, and Windows
+    python-agent_features-{py37,py38,py39,py310,py311,py312,py313}-{with,without}_extensions,


Didn't we want to only run on 3.13?

Confusing change, but the python- prefix now denotes default python runners with no dependencies, which is linux. The envs with windows- as a prefix are for running on windows runners specifically. The line you're looking at is a linux only runner set, there's a windows env for agent features on just Python 3.13 down below.

tests/agent_unittests/test_encoding_utils.py

hmstepanek · 2025-08-15T20:10:41Z

tests/agent_unittests/test_agent_protocol.py

@@ -522,7 +522,8 @@ def test_audit_logging():
    protocol = AgentProtocol(settings, client_cls=HttpClientRecorder)
    protocol.send("preconnect")

-    with Path(f.name).open() as f:
+    audit_log_path = Path(f.name)
+    with audit_log_path.open() as f:


Should this have an encoding on it?

No, the audit log file is being written to in the actual agent code in the preferred encoding as returned by locale.getencoding(). We read it back in using the same default encoding.

The other tests that needed encoding specified are pre-written files from Mac/Linux machines that encode unicode characters in UTF-8, which is not the default for Windows and produces errors.

hmstepanek · 2025-08-15T20:18:42Z

tests/agent_features/test_async_generator_trace.py

@@ -13,7 +13,6 @@
 # limitations under the License.

 import functools
-import sys


I don't understand how we are still finding these. Must have been missed in the original linted PR.

I turned off the rule for unused imports in the tests folder because of fixtures commonly causing that to trip and get removed unless we mark them with # noqa markers. Unfortunately, that means we don't catch unused imports in tests files but they're not run in production so I wasn't too concerned with it.

hmstepanek · 2025-08-15T20:23:31Z

tests/agent_features/test_agent_control_health_check.py

+
+@pytest.mark.skipif(sys.platform != "win32", reason="Only valid for Windows")
+@pytest.mark.parametrize("leading_slash", [True, False], ids=["leading_slash", "no_leading_slash"])
+def test_inconsistent_paths_on_windows(monkeypatch, tmp_path, leading_slash):


I feel like you went over this in mobbing but I can't remember now. Is this just because windows allows both /// and // so it must handle both?

There are inconsistencies with handling the root path on windows that caused a ton of headaches due to mixing which library was creating vs parsing the file URIs.

A path for the file C:\\Users\\Reli\\myapp\\main.py will end up represented as a URI with the file:// but both of these are valid for after the prefix:

c:/user/reli/myapp/main.py and /c:/user/reli/myapp/main.py.

It's hard to spot when it's file://c:/user/reli/myapp/main.py vs file:///c:/user/reli/myapp/main.py but the extra slash causes SOME parsers to fail due to interpreting the leading slash as "relative to the root", but there's no folder named c: under the root, since that's a drive letter. It's effectively the same as looking for C:\\C:. So the Path object produced from parsing that doesn't exist.

That being said, it SHOULD be valid on Windows as a URI but not every parser interprets it correctly. The parser with correct behavior is once again pathlib, but the URI parser wasn't added until Python 3.13. Since we're now not aiming to support lower versions on Windows yet, I could comment out that whole section and call Path.from_uri() on 3.13+ to sidestep this whole thing.

hmstepanek · 2025-08-15T20:29:03Z

newrelic/core/agent_control_health.py

            os.environ.get("NEW_RELIC_AGENT_CONTROL_HEALTH_DELIVERY_LOCATION", "") or "file:///newrelic/apm/health"
        )

-        return health_file_location
+        # Return from cache if already parsed


I'm gonna have to look at this part a little more in depth later.

The health stuff is a significant enough change that I could pull it out into a separate PR and mark these tests as XFAIL for now if you want.

TimPansino force-pushed the feature-windows-ci-tests branch from 1630b42 to 32eb0f7 Compare July 31, 2025 18:29

mergify bot added the tests-failing Tests failing in CI. label Jul 31, 2025

TimPansino force-pushed the feature-windows-ci-tests branch from c9a0c97 to e1771d6 Compare July 31, 2025 20:13

TimPansino force-pushed the feature-windows-ci-tests branch 9 times, most recently from 83b2d1f to f660276 Compare August 13, 2025 22:06

TimPansino added 16 commits August 13, 2025 15:06

Don't build some extensions on windows

ecd4d75

Add windows test runners for all core test suites

72e9ecb

Update get-envs script to replace grep

c2143de

Fix missing lambda handler deprecation

e5b29a4

Fix paths in logs in context tests

5689056

agent_features: Skip all failing windows tests for now

24cd182

agent_unittests: Skip all failing windows tests for now

abd75b6

cross_agent: Skip all failing windows tests for now

75217c1

Convert username lookup to more portable option

3a97d9f

Fix handling of health dir paths on Windows

58e1d8e

Fix issues related to tempfile on Windows

35d9519

Fix encoding in test fixtures

5499fd8

Fix globbing on Windows

ff449b1

Format

fcc715e

Fix Python 3.7 compat

d402d74

Drop windows py37 tests

ff6be99

TimPansino force-pushed the feature-windows-ci-tests branch from f660276 to ff6be99 Compare August 13, 2025 22:07

Update build_ext_errors in setup.py for modern versions

799810b

TimPansino marked this pull request as ready for review August 13, 2025 22:14

TimPansino requested a review from a team as a code owner August 13, 2025 22:14

TimPansino added 2 commits August 14, 2025 11:07

Reduce windows testing scope to 3.13

077f8e3

Change out windows xfails

0052853

mergify bot removed the tests-failing Tests failing in CI. label Aug 14, 2025

Enable colors in CI logs on Windows

d041027

TimPansino force-pushed the feature-windows-ci-tests branch 3 times, most recently from c33a036 to 29e450e Compare August 14, 2025 19:43

mergify bot added the tests-failing Tests failing in CI. label Aug 14, 2025

TimPansino added 2 commits August 14, 2025 12:53

Update all python-version specs in CI

fece9a8

Update coverage source paths

3e3a2c3

TimPansino force-pushed the feature-windows-ci-tests branch from 29e450e to 3e3a2c3 Compare August 14, 2025 19:54

mergify bot removed the tests-failing Tests failing in CI. label Aug 14, 2025

hmstepanek reviewed Aug 15, 2025

View reviewed changes

Add inline pytest ids to unittests

33d598a

mergify bot added the tests-failing Tests failing in CI. label Aug 15, 2025

TimPansino force-pushed the feature-windows-ci-tests branch from 281960c to 346bdba Compare August 16, 2025 00:26

Add tests for all OS and arch combinations

357154a

TimPansino force-pushed the feature-windows-ci-tests branch from 346bdba to 357154a Compare August 16, 2025 00:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Windows Test Runners for All Core Test Suites #1439

Add Windows Test Runners for All Core Test Suites #1439

Uh oh!

TimPansino commented Jul 31, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 31, 2025 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 31, 2025 •

edited

Loading

Uh oh!

hmstepanek left a comment

Uh oh!

hmstepanek Aug 15, 2025

Uh oh!

TimPansino Aug 15, 2025

Uh oh!

Uh oh!

hmstepanek Aug 15, 2025

Uh oh!

TimPansino Aug 15, 2025

Uh oh!

hmstepanek Aug 15, 2025

Uh oh!

TimPansino Aug 15, 2025

Uh oh!

hmstepanek Aug 15, 2025

Uh oh!

TimPansino Aug 15, 2025 •

edited

Loading

Uh oh!

hmstepanek Aug 15, 2025

Uh oh!

TimPansino Aug 15, 2025

Uh oh!

Uh oh!

Add Windows Test Runners for All Core Test Suites #1439

Are you sure you want to change the base?

Add Windows Test Runners for All Core Test Suites #1439

Uh oh!

Conversation

TimPansino commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Uh oh!

github-actions bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦙 MegaLinter status: ❌ ERROR

Uh oh!

codecov-commenter commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hmstepanek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TimPansino Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

TimPansino commented Jul 31, 2025 •

edited

Loading

github-actions bot commented Jul 31, 2025 •

edited

Loading

codecov-commenter commented Jul 31, 2025 •

edited

Loading

TimPansino Aug 15, 2025 •

edited

Loading