Currently, we run unit tests on Ubuntu only in CI.
We do now have some unit tests that seem to only work on Windows due to some upstream choices (#9886)
We should consider running them on windows as well. This might uncover some unit test failures we haven't seen before and might have to deal with.