Skip to content

Intermittent AddressSanitizer errors on Mac OS when optimizations are turned on #924

@robertbartel

Description

@robertbartel

As discussed extensively in #913, AddressSanitizer began throwing some errors that could only be produced in GitHub Actions Mac OS runners. That issue really needs to be read fully and carefully to understand the complete context here, but there are two key points:

  • it appeared, under reasonably careful analysis, that at least one of the errors could not be happening as reported
  • it was not possible to reproduce the errors either in Linux runner or development environments, or (at least at first) even any Mac development environments outside the runners

Eventually, we discovered that the compiler optimization setting was crucial. With -O0 set (the default for the ngen build config), the errors seem to never occur, but the Actions checks were explicitly configured with -O1. Once this was also applied on a personal Mac development environment, we could reproduce the AddressSanitizer errors. Not every time, though: the errors were intermittent, occurring only about ~10% of the time.

This is peculiar and likely is going to require deep analysis. For expedience in getting working CI/CD checks back, #921 was created to address the immediate problem by turning off optimizations on just the Mac Action runners. This should be fine (I've found nothing indicating using -O0 with ASan is problematic beyond things running more slowly), but it is not a permanent solution. We need to get to the bottom of what is going on and ensure errors are consistent across platforms and optimization levels (or, if against all odds there is actually an ASan bug here, at least reference it).

Metadata

Metadata

Assignees

No one assigned

    Labels

    CI/CDQA/QCbugSomething isn't workingbuildIssues related to CMake and building ngen

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions