[ASan] Disable test that sets call stack on Darwin

[boomanaiden154]

This test fails with the internal shell as we implement ulimit with a python wrapper. This python wrapper fails on recent versions of MacOS to set the stack size limit lower, or even to the current values. More discussion is in python/cpython#78783 towards the bottom (it seems like a new issue should be opened).

It does not seem like we lose significant test coverage by disabling this on MacOS as the test running on Linux should catch any major regressions. We can also simply reenable once the issue is fixed (although writing a simple c program with calls setrlimit directly also fails in the smae manner).

[llvmbot]

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: Aiden Grossman (boomanaiden154)

Changes

This test fails with the internal shell as we implement ulimit with a python wrapper. This python wrapper fails on recent versions of MacOS to set the stack size limit lower, or even to the current values. More discussion is in python/cpython#78783 towards the bottom (it seems like a new issue should be opened).

It does not seem like we lose significant test coverage by disabling this on MacOS as the test running on Linux should catch any major regressions. We can also simply reenable once the issue is fixed (although writing a simple c program with calls setrlimit directly also fails in the smae manner).

---

Full diff: https://github.com/llvm/llvm-project/pull/170786.diff

1 Files Affected:

• (modified) compiler-rt/test/asan/TestCases/Posix/deep_call_stack.cpp (+4)

diff --git a/compiler-rt/test/asan/TestCases/Posix/deep_call_stack.cpp b/compiler-rt/test/asan/TestCases/Posix/deep_call_stack.cpp
index 1342eae927794..c4e519fa61830 100644
--- a/compiler-rt/test/asan/TestCases/Posix/deep_call_stack.cpp
+++ b/compiler-rt/test/asan/TestCases/Posix/deep_call_stack.cpp
@@ -1,5 +1,9 @@
 // Check that UAR mode can handle very deep recursion.
 // REQUIRES: shell
+// TODO(boomanaiden154): This test currently fails with the internal
+// shell because python is not able to set RLIMIT_STACK. We should
+// reenable this when the behavior is fixed.
+// UNSUPPORTED: system-darwin
 // RUN: %clangxx_asan -O2 %s -o %t
 // RUN: ulimit -s 4096
 // RUN: %env_asan_opts=detect_stack_use_after_return=1 %run %t 2>&1 | FileCheck %s

[boomanaiden154]

I run into the same issue when calling setrlimit explicitly, so it would be hard to get around this with any sort of wrapper redesign. I'm honestly not sure how the default shell on MacOS is able to get around this.

aidengrossman-mac2:runtimes-bins aidengrossman$ cat /tmp/test.cpp
#include <stdio.h>
#include <errno.h>
#include <sys/resource.h>

int main(void)
{
    struct rlimit limits = {
        8372224,
        60792480
    };

    int r = setrlimit(RLIMIT_AS, &limits);
    if (r == -1) {
        perror("setrlimit");
        return 1;
    }
    return 0;
}
aidengrossman-mac2:runtimes-bins aidengrossman$ clang++ /tmp/test.cpp -o /tmp/test
aidengrossman-mac2:runtimes-bins aidengrossman$ /tmp/test
setrlimit: Invalid argument
aidengrossman-mac2:runtimes-bins aidengrossman$ python3
Python 3.9.6 (default, Apr 30 2025, 02:07:17)
[Clang 17.0.0 (clang-1700.0.13.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import resource
>>> print(resource.getrlimit(resource.RLIMIT_STACK))
(8372224, 67092480)
>>> exit()

[ndrewh]

Thanks for all the work on the internal shell! I'm planning to look at fixing the remaining sanitizer internal shell failures on Darwin later today (but if you're already working on some of them lmk).

I think disabling this test for now is reasonable.

[DanBlackwell]

@boomanaiden154 fwiw I also have the test failure using the lit internal shell but this program works and shows the limits correctly:

#include <stdio.h>
#include <errno.h>
#include <sys/resource.h>
#include <unistd.h>
#include <stdlib.h>
#include <unistd.h>

int main() {
  struct rlimit limits = { 4096 * 1024, 4096 * 1024 };

  int r = setrlimit(RLIMIT_STACK, &limits);
  if (r == -1) {
    perror("setrlimit");
    return 1;
  }

  int link[2];
  pid_t pid;
  char foo[4096];

  pipe(link);
  if (!fork()) {
    dup2 (link[1], STDOUT_FILENO);
    close(link[0]);
    close(link[1]);
    execl("/opt/homebrew/bin/python3", "python3", "-c", "import resource; print(resource.getrlimit(resource.RLIMIT_STACK))", NULL);
  } else {
    close(link[1]);
    int nbytes = read(link[0], foo, sizeof(foo));
    printf("Output: (%.*s)\n", nbytes, foo);
    wait(NULL);
  }
  return 0;
}

I am slightly surprised that setrlimit is getting EINVAL if it really is the same function call - which this suggests it should be:

https://github.com/python/cpython/blob/1173f8068b9a8c1168780532c466c391be10bf67/Modules/resource.c#L249-L252

[boomanaiden154]

Thanks for all the work on the internal shell! I'm planning to look at fixing the remaining sanitizer internal shell failures on Darwin later today (but if you're already working on some of them lmk).

This should be the last one looking at the previous failure logs.

[boomanaiden154]

fwiw I also have the test failure using the lit internal shell but this program works and shows the limits correctly:

Yeah, looks like in my case the example I copied and pasted from the cpython thread used RLIMIT_AS instead of RLIMIT_STACK. Switching to RLIMIT_STACK there fixes the problem. Either way, this does seem like a bug in cpython. I would expect that running

python3 -c "import resource; resource.setrlimit(resource.RLIMIT_STACK, resource.getrlimit(resource.RLIMIT_STACK))"

would still fail for you. More investigation is needed, and I'll file a bug with upstream python assuming it reproduces with Python 3.14. I will land this patch though to unblock enabling the internal shell by default.

[boomanaiden154]

I filed python/cpython#142317 to at least track this.

[ndrewh]

hi @boomanaiden154 I was wondering about the test you disabled in 75aa7bd, it already has

REQUIRES: shell

Are you intentionally setting leaving the shell feature included somewhere? I think it may be included because of the logic here but I am not sure.

[boomanaiden154]

Are you intentionally setting leaving the shell feature included somewhere? I think it may be included because of the logic here but I am not sure.

The shell feature should get added everywhere except Windows. We really should replace all of the REQUIRE: shell lines with UNSUPPORTED: Windows lines, and I've been doing that as I have time. My hope is to completely get rid of REQUIRES: shell now that everything users the internal shell.