Maven surefire fork error when running large test suites with OpenJ9

[kathrynkodama]

Tested with both OpenJ9 8 and OpenJ9 11. Large test suites ( > 1000 tests) with TestNg result in a Maven Surefire error:

Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 255
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?

openjdk version "1.8.0_265"
OpenJDK Runtime Environment (build 1.8.0_265-b01)
Eclipse OpenJ9 VM (build openj9-0.21.0, JRE 1.8.0 Mac OS X amd64-64-Bit Compressed References 20200728_646 (JIT enabled, AOT enabled)
OpenJ9   - 34cf4c075
OMR      - 113e54219
JCL      - c82ff0c20f based on jdk8u265-b01)

This error does occur with HotSpot.