Increased memory usage after 2.0.1 bump on Android Gradle builds

[remcomokveld]

Last friday our CI suddenly started failing with Java heap space and it took a couple of days to figure out what was the cause.

After a lot of investigation it turns out that, because we were using the action as datadog/test-visibility-github-action@v2 our CI instances started pulling in 2.0.1 when it was released and apparently that adds so much memory overhead to the gradle daemon process that it suddenly didn't have enough RAM to execute the build.

Maybe this is expected because the javaagent is now added to more processes but I would not expect a monitoring tool to add that much overhead that a process which usually runs at ~9GB of RAM would start throwing OOM errors.

It's a bit of a mystery overall and hard to reproduce, but reverting back to 2.0.0 completely fixed the issue for us so it may be worth an further investigation.

Please let me know if I can help you in any way with the investigation

[nikita-tkachenko-datadog]

Hi @remcomokveld, thank you for reporting this!

Could you please share the output of the action and the output of the step that runs your tests?
If possible, both with the 2.0.0 and with the 2.0.1 of the action.

[remcomokveld]

This is a proprietary project so unfortunately I can't just share logs, but I can look for stuff in the logs. Anything specific you'd like me to look out for?

[nikita-tkachenko-datadog]

Would it be possible for you to create a support issue with Datadog and share the logs there?
If not, it'd be helpful to see the contents of GRADLE_OPTS and JAVA_TOOL_OPTIONS environment variables before the Datadog config action and before the action that runs your tests. Also, if possible, the contents of the gradle.properties file of your project, in particular org.gradle.jvmargs, if specified.

[nikita-tkachenko-datadog]

Hi @remcomokveld, thank you for sharing the logs. Does your project happen to contain a gradle.properties file? If yes, does the file contain a org.gradle.jvmargs property?

[nikita-tkachenko-datadog]

Hi @remcomokveld, my hypothesis is that you have a max heap size for the Gradle Daemon process configured with the org.gradle.jvmargs setting. There was a bug in the tracer instrumentation logic that caused org.gradle.jvmargs to be overwritten, which in your case would mean resetting max heap size back to its default value, which could be the cause of the OOM errors you were seeing.
A new version of the Test Visibility action has been released today (v2.1.1), which should fix the org.gradle.jvmargs problem. Would you be willing to try it out and let me know if that fixes your problem? Thanks!

[remcomokveld]

Yeah we are setting a max heap size of 6gb in the project gradle properties and override that to a higher value on CI machines. If those value were overridden to not setting a max and thus using Gradle's default of 512MB I'm surprised our builds got as far as they did. We did attempt to read the org.gradle.jvmargs from inside Gradle though and it was reporting the values that we were setting. Will try out the new version, thanks!

[nikita-tkachenko-datadog]

Hi @remcomokveld, have you had the chance to try out the new version yet?

[remcomokveld]

Sorry I've been on paternity leave. We were unable to verify the fix and after moving to different builders the memory overhead became even worse, also on 2.0.0. We've now implemented custom gradle tasks that use datadog-ci to upload junit xml reports

[nikita-tkachenko-datadog]

the memory overhead became even worse, also on 2.0.0

Just to double-check, does the problem also happen with v2.1.1?

If it does, to debug it further could you please try running your build with debug logging and tracer diagnostic flare enabled? To do so you would need to set the following environment variables:

• DD_TRACE_DEBUG=true to enable debug logging
• DD_TRIAGE_REPORT_TRIGGER=5s to enable tracer flare generation 5 seconds after JVM startup
• DD_TRIAGE_REPORT_DIR=... to set the folder where tracer flares are stored

To be able to download the flares you would have to add them to the set of artifacts of your GitHub Actions workflow. Below is an example of such configuration:

- name: Configure Datadog Test Visibility
  uses: datadog/test-visibility-github-action@v2
  with:
    languages: java
    service-name: my-service
    api-key: ${{ secrets.DD_API_KEY }}

- name: Run tests
  run: ./gradlew test
  env:
    DD_TRACE_DEBUG=true
    DD_TRIAGE_REPORT_TRIGGER: 5s
    DD_TRIAGE_REPORT_DIR: /tmp/dd-java-agent-triage-report/

- name: Add DD flare flare artifact
  uses: actions/upload-artifact@v4
  with:
    name: dd-agent-flare
    path: /tmp/dd-java-agent-triage-report/