CPU and memory performance issue with PHP agent enabled

[ts-mothership]

Description

We're using NewRelics PHP agent on our server. When executing a script with disabled metric collection via newrelic_ignore_transaction() the script runs for ~4min with a constant memory usage of 150-200 MB. However with NewRelic collecting the metrics, the script OOMs after taking all 30 GB memory available on the server.

We checked relevant documentation, tried to improve configuration values. But memory consumption did not improve even with changed php.ini values:

• https://docs.newrelic.com/docs/apm/agents/php-agent/troubleshooting/segmentation-faults/
• https://docs.newrelic.com/docs/apm/agents/php-agent/troubleshooting/performance-issues-long-running-task/
• https://docs.newrelic.com/docs/apm/agents/php-agent/troubleshooting/agent-overhead-reduction-tips/

An strace log is attached, and we see that the PHP agent tries to allocate memory in a loop. Here our knowledge ends. Please let us know if you need anything else.

Steps to Reproduce

1. Calling a shell script in terminal for our application
   vendor/bin/phpunit
2. Watching memory consumption with htop

Expected Behavior

Constant memory consumption. It may increase a little but not more than 1 GB.

Relevant Logs / Console output

strace-1.log.gz
strace-2-detailed.log.gz

Your Environment

• OS Ubuntu 22.04.5 LTS
• PHP 8.2.28
• New Relic Version => 11.8.0.22
• It's a Shopware 6.6 application (based on Symfony)

php.ini config:

newrelic.daemon.address => @newrelic
newrelic.daemon.app_connect_timeout => no value
newrelic.daemon.app_timeout => no value
newrelic.daemon.auditlog => no value
newrelic.daemon.collector_host => no value
newrelic.daemon.dont_launch => no value
newrelic.daemon.location => /usr/bin/newrelic-daemon
newrelic.daemon.logfile => /var/log/newrelic/newrelic-daemon.log
newrelic.daemon.loglevel => no value
newrelic.daemon.pidfile => no value
newrelic.daemon.port => @newrelic
newrelic.daemon.proxy => no value
newrelic.daemon.ssl_ca_bundle => no value
newrelic.daemon.ssl_ca_path => no value
newrelic.daemon.start_timeout => no value
newrelic.daemon.utilization.detect_aws => enabled
newrelic.daemon.utilization.detect_azure => enabled
newrelic.daemon.utilization.detect_docker => enabled
newrelic.daemon.utilization.detect_gcp => enabled
newrelic.daemon.utilization.detect_kubernetes => enabled
newrelic.daemon.utilization.detect_pcf => enabled
newrelic.feature_flag => no value
newrelic.high_security => 0
newrelic.logfile => /var/log/newrelic/php_agent.log
newrelic.loglevel => info
newrelic.preload_framework_library_detection => 1
newrelic.transaction_tracer.internal_functions_enabled => disabled
newrelic.allow_raw_exception_messages => 1 => 1
newrelic.analytics_events.capture_attributes => enabled => enabled
newrelic.analytics_events.enabled => enabled => enabled
newrelic.application_logging.enabled => enabled => enabled
newrelic.application_logging.forwarding.context_data.enabled => disabled => disabled
newrelic.application_logging.forwarding.context_data.exclude => no value => no value
newrelic.application_logging.forwarding.context_data.include => no value => no value
newrelic.application_logging.forwarding.enabled => enabled => enabled
newrelic.application_logging.forwarding.labels.enabled => disabled => disabled
newrelic.application_logging.forwarding.labels.exclude => no value => no value
newrelic.application_logging.forwarding.log_level => WARNING => WARNING
newrelic.application_logging.forwarding.max_samples_stored => 10000 => 10000
newrelic.application_logging.local_decorating.enabled => disabled => disabled
newrelic.application_logging.metrics.enabled => enabled => enabled
newrelic.appname => <removed>
newrelic.attributes.enabled => enabled => enabled
newrelic.attributes.exclude => no value => no value
newrelic.attributes.include => no value => no value
newrelic.browser_monitoring.attributes.enabled => disabled => disabled
newrelic.browser_monitoring.attributes.exclude => no value => no value
newrelic.browser_monitoring.attributes.include => no value => no value
newrelic.browser_monitoring.auto_instrument => enabled => enabled
newrelic.browser_monitoring.capture_attributes => disabled => disabled
newrelic.browser_monitoring.debug => disabled => disabled
newrelic.browser_monitoring.loader => rum => rum
newrelic.capture_params => off => off
newrelic.cloud.aws.account_id => no value => no value
newrelic.code_level_metrics.enabled => enabled => enabled
newrelic.cross_application_tracer.enabled => disabled => disabled
newrelic.custom_events.max_samples_stored => 30000 => 30000
newrelic.custom_insights_events.enabled => enabled => enabled
newrelic.custom_parameters_enabled => 1 => 1
newrelic.datastore_tracer.database_name_reporting.enabled => enabled => enabled
newrelic.datastore_tracer.instance_reporting.enabled => enabled => enabled
newrelic.distributed_tracing.pad_trace_id => disabled => disabled
newrelic.distributed_tracing_enabled => 0 => 0
newrelic.distributed_tracing_exclude_newrelic_header => 0 => 0
newrelic.enabled => yes => yes
newrelic.error_collector.attributes.enabled => enabled => enabled
newrelic.error_collector.attributes.exclude => no value => no value
newrelic.error_collector.attributes.include => no value => no value
newrelic.error_collector.capture_attributes => enabled => enabled
newrelic.error_collector.capture_events => enabled => enabled
newrelic.error_collector.enabled => enabled => enabled
newrelic.error_collector.ignore_errors => no value => no value
newrelic.error_collector.ignore_exceptions => no value => no value
newrelic.error_collector.ignore_user_exception_handler => no => no
newrelic.error_collector.prioritize_api_errors => no => no
newrelic.error_collector.record_database_errors => yes => yes
newrelic.framework => auto-detect => auto-detect
newrelic.framework.drupal.modules => on => on
newrelic.framework.wordpress.hooks => on => on
newrelic.framework.wordpress.hooks.options => plugin_callbacks => plugin_callbacks
newrelic.framework.wordpress.hooks.threshold => 1ms => 1ms
newrelic.framework.wordpress.hooks_skip_filename => no value => no value
newrelic.guzzle.enabled => 1 => 1
newrelic.ignored_params => no value => no value
newrelic.infinite_tracing.span_events.agent_queue.size => 1000 => 1000
newrelic.infinite_tracing.span_events.agent_queue.timeout => 1s => 1s
newrelic.infinite_tracing.span_events.queue_size => 100000 => 100000
newrelic.infinite_tracing.trace_observer.host => no value => no value
newrelic.infinite_tracing.trace_observer.port => 443 => 443
newrelic.labels => no value => no value
newrelic.license => <removed>
newrelic.message_tracer.segment_parameters.enabled => enabled => enabled
newrelic.phpunit_events.enabled => disabled => disabled
newrelic.process_host.display_name => no value => no value
newrelic.security_policies_token => no value => no value
newrelic.span_events.attributes.enabled => enabled => enabled
newrelic.span_events.attributes.exclude => no value => no value
newrelic.span_events.attributes.include => no value => no value
newrelic.span_events.max_samples_stored => 2000 => 2000
newrelic.span_events_enabled => 1 => 1
newrelic.special.max_nesting_level => -1 => -1
newrelic.synthetics.enabled => enabled => enabled
newrelic.transaction_events.attributes.enabled => enabled => enabled
newrelic.transaction_events.attributes.exclude => no value => no value
newrelic.transaction_events.attributes.include => no value => no value
newrelic.transaction_events.enabled => enabled => enabled
newrelic.transaction_tracer.attributes.enabled => enabled => enabled
newrelic.transaction_tracer.attributes.exclude => no value => no value
newrelic.transaction_tracer.attributes.include => no value => no value
newrelic.transaction_tracer.capture_attributes => enabled => enabled
newrelic.transaction_tracer.custom => no value => no value
newrelic.transaction_tracer.detail => 1 => 1
newrelic.transaction_tracer.enabled => enabled => enabled
newrelic.transaction_tracer.explain_enabled => enabled => enabled
newrelic.transaction_tracer.explain_threshold => 500 => 500
newrelic.transaction_tracer.gather_input_queries => 1 => 1
newrelic.transaction_tracer.max_segments_cli => 100000 => 100000
newrelic.transaction_tracer.max_segments_web => 0 => 0
newrelic.transaction_tracer.record_sql => obfuscated => obfuscated
newrelic.transaction_tracer.slow_sql => enabled => enabled
newrelic.transaction_tracer.stack_trace_threshold => 500 => 500
newrelic.transaction_tracer.threshold => apdex_f => apdex_f
newrelic.vulnerability_management.composer_api.enabled => disabled => disabled
newrelic.vulnerability_management.package_detection.enabled => enabled => enabled
newrelic.webtransaction.name.files => no value => no value
newrelic.webtransaction.name.functions => no value => no value
newrelic.webtransaction.name.remove_trailing_path => no => no

Additional context

No other newrelic_ function is executed in this script. Only for testing we included the newrelic_ignore_transaction to see memory usage there.

[newrelic-php-agent-bot]

Thank you for your issue report. A member of the New Relic PHP Agent Team will review your report and endeavor to respond in a timely manner.
Need troubleshooting help? Please review some of our common troubleshooting documentation, starting with generating debug logs. From there, additional resources and tips are available in our troubleshooting section, accessible from the left-hand panel of our documentation site. Additionally, please be sure to visit our Explorer's Hub, where other members of the community may have solved an issue similar to yours with the help of our Support team members.
Have a question regarding support for an Operating System, PHP Version, Framework, or Library? Please take a look at our Compatibility Docs.
Binary support for ARM-based CPU architecture is now available via the tarball installation method for PHP Versions 8.0+!
General installation instructions can be found on our Installation Overview page.
Need more information about an Agent INI setting? Our Configuration page contains an explanation for each INI setting and how to modify the defaults.
Looking for something else? All New Relic PHP Agent Documentation can be found under our Introduction page.

[newrelic-php-agent-bot]

This issue has been identified as requiring additional information to debug. Due to the potentially sensitive nature of the information we may have to collect, such as PHPInfo and Debug Logs, please follow these steps to create a support case, where a member of our New Relic Support Team will work with you to gather the necessary information securely and help debug your issue.
If your subscription level does not include technical support, the other option available to customers is to use New Relic’s community support channel: Explorers Hub. From there, a member of our New Relic Support Team will work with you to gather the necessary information securely and help debug your issue. For all available options on how to get support, please see these resources.

[bduranleau-nr]

Hi @ts-mothership,
I've marked this issue as needing additional, typically sensitive information to debug. If you follow the instructions in the above bot message, you can get connected with our Tech Support engineers who have a secure way to gather this information and track everything in an internal ticket. I'll reach out to them to let them know to expect to hear from you.

[ts-mothership]

We got feedback from NewRelic which explains this behavior. For anyone who has memory issues for PHPUnit with NewRelic enabled, this answer might help:

newrelic.phpunit_events.enabled indeed disables instrumentation of PHPUnit (Test and TestSuite events are not generated - see https://docs.newrelic.com/docs/apm/agents/php-agent/frameworks-libraries/analyze-phpunit-test-data-new-relic/ for details on what functionality is controlled by newrelic.phpunit_events.enabled). However it’s not the instrumentation of PHPUnit that’s causing the increased memory problem - it is the incompatibility between PHPUnit and New Relic PHP Agent. The reason for incompatibility between PHPUnit version 11 on PHPs 8.3+ is that in PHPUnit 11.x new code was added in Reapply "Check and restore error/exception global handlers" · sebastianbergmann/phpunit@0214cf8, which attempts to retrieve the list of user exception handlers in the method vendor/phpunit/phpunit/src/Framework/TestCase.php::activeExceptionHandlers(). It pops each handler off the stack, then restores them and returns the list of popped handlers. The issue is that by default New Relic PHP agent will restore newrelic_exception_handler if it get removed. So activeExceptionHandlers method can never finish popping the handlers off as the agent keeps putting one back on! These are the only possible workarounds:

• disable New Relic PHP agent when running unit tests which use PHPUnit version 11 on PHPs 8.3+. This workaround disables all instrumentation of unit tests.
• disable New Relic PHP agent’s exception handler by adding newrelic.special=no_exception_handler to INI. NThis will prevent the agent from reporting any exceptions which occur during a test run, but perhaps will be a suitable workaround.

Since we don't need NewRelic when running PHPUnit, we'll keep our workaround by disabling metric collection via newrelic_ignore_transaction in PHPUnit's bootstrap.php.