Performance issue in LogEntryEventHandler

[rmccu]

I apologize for all of the discussion posts recently with suggestions for the LogEntryEventHandler class. I was reviewing the class as part of a performance investigation and as I was in there I figured I would ask some questions about the code.

We are seeing a performance issue that appears to be related to LogEntryEventHandler. It is consuming 50% of the DB CPU for 1 node and is causing us to get throttled. My suspicion is that it has to do with the Logger Tags.

We also have a bulk delete operation running in the background as well that was encountering a lot of errors. I met with that team today and changed how we are deleting from deleting the Log records (and relying on the cascading delete to Log Entry and Log Entry Tag), to instead delete the Log Entry Tag records first (sorted by Tag) in serial mode, then I think they can delete the Log records and cascade into Log Entry. As a fallback, I will have them delete the Log Entry records in parallel before deleting Log records in parallel.

Does this sound like it is the culprit? Or is there something additional I should look at in the LogEntryEventHandler class? I think we are generating around 2M Log records a day.

[jongpie]

Hi @rmccu, it sounds like you've got 2 issues going on: possible performance issues in LogEntryEventHandler, and bulk delete operations:

1. For the possible performance issues in LogEntryEventHandler, I think I need a little more detail about your use case to help troubleshoot.

   • Do you have any details you can share about how you're logging data? (How many log entries are generated in a transaction, per day, etc.)
   • What makes you think that the issue is related to Logger Tags?
   • Are you heavily using methods like Logger.info('some message, recordId) or Logger.info('some message, record) where you provide a record ID or SObject instance?
   • How do you know that it's consuming 50% of the DB CPU?

2. For bulk delete operations, it sounds like you're using a custom solution for deleting logging data. Is there a particular reason you're using a custom solution, instead of using the included batch purge job? The included batch class LogBatchPurger (and the schedulable class LogBatchPurgeScheduler) has been designed to handle that situation - instead of relying on cascading deletes (which can fail, as you're experiencing), it will first delete LogEntryTag__c records, then restart itself to delete LogEntry__c records, and then finally restart itself to delete Log__c records.

[rmccu]

1a. I don't have this, but can ask the dev team. I was just told that we are generating ~2M logs a day
1b. I don't know that this is the issue, it is just a suspicion. It could be coming from something else, like the SOQL queries that try to find the API version.
1c. I don't know, I would need to ask.
1d. Salesforce support told me this. It is limited to 1 node, which is good, but it is still consuming more DB CPU than we would like to see, especially since we are still adding more users every few weeks.

2. I wasn't aware of this, I will ask the DevOps resources if they have looked at using this. They may not be aware of it either.