Recalculate Rollups Full Calculate only partially completing

[louisskelton]

Hi there! I hope you're well! ☺️

I'm keen to consider apex-rollup as a replacement to DLRS but I seem to be falling at the first hurdle.

In a Sandbox, using apex-rollup 1.7.5, I've created a custom object named Line_Item__c and have created 6 million Line Item records related to 2 million Opportunities. There are are 3 Line Items for every Opportunity - i.e no data skew.

I'm using the 'Recalculate Rollups' tab to to an initial SUM rollup. It gets through about 500k Opportunities and then seems to run out of puff. Compared to the video tutorial, the Apex Jobs are showing it's used a Queueable process rather than batches. I've tried replicating the process using CMT Rollup pointing to a Rollup Control using 'Batchable' instead of Queueable but it still seems to be launching a Queueable process... Again, seems to run out of puff after 500k Opportunities.

I know these data volumes are quite high, but DLRS seems to crunch through very quickly, so I feel like I must be doing something wrong! Very keen to move away from DLRS so I'm anxious to get apex-rollups to work.

Thanks!

Louis

[jamessimone]

@louisskelton that's not that much data volume at all, but in 1.7 the full recalculations moved over to always using Queueables to get around some limitations with Batch Apex.

Do you happen to have an Apex trace on the user running the job? I'm wondering if there's some kind of exception being thrown. I'll kick off a LDV recalc this morning to see if there's anything unusual occurring.

[jamessimone]

@louisskelton based on my initial experiments, this is probably due to large swathes of the "batch" not requiring state records. It's hard to track the ongoing status of the job as a result - so it's reporting as "completed" but still running the in background. You should be able to track the ongoing actual status in the Apex Jobs page. I will definitely be looking into improvements within the full recalc page to track the overall status of the job at a better level.

Edit - also, just to give some additional context for posterity: using the standard 500 items per "batch", this job has processed 1.6 million records in two hours (about 60% complete). That's much faster than any batch process I've seen.

[louisskelton]

Hi @jamessimone - thanks so much for taking a look! Really appreciate it. I added a trace and I think I caught it just in time! The queueable jobs were still going but about half an hour later they stopped with an error "Batchable instance is too big". Here are the logs:

62.0 APEX_CODE,WARN;APEX_PROFILING,NONE;CALLOUT,NONE;DATA_ACCESS,NONE;DB,NONE;NBA,NONE;SYSTEM,NONE;VALIDATION,NONE;VISUALFORCE,NONE;WAVE,NONE;WORKFLOW,NONE
17:56:35.1 (1780356)|USER_INFO|[EXTERNAL]|BLAHBLAHBLAH|(GMT+00:00) Greenwich Mean Time (Europe/London)|GMTZ
17:56:35.1 (1806828)|EXECUTION_STARTED
17:56:35.1 (1814584)|CODE_UNIT_STARTED|[EXTERNAL]|01pPv000003G1t5|RollupFullBatchRecalculator.FullRecalcFinalizer
17:56:36.167 (1167968946)|FATAL_ERROR|System.LimitException: Batchable instance is too big: RollupFullBatchRecalculator

External entry point
Class.RollupFullBatchRecalculator.startAsyncWork: line 70, column 1
Class.RollupFullBatchRecalculator.FullRecalcFinalizer.handleSuccess: line 130, column 1
Class.RollupFinalizer.execute: line 36, column 1
17:56:36.167 (1168008887)|CODE_UNIT_FINISHED|RollupFullBatchRecalculator.FullRecalcFinalizer
17:56:36.167 (1168015853)|EXECUTION_FINISHED

[louisskelton]

Screenshot below is the Rollup Control I used. But I had the same issue running on an unedited Org_Defaults Control too. I've also used both the non-CMT full calculate and the Rollup CMT full calculate. But both launched from the 'Recalculate Rollups' tab.

[jamessimone]

@louisskelton something that might help is enabling the Should Skip Resetting Parent Fields flag on the org default rollup control - you can also try lowering the Batch Chunk Size a bit more from the default of 500. To be honest, the error message is a bit of a red herring - the RollupFullBatchRecalculator is no longer a Batchable instance.

My 2.5 million record recalculation is about to finish, so I'm going to try upping the number to something in the tens of millions to see if I can reproduce what you're experiencing.

[louisskelton]

Thanks - I'll try that now. So I have below for Org_Defaults:

And this is the rollup config I have:

Do they look okay before I git 'go'?!

[jamessimone]

Additional info - I added some logging into my own RollupFullBatchRecalculator instance just to see what was going on with both of the concurrent calculations I have running - one for 2.25 million records, one for nearly 10 million. Here are some examples:

For the ~10 mm record job:

Count of records to return: 500
Current position in cursor: 2622500
Cursor calculated number of records: 9701985
New position in cursor: 2623000
Heap size during fetch: 146264
Heap limits as a reminder: 12000000
|CODE_UNIT_FINISHED|RollupFullBatchRecalculator.FullRecalcFinalizer

For the 2.5 mm record job:

Count of records to return: 500
Current position in cursor: 2208000
Cursor calculated number of records: 2251882
New position in cursor: 2208500
Heap size during fetch: 1406885
Heap limits as a reminder: 12000000
|CODE_UNIT_FINISHED|RollupFullBatchRecalculator.FullRecalcFinalizer

This is where I added those lines within RollupFullBatchRecalculator, if you'd like to do the same:

  private class CursorBasedRetriever implements CalcItemRetriever {
    private final Database.Cursor cursor;
    private Integer countOfRecordsToReturn;
    private Integer currentPosition = 0;

    public CursorBasedRetriever(RollupRepository repo, RollupControl__mdt control) {
      this.cursor = repo.getCursor();
      this.countOfRecordsToReturn = control.BatchChunkSize__c.intValue();
    }

    public List<SObject> fetch() {
      System.debug('Count of records to return: ' + this.countOfRecordsToReturn);
      System.debug('Current position in cursor: ' + this.currentPosition);
      System.debug('Cursor calculated number of records: ' + this.cursor.getNumRecords());
      if (this.countOfRecordsToReturn + this.currentPosition > this.cursor.getNumRecords()) {
        this.countOfRecordsToReturn = this.cursor.getNumRecords() - this.currentPosition;
      }
      List<SObject> fetchedRecords = this.cursor.fetch(this.currentPosition, this.countOfRecordsToReturn);
      this.currentPosition += this.countOfRecordsToReturn;
      System.debug('New position in cursor: ' + this.currentPosition);
      System.debug('Heap size during fetch: ' + Limits.getHeapSize());
      System.debug('Heap limits as a reminder: ' + Limits.getLimitHeapSize());
      return fetchedRecords;
    }

So even the ~10mm record job has only taken up 1.4mb of space, which is just over 10% of the space available to the job as a whole. Maybe you can add those logging lines in and monitor how things progress - also, you should be able to use the toggle at the top of the recalc app so you can just select your CMDT record (so you don't have to type out all of your info each time). Hopefully that helps!

[louisskelton]

Hi @jamessimone - still having no luck here. I uninstalled apex-rollup and reinstalled, edited only the settings per your suggestion. It took a few days to crunch through the records, but this time left ~200k uncalculated. I'm struggling to debug because the quantity of queueable jobs and the durations involved. Any ideas what I might be doing wrong? I'm still running from the Recalculate Rollup tab without using CMT. This is in an effort to minimise things I might be doing wrong! It is a Partner Trial org that I'm demoing. But I tried previously with that flag checked (Is Dev Ed Or Trial Org) and it didn't seem to make a difference.

Very keen to move away from DLRS but it was able to get through the data in a few hours. Feels like something must be up on my side but can't work it out! I've deactivated flows/triggers etc.

[jamessimone]

@louisskelton are there ever instances where the lookup field on that child item is null? Line items without opportunities? Just thinking out loud here about possible where clauses that might restrict the number of records to a subset that makes debugging easier.

I haven't been able to repro this issue, though. I've run several recalcs in the tens of millions of records since we last spoke without issue.

DLRS, last I checked, uses a true batch apex process to do bulk calculations. Unless you're running 2k records through each batch chunk - which you could also do with apex rollup - I'm not sure how it could be faster to chunk through the same amount of records without more context.

I don't think you're doing anything complicated here. I'm happy to try to assist with debugging in any way I can; just let me know what you think would be helpful!

[jamessimone]

@louisskelton haven't heard from you on this issue in a while - is this still a problem?

[louisskelton]

Hi @jamessimone! I'm afraid we had to abandon ship on this and roll our own nightly batch thing. I'm very keen to give this a try again in the future but at the time I couldn't get consistent results - likely due to a misconfiguration on our end. Thanks for checking back in - I'll update if I ever get a chance to give this another go.