[BUG] XML data imports fail for large datasets

[skarpik]

Bug Affects...

[x] admin experience UI
[x] Content Types or data management

Current Behavior / Expected Behavior

For small datasets (500 or fewer) records, importing data through XML templating works well. For my project, I had to work with a dataset that included 8028 records consisting of 4014 en-CA records and 4014 fr-CA records).

Directly importing the data from one XML file fails - the operation never completes. To work around this, I created 17 files each with no more than 500 records (250 en-CA, 250 fr-CA). Here is the summary for one of the batches:

Importing the data one by one went fine until batch number 9 when an error occurred. Here is a screen shot of the error that pops up:

To Reproduce (Steps, Videos, Screenshots, Apps)

1. Created an app with data called RollOfHonourEntry (data template just below)

2sxc RollOfHonourEntry Template 20251111194525.xml

2. Attempted to import 17 XML files - the maximum number of records in each is 500. Failed on number 10.
3. In the DNN Admin log, there was the following error:

errorlog.xml

5. The JavaScript console contained this information:

JavaScript Console.txt

Your environment

• 2sxc version(s): 20.00.08
• Browser: all
• DNN : 10.1.2
• Hosting platform: IIS
• Ui Languages: English & French

Additional context

It seemed that each subsequent batch took longer to process but I may be imagining this.

I can supply data and the data template if that is helpful.

[skarpik]

Here is an interesting fact... The uploads would get through 9 batches of 500 records and then it would fail on number 10. Cutting number 10 into 2 imports of 250 records did nothing - it failed again. There seems to be something bigger at play than just the size of the bundles.

[iJungleboy]

could it be that your number 10 has reference to data which does not yet exist?

[skarpik]

I was suspicious of bad data as well. However, as I did more testing, I noticed that the import did not always fail at the same batch number. In fact, the failure started to occur earlier. I began to suspect some sort of caching issue (i.e. cache was not getting disposed of quickly enough, thereby causing the failure).

To investigate this theory, every 3 files, I logged out from the website and then cycled the IIS server. This allowed me to successfully all the data - 16 files with 250 entries (250 en-CA and 250 fr-CA) and a final batch with just 14 records for a total of 4014 records (4014 en-CA and 4014 fr-CA). I compared the imported data against the original and it all looks good.

Stopping and starting the IIS server was essential to getting all the data ingested. Possibly I could have done a restart every 4 files but I didn't want to chance it. Either some cache is growing beyond usefulness and/or old data is not getting garbage collected (that's my theory).

I should also mention that I cleaned the 2sxc cache each time I restarted IIS.

[skarpik]

I mentioned this issue in the DNN Discord group and Alex Pommier recalled facing a similar issue with XML imports. If I understand what he did to work around the problem, he only restarted IIS when he ran into problems with a batch.

[iJungleboy]

Any chance we could have the data you were working on, so we could reproduce it?
I assume you know how to reach me... info - 2sxc - you know the gist

[skarpik]

I found some errors in the data that was preventing import of the data in batches. Those fixed, import by batches worked but import of the whole data set in one action failed. @iJungleboy - I will send you corrected data and an export of the app (the app is very simple, just 3 data tables and no real functionality).

[iJungleboy]

thx, I'll try to prioritize