AD1 with many files

[gfd2020]

The initial implementation of the AD1 extractor is done recursively (I don't know if the modifications that @lfcnassif made changed this). That said, if the AD1 file has many files, a stack overflow may occur. This has already occurred in some cases, the solution was to increase the stack (100 MB is enough for all cases I tested).

The question is: document this and warn users about it or change the extractor logic to be iterative?

[lfcnassif]

Hi @gfd2020! I don't remember either if I changed the recursive logic... Increasing the stack size is not too intuitive for users, I think it would be better to avoid the recursion if possible. If not, maybe catch the StackOverflowException and print a message telling users how to increase the stack size?

[gfd2020]

Hi @lfcnassif . Removing recursion is possible but I think it will take a little work. It's a good idea to handle the stack overflow exception and leave a warning.
I was looking at the code a little and I think it is possible to make some optimizations to use less stack memory, reducing the type of primitives used as in the example below.

// Recursive method called a lot, each file calls this method multiple times
public readProperty(){
long tamanho_propriedade = 20;
vector_propriedade = new byte[(int) tamanho_propriedade];
PC += tamanho_propriedade;
...
}

Perhaps, "tamanho_propriedade" does not need to be long (short or int) and does not need to be declared within the method.

I have a 7TB AD1 which I think would be a good use case. I'll try to make some optimizations and deal with the stack overflow error.

[lfcnassif]

Wow, that's a huge AD1!

[gfd2020]

I did a small test with a 4GB AD1 (64k files and 10k folders) and I was unable to reproduce a stack overflow, even reducing the stack size to 180kbytes (minimum possible) in IPED 3.18 and 4.1.
Already in my original test version AD1 extractor already caused stack overflow at first in this same case. I think the way @lfcnassif implemented it is well done and there won't be any problems. It's a concern I've always had, I think I can put it aside.

PS: When I tried to test reduce the stack to 180K, I had the error below in regextask. I had to disable it to make iped run normally.
java.lang.StackOverflowError: null
at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:245) ~[fst-2.57.jar:?]
at org.nustaq.serialization.FSTConfiguration.asObject(FSTConfiguration.java:1158) ~[fst-2.57.jar:?]
at iped.engine.task.regex.RegexTask.loadCache(RegexTask.java:225) ~[iped-engine-4.2-snapshot.jar:?]
at iped.engine.task.regex.RegexTask.init(RegexTask.java:158) ~[iped-engine-4.2-snapshot.jar:?]
at iped.engine.core.Worker.initTasks(Worker.java:128) ~[iped-engine-4.2-snapshot.jar:?]
at iped.engine.core.Worker.(Worker.java:110) ~[iped-engine-4.2-snapshot.jar:?]
at iped.engine.core.Manager.initWorkers(Manager.java:551) ~[iped-engine-4.2-snapshot.jar:?]
at iped.engine.core.Manager.process(Manager.java:273) ~[iped-engine-4.2-snapshot.jar:?]
at iped.app.processing.Main.startManager(Main.java:178) [iped.jar:?]
at iped.app.processing.Main.execute(Main.java:240) [iped.jar:?]
at iped.app.processing.Main.main(Main.java:308) [iped.jar:?]