Slow Single Thread Performance for Large Lists

The HostlistCompiler does not perform well with large lists e.g. 10 million rules across around 25 sources, running for longer than 1 hour.

[jaypatrick]

Are you consolidating them all into one main rules list? If so, by the nature of how threading works with streaming to files, it's not a trivial fix. You could try to introduce intermediate steps that chunk files up into multiple smaller files, and then run the compiler again to consolidate, as many times as it takes. You could start multiple instances of the compiler up, let's say you want to keep each rules list to 100,000 rules each per file. Then just iteratively consolidate. This would simulate multi-threading. It wouldn't be hard to write a powershell script that does this.

[jaypatrick]

I decided to expand the wrappers I've been working on to include support for multithreading. You can also chunk to number files, or specify how many rules should be in each intermediate list. Ultimately, I will include steps to compile the intermediate lists, but for now it's just one pass. A "multiple pass" compiler makes sense, b/c that's how language compilers work as well.

The code is on my profile, and extensively documented. I'm working on other tools as well.d