[Feat]: Faster IFC parsing via preallocation and sorted storage

[barvirm]

What is your idea?

Hello,

I’ve been investigating performance in web-ifc, specifically around parsing and IfcLine storage.

Test data

• File size: ~420 MB
• Lines: 7,413,409

Your current workflow

1. Read file in chunks
2. Parse each line lazily
3. Store parsed results in std::unordered_map

In this workflow, allocation and reallocation of std::unordered_map takes ~2.3s, which is a significant bottleneck.

Proposed approach

Scan the file once to count the number of ; delimiters (used as a aproximation number of IfcLines)

• Allocate a array of that size
• Parse lines and fill the array
• Sort the array by 'ref number'

This also enables easy parallel line parsing.

Perform lookups using binary search (or better: since IFC reference numbers are roughly ordered, the target index is usually near the reference value)

Performance measurements

Test machine:

• CPU: AMD Ryzen 5 3600 3.6GHz

Counting ; in the file takes:

• SIMD 128: 72.6 ms
• 4 threads (native): 20.9 ms
• Single thread: 83.1 ms

Additional costs:

• Array allocation: ~20 µs
• Sorting: ~140 ms

Even in single-threaded mode, total time is significantly lower than the current unordered_map allocation cost alone.

Questions

1. Are there known constraints (e.g. memory limits, streaming requirements) that would make full-file buffering unacceptable in some cases?
2. Would you be interested in discussing this approach further?

[beachtom]

1. So in cases where web-ifc is being used in wasm - it has a maximum of 4gb of memory available - this is the reason for the chunking approach used.
2. Yes definetly.

I saw your comment about the lines being roughly ordered - we cannot assume that. In many cases they are - but in some cases they are in reverse order - or even sparse - i.e. line numbers jumping . I tried this - it gave performance improvement on many models - but also killed the performance on other models.

but the counting of the number of ; - is an interesting idea