Poor performance when "drawing" many instances containing hidden layers

[anewusername]

Layouts containing a large number of instances can be very slow to redraw (and cause freezes / crashes), even in cases where those instances contain only a single, hidden, layer.

I've attached a file (tiles.gds.gz) that can help to show the effect -- to reproduce, open the file, hide layer 1/0, show the full hierarchy (to avoid drawing cell boxes), then zoom in and out and watch the other layers get drawn one-by-one.

The effect goes away if you flatten the layout -- based on this, I assume the drawing code is visiting each instance (but drawing nothing as the layer is hidden). Is there an easy opportunity to do some caching (e.g. "nothing to draw for this cell, skip its instances"), or are instances stored in a way that wouldn't help?

Happy to try to build something myself, but figured I'd ask before getting too deep into the code (so far, some quick profiling pointed me towards layRedrawThreadWorker.cc).

[stefanottili]

If you only want to view this, use strm2oas tiles.gds.gz tiles.oas; klayout -ne tiles.oas.
klayout will take advantage of the "repetition 10/11" feature of oasis and will draw the 4M tile references very fast.
But only in -ne viewer mode. klayout tiles.oas behaves the same as klayout tiles.gds.gz.

[klayoutmatthias]

Hi @anewusername,

The effect in your case is a different one: it's not the hidden layers, but the cells which do contain only a small subset of the layer set. The problem is that a cell is defined first of all by its bounding box, so from that perspective for every layer, the cell could have some content and needs to be investigated. More specifically it's the instance quad tree which needs to be iterated for every layer. Caching does not help much here - the iteration itself is responsible for the overhead. The solution were per-layer quad trees, but their memory overhead and even more the overhead to build and maintain them is prohibitive.

The ideal case is a layout with cells that use every layer (a fully populated layer/cell matrix). The worst case is a layout with many layers and only one layer used in every cell (a sparse layer/cell matrix). KLayout is optimized for the first case. In a P&R block for example, you are closer to the ideal case as standard cells typically hold shapes for maybe half the design layers.

I don't understand why you say "freeze/crash". Drawing is done in a background thread, so it should not freeze the application, neither crash it. If there is a way to freeze the application, that's a gap in the thread resume logic, so I would like to know about it. However I am not aware of such a case. In you test case, the application remains responsive for me. It's possible to turn off background drawing by setting the number of threads to zero in File/Setup, Display/Optimization. Is that the case for you maybe? BTW: you can also use more than one thread to improve rendering performance.

Also, the rendering thread will first render all the visible layers, so if you hide the 0/* layers of your testcase, rendering is rather fast. It renders the hidden layers in the background, so they are ready when you show them. So again, hidden layers are not the issue. They are processed in the background.

Nevertheless, your test case points to a simple optimization opportunity (see 92c08a3). It's just very easy to modify the test case, so the optimization is no longer effective. As the cost of this optimization is basically zero, I tend to include it. Yet it needs some testing.

Matthias

[anewusername]

Hi Matthias,

KLayout is optimized for [cells that use every layer]

Definitely appreciate that this is the right approach -- my goal is to recover the usually-excellent performance for a troublesome subset of layouts :).

The solution were per-layer quad trees, but their memory overhead and even more the overhead to build and maintain them is prohibitive.

Would you necessarily need per-layer quadtrees? Could you use a "non-hidden" tree (all objects with any non-hidden layers) and rebuild on layer visibility changes, or is that already too expensive?

Drawing is done in a background thread, so it should not freeze the application, neither crash it.

I've had the app lock up, gotten "soft" errors (can continue via the dialog box) and hard crashes, but I haven't been able to reproduce them reliably. I'll gather some logs/dumps next time it happens -- possible it's been exacerbated by other resource pressure on my system. In any case, I've been running with 1 background thread.

BTW: you can also use more than one thread to improve rendering performance.

I wasn't aware of this option, and it does help significantly -- thanks!!

(see 92c08a3).

Thanks -- I wasn't expecting any code this quickly! I'll check for any improvement on real layouts, and poke around a bit to get a better understanding of the relevant functions -- your explanations have already been very helpful. This isn't super urgent; I've lived with it for many years and only now bothered to investigate deeper :)

-Jan