OpenROAD bazel test running times

[oharboe]

In bazel, we now get a trace of the critical path. There are lots of activities running in parallel in a DAG in bazel, so critical path is what constitutes the longst running time.

A neat testing feature that has been added to OpenROAD and where Bazel shines is to run testing of power per module using the -hier feature and reading in a .vcd file from a simulation of the gate netlist (after cts, estimated parasitics, as well as final real parasitics). This is a test that is quite involved in number of dependencies that is has on tooling and there is no other automated test that brings this all together. This test case is optimized for maximum test coverage and minimum running time. Getting power for a non-trivial workload for a real design could easily be on the order of 24 hours, so 5 minutes is pretty good for the coverage it gives. mock-array has a long history in ORFS as a productive test case and has it is the culmination of many tweaks to cover various aspects(such as having pins on multiple metal levels, automated macro layout with routing by abutment and whatnot).

Running time wise, it does mean that detailed routing has to run, which is the slowest step for mock-array. mock-array has a good floorplan now(space for flip flops and buffers used on IOs).

It is a bit surprising that detailed routing isn't faster when there is so much routing by abutment going on and the rest of the routing is essentially routing horizontal and vertical wires.

These are running times from a local build bazelisk test test/orfs/mock-array/... --profile=build.profile && bazelisk analyze-profile build.profile:

=== PHASE SUMMARY INFORMATION ===

Total launch phase time                              1.343 s    0.14%
Total init phase time                                1.430 s    0.15%
Total target pattern evaluation phase time           0.414 s    0.04%
Total interleaved loading, analysis and execution phase time  942.414 s   99.65%
Total finish phase time                              0.089 s    0.01%
---------------------------------------------------------------------
Total run time                                     945.691 s  100.00%

Critical path (676.351 s):
       Time Percentage   Description
    75.9 ms    0.01%   action 'Writing script src/odb/odb.cppmap [for tool]'
  259.580 s   38.38%   action 'Compiling src/drt/src/io/io.cpp [for tool]'
    2.553 s    0.38%   action 'Linking openroad [for tool]'
    1.31 ms    0.00%   runfiles for // openroad
    6.943 s    1.03%   action 'Action test/orfs/mock-array/results/asap7/Element/base/1_synth.v'
    7.418 s    1.10%   action 'Action test/orfs/mock-array/results/asap7/Element/base/2_floorplan.odb'
   12.193 s    1.80%   action 'Action test/orfs/mock-array/results/asap7/Element/base/3_place.odb'
    5.074 s    0.75%   action 'Action test/orfs/mock-array/results/asap7/Element/base/4_cts.odb'
    7.225 s    1.07%   action 'Action test/orfs/mock-array/results/asap7/Element/base/5_1_grt.odb'
   32.525 s    4.81%   action 'Action test/orfs/mock-array/results/asap7/Element/base/5_route.odb'
    6.679 s    0.99%   action 'Action test/orfs/mock-array/results/asap7/Element/base/6_final.gds'
    2.195 s    0.32%   action 'Action test/orfs/mock-array/results/asap7/Element/base/Element.lef'
    2.196 s    0.32%   action 'Action test/orfs/mock-array/results/asap7/MockArray/base/1_1_yosys_canonicalize.rtlil'
    4.338 s    0.64%   action 'Action test/orfs/mock-array/results/asap7/MockArray/base/1_synth.v'
   12.621 s    1.87%   action 'Action test/orfs/mock-array/results/asap7/MockArray/base/2_floorplan.odb'
   14.826 s    2.19%   action 'Action test/orfs/mock-array/results/asap7/MockArray/base/3_place.odb'
    4.789 s    0.71%   action 'Action test/orfs/mock-array/results/asap7/MockArray/base/4_cts.odb'
    8.084 s    1.20%   action 'Action test/orfs/mock-array/results/asap7/MockArray/base/5_1_grt.odb'
  255.267 s   37.74%   action 'Action test/orfs/mock-array/results/asap7/MockArray/base/5_route.odb'
    9.347 s    1.38%   action 'Action test/orfs/mock-array/results/asap7/MockArray/base/6_final.gds'
    6.385 s    0.94%   action '[Verilator] Compiling @@//test/orfs/mock-array array_verilator_final'
     515 ms    0.08%   action 'VerilatorCopyTree test/orfs/mock-array/array_verilator_final_cpp'
    3.64 ms    0.00%   action 'Writing script test/orfs/mock-array/array_verilator_final.cppmap'
    3.766 s    0.56%   action 'Compiling test/orfs/mock-array/simulate.cpp'
     720 ms    0.11%   action 'Linking test/orfs/mock-array/simulator_final'
     370 ms    0.05%   action 'Executing genrule //test/orfs/mock-array vcd_final'
   10.482 s    1.55%   action 'Action test/orfs/mock-array/power_final.txt'
    0.05 ms    0.00%   runfiles for //test/orfs/mock-array MockArray_power_final_test
     179 ms    0.03%   action 'Testing //test/orfs/mock-array MockArray_power_final_test'

If I run ``bazelisk test test/orfs/mock-array/... --profile=build.profile && bazelisk analyze-profile build.profile` again, I get the running time, considering that everything is cached.

I don't think bazel can tell me what the critical path is for the running times of the cached items...

=== PHASE SUMMARY INFORMATION ===

Total launch phase time                              0.006 s    1.27%
Total init phase time                                0.406 s   86.21%
Total target pattern evaluation phase time           0.001 s    0.28%
Total interleaved loading, analysis and execution phase time    0.056 s   12.07%
Total finish phase time                              0.000 s    0.16%
---------------------------------------------------------------------
Total run time                                       0.471 s  100.00%

Critical path (1.10 ms):
       Time Percentage   Description
    1.10 ms  100.00%   action 'Testing //test/orfs/mock-array MockArray_power_instances_cts_test'

Looking at a recent build on the server, I get some suprising running times.... https://jenkins.openroad.tools/job/OpenROAD-Public/job/PR-8543-merge/1/pipeline-overview/?selected-node=57

Why does the below take the most time???

What is going on here? This takes 0 seconds locally:

Total interleaved loading, analysis and execution phase time  529.300 s

I suspect that short running times here are explained by caching. This is a python test case that failed(?) in the first iteration of this PR?

 === PHASE SUMMARY INFORMATION ===
 
 Total launch phase time                              0.000 s    0.00%
 Total init phase time                               10.385 s    1.54%
 Total target pattern evaluation phase time         132.163 s   19.57%
 Total interleaved loading, analysis and execution phase time  529.300 s   78.37%
 Total finish phase time                              0.026 s    0.00%
 Total unknown phase time                             3.482 s    0.52%
 ---------------------------------------------------------------------
 Total run time                                     675.358 s  100.00%
 
 Critical path (112.855 s):
        Time Percentage   Description
     1.126 s    1.00%   action 'Executing genrule //src/sta StaConfig'
    27.818 s   24.65%   action 'Compiling openroad_swig-py.cc'
     58.6 ms    0.05%   action 'Linking _openroadpy.so'
     8.29 ms    0.01%   runfiles for //src/gpl/test simple10
    83.845 s   74.29%   action 'Testing //src/gpl/test simple10'

[oharboe]

I believe what bazel is trying to tell me is that with infinite parallelism, the tests could have run 5x faster(529/112). I don't understand why mock-array isn't showing up as a critical path.

Is it because the build server is running on some highly parallel machine and that the detailed routing in mock-array is sped up enormously meaning that it is not in the critical path?

Total interleaved loading, analysis and execution phase time  529.300 s   78.37%
 Critical path (112.855 s):

[oharboe]

When going from 16 threads to 48 threads CPU on a workstation(nothing else is running), the total / critical is ca 1x.

Python doesn't work on my machine, so I can't run bazelisk test ... like on the server. Ubuntu 25.04 is paria :-). I'm not using Ubuntu 25, non-LTS, by choice. 2 years is an eternity and not all dependencies tolerate waiting that long, whereas others don't want to support non-LTS.... With infinite resources for everybody, this tension wouldn't exist of course...

=== PHASE SUMMARY INFORMATION ===

Total launch phase time                              0.866 s    0.19%
Total init phase time                                1.458 s    0.32%
Total target pattern evaluation phase time           0.353 s    0.08%
Total interleaved loading, analysis and execution phase time  458.725 s   99.42%
Total finish phase time                              0.016 s    0.00%
---------------------------------------------------------------------
Total run time                                     461.419 s  100.00%

Critical path (435.145 s):
       Time Percentage   Description
    11.2 ms    0.00%   action 'Expanding template external/toolchains_llvm++llvm+llvm_toolchain/module-x86_64-linux.modulemap [for tool]'
     280 ms    0.06%   action 'Compiling Source/Modules/python.cxx [for tool]'
     448 ms    0.10%   action 'Linking external/org_swig/swig [for tool]'
   33.587 s    7.72%   action 'Action src/odb/swig.cc [for tool]'
   96.326 s   22.14%   action 'Compiling src/odb/swig.cc [for tool]'
    1.207 s    0.28%   action 'Linking openroad [for tool]'
    1.01 ms    0.00%   runfiles for // openroad
    6.366 s    1.46%   action 'Action test/orfs/mock-array/results/asap7/Element/base/1_synth.v'
    9.543 s    2.19%   action 'Action test/orfs/mock-array/results/asap7/Element/base/2_floorplan.odb'
   15.373 s    3.53%   action 'Action test/orfs/mock-array/results/asap7/Element/base/3_place.odb'
    6.191 s    1.42%   action 'Action test/orfs/mock-array/results/asap7/Element/base/4_cts.odb'
    6.466 s    1.49%   action 'Action test/orfs/mock-array/results/asap7/Element/base/5_1_grt.odb'
   35.524 s    8.16%   action 'Action test/orfs/mock-array/results/asap7/Element/base/5_route.odb'
    8.249 s    1.90%   action 'Action test/orfs/mock-array/results/asap7/Element/base/6_final.gds'
    2.417 s    0.56%   action 'Action test/orfs/mock-array/results/asap7/Element/base/Element.lef'
    1.485 s    0.34%   action 'Action test/orfs/mock-array/results/asap7/MockArray/base/1_1_yosys_canonicalize.rtlil'
    5.613 s    1.29%   action 'Action test/orfs/mock-array/results/asap7/MockArray/base/1_synth.v'
   15.591 s    3.58%   action 'Action test/orfs/mock-array/results/asap7/MockArray/base/2_floorplan.odb'
   21.528 s    4.95%   action 'Action test/orfs/mock-array/results/asap7/MockArray/base/3_place.odb'
    6.963 s    1.60%   action 'Action test/orfs/mock-array/results/asap7/MockArray/base/4_cts.odb'
   10.080 s    2.32%   action 'Action test/orfs/mock-array/results/asap7/MockArray/base/5_1_grt.odb'
  119.932 s   27.56%   action 'Action test/orfs/mock-array/results/asap7/MockArray/base/5_route.odb'
   11.186 s    2.57%   action 'Action test/orfs/mock-array/results/asap7/MockArray/base/6_final.gds'
    7.899 s    1.82%   action '[Verilator] Compiling @@//test/orfs/mock-array array_verilator_final'
     238 ms    0.05%   action 'VerilatorCopyTree test/orfs/mock-array/array_verilator_final_cpp'
    0.36 ms    0.00%   action 'Writing script test/orfs/mock-array/array_verilator_final.cppmap'
    17.6 ms    0.00%   action 'Compiling test/orfs/mock-array/simulate.cpp'
     267 ms    0.06%   action 'Linking test/orfs/mock-array/simulator_final'
     246 ms    0.06%   action 'Executing genrule //test/orfs/mock-array vcd_final'
   12.103 s    2.78%   action 'Action test/orfs/mock-array/power_final.txt'
    0.05 ms    0.00%   runfiles for //test/orfs/mock-array MockArray_power_final_test
    6.15 ms    0.00%   action 'Testing //test/orfs/mock-array MockArray_power_final_test'