Skip to content

Commit aa3b0e1

Browse files
Process TFM_MIG entries in parallel; add --profile flag (#338)
This PR adds a `--profile` flag to run_benchmarks.py that produces a cProfile file `xl2times.prof` that can be visualized with e.g. [snakeviz](https://jiffyclub.github.io/snakeviz/). It also parallelizes the querying and update creation of TFM_MIG rows in order to speed up the tool on TIMES-GEO. Before (on `208aa4e`): ``` Ran TIMES-GEO in 838.50s. 96.1% (823412 correct, 14985 additional). ``` After (on `616ae9a`): ``` Ran TIMES-GEO in 228.88s. 96.1% (823412 correct, 14985 additional). ``` A 3.7x speedup on my 2.3 GHz 8-Core Intel Core i9 laptop! (I created a new test file to test different approaches to speeding up the querying. It's not really testing anything at the moment, but I've kept it in because it'll be useful if we need to experiment with other optimizations for the querying and speed this up even further.) --------- Co-authored-by: Olexandr Balyk <ob@facilitate.energy>
1 parent 84d0e22 commit aa3b0e1

File tree

7 files changed

+516
-56
lines changed

7 files changed

+516
-56
lines changed

.github/workflows/ci.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ jobs:
2121

2222
env:
2323
PY_VERSION: "3.11"
24-
CACHE_KEY: 4 # Use this for manual cache key bumps, e.g., when caching code changes
24+
CACHE_KEY: 5 # Use this for manual cache key bumps, e.g., when caching code changes
2525

2626
steps:
2727
- uses: actions/checkout@v4

setup-benchmarks.sh

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ REF_demos_dd="c3b53ceae1de72965559f95016171ec67f333fc7"
1111
REF_tim_xlsx="e820d8002adc6b1526a3bffcc439219b28d0eed5"
1212
REF_tim_gams="cfe2628dbb5974b99c8a5664a9358849324e31ac"
1313
REF_TIMES_NZ="64c48461690fd2458d112b97c68e788c63490461"
14+
REF_times_geo_xlsx="48e053ad4c287f21c03de56c759d83d408636ee2"
15+
REF_times_geo_gams="3bf609f24f27710372bc0631c75af83df05819d1"
1416

1517
# If no GitHub token is provided, try to clone using SSH
1618
if [ -z "$GH_PAT_DEMOS_XLSX" ]; then
@@ -62,6 +64,8 @@ repositories=(
6264
"esma-cgep/tim|benchmarks/xlsx/TIM|$REF_tim_xlsx"
6365
"esma-cgep/tim-gams|benchmarks/dd/TIM|$REF_tim_gams"
6466
"olejandro/TIMES-NZ-Model-Files|benchmarks/TIMES-NZ|$REF_TIMES_NZ"
67+
"olejandro/times-geo|benchmarks/xlsx/TIMES-GEO|$REF_times_geo_xlsx"
68+
"olejandro/times-geo-gams|benchmarks/dd/TIMES-GEO|$REF_times_geo_gams"
6569
)
6670

6771
# Setup / update the repositories

0 commit comments

Comments
 (0)