Compute region allocations on the fly

[sverhoeven]

Refs #95

Run instructions

First install ws.py deps in effort sharing environment.

mamba activate effort-sharing
pip install -r requirements.txt
# then run gunicorn / flask from effort-sharing environment.

To compare values with cabe website the xr_dataread.nc of the website was copied to th effort-sharring data output with

cp website-carbon-budget-explorer/data/Version_0_4_2/startyear_2021/xr_dataread.nc effort-sharing/data/output/startyear_2021/

The `../effort-sharing/notebooks/input.yml` is used by default and should have absolute paths

On my machine

paths:
  data:
    external: /home/verhoes/git/carbon-budget-explorer/effort-sharing/data/input/
    baseline: /home/verhoes/git/carbon-budget-explorer/effort-sharing/data/input/
    # internal: Data/
    datadrive: /home/verhoes/git/carbon-budget-explorer/effort-sharing/data/output/
    export: /home/verhoes/git/carbon-budget-explorer/effort-sharing/data/output/
    lambollrepo: /home/verhoes/git/carbon-budget-explorer/AR6CarbonBudgetCalc/  # clone from https://github.com/Rlamboll/AR6CarbonBudgetCalc
...

TODO

• takes 1-2 seconds to fill allocator for a region. Find way to only calculate for current global setting
• allocator constructor needs input.yml path, for now we expect effort-sharing to be cloned next to this repo in same root dir. Should use env var to tell where that file.
• Add effort-sharing in requirements.txt
• During deployment make sure effort-sharing data files and its input.yml are put in right place.
• Test all regions using tests/region.test.ts
• renable sentry/glitchtip

NICE

• on the fly calc of Version_0_4_2/startyear_2021/GHG_incl/Aggregated_files/xr_alloc_20?0.nc files in memory using effort sharing package. infeasable at short term see Compute region allocations on the fly #96 (comment)
• compute Version_0_4_2/startyear_2021/xr_dataread.nc in memory on startup of web service from effort sharing input files. Preferably showing some busy page while computing.
• compute Version_0_4_2/xr_policyscen.nc

With all the nice to haves implemented we no longer need the output of the effort-sharing, but can directly use the effort sharing input files.

[sverhoeven]

Tested all regions
screenshots.zip

timings

36mslocator.getByText('Select country ▼').click— region.test.ts:34
2.2sAfrican Group— region.test.ts:37
2.4sAustralasia— region.test.ts:37
2.4sEU— region.test.ts:37
2.4sG20— region.test.ts:37
2.4sG7— region.test.ts:37
2.4sLDC— region.test.ts:37
2.4sFSM— region.test.ts:37
3.4sNorthern America— region.test.ts:37
2.4sSIDS— region.test.ts:37
2.4sUmbrella— region.test.ts:37
3.3sAFG— region.test.ts:37
3.4sALB— region.test.ts:37
2.4sDZA— region.test.ts:37
2.4sAND— region.test.ts:37
2.3sAGO— region.test.ts:37
2.4sATG— region.test.ts:37
3.4sARG— region.test.ts:37
2.4sARM— region.test.ts:37
2.4sAUS— region.test.ts:37
2.4sAUT— region.test.ts:37
2.4sAZE— region.test.ts:37
2.4sBHS— region.test.ts:37
3.4sBHR— region.test.ts:37
3.4sBGD— region.test.ts:37
2.3sBRB— region.test.ts:37
2.4sBLR— region.test.ts:37
3.4sBEL— region.test.ts:37
3.3sBLZ— region.test.ts:37
3.4sBEN— region.test.ts:37
2.4sBTN— region.test.ts:37
2.4sBOL— region.test.ts:37
2.4sBIH— region.test.ts:37
2.4sBWA— region.test.ts:37
3.4sBRA— region.test.ts:37
3.4sBRN— region.test.ts:37
2.4sBGR— region.test.ts:37
2.4sBFA— region.test.ts:37
2.4sBDI— region.test.ts:37
3.4sKHM— region.test.ts:37
2.4sCMR— region.test.ts:37
2.4sCAN— region.test.ts:37
2.4sCPV— region.test.ts:37
2.3sCAF— region.test.ts:37
2.4sTCD— region.test.ts:37
3.3sCHL— region.test.ts:37
3.4sCHN— region.test.ts:37
2.4sCOL— region.test.ts:37
2.4sCOM— region.test.ts:37
2.4sCOG— region.test.ts:37
3.3sCOK— region.test.ts:37
2.4sCRI— region.test.ts:37
3.4sHRV— region.test.ts:37
2.4sCUB— region.test.ts:37
2.3sCYP— region.test.ts:37
2.4sCZE— region.test.ts:37
3.4sCIV— region.test.ts:37
2.4sCOD— region.test.ts:37
3.4sDNK— region.test.ts:37
2.4sDJI— region.test.ts:37
2.3sDMA— region.test.ts:37
2.4sDOM— region.test.ts:37
3.4sEARTH— region.test.ts:37
2.4sECU— region.test.ts:37
3.4sEGY— region.test.ts:37
2.3sSLV— region.test.ts:37
2.4sGNQ— region.test.ts:37
3.4sERI— region.test.ts:37
3.4sEST— region.test.ts:37
2.4sETH— region.test.ts:37
2.4sFJI— region.test.ts:37
3.4sFIN— region.test.ts:37
2.4sFRA— region.test.ts:37
3.4sGAB— region.test.ts:37
2.4sGMB— region.test.ts:37
2.4sGEO— region.test.ts:37
2.4sDEU— region.test.ts:37
3.4sGHA— region.test.ts:37
2.3sGRC— region.test.ts:37
2.4sGRD— region.test.ts:37
2.4sGTM— region.test.ts:37
2.4sGIN— region.test.ts:37
3.4sGNB— region.test.ts:37
3.4sGUY— region.test.ts:37
3.4sHTI— region.test.ts:37
3.4sHND— region.test.ts:37
2.4sHUN— region.test.ts:37
2.4sISL— region.test.ts:37
2.4sIND— region.test.ts:37
2.3sIDN— region.test.ts:37
3.4sIRN— region.test.ts:37
2.3sIRQ— region.test.ts:37
2.4sIRL— region.test.ts:37
2.4sISR— region.test.ts:37
2.4sITA— region.test.ts:37
2.4sJAM— region.test.ts:37
2.4sJPN— region.test.ts:37
2.4sJOR— region.test.ts:37
2.4sKAZ— region.test.ts:37
2.4sKEN— region.test.ts:37
3.4sKIR— region.test.ts:37
3.4sKWT— region.test.ts:37
2.4sKGZ— region.test.ts:37
2.4sLAO— region.test.ts:37
3.4sLVA— region.test.ts:37
2.4sLBN— region.test.ts:37
2.3sLSO— region.test.ts:37
3.4sLBR— region.test.ts:37
2.4sLBY— region.test.ts:37
2.4sLIE— region.test.ts:37
2.4sLTU— region.test.ts:37
2.4sLUX— region.test.ts:37
2.4sMDG— region.test.ts:37
2.4sMWI— region.test.ts:37
2.4sMYS— region.test.ts:37
2.4sMDV— region.test.ts:37
2.4sMLI— region.test.ts:37
2.4sMLT— region.test.ts:37
3.4sMHL— region.test.ts:37
2.4sMRT— region.test.ts:37
3.4sMUS— region.test.ts:37
2.4sMEX— region.test.ts:37
2.4sMDA— region.test.ts:37
2.4sMCO— region.test.ts:37
3.4sMNG— region.test.ts:37
3.4sMNE— region.test.ts:37
3.4sMAR— region.test.ts:37
2.4sMOZ— region.test.ts:37
3.4sMMR— region.test.ts:37
2.4sNAM— region.test.ts:37
2.4sNRU— region.test.ts:37
2.4sNPL— region.test.ts:37
2.4sNLD— region.test.ts:37
2.4sNZL— region.test.ts:37
3.4sNIC— region.test.ts:37
3.4sNER— region.test.ts:37
3.4sNGA— region.test.ts:37
3.4sNIU— region.test.ts:37
2.4sPRK— region.test.ts:37
3.4sMKD— region.test.ts:37
2.4sNOR— region.test.ts:37
3.3sOMN— region.test.ts:37
2.4sPAK— region.test.ts:37
3.3sPLW— region.test.ts:37
2.4sPSE— region.test.ts:37
2.4sPAN— region.test.ts:37
2.4sPNG— region.test.ts:37
3.4sPRY— region.test.ts:37
3.3sPER— region.test.ts:37
2.4sPHL— region.test.ts:37
2.4sPOL— region.test.ts:37
2.4sPRT— region.test.ts:37
3.3sQAT— region.test.ts:37
2.4sROU— region.test.ts:37
2.4sRUS— region.test.ts:37
2.4sRWA— region.test.ts:37
2.4sSSD— region.test.ts:37
2.4sKNA— region.test.ts:37
3.4sLCA— region.test.ts:37
2.4sVCT— region.test.ts:37
2.4sWSM— region.test.ts:37
2.4sSMR— region.test.ts:37
3.4sSAU— region.test.ts:37
3.4sSEN— region.test.ts:37
3.4sSRB— region.test.ts:37
3.4sSYC— region.test.ts:37
3.4sSLE— region.test.ts:37
3.4sSGP— region.test.ts:37
2.4sSVK— region.test.ts:37
2.4sSVN— region.test.ts:37
2.4sSLB— region.test.ts:37
2.4sSOM— region.test.ts:37
2.4sZAF— region.test.ts:37
2.4sKOR— region.test.ts:37
3.3sESP— region.test.ts:37
2.4sLKA— region.test.ts:37
2.4sSDN— region.test.ts:37
2.4sSUR— region.test.ts:37
2.4sSWE— region.test.ts:37
3.4sCHE— region.test.ts:37
3.4sSYR— region.test.ts:37
2.4sSTP— region.test.ts:37
2.4sTJK— region.test.ts:37
2.3sTZA— region.test.ts:37
2.4sTHA— region.test.ts:37
3.4sTLS— region.test.ts:37
2.3sTGO— region.test.ts:37
2.4sTON— region.test.ts:37
2.4sTTO— region.test.ts:37
2.4sTUN— region.test.ts:37
2.4sTUR— region.test.ts:37
2.4sTKM— region.test.ts:37
2.4sTUV— region.test.ts:37
2.4sUGA— region.test.ts:37
2.4sUKR— region.test.ts:37
3.4sARE— region.test.ts:37
2.4sGBR— region.test.ts:37
2.4sUSA— region.test.ts:37
2.4sURY— region.test.ts:37
3.4sUZB— region.test.ts:37
3.4sVUT— region.test.ts:37
3.4sVAT— region.test.ts:37
2.4sVEN— region.test.ts:37
2.4sVNM— region.test.ts:37
2.4sYEM— region.test.ts:37
2.4sZMB— region.test.ts:37
3.4sZWE— region.test.ts:37
2.4sSWZ— region.test.ts:37
7msAfter Hooks

[sverhoeven]

xr_alloc_20?0.nc files are made in https://github.com/imagepbl/effort-sharing/blob/main/notebooks/Aggregator.ipynb which uses all the region nc files as input. It will be too slow and big on the web app to first generate all the region datasets and then merged on just a year. As each region ds must be generated with effortsharing.allocation.allocation which takes 1-2s per region and we have +200 regions.

So without a full rewrite of effort sharing package this can not be computed on the fly.