Skip to content

Add CML GitHub Action#7

Draft
gchhablani wants to merge 72 commits intomoja-global:masterfrom
gchhablani:add_cml_test
Draft

Add CML GitHub Action#7
gchhablani wants to merge 72 commits intomoja-global:masterfrom
gchhablani:add_cml_test

Conversation

@gchhablani
Copy link

@gchhablani gchhablani commented Feb 24, 2022

This PR is a work-in-progress. Current action items include:

  • Use the container from moja global's repo directly instead of pulling and creating a container.
  • Look into self-hosted runners on CML. Current CML run takes 2+ hrs.
  • Run post-process scripts and get results in the report.

@gchhablani gchhablani changed the title Add cml test Add CML action Feb 24, 2022
@gchhablani gchhablani changed the title Add CML action Add CML GitHub Action Feb 24, 2022

jobs:
run:
runs-on: [ubuntu-latest]
Copy link

@aornugent aornugent Feb 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HarshCasper - I think we can add the FLINTcloud containers here. Docs on Running Jobs in a Container

-runs-on: [ubuntu-latest]
+container: ghcr.io/moja-global/rest_api_gcbm:master

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I will add the container for the workflow.

@aornugent
Copy link

I've been tinkering with the post-processing tools to get some nice summaries. We can add this step to our CML Action:

# from the gcbm_project/ folder
python3 ../tools/CompileGCBMResults/compileresults.py sqlite:///output/gcbm_output.db --output_db sqlite:///output/compiled_gcbm_output.db

Then we want to pull out annual summaries. Here's an example that returns the total forest biomass in megatonnes:

import sqlite3
import pandas

conn = sqlite3.connect("output/compiled_gcbm_output.db")

query = f"""
        SELECT years.year, COALESCE(SUM(i.pool_tc), 0) / 1e6 AS total_biomass_mt
        FROM (SELECT DISTINCT year FROM v_age_indicators ORDER BY year) AS years
        LEFT JOIN v_pool_indicators i
            ON years.year = i.year
        WHERE i.indicator = 'Total Biomass'
            AND (years.year BETWEEN 1920 AND 2020)
        GROUP BY years.year
        ORDER BY years.year
        """

df = pd.read_sql_query(query, conn)
ax = df.plot.line("year") # x-axis
ax.figure.savefig('output/total_biomass_mt.png', dpi=300)

total_biomass_mt

@yodavo might have some good suggestions for relevant results summaries!

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml

Update cml.yaml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants