Skip to content

Postprocessing, DVC pipeline and Cml workflow#2

Open
aldeav wants to merge 8 commits intomoja-global:mainfrom
aldeav:cml-workflow
Open

Postprocessing, DVC pipeline and Cml workflow#2
aldeav wants to merge 8 commits intomoja-global:mainfrom
aldeav:cml-workflow

Conversation

@aldeav
Copy link

@aldeav aldeav commented Mar 3, 2023

Pull Request Template

Description

This PR adds 3 major components:

  1. Postprocessing in R
  2. DVC pipeline (including postprocessing and metrics)
  3. CML workflow

Modifications:

  1. Standalone_GCBM/gcbm_project/run_gcbm.bat - to include run path for windows runner in CML workflow
  2. Standalone_GCBM/tools/CompileGCBMResults/compileresults.py - to create compile_results.log file
  3. Standalone_GCBM/input_database/run_recliner2gcbm.bat - to create recliner_log.log file

Note: DVC pipeline uses batch files for all 6 stages.
Note 2: Added 2 kinds of CML workflow files:

  1. Triggered when the compiled db is pushed and hence only the postprocessing part is executed. The logs are here.
  2. Triggered when the input db is pushed and hence, it runs gcbm, then compiles the results and performs postprocessing. The 2 approaches used:
    a. Using the container rest_api_gcbm to run gcbm and compile results. Then switching to another job for the postprocessing. Logs here - container logs
    b. Using a windows runner to install deps using a requirements file (+ the GDAL whl file), run the gcbm and compile results. Next, postprocessing is performed in the next job which using a ubuntu runner. Logs here - windows logs

Type of change

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

How Has This Been Tested?

Refer to Note 2 in the description.

aldeav added 8 commits March 2, 2023 17:15
Signed-off-by: aldeav <akandoria@gmail.com>
Signed-off-by: aldeav <akandoria@gmail.com>
Signed-off-by: aldeav <akandoria@gmail.com>
Signed-off-by: aldeav <akandoria@gmail.com>
Signed-off-by: aldeav <akandoria@gmail.com>
Signed-off-by: aldeav <akandoria@gmail.com>
Signed-off-by: aldeav <akandoria@gmail.com>
@Simpleshell3
Copy link

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants