Skip to content

Commit 86b7d50

Browse files
hvanhaevhvanhaev
authored
Merge pull request #1 from cms-legacydata-validation/add-workflow
Add workflow
2 parents 761aee4 + 904db33 commit 86b7d50

File tree

4 files changed

+181
-2
lines changed

4 files changed

+181
-2
lines changed

‎.github/workflows/main.yml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
name: Test run on minikube
2+
on:
3+
- pull_request
4+
jobs:
5+
job1:
6+
runs-on: ubuntu-latest
7+
name: CASTOR argo workflow test on minikube
8+
steps:
9+
- uses: actions/checkout@v2
10+
- name: Start minikube
11+
uses: medyagh/setup-minikube@master
12+
- name: Set up persistent volume and storage pod
13+
run: |
14+
kubectl apply -f volumes.yaml
15+
- name: Set up argo
16+
run: |
17+
kubectl create ns argo
18+
kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo/stable/manifests/quick-start-postgres.yaml
19+
curl -sLO https://github.com/argoproj/argo/releases/download/v2.10.0-rc3/argo-linux-amd64
20+
chmod +x argo-linux-amd64
21+
mv ./argo-linux-amd64 $HOME/bin/argo
22+
argo version
23+
- name: Submit the argo test workflow
24+
run: |
25+
argo submit -n argo --wait argo-workflow.yaml
26+
argo get -n argo @latest
27+
- name: Check the output
28+
run: |
29+
argo logs -n argo @latest
30+
mkdir outputs
31+
echo Copy file with kubectl cp task-pv-pod:/mnt/data/ outputs/ :
32+
kubectl cp task-pv-pod:/mnt/data/ outputs/
33+
echo ls -l :
34+
ls -l
35+
- name: Upload the ouput as a github artifact
36+
uses: actions/upload-artifact@v2
37+
with:
38+
name: output
39+
path: outputs/

‎README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,15 @@
44

55
This repository contains code that needs to be used to properly read out CASTOR OpenData of Commissioning10, Run2010A, and Run2010B datasets. It includes two EDAnalyzer plugins that are created to correctly read the reconstructed CASTOR data objects and make validation plots. The needed python configuration files are also included, as well as a plotting script + ROOT files to check the output histograms.
66

7+
This repository containes also [a github action](.github/workflows/main.yml), which runs the test workflow on the CMS open data container using github free resources. It sets up a minikube environment and runs a workflow defined with argo workflow engine. The ouput is returned as a github artifcat. The workflow is triggered by a pull request. The number of events can be modified in [argo-workflow.yaml](argo-workflow.yaml).
8+
79
## Requirements
810

9-
The analysis needs to be run within a [CMS VM 2010](http://opendata.cern.ch/docs/cms-virtual-machine-2010) and the CMSSW_4_2_8_lowpupatch1 release.
11+
The analysis needs to be run within a [CMS VM 2010](http://opendata.cern.ch/docs/cms-virtual-machine-2010) and the CMSSW_4_2_8_lowpupatch1 release or in the CMS open data docker container with that release installed. For the usage in the container e.g. in a minikube environment, see defining and using a persistent volume and setting up argo in [the github action definition](.github/workflows/main.yml) and setting up the workflow definition in [argo-workflow.yaml](argo-workflow.yaml).
1012

1113
## Installation
1214

13-
After installing the CERN OpenData VM for 2010 data (version CMS-OpenData-1.1.2) you need to start up your VM and open the 'CMS shell'.
15+
After installing the CERN OpenData VM for 2010 data (version CMS-OpenData-1.1.4) you need to start up your VM and open the 'CMS shell'.
1416
First install the correct version of CMSSW and activate it:
1517

1618
cmsrel CMSSW_4_2_8_lowpupatch1

‎argo-workflow.yaml‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
apiVersion: argoproj.io/v1alpha1
2+
kind: Workflow
3+
metadata:
4+
generateName: castor-argo-
5+
spec:
6+
entrypoint: castor-argo
7+
volumes:
8+
- name: workdir
9+
hostPath:
10+
path: /mnt/data
11+
type: DirectoryOrCreate
12+
templates:
13+
- name: castor-argo
14+
# the order and inputs to the different steps:
15+
dag:
16+
tasks:
17+
- name: step1-data
18+
template: castor-step1
19+
arguments:
20+
parameters:
21+
- name: config
22+
value: "analyzer_cfg_Commissioning10.py"
23+
- name: dir
24+
value: "Commissioning10Analyzer"
25+
- name: step1-mc
26+
template: castor-step1
27+
arguments:
28+
parameters:
29+
- name: config
30+
value: "analyzer_cfg_Comm10MC.py"
31+
- name: dir
32+
value: "Commissioning10Analyzer"
33+
- name: step2-plot
34+
dependencies: [step1-mc, step1-data]
35+
template: castor-step2
36+
arguments:
37+
parameters:
38+
- name: config
39+
value: "drawValidationPlots_Commissioning10.py"
40+
41+
# analysis step definition
42+
- name: castor-step1
43+
inputs:
44+
parameters:
45+
- name: config
46+
- name: dir
47+
script:
48+
image: gitlab-registry.cern.ch/clange/cmssw-docker/cmssw_4_2_8_lowpupatch1
49+
command: [sh]
50+
source: |
51+
sudo chown $USER /mnt/vol
52+
source /opt/cms/entrypoint.sh
53+
git clone git://github.com/cms-legacydata-validation/CastorDataValidation.git
54+
mv CastorDataValidation/CMSSW_additional_packages.tar .
55+
tar -xvf CMSSW_additional_packages.tar
56+
scram b
57+
cd CastorDataValidation/{{inputs.parameters.dir}}/
58+
nev=10000
59+
eventline=$(grep maxEvents {{inputs.parameters.config}})
60+
sed -i "s/$eventline/process.maxEvents = cms.untracked.PSet( input = cms.untracked.int32($nev) )/g" {{inputs.parameters.config}}
61+
#comment the connection to the condition database on cvmfs, the condition data is read differently in the container
62+
sed -i "s/process.GlobalTag.connect/\#process.GlobalTag.connect/g" {{inputs.parameters.config}}
63+
cmsRun {{inputs.parameters.config}}
64+
ls -l
65+
cp CASTOR*.root /mnt/vol
66+
volumeMounts:
67+
- name: workdir
68+
mountPath: /mnt/vol
69+
70+
# plotting step definition
71+
- name: castor-step2
72+
inputs:
73+
parameters:
74+
- name: config
75+
script:
76+
image: gitlab-registry.cern.ch/clange/cmssw-docker/cmssw_4_2_8_lowpupatch1
77+
command: [sh]
78+
source: |
79+
sudo chown $USER /mnt/vol
80+
source /opt/cms/entrypoint.sh
81+
git clone git://github.com/cms-legacydata-validation/CastorDataValidation.git
82+
mv CastorDataValidation/CMSSW_additional_packages.tar .
83+
tar -xvf CMSSW_additional_packages.tar
84+
scram b
85+
cd CastorDataValidation/Plots/
86+
mv /mnt/vol/CASTOR*.root .
87+
sed -i "s/preventExit()/\#preventExit()/g" {{inputs.parameters.config}}
88+
python {{inputs.parameters.config}}
89+
ls -l *.pdf
90+
mv *.pdf /mnt/vol
91+
volumeMounts:
92+
- name: workdir
93+
mountPath: /mnt/vol
94+
95+

‎volumes.yaml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
apiVersion: v1
2+
kind: PersistentVolume
3+
metadata:
4+
name: task-pv-volume
5+
labels:
6+
type: local
7+
spec:
8+
storageClassName: manual
9+
capacity:
10+
storage: 10Gi
11+
accessModes:
12+
- ReadWriteOnce
13+
hostPath:
14+
path: "/mnt/data"
15+
---
16+
apiVersion: v1
17+
kind: PersistentVolumeClaim
18+
metadata:
19+
name: task-pv-claim
20+
spec:
21+
storageClassName: manual
22+
accessModes:
23+
- ReadWriteOnce
24+
resources:
25+
requests:
26+
storage: 3Gi
27+
---
28+
apiVersion: v1
29+
kind: Pod
30+
metadata:
31+
name: task-pv-pod
32+
spec:
33+
volumes:
34+
- name: task-pv-storage
35+
persistentVolumeClaim:
36+
claimName: task-pv-claim
37+
containers:
38+
- name: task-pv-container
39+
image: busybox
40+
command: ["tail", "-f", "/dev/null"]
41+
volumeMounts:
42+
- mountPath: /mnt/data
43+
name: task-pv-storage

0 commit comments

Comments
 (0)