Skip to content

Commit edfcee8

Browse files
arooshaparooshap
authored
Merge pull request #1545 from novicecpp/helm_crabserver_resource_groups_pr
CRABServer helm: deploy all crab services in single helm chart
2 parents 5d3c55c + 5001477 commit edfcee8

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

47 files changed

+1265
-311
lines changed

‎.gitignore‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ hmac
1515
# 3rd party packages
1616
kubernetes-prometheus
1717

18-
# docker area
18+
# docker area
1919
docker/sqoop/awg-ETL-crons/
2020

2121
# skip configs/secrets
@@ -81,3 +81,5 @@ kubernetes/rucio/futureprod-db.yaml
8181

8282

8383
kubernetes/rucio/dev_grafana_password.txt
84+
85+
!helm/crabserver/config

‎helm/crabserver/Chart.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,9 @@ type: application
1515
# This is the chart version. This version number should be incremented each time you make changes
1616
# to the chart and its templates, including the app version.
1717
# Versions are expected to follow Semantic Versioning (https://semver.org/)
18-
version: 0.1.11
18+
version: 0.2.0
1919

2020
# This is the version number of the application being deployed. This version number should be
2121
# incremented each time you make changes to the application. Versions are not expected to
2222
# follow Semantic Versioning. They should reflect the version the application is using.
23-
appVersion: "v3.240731"
23+
appVersion: "v3.240809"

‎helm/crabserver/README.md‎

Lines changed: 56 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,63 @@
1-
## Deploy CRAB
1+
# Deploy CRAB
22

3-
At the moment CRAB team is transitioning to using helm. However, we do noy like
4-
to use helm for deployment. See this
5-
[comment](https://github.com/dmwm/CRABServer/issues/7843#issuecomment-2025085120)
6-
for some context.
3+
In this helm chart, we separate k8s resources into 3 resource groups:
74

8-
We like to use helm for templating, using the helm charts to generate the
9-
manifest file, then applying it with `kubectl apply`.
5+
1. `crabserver` manifest for CRAB REST, including `crabserver-canary`.
6+
2. `logPipline`, the log pipeline which have logstash, filebeat and it config.
7+
3. `ciServiceAccount` with deployment permission, for deploying new image from CI. Note that only cluster admin could not `rolebinding` in testbed and production cluster.
108

11-
We therefore ask cmsweb operators to deploy crab following the procedure at
12-
[1].
9+
CRAB team will only handle the first resources group, `crabserver`, to deploy a new service version or modify how we start our service. The rest we kindly ask cluster admin to take care of.
1310

14-
Changes to `../../kubernetes/cmsweb/services/crabserver.yaml` should not be committed.
11+
## Configuration (helm's value)
1512

16-
---
13+
There are helm value, `enabled`, that is used to control which resource group will be generated.
1714

18-
[1] https://cmscrab.docs.cern.ch/technical/crab-rest/deploy.html#deploy-on-kubernetes-use-helm-template-to-generate-manifest-preferred
15+
#### enabled
1916

17+
Default: `true`
18+
19+
Generate `crabserver` resources group.
20+
21+
#### canary.enabled
22+
23+
Default: `false`
24+
25+
Enable canary deployment, a.k.a. the `crabserver-canary` Deployment manifest.
26+
27+
#### logPipeline.enabled
28+
29+
Default: `false`
30+
31+
Generate `logPipeline` resources group.
32+
33+
#### ciServiceAccount.enabled
34+
35+
Generate `ciServiceAccount` resources group. Need cluster admin for applying `role` and `rolebinding` of k8s service account.
36+
37+
## Deploy
38+
39+
### For cluster admins
40+
41+
To deploy everything on new testbed and production cluster:
42+
43+
1. Deploy `crabserver-secrets` credential.
44+
45+
2. Deploy all services:
46+
```
47+
helm install crabserver . -f values.yaml -f values-clusteradmin.yaml --set environment=preprod
48+
```
49+
Please change `environment=preprod` to the cluster you are deploying (`preprod` for testbed and preprod cluster, `prod` for production cluster).
50+
51+
Please leave test cluster to CRAB operators.
52+
53+
### For CRAB operators
54+
55+
Please consult [Deploying CRAB REST](https://cmscrab.docs.cern.ch/crab-components/crab-rest/deploy.html).
56+
57+
## Regarding deploy with `helm install` command
58+
59+
CRAB team does not use helm for deployment. We like to use helm for templating, using the helm charts to generate the manifest file, then applying it with `kubectl apply`. See this [comment](https://github.com/dmwm/CRABServer/issues/7843#issuecomment-2025085120) for some context.
60+
61+
However, it should not have any conflict if cluster admins run `helm install` **before** CRAB operators apply new manifests with `kubectl apply`.
62+
63+
In case of conflict, feel free to purge all resources and reinstall with helm again.

‎helm/crabserver/apply_logstash.sh‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#!/bin/bash
2+
set -x
3+
set -euo pipefail
4+
ENV=test
5+
kubectl create configmap logstash-crab --from-file=config/${ENV}/logstash/logstash.conf --from-file config/${ENV}/logstash/logstash.yml --dry-run=client -oyaml | kubectl apply -f -
6+
kubectl delete pod $(kubectl get pod --no-headers -o custom-columns=":metadata.name" | grep logstash)

‎helm/crabserver/config/preprod/config.py‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
from WMCore.Configuration import Configuration
2+
import re
3+
import socket
4+
import time
5+
import os
6+
7+
myhost = socket.getfqdn().lower()
8+
9+
conf = Configuration()
10+
main = conf.section_('main')
11+
srv = main.section_('server')
12+
srv.thread_pool = int(os.getenv('CRABSERVER_THREAD_POOL', 15))
13+
main.application = 'crabserver'
14+
main.port = 8270
15+
main.index = 'data'
16+
17+
main.authz_defaults = { 'role': None, 'group': None, 'site': None }
18+
main.section_('tools').section_('cms_auth').key_file = "%s/auth/crabserver/header-auth-key" % __file__.rsplit('/', 3)[0]
19+
20+
app = conf.section_('crabserver')
21+
app.admin = 'cms-service-webtools@cern.ch'
22+
app.description = 'CRABServer RESTFull API'
23+
app.title = 'CRABRESTFull'
24+
25+
views = conf.section_('views')
26+
ui = views.section_('ui')
27+
ui.object = 'CRABInterface.Pages.FrontPage.FrontPage'
28+
29+
data = views.section_('data')
30+
data.object = 'CRABInterface.RESTBaseAPI.RESTBaseAPI'
31+
data.phedexurl = 'https://cmsweb.cern.ch/phedex/datasvc/xml/prod/'
32+
data.dbsurl = 'http://cmsdbsprod.cern.ch/cms_dbs_prod_global/servlet/DBSServlet'
33+
data.defaultBlacklist = ['T0_CH_CERN']
34+
data.serverhostcert = "%s/auth/crabserver/dmwm-service-cert.pem" % __file__.rsplit('/', 3)[0]
35+
data.serverhostkey = "%s/auth/crabserver/dmwm-service-key.pem" % __file__.rsplit('/', 3)[0]
36+
data.credpath = '%s/state/crabserver/proxy/' % __file__.rsplit('/', 4)[0]
37+
data.backend = 'oracle'
38+
data.db = 'CRABServerAuth.dbconfig'
39+
data.s3 = 'CRABServerAuth.s3'
40+
data.workflowManager = 'HTCondorDataWorkflow'
41+
42+
data.extconfigurl = 'http://gitlab.cern.ch/crab3/CRAB3ServerConfig/raw/master/cmsweb-rest-config.json'
43+
44+
data.loggingLevel = 10
45+
data.loggingFile = '%s/logs/crabserver/CRAB-%s.log' % (__file__.rsplit('/', 4)[0], myhost)
46+
data.keptLogDays = 7
47+
data.mode = "cmsweb-preprod"
48+
49+
data.delegateDN = "/DC=ch/DC=cern/OU=computers/CN=crab-(preprod|prod)-tw(01|02).cern.ch|/DC=ch/DC=cern/OU=computers/CN=crab-dev-tw(01|02|03|04).cern.ch|/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=cmscrab/CN=(817881|373708)/CN=Robot: cms crab|/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=crabint1/CN=373708/CN=Robot: CMS CRAB Integration 1"

kubernetes/cmsweb/monitoring/crab/logstash.conf renamed to helm/crabserver/config/preprod/logstash/logstash.conf

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -176,8 +176,8 @@ filter {
176176
}
177177

178178
grok {
179-
# [07/Dec/2022:20:06:47] crabserver-56c69685b6-477jr 188.184.91.103:41924 "GET /crabserver/devthree/info HTTP/1.1" 200 OK [data: 9354 in 103 out 4870 us ] [auth: ok "/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=tseethon/CN=856006/CN=Thanayut Seethongchuen" "" ] [ref: "https://cmsweb-test12.cern.ch" "Mozilla/5.0 (X11; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0" ] Type=cherrypylog
180-
+ match => { "message" => '\[%{NOTSPACE:timestamp_temp}\] %{DATA:backend} %{IPORHOST:clientip}(|:%{NUMBER}) "%{WORD:method} %{NOTSPACE:request} %{DATA:httpversion}" %{NUMBER:code:int} %{DATA} \[data: (-|%{NUMBER:bytes_sent:int}) in (-|%{NUMBER:bytes_received:int}) out (-|%{NUMBER:time_spent_ms:int}) us \] \[auth: %{DATA} "%{DATA:dn}".*\] \[ref: "%{DATA}.*" "%{DATA:client}" \]' }
179+
# [07/Dec/2022:20:06:47] crabserver-56c69685b6-477jr 188.184.91.103:41924 "GET /crabserver/devthree/info HTTP/1.1" 200 OK [data: 9354 in 103 out 4870 us ] [auth: ok "/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=tseethon/CN=856006/CN=Thanayut Seethongchuen" "" ] [ref: "https://cmsweb-test12.cern.ch" "Mozilla/5.0 (X11; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0" ] - Podname=crabserver-6cf49bcfff-bxvb6 Type=cherrypylog
180+
match => { "message" => '\[%{NOTSPACE:timestamp_temp}\] %{DATA:backend} %{IPORHOST:clientip}(|:%{NUMBER}) "%{WORD:method} %{NOTSPACE:request} %{DATA:httpversion}" %{NUMBER:code:int} %{DATA} \[data: (-|%{NUMBER:bytes_sent:int}) in (-|%{NUMBER:bytes_received:int}) out (-|%{NUMBER:time_spent_ms:int}) us \] \[auth: %{DATA} "%{DATA:dn}".*\] \[ref: "%{DATA}.*" "%{DATA:client}" \]' }
181181
}
182182
grok {
183183
match => { "request" => '/%{WORD:system}%{UNIXPATH:uri_path}%{URIPARAM:uri_params}?' }

kubernetes/cmsweb/monitoring/crab/logstash.yml renamed to helm/crabserver/config/preprod/logstash/logstash.yml

File renamed without changes.

‎helm/crabserver/config/prod/config.py‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
from WMCore.Configuration import Configuration
2+
import re
3+
import socket
4+
import time
5+
import os
6+
7+
myhost = socket.getfqdn().lower()
8+
9+
conf = Configuration()
10+
main = conf.section_('main')
11+
srv = main.section_('server')
12+
srv.thread_pool = int(os.getenv('CRABSERVER_THREAD_POOL', 15))
13+
main.application = 'crabserver'
14+
main.port = 8270
15+
main.index = 'data'
16+
17+
main.authz_defaults = { 'role': None, 'group': None, 'site': None }
18+
main.section_('tools').section_('cms_auth').key_file = "%s/auth/crabserver/header-auth-key" % __file__.rsplit('/', 3)[0]
19+
20+
app = conf.section_('crabserver')
21+
app.admin = 'cms-service-webtools@cern.ch'
22+
app.description = 'CRABServer RESTFull API'
23+
app.title = 'CRABRESTFull'
24+
25+
views = conf.section_('views')
26+
ui = views.section_('ui')
27+
ui.object = 'CRABInterface.Pages.FrontPage.FrontPage'
28+
29+
data = views.section_('data')
30+
data.object = 'CRABInterface.RESTBaseAPI.RESTBaseAPI'
31+
data.phedexurl = 'https://cmsweb.cern.ch/phedex/datasvc/xml/prod/'
32+
data.dbsurl = 'http://cmsdbsprod.cern.ch/cms_dbs_prod_global/servlet/DBSServlet'
33+
data.defaultBlacklist = ['T0_CH_CERN']
34+
data.serverhostcert = "%s/auth/crabserver/dmwm-service-cert.pem" % __file__.rsplit('/', 3)[0]
35+
data.serverhostkey = "%s/auth/crabserver/dmwm-service-key.pem" % __file__.rsplit('/', 3)[0]
36+
data.credpath = '%s/state/crabserver/proxy/' % __file__.rsplit('/', 4)[0]
37+
data.backend = 'oracle'
38+
data.db = 'CRABServerAuth.dbconfig'
39+
data.s3 = 'CRABServerAuth.s3'
40+
data.workflowManager = 'HTCondorDataWorkflow'
41+
42+
data.extconfigurl = 'http://gitlab.cern.ch/crab3/CRAB3ServerConfig/raw/master/cmsweb-rest-config.json'
43+
44+
data.loggingLevel = 10
45+
data.loggingFile = '%s/logs/crabserver/CRAB-%s.log' % (__file__.rsplit('/', 4)[0], myhost)
46+
data.keptLogDays = 7
47+
data.mode = "cmsweb-prod"

0 commit comments

Comments
 (0)