Skip to content

Commit e27b687

Browse files
HumairAKHumairAK
authored
Merge pull request #76 from rimolive/rmartine-odh
UPSTREAM: <carry>: Add the upstream code rebase document
2 parents 49cab50 + 96941d2 commit e27b687

File tree

1 file changed

+186
-0
lines changed

1 file changed

+186
-0
lines changed

‎REBASE.opendatahub.md

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
# KFP -> DSP Rebase process
2+
3+
This document describes the process to upgrade Data Science Pipelines (DSP) code from a specific Kubeflow Pipelines (KFP) version tag. The following repositories must be updated, in the following order:
4+
5+
- https://github.com/opendatahub-io/data-science-pipelines
6+
- https://github.com/opendatahub-io/argo-workflows
7+
- https://github.com/opendatahub-io/data-science-pipelines-operator
8+
9+
## Checklist
10+
11+
- The new rebase branch has been created from the upstream tag
12+
- The new rebase branch includes relevant carries from target branch
13+
- The upstream tag is pushed to kubeflow/pipelines to ensure that build artifacts are versioned correctly
14+
15+
## Getting Started
16+
17+
### Data Science Pipelines repository
18+
19+
Preparing the local repo clone
20+
21+
Clone from a personal fork, and add the remote for upstream and opendatahub, fetching its branches:
22+
23+
```
24+
git remote add --fetch kubeflow https://github.com/kubeflow/pipelines
25+
git remote add --fetch opendatahub https://github.com/opendatahub-io/data-science-pipelines
26+
```
27+
28+
### Creating a new local branch for the new rebase
29+
30+
Branch the target release to a new branch:
31+
32+
```
33+
TAG=2.2.0
34+
git checkout -b rebase-$TAG $TAG
35+
```
36+
37+
Merge opendatahub(master) branch into the `rebase-\$TAG` branch with merge strategy ours. It discards all changes from the other branch (opendatahub/master) and create a merge commit. This leaves the content of your branch unchanged, and when you next merge with the other branch, Git will only consider changes made from this point forward. (Do not confuse this with ours conflict resolution strategy for recursive merge strategy, -X option.)
38+
39+
```
40+
git merge opendatahub/master
41+
```
42+
43+
This action will need to resolve some conflicts manually. Some recommentations when working in this task are:
44+
45+
* Dockerfiles are not expected to have any merge conflicts. We should have our dsp images stored in a separate path from the kfp ones.
46+
* Any changes in generated files (go.mod, go.sum, package.json, package-lock.json) should always prioritize upstream changes.
47+
* In case of changes in backend code that diverges completelly between kfp and dsp, you should use `git blame` to find the author(s) of the changes and work together to fix the conflicts. Do not try to fix by yourself, you are not alone in this.
48+
49+
After resolving all conflicts, remember to run a pre-flight check before going to the next task.
50+
51+
```
52+
pre-commit
53+
make unittest
54+
make functest
55+
```
56+
57+
### Create the Pull-Request in opendatahub-io/data-science-pipelines repository
58+
59+
Create a PR with the result of the previous tasks with the following description: `UPSTREAM <carry>: Rebase code to kfp x.y.z`
60+
61+
## Argo Workflows Repo
62+
63+
If the kfp code you are rebasing uses a newer Argo workflows version, you must update opendatahub-io/argo-workflows repository.
64+
65+
### Preparing the local repo clone
66+
67+
Clone from a personal fork, and add the remote for upstream and opendatahub, fetching its branches:
68+
69+
```
70+
git clone [email protected]:<user id>/argo-workflows
71+
git remote add --fetch argo https://github.com/argoproj/argo-workflows
72+
git remote add --fetch opendatahub https://github.com/opendatahub-io/argo-workflows
73+
```
74+
75+
### Creating a backup branch form the dsp repo
76+
77+
Argo Workflows git history diverges completely across versions, so it's important to create a backup branch from the current dsp repo in case we need to revert changes.
78+
79+
```
80+
git fetch opendatahub main
81+
git checkout -b dsp-backup opendatahub/main
82+
```
83+
84+
**NOTE:** Keep this branch for as long time as possible after Argo rebase, just in case we need to revert some changes.
85+
86+
### Creating a new local branch for the new rebase
87+
88+
Branch the target release to a new branch:
89+
90+
```
91+
TAG=v3.4.17
92+
git checkout -b argo-upgrade $TAG
93+
```
94+
95+
### Create the Pull-Request in opendatahub-io/argo-workflows repository
96+
97+
Create a PR with the result of the previous tasks with the following description: `Upgrade argo-workflows code to x.y.z`
98+
99+
## Data Science Pipelines Operator repository
100+
101+
### Apply the DataSciencePipelinesApplication CustomResource from the opendatahub-io/data-science-pipelines Pull-Request
102+
103+
With the Pull-Request opened in opendatahub-io/data-science-pipelines repository, you can get a DataSciencePipelinesApplication (DSPA) CustomResource with the resulting image builds from the bot comment like this.
104+
105+
```
106+
An OCP cluster where you are logged in as cluster admin is required.
107+
108+
The Data Science Pipelines team recommends testing this using the Data Science Pipelines Operator. Check here for more information on using the DSPO.
109+
110+
To use and deploy a DSP stack with these images (assuming the DSPO is deployed), first save the following YAML to a file named dspa.pr-76.yaml:
111+
112+
apiVersion: datasciencepipelinesapplications.opendatahub.io/v1alpha1
113+
kind: DataSciencePipelinesApplication
114+
metadata:
115+
name: pr-76
116+
spec:
117+
dspVersion: v2
118+
apiServer:
119+
image: "quay.io/opendatahub/ds-pipelines-api-server:pr-76"
120+
argoDriverImage: "quay.io/opendatahub/ds-pipelines-driver:pr-76"
121+
argoLauncherImage: "quay.io/opendatahub/ds-pipelines-launcher:pr-76"
122+
persistenceAgent:
123+
image: "quay.io/opendatahub/ds-pipelines-persistenceagent:pr-76"
124+
scheduledWorkflow:
125+
image: "quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-76"
126+
mlmd:
127+
deploy: true # Optional component
128+
grpc:
129+
image: "quay.io/opendatahub/mlmd-grpc-server:latest"
130+
envoy:
131+
image: "registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2"
132+
mlpipelineUI:
133+
deploy: true # Optional component
134+
image: "quay.io/opendatahub/ds-pipelines-frontend:pr-76"
135+
objectStorage:
136+
minio:
137+
deploy: true
138+
image: 'quay.io/opendatahub/minio:RELEASE.2019-08-14T20-37-41Z-license-compliance'
139+
Then run the following:
140+
141+
cd $(mktemp -d)
142+
git clone [email protected]:opendatahub-io/data-science-pipelines.git
143+
cd data-science-pipelines/
144+
git fetch origin pull/76/head
145+
git checkout -b pullrequest f5a03d13022b1e1ba3ba09129e840633982522ac
146+
oc apply -f dspa.pr-76.yaml
147+
More instructions here on how to deploy and test a Data Science Pipelines Application.
148+
```
149+
150+
### Fix the Data Science Pipelines Operator code
151+
152+
Check if there are any breaking changes, and fix the code whenever is needed
153+
One obvious change would be the tag references in params.env file
154+
155+
### Create the Pull-Request in opendatahub-io/argo-workflows repository
156+
157+
Create a PR with the result of the previous tasks with the following description: `Upgrade argo-workflows code to x.y.z`
158+
159+
### Create a Draft/WIP Pull Request in opendatahub-io/argo-workflows repository
160+
161+
**NOTE**: This is only to show the diff/changeset and accept feedback from the team before proceeding. DO NOT ACTUALLY MERGE THIS
162+
163+
Create a PR with the result of the previous tasks with the following description: `Upgrade argo-workflows code to x.y.z`
164+
165+
### Force-push Version Upgrade branch to main
166+
167+
Upon acceptance of the Draft PR (again, do not actually merge this), force the `opendatahub/main` branch to now match the upgrade version 'feature' branch:
168+
169+
```bash
170+
git push -f origin argo-upgrade:main
171+
```
172+
173+
Obviously, this will completely overwrite the git history of the `opendatahub/main` remote branch so please ensure a backup branch (`dsp-backup`) was created as instructed above
174+
175+
### Disclaimer / Future Work
176+
177+
This process this obviously very heavy-handed and destructive, and depends on there being no carries or downstream-only commits. We should adjust the procedure to account for this
178+
179+
## Followup work
180+
QE team has a Jenkins Job that can help test some basic features. Ask Diego Lovison to trigger the Jenkins job. You might need to create a separate branch DSPO from the code rebase to change `params.env` file with the values of the generated images from the previous PRs to run this Jenkins job.
181+
182+
It is also good creating a follow-up task in JIRA to coordinate with QE to run some regression tests before merging the PRs.
183+
184+
## Updating with rebase.sh (experimental)
185+
WIP
186+

0 commit comments

Comments
 (0)