Using git to cherry-pick a set of changes from a branch to a new branch on b4b-dev for only some files

[ekluzek]

Introduction

Making small commits can make it easy to move just a few changes from one branch to another without bringing the entire branch of changes in. When it's just a few commits this is easy to do. You want to go through them in reverse order, so oldest first.

However, you might have a case where a longer term branch has a lot of changes and a lot of commits -- and you only want to bring changes for a few select files -- cherry-picking the right commits is a bigger deal.

Example, cherry-picking commits for some files on mpi_scan branch to another b4b-dev branch

Here I give the process I used for cherry-picking the commits for a few specific files from my mpi_scan ($BRANCH) 00branch to a new b4b-dev ($BASE) branch I'll call "decomp_mpi_scan_move_to_b4b" ($MERGE_BRANCH). Here I want a new branch to $BASE that brings in some of the changes on the $BRANCH branch for just some files. There's 169 commits on the $BRANCH branch, but I only want some of those to a few of the files. As you see below the list of commits I want to bring over is 42, which is still a bunch. And the reason to have some automated process to do it rather than identify all of them by hand.

Here's the basic sequence:

1. Environment variables to use in later steps
2. Merge branches to the same point
3. Figure out list of commits
4. Verify the list is right, and remove merges
5. Do cherry-pick on list in reverse order
6. Do continue and/or skip to move along the list of commits
7. Verify the changes are correct and just to the files you wanted

Environment variables to use in later steps

# This is for tcsh
setenv BRANCH mpi_scan
setenv BASE b4b-dev
setenv CHERRYPICK_BRANCH decomp_mpi_scan_move_to_b4b
# The name of the remote for my fork
setenv MYREMOTE ekluzek
# The files that I want to do the cherry-pick for
set MYFILES = ( src/main/decomp* src/cpl/share_esmf/lnd_set_decomp_and_domain.F90 )

Merge branches to the same point

I want the new branch to come to b4b-dev, so I'm going to update the base branch to it, and then use the latest b4b-dev as the comparison.

# Make sure $BASE branch is updated to the latest
git checkout $BASE
git pull escomp $BASE
# Make sure the $BRANCH is updated to the latest on the fork as well as to $BASE
git checkout $BRANCH
git pull MYREMOTE $BRANCH
git merge $BASE
# Checkout the branch to do the cherry-pick on
git checkout -b $CHERRYPICK_BRANCH $BASE

Figure out list of commits

Do a git log for the files on the branch and on the base branch and compare

git log $BASE $MYFILES > base.log
git log $BRANCH $MYFILES > branch.log
diff branch.log base.log > branchlog.diff

The branchlog.diff file then has the list of commits that are different in the branch.

Verify the list is right, and remove merges

The branchlog.diff file should just have a list of "<" differences. So you can go through the file and make sure that's the case. The other thing that might be in there are merges to the $BASE branch and/or other branches that you probably don't want in your cherry-pick operation. To remove them you can search for 'Merge: " and remove the commits that have to do with a merge. There might be merges you want to keep, but you'll need to include an "-m" option in the final cherry-pick operation below. The kind of merges you might keep are for branches that weren't merged to $BASE and are only on the $BRANCH.

At this step I also removed commits that I didn't want to come into $BASE. For example, I removed several commits that had to do with memory handling that I didn't want to bring into $BASE.

$EDITOR branchlog.diff.     # get rid of the merge commits and any commits that you can tell you don't want in the cherry-pick
grep commit branchlog.diff | awk '{print $3}' > commitlist.txt. # This is the list of hashes for the commits you want to cherry-pick

One way to check the list of commits is to use "git show" to show what files are changed, and/or show the files modified to make sure you are getting the right changes.

# This is for tcsh (NOTE tac is used to reverse the order)
foreach hash ( `tac commitlist.txt` )
   git show --stat $hash
end

[! NOTE]
While doing the above I noticed a commit that also editted clm_initialize so I used the show on it to make sure I wanted those changes to clm_initializeMod.F90.

git show 4bc13a36c11833aa7769aeb41472a014ac4edc08

There were also two commits that did something to the TestDecompInit.F90 self tester, so I made sure both of those were fine.

git show 469c1083163a1c9f48e281a3d70a5144da9bf101
git show c87adff765126e2d3523b6b33828e9b55988e622

With these I decided that I'll leave these commits in. But, may cancel the changes to this file in case they cause problems. The other changes are important so I wanted to make sure they were in place.

Do cherry-pick on list in reverse order

My final list of commits was 34, which is a fairly reasonable list to deal with. To do the cherry-pick in reverse order I use "tac" (the reverse order of UNIX cat) as follows.

git cherry-pick `tac commitlist.txt`

Do continue and/or skip to move along the list of commits

After doing the initial cherry-pick above I see the following output:

Auto-merging src/cpl/share_esmf/lnd_set_decomp_and_domain.F90
CONFLICT (content): Merge conflict in src/cpl/share_esmf/lnd_set_decomp_and_domain.F90
error: could not apply 3a3251981... Start adding timers to lnd_set_decomp_and_domain_from_readmesh
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".

This means I need to edit the file "src/cpl/share_esmf/lnd_set_decomp_and_domain.F90" for conflicts. I then continue along with...

$EDITOR src/cpl/share_esmf/lnd_set_decomp_and_domain.F90
git add src/cpl/share_esmf/lnd_set_decomp_and_domain.F90
git cherry-pick --continue

!NOTE: git status also shows you what you need to do:

git status
On branch decomp_mpi_scan_move_to_b4b
Cherry-pick currently in progress.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   src/cpl/share_esmf/lnd_set_decomp_and_domain.F90

As you go along you may run into commits that end up being empty, in these cases you'll use the "--skip" option to remove those commits.

For this case, probably especially because there were changes I didn't want to bring in there were over 20 continues, and 3 skips, and took about 3 hours.

Verify the changes are correct and just to the files you wanted

Steps for this are:

1. Compare to $BRANCH for $MYFILES
2. Compare other files changed
3. Compare to $BASE
4. Run normal testing on the new branch

First we want to compare the resulting cherry-pick to the $BRANCH and make sure the changes are correct for the files you intended to change and that any differences with $BRANCH are expected.

git diff $BRANCH $MYFILES

The differences I saw there had to do with comments, memory checking and for code timers. I brought in some code comments by hand that I saw missing, and I made sure the code timers were correct (they have a start and stop for a code section). So I committed those changes.

Next compare the other file that was changed in the cherry-pick. To first get the list and then compare do a git diff from $BASE to $CHERRYPICK_BRANCH to show that clm_initializeMod.F90 also had a few changes. And then do a diff on it

git diff --stat $BASE...$CHERRYPICK_BRANCH | head
 src/cpl/share_esmf/lnd_set_decomp_and_domain.F90 |  70 ++++++-
 src/main/clm_initializeMod.F90                   |   6 +-
 src/main/decompInitMod.F90                       | 255 ++++++++++++++++-------
 src/main/decompMod.F90                           | 191 ++++++++++++++++-
 4 files changed, 428 insertions(+), 94 deletions(-)

git diff $BRANCH src/main/clm_initializeMod.F90

The differences were mostly what I wanted, there was just one change I moved by hand to the new branch.

The third step is to compare to the $BASE branch and make sure it looks like the correct changes are there and nothing that shouldn't be.

git diff $BASE

Here I noticed that I didn't want all the changes to the timers and a few other changes that I didn't want to pull in right now.