Skip to content

Commit f6d1f49

Browse files
authored
Merge pull request #125 from HarperFast/dev/add-harperdb-sync-helper-script
Add harperdb repo sync helper script
2 parents fd2e496 + 249b77d commit f6d1f49

File tree

2 files changed

+189
-21
lines changed

2 files changed

+189
-21
lines changed

CONTRIBUTING.md

Lines changed: 15 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -55,28 +55,20 @@ These are the steps @Ethan-Arrowood has been following to synchronize the reposi
5555
>
5656
> # Only fetch `main` branch
5757
> git config remote.old.fetch '+refs/heads/main:refs/remotes/old/main'
58-
>
59-
> git fetch old
6058
> ```
6159
62-
1. Ensure we have the latest commits from the old repository with `git fetch old`
63-
2. Make sure local `main` branch is up to date `git pull origin main`
64-
3. Create a new branch for the synchronization `git checkout -b sync-mmddyyyy`
65-
4. Create list of commits using `git rev-list --reverse --first-parent $(start-commit)..old/main > commits-to-pick.txt`
66-
- The start-commit will be excluded so always include the last commit from `old/main` that was synchronized previously.
67-
- The hash for that commit is recorded at the end of this section and should be updated after each synchronization
68-
5. Go through commits one-by-one, from the top to the bottom of the `commits-to-pick.txt` file, following these steps:
69-
1. If `git rev-parse $(commit)^2 &>/dev/null 2>&1` exits with 0 then it's a merge commit
70-
1. Execute `git cherry-pick -m 1 $(commit)`
71-
2. Else it's a regular commit
72-
1. Execute `git cherry-pick $(commit)`
73-
3. If either cherry-pick command results in a non-zero exit code that means there is a merge conflict
74-
1. If the conflict is a content, resolve it manually and `git add` the file
75-
- Example: `CONFLICT (content): Merge conflict in package.json`
76-
2. Else if the conflict is a modify/delete then likely `git rm` the file
77-
- Example: `CONFLICT (modify/delete): unitTests/bin/copyDB-test.js deleted in HEAD and modified in f75d9170b`
78-
3. Then check `git status`, if there is nothing you can `git cherry-pick --skip`
79-
- Note: in this circumstance, running `git cherry-pick --continue` results in a non-zero exit code with the message `The previous cherry-pick is now empty, possibly due to conflict resolution.` Maybe we use this to then run `--skip`? Or maybe there is a way to parse the output of previous `git status` step?
60+
1. Make sure local `main` branch is checked out and clean `git checkout main && git status`.
61+
2. Copy the [latest previously-synced commit hash from this file](#last-synchronized-commit).
62+
3. Run the sync-commits helper script: `dev/sync-commits.js <previously-synced-commit-hash>`
63+
4. For each commit the script lists, run the `git cherry-pick ...` command it suggests.
64+
- NB: Some of these may have `-m 1` params to handle merge commits correctly.
65+
5. If either cherry-pick command results in a non-zero exit code that means there is a merge conflict.
66+
1. If the conflict is a content, resolve it manually and `git add` the file
67+
- Example: `CONFLICT (content): Merge conflict in package.json`
68+
2. Else if the conflict is a modify/delete then likely `git rm` the file
69+
- Example: `CONFLICT (modify/delete): unitTests/bin/copyDB-test.js deleted in HEAD and modified in f75d9170b`
70+
3. Then check `git status`, if there is nothing you can `git cherry-pick --skip`
71+
- Note: in this circumstance, running `git cherry-pick --continue` results in a non-zero exit code with the message `The previous cherry-pick is now empty, possibly due to conflict resolution.` Maybe we use this to then run `--skip`? Or maybe there is a way to parse the output of previous `git status` step?
8072
6. After all commits have been picked, manually check that everything brought over was supposed to be. Look out for any source code we do not want open-sourced or things like unit tests which we are actively migrating separately (and will eventually include as part of the synchronization process)
8173
- The GitHub PR UI is useful for this step; but make sure to leave the PR as a draft until all synchronization steps are complete
8274
7. Once everything looks good, run `npm run format:write` to ensure formatting is correct
@@ -87,7 +79,9 @@ These are the steps @Ethan-Arrowood has been following to synchronize the reposi
8779
12. Push all changes and open the PR for review
8880
13. Merge using a Merge Commit so that all relative history is retained and things like the formatting change hash stays the same as recorded.
8981
90-
Last Synchronized Commit: `e1ea920d74e919140ae89d5ca4d75614c10c2925`
82+
### Last Synchronized Commit
83+
84+
`e1ea920d74e919140ae89d5ca4d75614c10c2925`
9185
9286
## Code of Conduct
9387

dev/sync-commits.js

Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
#!/usr/bin/env node
2+
3+
const { execSync, exec } = require('node:child_process');
4+
const fs = require('node:fs');
5+
6+
/* This script should be deleted someday. It is for syncing commits from the
7+
* old HarperDB closed-source repository while the Harper devs were
8+
* transitioning the platform to open source. See CONTRIBUTING.md for more
9+
* details. - WSM 2026-01-20
10+
*/
11+
12+
function letsBail(exitCode, syncBranch = null) {
13+
execSync('git checkout main', { stdio: 'ignore' });
14+
if (syncBranch) {
15+
execSync(`git branch -D ${syncBranch}`, { stdio: 'ignore' });
16+
}
17+
process.exit(exitCode);
18+
}
19+
20+
function gitRemotes() {
21+
let remotesList = execSync('git remote -v')
22+
.toString()
23+
.trim()
24+
.split('\n')
25+
.map((r) => r.split('\t'));
26+
let remotes = {};
27+
remotesList.forEach(([name, urlAndType]) => {
28+
if (remotes[name] == null) {
29+
remotes[name] = {};
30+
}
31+
let [url, type] = urlAndType.split(' ');
32+
type = type.replace('(', '').replace(')', '');
33+
remotes[name][type] = url;
34+
});
35+
return remotes;
36+
}
37+
38+
function verifyRemote(remoteName, remoteUrl) {
39+
let remotes = gitRemotes();
40+
if (!Object.hasOwn(remotes, remoteName)) {
41+
return false;
42+
}
43+
if (!(Object.hasOwn(remotes[remoteName], 'fetch') && Object.hasOwn(remotes[remoteName], 'push'))) {
44+
return false;
45+
}
46+
return remotes[remoteName]['fetch'] === remoteUrl && remotes[remoteName]['push'] === remoteUrl;
47+
}
48+
49+
function isOldRemoteConfigured() {
50+
return verifyRemote('old', 'git@github.com:HarperFast/harperdb.git');
51+
}
52+
53+
function isOriginRemoteConfigured() {
54+
return verifyRemote('origin', 'git@github.com:HarperFast/harper.git');
55+
}
56+
57+
function isBranchCheckedOut(branchName) {
58+
let branch = execSync(`git branch --show-current`).toString().trim();
59+
return branch === branchName;
60+
}
61+
62+
function fetchCommits(remoteName) {
63+
exec(`git fetch ${remoteName}`, (error, _stdout, _stderr) => {
64+
// Note that git outputs all kinds of non-errors on stderr, so we don't
65+
// want to assume something went wrong if there's anything written there.
66+
if (error) {
67+
console.error(`git exited with error '${error.message}' fetching ${remoteName} commits`);
68+
letsBail(error.code);
69+
}
70+
});
71+
}
72+
73+
function pullRemoteBranch(remoteName, branchName) {
74+
fetchCommits(remoteName);
75+
exec(`git merge ${remoteName}/${branchName}`, (error, _stdout, stderr) => {
76+
if (error) {
77+
console.error(`git exited with error '${error.message}' merging origin/main`);
78+
letsBail(error.code);
79+
}
80+
if (stderr) {
81+
console.error(`git error merging origin/main: ${stderr}`);
82+
letsBail(6);
83+
}
84+
});
85+
}
86+
87+
function checkoutNewBranch(branchName) {
88+
exec(`git checkout -b ${branchName}`, (error, _stdout, stderr) => {
89+
if (error) {
90+
console.error(`git exited with error '${error.message}' creating branch ${branchName}`);
91+
letsBail(error.code, branchName);
92+
}
93+
if (stderr && !stderr.startsWith('Switched to a new branch')) {
94+
console.error(`git error creating branch ${branchName}: ${stderr}`);
95+
letsBail(7, branchName);
96+
}
97+
});
98+
}
99+
100+
function ensureValidConfig() {
101+
process.stdout.write('Verifying git config... ');
102+
if (!isOldRemoteConfigured()) {
103+
process.stdout.write('❌');
104+
console.error('old remote not configured correctly.');
105+
console.error(
106+
'Run `git remote add old git@github.com:HarperFast/harperdb.git` to configure it (you may have to remove the old remote first with `git remote rm old`).'
107+
);
108+
process.exit(2);
109+
}
110+
if (!isOriginRemoteConfigured()) {
111+
console.log('❌');
112+
console.error('origin remote not configured correctly.');
113+
console.error(
114+
'Run `git remote add origin git@github.com:HarperFast/harper.git` to configure it (you may have to remove the origin remote first with `git remote rm origin`).'
115+
);
116+
process.exit(3);
117+
}
118+
if (!isBranchCheckedOut('main')) {
119+
console.log('❌');
120+
console.error('main branch not checked out. Run `git checkout main` to check it out.');
121+
process.exit(4);
122+
}
123+
console.log('✅');
124+
}
125+
126+
function generateCommitsToPick(startCommit) {
127+
const commits = execSync(`git rev-list --reverse --first-parent ${startCommit}..old/main`)
128+
.toString()
129+
.trim()
130+
.split('\n');
131+
// write to file in case a human needs to take over
132+
fs.writeFileSync('commits-to-pick.txt', commits.join('\n') + '\n');
133+
return commits;
134+
}
135+
136+
function isMergeCommit(commit) {
137+
try {
138+
execSync(`git rev-parse ${commit}^2`, { stdio: 'ignore' });
139+
} catch {
140+
return false;
141+
}
142+
return true;
143+
}
144+
145+
function doItRockapella(startCommit) {
146+
process.stdout.write('Finding commits to sync... ');
147+
fetchCommits('old');
148+
pullRemoteBranch('origin', 'main');
149+
const syncDate = new Date();
150+
const month = String(syncDate.getMonth() + 1).padStart(2, '0');
151+
const day = String(syncDate.getDate()).padStart(2, '0');
152+
checkoutNewBranch(`sync-${month}${day}${syncDate.getFullYear()}`);
153+
const commits = generateCommitsToPick(startCommit);
154+
console.log('✅');
155+
console.log(`\n${commits.length} commits found:`);
156+
for (const commit of commits) {
157+
if (isMergeCommit(commit)) {
158+
console.log(`${commit} (merge): git cherry-pick -m 1 ${commit}`);
159+
} else {
160+
console.log(`${commit}: git cherry-pick ${commit}`);
161+
}
162+
}
163+
}
164+
165+
function run(startCommit) {
166+
if (!startCommit) {
167+
console.error(`No start commit specified. Specify a commit hash or tag: sync-commits.js <commit hash or tag>`);
168+
letsBail(1);
169+
}
170+
ensureValidConfig();
171+
doItRockapella(startCommit);
172+
}
173+
174+
run(process.argv[2]);

0 commit comments

Comments
 (0)