docker: project layout updates (bug 1940614) by shtrom · Pull Request #23 · mozilla-conduit/git-hg-sync

shtrom · 2025-02-04T01:22:29Z

Setup and document the project so a standalone syncer can be stood up and manually tested in isolation.

This also adds support for overriding the parameters of the Pulse queue via environment variables, as needed by mozilla-conduit/suite#66.

Looking forwards: more generally, we'll have to override repository and branch configuration per deployment environment. Doing so via env variables will quickly explode. Another option would be to bake (all) per-env configuration files within the source here and into the container image, and select which configuration to use via a single env variable when deploying the container.

This is left for future work, in #24.

shtrom · 2025-02-04T05:55:03Z

config-docker.toml

+[[tracked_repositories]]
+name = "firefox-releases"
+url = "/home/fbessou/dev/MOZI/fake-forge/git/firefox-releases"
+
+[[branch_mappings]]
+source_url = "/home/fbessou/dev/MOZI/fake-forge/git/firefox-releases"
+branch_pattern = "^(esr\\d+)$"
+destination_url = "/home/fbessou/dev/MOZI/fake-forge/hg/mozilla-\\1"
+destination_branch = "default"
+
+[[tag_mappings]]
+source_url = "/home/fbessou/dev/MOZI/fake-forge/git/firefox-releases"
+tag_pattern = "^FIREFOX_BETA_(\\d+)_(BASE|END)$"
+destination_url = "/home/fbessou/dev/MOZI/fake-forge/hg/mozilla-beta"
+tags_destination_branch = "tags"


We'll need to populate something more local at some point.

cgsheeh

Looks great!

README.md

shtrom · 2025-02-07T05:59:03Z

@cgsheeh git-hg-sync PR

shtrom · 2025-03-18T05:55:24Z

@cgsheeh @zzzeid ok, I think we're good for this one.

create_clones.sh

group branch and tag mappings by target

README.md

zzzeid · 2025-03-21T18:20:43Z

config-docker.toml

+[[branch_mappings]]
+source_url = "/clones/test-repo-git"
+branch_pattern = "beta"
+destination_url = "/clones/mozilla-beta" # also, m-c


We do have a dev conduit-testing/m-c repo, so would be good to be more explicit about this here. But not sure what this comment means?

Hum, I think this was a note-to-self more than anything else. It evolved into #24.

The docker-local test setup uses http://github.com/mozilla-conduit/test-repo as a base, and create all the test repos, so it's not really M-C. My reasoning was that it would be overkill [most of the time] to bandy around multiple copies of M-C.

zzzeid · 2025-03-21T18:22:48Z

create_clones.sh

+# make git status and friends faster:
+# mv .git/hooks/fsmonitor-watchman.sample .git/hooks/query-watchman
+# git config core.fsmonitor .git/hooks/query-watchman
+# add git origin
+# git remote add git_chatzilla http://github.com/djangoliv/chatzilla.git


Should this code be removed?

Fixed. I left the fsmonitor bit as I suspect it could be useful later, but the rest of the comments are no longer needed.

zzzeid · 2025-03-21T18:28:02Z

git_hg_sync/pulse_worker.py

+                return
+
+        if not isinstance(body, dict):
+            breakpoint()


Forgotten breakpoint?

zzzeid · 2025-03-21T18:29:48Z

git_hg_sync/pulse_worker.py

+        try:
+            if self.event_handler:
+                self.event_handler(event)
+        except:  # noqa: E722


Is there something better here we could do to handle the exception?

Not really, this is a last-resort catch-all in case we don't know what led to the exception (and therefore how to handle it).

The goal is to make sure we reject the message and re-queue it, rather than dropping it and forgetting all about it, which could lead to inconsistencies in the synced data.

zzzeid · 2025-03-21T18:31:03Z

rabbitmq/definitions.json

+      "hashing_algorithm": "rabbit_password_hashing_sha256",
+      "limits": {},
+      "name": "guest",
+      "password_hash": "ywxQjVk0n/qwUfbLIf7L2tJIXrNLfM1bd1GoMeJSzZiRMWSd",


How are these values overridden?

RabbitMQ is only here for local testing, so I don't think we need to.

Co-authored-by: Zeid <2043828+zzzeid@users.noreply.github.com>

Addressed/answered.

and other conditions

shtrom · 2025-03-28T05:14:09Z

ok @cgsheeh @zzzeid I think this is good to review for real.

put entrypoint outside of app code

cgsheeh · 2025-04-01T03:23:48Z

Dockerfile

+RUN pip-compile --verbose pyproject.toml \
+    && pip install -r requirements.txt
+
+RUN echo '[ui]\nssh = ssh -oStrictHostKeyChecking=accept-new' >> /etc/mercurial/hgrc \


Nit: we could put the hgrc in a separate file and COPY it into the image.

cgsheeh · 2025-04-01T03:51:05Z

git_hg_sync/config.py

+        # Allow to override Pulse parameters via environment.
+        for config in self.pulse.model_fields:
+            env_var = f"PULSE_{config}".upper()
+            if value := os.getenv(env_var):
+                setattr(self.pulse, config, value)


We could log updates here, to avoid confusion if it breaks something.

cgsheeh · 2025-04-01T03:52:01Z

git_hg_sync/events.py

-    user: str
-    push_json_url: str
+
+Event = Push


Any reason not rename Push to Event in the class definition?

I'd like to retain the ability to handle other events, even if we are down to 1 at the moment.

pyproject: add project.script for git-hg-sync

cd720eb

shtrom force-pushed the 1940611/pulse-message-update branch 2 times, most recently from 8ec0efd to 779e040 Compare February 4, 2025 05:52

shtrom commented Feb 4, 2025

View reviewed changes

shtrom requested review from cgsheeh, globau and zzzeid February 4, 2025 05:56

cgsheeh previously approved these changes Feb 4, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

shtrom force-pushed the 1940611/pulse-message-update branch 4 times, most recently from 3babf19 to 1d72c6e Compare February 5, 2025 05:35

shtrom changed the title ~~docker: project layout updates (bug 1940611)~~ docker: project layout updates (bug 1940614) Feb 5, 2025

shtrom mentioned this pull request Feb 7, 2025

docker: add support for pulse.test (bug 1940614) mozilla-conduit/suite#66

Merged

shtrom requested a review from cgsheeh February 7, 2025 00:34

shtrom added 4 commits March 12, 2025 16:03

docker: run sync by default, and add send/recv modes

24dafdf

docker: default rabbitmq config

0b95bda

application: typo in sighandler

4a9ba98

main: test connection at startup

64177bd

shtrom force-pushed the 1940611/pulse-message-update branch 3 times, most recently from 4db0574 to 831f18a Compare March 17, 2025 03:09

shtrom added 4 commits March 18, 2025 15:23

pulse_worker: error handling for payload

b4dbfe6

config: allow to override pulse parameter via env

4394271

docker: add create_clones.sh

c3b8287

docker: init clones on up

3adab72

shtrom force-pushed the 1940611/pulse-message-update branch 3 times, most recently from e8daffa to 70bc43b Compare March 18, 2025 05:22

shtrom marked this pull request as ready for review March 18, 2025 05:55

readme: document configuration

0c61c77

globau reviewed Mar 18, 2025

View reviewed changes

create_clones.sh Outdated Show resolved Hide resolved

shtrom added 2 commits March 19, 2025 08:37

fixup! docker: add create_clones.sh

9e909cf

squash! docker: update config to use local repos

e63f133

group branch and tag mappings by target

zzzeid previously requested changes Mar 21, 2025

View reviewed changes

shtrom and others added 4 commits March 24, 2025 14:28

Update README.md

1ebf975

Co-authored-by: Zeid <2043828+zzzeid@users.noreply.github.com>

fixup! pulse_worker: error handling for payload

f86b3d1

fixup! squash! docker: update config to use local repos

6af77c5

fixup! docker: add create_clones.sh

995aee0

shtrom requested a review from zzzeid March 24, 2025 03:41

shtrom added 2 commits March 26, 2025 11:58

push_event: rename pushid to push_id for consistency

5f5274c

app: better handling of Config ValidationError

b91fe2c

shtrom force-pushed the 1940611/pulse-message-update branch from 7a9af4c to b91fe2c Compare March 26, 2025 01:01

shtrom added 5 commits March 26, 2025 12:03

fixup! fixup! docker: add create_clones.sh

48e7a19

docker: add entrypoint to set SSH_PRIVATE_KEY

6e78933

application: warn if no operations were found

a7c4a11

fixup! docker: add entrypoint to set SSH_PRIVATE_KEY

3929404

squash! application: warn if no operations were found

c016538

and other conditions

shtrom requested a review from Copilot March 28, 2025 05:11

squash! fixup! docker: add entrypoint to set SSH_PRIVATE_KEY

737e57e

put entrypoint outside of app code

cgsheeh approved these changes Apr 1, 2025

View reviewed changes

shtrom requested a review from cgsheeh April 1, 2025 05:42

shtrom added 2 commits April 1, 2025 17:06

docker: add template hgrc

7162236

config: log when using environment variable

8246d4c

shtrom merged commit 388be2c into main Apr 2, 2025
3 checks passed

shtrom deleted the 1940611/pulse-message-update branch April 16, 2025 04:23

Conversation

shtrom commented Feb 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shtrom Feb 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgsheeh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shtrom commented Feb 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shtrom commented Mar 18, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shtrom Mar 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shtrom commented Mar 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shtrom commented Feb 4, 2025 •

edited

Loading

shtrom Feb 4, 2025 •

edited

Loading

shtrom commented Feb 7, 2025 •

edited

Loading

shtrom Mar 24, 2025 •

edited

Loading