-
Notifications
You must be signed in to change notification settings - Fork 83
feat(deployment)!: Migrate package orchestration to Docker Compose (resolves #1177); Temporarily remove support for multi-node deployments. #1178
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
261 commits
Select commit
Hold shift + click to select a range
934a83c
remove FIXME
junhaoliao 5bf23c2
Merge branch 'main' into docker-compose
junhaoliao 83cc9d1
reformat
junhaoliao 82abc07
reformat
junhaoliao 0fb2294
Update garbage collector logs directory mapping
junhaoliao c8ffb94
Remove unused component argument parsers
junhaoliao 83e902a
Remove unused component argument parsers
junhaoliao 5f2e5cd
Refactor dependency checks to include docker-compose status validation
junhaoliao edfa9c9
Refactor log directory handling to use constant path definitions
junhaoliao 7b3965e
Add constants for archive and stream directory paths
junhaoliao cd84be8
remove unused component groups and functions
junhaoliao aa12bdb
Remove unused CONTROLLER_TARGET_NAME constant from start_clp.py
junhaoliao 5365722
fix staging dirs
junhaoliao 21ef703
fix: update command to check if Docker Compose is running
junhaoliao d2cdfbc
add AWS env credentials support
junhaoliao 4f56709
Merge branch 'main' into docker-compose
junhaoliao f0db07f
Update container image name in start_clp.py
junhaoliao 655600d
Merge remote-tracking branch 'origin/main' into docker-compose
junhaoliao 5f24ce7
add support for configurable CLP WebUI rate limiting
junhaoliao 3df20fc
update WebUI server path in start_clp.py and docker-compose configura…
junhaoliao c6f81ad
copy docker-compose.yml in package task
junhaoliao db9c20f
use absolute paths in archive and stream storage configurations
junhaoliao ea03e17
refactor: centralize environment variable management and enhance vali…
junhaoliao 60994ee
fix: use List[str] type hint for command parameter in start_clp.py
junhaoliao 7e25d75
refactor: remove `dump_to_env_vars_dict` methods and centralize envir…
junhaoliao 3e24e4e
lint
junhaoliao cbb9ce1
refactor: modularize and simplify start_clp.py by introducing DockerC…
junhaoliao 3eb8dfe
refactor: remove obsolete node-specific directory configuration comments
junhaoliao 669fa9c
refactor: remove redundant `conf_dir` parameter and use centralized c…
junhaoliao acee071
refactor: rename `controllers` module to `controller` and update impo…
junhaoliao 3c45cfa
refactor: make `validate_log_directory` private and update references…
junhaoliao ed20110
refactor: rename `conf_dir` to `_conf_dir` and update references to r…
junhaoliao 4d1f5aa
refactor: make `get_ip_from_hostname` private and update all references
junhaoliao 3a698bb
refactor: extract `transform_for_container_config` method to simplify…
junhaoliao cca84f4
refactor: centralize and simplify path definitions and container conf…
junhaoliao 537398a
refactor: streamline container service configuration with `transform_…
junhaoliao 42442b8
fix imports
junhaoliao a3288ae
reorganize imports
junhaoliao 5b36779
fix: adjust volume path formatting in docker-compose
junhaoliao 93882af
fix: correct typo in comment for scheduler healthcheck in docker-compose
junhaoliao a4d8e1f
refactor: unify volume path definitions and remove redundant staging …
junhaoliao 63e6d72
remove comment
junhaoliao 0531a7b
feat: implement `stop` method in `DockerComposeController` and update…
junhaoliao f93aec7
refactor: reorder volume definitions in docker-compose for consistenc…
junhaoliao 6a20628
revert error message in start_clp.py
junhaoliao 0935658
refactor: standardize logger messages from "Initializing" to "Provisi…
junhaoliao ad6192a
remove comment
junhaoliao 9469db4
docs: update multi-node deployment guide and add Docker Compose desig…
junhaoliao 110e9fa
docs: refine Docker Compose design doc for clarity and consistency
junhaoliao 045dde6
refactor: update CLP container configuration to use execution contain…
junhaoliao fa87d92
refactor(controller): move directory creation logic from `_provision`…
junhaoliao b6ac2c6
feat(controller): add function to dump shared container configuration…
junhaoliao bfd8c7b
lint
junhaoliao 2b7959b
refactor(clp-py-utils): update staging directory handling in S3 stora…
junhaoliao f9eb88c
docs(design): Enhance Docker Compose design doc with diagrams and det…
junhaoliao 0f5bd45
fix title case
junhaoliao 0656928
Update references to docker-compose.yml to docker-compose.yaml.
junhaoliao 66dcbb2
feat(docker): add project name to docker-compose and enhance running …
junhaoliao 09ef298
fix lint
junhaoliao d3b6a67
fix(taskfile): update default task to `docker-images:package`
junhaoliao 6148a65
feat: reset default ports for container configs.
junhaoliao 0ab99f0
fix(docker): update MongoDB connection string to use internal port
junhaoliao 263be6f
fix(controller): update webui configuration to use container-specific…
junhaoliao 51d1b55
add ownership management for data and logs directories when running a…
junhaoliao 0066386
fix(controller, docker-compose): update user and group ID handling fo…
junhaoliao bc9ab99
fix(search): enable direct connection for MongoDB client in search.py
junhaoliao 8008138
revert direct connection option for MongoDB client in search.py
junhaoliao 34b60bd
refactor(controller): rename provision methods to set_up_env_for_* to…
junhaoliao 9f63481
reorder private functions
junhaoliao 2344db9
add documentation
junhaoliao 2ba93d7
change visibility of `set_up_env_for` function from public to private…
junhaoliao ce94225
Improve type hints.
junhaoliao 5225870
docs(clp-package-utils): add docstring for is_docker_compose_running …
junhaoliao cecf5b2
docs(clp-package-utils): add docstring for check_docker_dependencies …
junhaoliao 0c0c562
docs(clp-package-utils): add docstring for _validate_log_directory fu…
junhaoliao 762884b
remove `None` return type annotation from _validate_log_directory fun…
junhaoliao fc515db
Rename transform_for_container_config to transform_for_container; add…
junhaoliao 268c144
add docs for x-service-defaults & x-healthcheck-defaults in docker-co…
junhaoliao c1d7b23
Merge branch 'main' into docker-compose
junhaoliao fc7aae3
remove garbage-collector's dependency condition on query-scheduler.
junhaoliao cda0348
Split base services into a separate docker-compose.base.yaml; Launch …
junhaoliao e506f6d
feat(controller): Add instance ID support for Docker Compose project …
junhaoliao 93034fe
remove network to use default network with bridge driver
junhaoliao 7ebdbb6
Merge branch 'main' into docker-compose
junhaoliao ca62009
fix(controller): update clp_home to be private variable.
junhaoliao 82cc48c
Use unique ids to launch dev clp package.
junhaoliao 8f6ba1b
remove trailing space
junhaoliao 2aff301
remove `execution_container` field from clp-config.yml template
junhaoliao 5c91bf3
fix(docker-compose): update `include` array syntax to use [] format.
junhaoliao 049e269
update service name from `db` to `database`.
junhaoliao fc70c05
fix(docker-compose, controller): correct configuration file variable …
junhaoliao 2cb1807
add _HOST postfix to path env var names for consistency
junhaoliao 08472d1
Merge branch 'main' into docker-compose
junhaoliao 09658e5
Merge branch 'main' into docker-compose
junhaoliao e657feb
fix: the webui service should depend on db-table-creator rather than …
junhaoliao c79259a
Merge branch 'main' into docker-compose
junhaoliao af96fc8
lint
junhaoliao 320dd11
Merge branch 'main' into docker-compose
junhaoliao 08644f4
Merge branch 'main' into docker-compose
junhaoliao dd56d04
fix: Replace container_name with hostname for services to avoid name …
junhaoliao 5298540
refactor(docker-compose): move image definition to service defaults.
junhaoliao 140187f
refactor(docker-compose): Remove image entry from reducer service.
junhaoliao 68586f0
refactor(docker-compose): add default fallback for image and storage …
junhaoliao cba044a
improve docs
junhaoliao 8a6790f
Merge branch 'main' into docker-compose
junhaoliao f41e22a
refactor(docker-compose): use list syntax for healthcheck test command.
junhaoliao da6c17f
Merge branch 'main' into docker-compose
junhaoliao 02fcbe7
fix(docker-compose): revert workers log level to hardcoded "warning".
junhaoliao 6b0e1f2
fix(docker-compose): Fix environment variable name for results cache …
junhaoliao ddcf760
refactor(clp-package-utils): Move EnvVarsDict type alias below imports.
junhaoliao 4134b1a
refactor(clp-package-utils): Rename LOGS_FILE_MODE to LOG_FILE_ACCESS…
junhaoliao f483abd
reflow comments to 100 char per line
junhaoliao 7d4f47d
refactor(clp-package-utils): Rename deploy method to start and adjust…
junhaoliao 064f365
refactor(clp-package-utils): Rename _provision method to _set_up_env …
junhaoliao 7beb199
docs(clp-package-utils): Update docstrings to use “sets up” phrasing.
junhaoliao 3507a1b
docs(clp-package-utils): Standardize return docstrings; Remove return…
junhaoliao eb2754a
refactor(clp-package-utils): Add spacing before log and data director…
junhaoliao cae66df
refactor(clp-package-utils): Move logs directory assignment after dat…
junhaoliao dffc2f2
refactor(clp-package-utils): Rename logs_file -> log_file
junhaoliao c0ef4ff
refactor(clp-package-utils): Rename LOGS_FILE variables to LOG_FILE i…
junhaoliao 95404d0
refactor(clp-package-utils): Remove redundant flush after writing ins…
junhaoliao e1db192
Clarify resolving to IPv4 in the docstring - Apply suggestions from c…
junhaoliao 630329b
Remove redundant param description in `_get_ip_from_hostname()` - App…
junhaoliao d9da763
refactor(clp-package-utils): Rename db logging config variable and en…
junhaoliao c200502
use long form `:readonly` instead of short form `:ro` in binding moun…
junhaoliao 5751ddf
revert unintentional change
junhaoliao 3e9734e
chore(deployment): Add local logging driver to service defaults.
junhaoliao ab64f4a
Improve docs - Apply suggestions from code review
junhaoliao 2d9906f
chore(deployment): Adjust healthcheck defaults (start interval, perio…
junhaoliao 3635883
refactor(deployment): Use long form volume definitions and use bind o…
junhaoliao 5e0d0d2
refactor(deployment): Use long form port definitions; fix CLP_RESULTS…
junhaoliao e771800
refactor(deployment): Reorder port definitions to long form order.
junhaoliao 6e878cf
refactor(clp-package-utils): Replace deprecated Pydantic (V2) copy() …
junhaoliao f2b0967
feat(deployment): Add support for configurable logs input directory m…
junhaoliao 01558ba
lint
junhaoliao 8a3985b
refactor(deployment): Remove secrets usage; replace with environment …
junhaoliao c691735
docs(design): Add health check defaults.
junhaoliao 787c7d7
refactor(clp-package-utils): Improve error handling for config loadin…
junhaoliao 323cb58
Merge branch 'main' into docker-compose
junhaoliao 8b97bc2
refactor(deployment): Remove outdated comment in reducer healthcheck …
junhaoliao ccb8f66
refactor(deployment): Remove duplicated docstring from docker-compose…
junhaoliao 40b88a0
docs(design): Remove health check defaults section; add comments to h…
junhaoliao e58512a
docs(quick-start): Add required containerd, Docker CE, CLI, and compo…
junhaoliao d7b40b0
Merge branch 'main' into docker-compose
junhaoliao 37a0a8a
restore verbose logging option to start_clp script.
junhaoliao 3cdf245
Merge remote-tracking branch 'origin/main' into docker-compose
junhaoliao 95d900a
refactor(clp-py-utils): Use DEFAULT_PORT for WebUi port configuration.
junhaoliao f6cbc7f
apply docs suggestions
junhaoliao 5bf0887
refactor(controller): Rename clp_config -> _clp_config for consistent…
junhaoliao 26dfaca
refactor(deployment): Rename Redis environment variables for improved…
junhaoliao 99e983f
Apply suggestions - Reformat multiline statements and remove unnecess…
junhaoliao 0b028dc
Apply suggestions - Rename UID/GID environment variables for improved…
junhaoliao f11963c
Apply suggestions - Use constants for jobs table names.
junhaoliao 91cb9fc
refactor(controller): Remove redundant `stderr` redirection from subp…
junhaoliao 1d57a89
docs: move high-level comments to individual blocks
junhaoliao f4254fa
docs(controller): reference issue for revisiting worker count logic
junhaoliao 932c334
refactor(controller): add `--wait` flag to `docker compose up` comman…
junhaoliao a8cee64
refactor(controller): replace `Dict` with a custom `EnvVarsDict` for …
junhaoliao f0e1740
lint
junhaoliao 6679c2d
use local variable in assignment in _update_settings_object()
junhaoliao 416c31a
refactor(controller): remove redundant try-except blocks around `subp…
junhaoliao 9ab85d2
add warning log for failed Docker dependency check before attempting …
junhaoliao 1345de3
refactor(controller): add explicit return type annotations for all me…
junhaoliao 25c3812
Merge branch 'main' into docker-compose
junhaoliao 3c3e23d
refactor(scripts): move `instance_id` assignment inside `try` block i…
junhaoliao 7c43aad
revert `--config` argument removal in `stop_clp` to allow specifying …
junhaoliao 63c700a
refactor(config): Rename CLP_DEFAULT_ARCHIVE_* -> CLP_DEFAULT_ARCHIVE…
junhaoliao 0f7c78c
apply docs suggestions
junhaoliao 5e07756
docs(clp-package-utils): Update docstrings for clarity and consistency.
junhaoliao ff4f06c
remove return type from dump_shared_container_config()
junhaoliao e381c3d
refactor(clp-package-utils): Rename base_config -> component_config i…
junhaoliao f426c82
Simplify raised error message for missing configuration file: redis
junhaoliao 73ec68f
Apply suggestion - To alphabetize
junhaoliao 4e707ae
Remove individual log file paths and use shared logging volume
junhaoliao 6f53877
fix(controller): revert removal of "ClpQueryEngine" in server_setting…
junhaoliao e7addd3
refactor(clp-package-utils): Rename CLP_PACKAGE_CONTAINER -> CLP_PACK…
junhaoliao d4e81f3
fix(docker-compose): use `:?error` syntax to mark required non-empty …
junhaoliao a5a497d
disallow empty `username` and `password` in `Database` because our db…
junhaoliao 0dd11ce
fix(docker-compose): Update default database image to mariadb:10-jammy.
junhaoliao d27a0b1
fix(docker-compose): Add stop_grace_period for compression-worker and…
junhaoliao a6478e6
apply docs suggestions
junhaoliao 7bbd134
refactor(docker-compose): Rename volume_root_logs_readonly -> volume_…
junhaoliao 04e56f0
ensure `healthcheck` goes after `command`
junhaoliao 0fc91d7
fix(docker-compose): Add AWS credentials environment variables in web…
junhaoliao f307d94
fix(docker-compose): Update environment variables for archive and str…
junhaoliao e5a90dd
Replace "docker-compose" (usually refer to v1) with "Docker Compose" …
junhaoliao 3015ec4
lint
junhaoliao ff43c78
refactor(clp-package-utils): Reorder functions and rename is_docker_c…
junhaoliao 699979a
fix(docker-compose): Add mounts for archive and stream output in garb…
junhaoliao 092a86c
refactor(clp-package-utils): Rename _is_docker_compose_running -> _is…
junhaoliao 8d75171
fix(docker-compose): Rename CLP_AWS_ACCESS_KEY_ID -> CLP_STREAM_OUTPU…
junhaoliao f53339f
fix(docker-compose): Break long volume mount lines for improved reada…
junhaoliao 18d1109
lint
junhaoliao 9d9bd46
refactor(clp-package-utils): Rename should_compose_run -> should_comp…
junhaoliao 225c6c5
Merge branch 'main' into docker-compose
junhaoliao 127aacc
refactor(clp-package-utils): Reorder private functions
junhaoliao f5ea3e2
Replace EnvironmentError with OSError as per Ruff os-error-alias (UP024)
junhaoliao 22593d8
refactor(general): Improve error handling for Docker and Docker Compose
junhaoliao bc69b26
refactor(general): Update dump_container_config call to remove return…
junhaoliao 1d2ab61
refactor(docs): Update exception documentation to use 'raise' format
junhaoliao d4c921a
add missing `f` in f-string
junhaoliao e6bb002
Add retention periods check for garbage collector and disable the ser…
junhaoliao 90ee7da
Merge branch 'main' into docker-compose
junhaoliao b0d0bcc
refactor(clp-package-utils): Remove unused `pass` statements from abs…
junhaoliao 955a0ac
refactor(clp-config): Remove unused typing import 'Set'.
junhaoliao 106fdda
apply doc suggestions
junhaoliao 7a1ea6e
refactor(docker-compose): Update environment variable error messages …
junhaoliao a732939
Adjust volume bindings to fix directory mappings when CLP_STAGED_ARCH…
junhaoliao c687f78
fix string indents
junhaoliao 19e8a9d
apply doc suggestions
junhaoliao c53c0f1
apply doc suggestions
junhaoliao a2fa02c
fix(clp-package-utils): Remove extra space in `.env` file comment.
junhaoliao 05810be
apply doc suggestions
junhaoliao 2a02ecb
apply doc suggestions
junhaoliao 678de80
Update comment for `_get_num_workers` method - apply doc suggestion
junhaoliao c30adae
Simplify log message for CLP start action - apply suggestion
junhaoliao 07d1e35
Update comments for consistency in `controller.py`.
junhaoliao 0bf6b70
feat(clp-package-utils): Add custom exceptions for Docker dependency …
junhaoliao 2986082
fix(clp-package-utils): Use `docker --version` to check docker availa…
junhaoliao e499443
Avoid mapping ~/.aws on host when CLP_AWS_CONFIG_DIR_HOST is undefined
junhaoliao b788c33
alphabetize mounts
junhaoliao 6fc219c
Do not map logs input dir when CLP_LOGS_INPUT_DIR_HOST is undefined
junhaoliao 0926180
lint
junhaoliao e2daf32
lint
junhaoliao 736c805
Merge remote-tracking branch 'origin/main' into docker-compose
junhaoliao 4956106
fix: Add AWS config directory mount to garbage collector
junhaoliao 91cacb0
fix(clp-package-utils): Add return type annotations to custom excepti…
junhaoliao 3ad7f70
Merge branch 'main' into docker-compose
junhaoliao e8e4d58
Merge branch 'main' into docker-compose
junhaoliao c208942
Rename CLP_DB_IMAGE to CLP_DB_CONTAINER_IMAGE_REF.
kirkrodrigues 6cea88b
Simplify package start/stop logs.
kirkrodrigues 8c5a167
Refactor new Docker exceptions.
kirkrodrigues f02aff4
general.py: Minor refactoring.
kirkrodrigues 667d753
Rename CLP_GC_LOGGING_LEVEL to CLP_GARBAGE_COLLECTOR_LOGGING_LEVEL.
kirkrodrigues 930a2dd
Put operator on next line of multiline statement.
kirkrodrigues 340a5e7
Use term 'config' instead of less common 'params'; Add required newli…
kirkrodrigues 2bc2d2b
Correct AWS profile mount for webui - Apply suggestions from code review
junhaoliao f0fa5ae
Bump stop grace periods for various services.
junhaoliao 26c3e32
Rewrite design doc.
kirkrodrigues 164e27e
Rename design document file.
kirkrodrigues 7759c5b
Lint.
kirkrodrigues 0fdf31c
Split notes.
kirkrodrigues 7447fef
Tell readers how to find the instance ID.
kirkrodrigues 5227292
Rename Databases to Shared data stores.
kirkrodrigues a8ca533
Merge branch 'main' into docker-compose
junhaoliao ed25fc3
Update docs/src/dev-docs/design-deployment-orchestration.md
kirkrodrigues 4926450
building-package.md: Minor fixes.
kirkrodrigues 127cb79
Remove other details from multi-node doc until we add multi-node supp…
kirkrodrigues 649276f
Add comment to explain the `+/tmp` hack.
kirkrodrigues 7139740
docs(design-deployment): Rename "Shared data stores" subgraph to "Dat…
junhaoliao File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@coderabbitai please create an issue to track removal of this hack
an idea is to accept a
var/tmpdirectory config from the user, and we generate all temporary runtime data in that directory instead of the currentvar/datadirectory. then we don't have to add the/var/databind mount belowThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
Length of output: 42
@junhaoliao I've created issue #1478 to track the removal of this Docker Compose mount hack. The issue describes the current workaround and includes your proposed solution of using a separate
var/tmpdirectory for temporary runtime data.