Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
261 commits
Select commit Hold shift + click to select a range
934a83c
remove FIXME
junhaoliao Aug 24, 2025
5bf23c2
Merge branch 'main' into docker-compose
junhaoliao Aug 24, 2025
83cc9d1
reformat
junhaoliao Aug 24, 2025
82abc07
reformat
junhaoliao Aug 24, 2025
0fb2294
Update garbage collector logs directory mapping
junhaoliao Aug 24, 2025
c8ffb94
Remove unused component argument parsers
junhaoliao Aug 24, 2025
83e902a
Remove unused component argument parsers
junhaoliao Aug 24, 2025
5f2e5cd
Refactor dependency checks to include docker-compose status validation
junhaoliao Aug 24, 2025
edfa9c9
Refactor log directory handling to use constant path definitions
junhaoliao Aug 24, 2025
7b3965e
Add constants for archive and stream directory paths
junhaoliao Aug 24, 2025
cd84be8
remove unused component groups and functions
junhaoliao Aug 24, 2025
aa12bdb
Remove unused CONTROLLER_TARGET_NAME constant from start_clp.py
junhaoliao Aug 24, 2025
5365722
fix staging dirs
junhaoliao Aug 24, 2025
21ef703
fix: update command to check if Docker Compose is running
junhaoliao Aug 24, 2025
d2cdfbc
add AWS env credentials support
junhaoliao Aug 24, 2025
4f56709
Merge branch 'main' into docker-compose
junhaoliao Aug 25, 2025
f0db07f
Update container image name in start_clp.py
junhaoliao Aug 26, 2025
655600d
Merge remote-tracking branch 'origin/main' into docker-compose
junhaoliao Sep 3, 2025
5f24ce7
add support for configurable CLP WebUI rate limiting
junhaoliao Sep 3, 2025
3df20fc
update WebUI server path in start_clp.py and docker-compose configura…
junhaoliao Sep 3, 2025
c6f81ad
copy docker-compose.yml in package task
junhaoliao Sep 3, 2025
db9c20f
use absolute paths in archive and stream storage configurations
junhaoliao Sep 3, 2025
ea03e17
refactor: centralize environment variable management and enhance vali…
junhaoliao Sep 4, 2025
60994ee
fix: use List[str] type hint for command parameter in start_clp.py
junhaoliao Sep 4, 2025
7e25d75
refactor: remove `dump_to_env_vars_dict` methods and centralize envir…
junhaoliao Sep 4, 2025
3e24e4e
lint
junhaoliao Sep 4, 2025
cbb9ce1
refactor: modularize and simplify start_clp.py by introducing DockerC…
junhaoliao Sep 4, 2025
3eb8dfe
refactor: remove obsolete node-specific directory configuration comments
junhaoliao Sep 4, 2025
669fa9c
refactor: remove redundant `conf_dir` parameter and use centralized c…
junhaoliao Sep 4, 2025
acee071
refactor: rename `controllers` module to `controller` and update impo…
junhaoliao Sep 4, 2025
3c45cfa
refactor: make `validate_log_directory` private and update references…
junhaoliao Sep 4, 2025
ed20110
refactor: rename `conf_dir` to `_conf_dir` and update references to r…
junhaoliao Sep 4, 2025
4d1f5aa
refactor: make `get_ip_from_hostname` private and update all references
junhaoliao Sep 4, 2025
3a698bb
refactor: extract `transform_for_container_config` method to simplify…
junhaoliao Sep 4, 2025
cca84f4
refactor: centralize and simplify path definitions and container conf…
junhaoliao Sep 4, 2025
537398a
refactor: streamline container service configuration with `transform_…
junhaoliao Sep 4, 2025
42442b8
fix imports
junhaoliao Sep 4, 2025
a3288ae
reorganize imports
junhaoliao Sep 4, 2025
5b36779
fix: adjust volume path formatting in docker-compose
junhaoliao Sep 4, 2025
93882af
fix: correct typo in comment for scheduler healthcheck in docker-compose
junhaoliao Sep 4, 2025
a4d8e1f
refactor: unify volume path definitions and remove redundant staging …
junhaoliao Sep 4, 2025
63e6d72
remove comment
junhaoliao Sep 4, 2025
0531a7b
feat: implement `stop` method in `DockerComposeController` and update…
junhaoliao Sep 4, 2025
f93aec7
refactor: reorder volume definitions in docker-compose for consistenc…
junhaoliao Sep 4, 2025
6a20628
revert error message in start_clp.py
junhaoliao Sep 4, 2025
0935658
refactor: standardize logger messages from "Initializing" to "Provisi…
junhaoliao Sep 4, 2025
ad6192a
remove comment
junhaoliao Sep 4, 2025
9469db4
docs: update multi-node deployment guide and add Docker Compose desig…
junhaoliao Sep 4, 2025
110e9fa
docs: refine Docker Compose design doc for clarity and consistency
junhaoliao Sep 4, 2025
045dde6
refactor: update CLP container configuration to use execution contain…
junhaoliao Sep 4, 2025
fa87d92
refactor(controller): move directory creation logic from `_provision`…
junhaoliao Sep 4, 2025
b6ac2c6
feat(controller): add function to dump shared container configuration…
junhaoliao Sep 4, 2025
bfd8c7b
lint
junhaoliao Sep 4, 2025
2b7959b
refactor(clp-py-utils): update staging directory handling in S3 stora…
junhaoliao Sep 4, 2025
f9eb88c
docs(design): Enhance Docker Compose design doc with diagrams and det…
junhaoliao Sep 4, 2025
0f5bd45
fix title case
junhaoliao Sep 4, 2025
0656928
Update references to docker-compose.yml to docker-compose.yaml.
junhaoliao Sep 4, 2025
66dcbb2
feat(docker): add project name to docker-compose and enhance running …
junhaoliao Sep 10, 2025
09ef298
fix lint
junhaoliao Sep 10, 2025
d3b6a67
fix(taskfile): update default task to `docker-images:package`
junhaoliao Sep 10, 2025
6148a65
feat: reset default ports for container configs.
junhaoliao Sep 17, 2025
0ab99f0
fix(docker): update MongoDB connection string to use internal port
junhaoliao Sep 17, 2025
263be6f
fix(controller): update webui configuration to use container-specific…
junhaoliao Sep 17, 2025
51d1b55
add ownership management for data and logs directories when running a…
junhaoliao Sep 17, 2025
0066386
fix(controller, docker-compose): update user and group ID handling fo…
junhaoliao Sep 17, 2025
bc9ab99
fix(search): enable direct connection for MongoDB client in search.py
junhaoliao Sep 20, 2025
8008138
revert direct connection option for MongoDB client in search.py
junhaoliao Sep 20, 2025
34b60bd
refactor(controller): rename provision methods to set_up_env_for_* to…
junhaoliao Sep 22, 2025
9f63481
reorder private functions
junhaoliao Sep 22, 2025
2344db9
add documentation
junhaoliao Sep 22, 2025
2ba93d7
change visibility of `set_up_env_for` function from public to private…
junhaoliao Sep 22, 2025
ce94225
Improve type hints.
junhaoliao Sep 22, 2025
5225870
docs(clp-package-utils): add docstring for is_docker_compose_running …
junhaoliao Sep 22, 2025
cecf5b2
docs(clp-package-utils): add docstring for check_docker_dependencies …
junhaoliao Sep 22, 2025
0c0c562
docs(clp-package-utils): add docstring for _validate_log_directory fu…
junhaoliao Sep 22, 2025
762884b
remove `None` return type annotation from _validate_log_directory fun…
junhaoliao Sep 22, 2025
fc515db
Rename transform_for_container_config to transform_for_container; add…
junhaoliao Sep 22, 2025
268c144
add docs for x-service-defaults & x-healthcheck-defaults in docker-co…
junhaoliao Sep 22, 2025
c1d7b23
Merge branch 'main' into docker-compose
junhaoliao Sep 24, 2025
fc7aae3
remove garbage-collector's dependency condition on query-scheduler.
junhaoliao Sep 24, 2025
cda0348
Split base services into a separate docker-compose.base.yaml; Launch …
junhaoliao Sep 24, 2025
e506f6d
feat(controller): Add instance ID support for Docker Compose project …
junhaoliao Sep 24, 2025
93034fe
remove network to use default network with bridge driver
junhaoliao Sep 24, 2025
7ebdbb6
Merge branch 'main' into docker-compose
junhaoliao Sep 25, 2025
ca62009
fix(controller): update clp_home to be private variable.
junhaoliao Sep 25, 2025
82cc48c
Use unique ids to launch dev clp package.
junhaoliao Sep 25, 2025
8f6ba1b
remove trailing space
junhaoliao Sep 25, 2025
2aff301
remove `execution_container` field from clp-config.yml template
junhaoliao Sep 25, 2025
5c91bf3
fix(docker-compose): update `include` array syntax to use [] format.
junhaoliao Sep 25, 2025
049e269
update service name from `db` to `database`.
junhaoliao Sep 25, 2025
fc70c05
fix(docker-compose, controller): correct configuration file variable …
junhaoliao Sep 25, 2025
2cb1807
add _HOST postfix to path env var names for consistency
junhaoliao Sep 25, 2025
08472d1
Merge branch 'main' into docker-compose
junhaoliao Sep 25, 2025
09658e5
Merge branch 'main' into docker-compose
junhaoliao Sep 26, 2025
e657feb
fix: the webui service should depend on db-table-creator rather than …
junhaoliao Sep 26, 2025
c79259a
Merge branch 'main' into docker-compose
junhaoliao Oct 1, 2025
af96fc8
lint
junhaoliao Oct 1, 2025
320dd11
Merge branch 'main' into docker-compose
junhaoliao Oct 2, 2025
08644f4
Merge branch 'main' into docker-compose
junhaoliao Oct 2, 2025
dd56d04
fix: Replace container_name with hostname for services to avoid name …
junhaoliao Oct 2, 2025
5298540
refactor(docker-compose): move image definition to service defaults.
junhaoliao Oct 2, 2025
140187f
refactor(docker-compose): Remove image entry from reducer service.
junhaoliao Oct 2, 2025
68586f0
refactor(docker-compose): add default fallback for image and storage …
junhaoliao Oct 2, 2025
cba044a
improve docs
junhaoliao Oct 3, 2025
8a6790f
Merge branch 'main' into docker-compose
junhaoliao Oct 3, 2025
f41e22a
refactor(docker-compose): use list syntax for healthcheck test command.
junhaoliao Oct 3, 2025
da6c17f
Merge branch 'main' into docker-compose
junhaoliao Oct 6, 2025
02fcbe7
fix(docker-compose): revert workers log level to hardcoded "warning".
junhaoliao Oct 6, 2025
6b0e1f2
fix(docker-compose): Fix environment variable name for results cache …
junhaoliao Oct 6, 2025
ddcf760
refactor(clp-package-utils): Move EnvVarsDict type alias below imports.
junhaoliao Oct 6, 2025
4134b1a
refactor(clp-package-utils): Rename LOGS_FILE_MODE to LOG_FILE_ACCESS…
junhaoliao Oct 6, 2025
f483abd
reflow comments to 100 char per line
junhaoliao Oct 6, 2025
7d4f47d
refactor(clp-package-utils): Rename deploy method to start and adjust…
junhaoliao Oct 6, 2025
064f365
refactor(clp-package-utils): Rename _provision method to _set_up_env …
junhaoliao Oct 6, 2025
7beb199
docs(clp-package-utils): Update docstrings to use “sets up” phrasing.
junhaoliao Oct 6, 2025
3507a1b
docs(clp-package-utils): Standardize return docstrings; Remove return…
junhaoliao Oct 6, 2025
eb2754a
refactor(clp-package-utils): Add spacing before log and data director…
junhaoliao Oct 6, 2025
cae66df
refactor(clp-package-utils): Move logs directory assignment after dat…
junhaoliao Oct 6, 2025
dffc2f2
refactor(clp-package-utils): Rename logs_file -> log_file
junhaoliao Oct 6, 2025
c0ef4ff
refactor(clp-package-utils): Rename LOGS_FILE variables to LOG_FILE i…
junhaoliao Oct 6, 2025
95404d0
refactor(clp-package-utils): Remove redundant flush after writing ins…
junhaoliao Oct 6, 2025
e1db192
Clarify resolving to IPv4 in the docstring - Apply suggestions from c…
junhaoliao Oct 6, 2025
630329b
Remove redundant param description in `_get_ip_from_hostname()` - App…
junhaoliao Oct 6, 2025
d9da763
refactor(clp-package-utils): Rename db logging config variable and en…
junhaoliao Oct 6, 2025
c200502
use long form `:readonly` instead of short form `:ro` in binding moun…
junhaoliao Oct 6, 2025
5751ddf
revert unintentional change
junhaoliao Oct 6, 2025
3e9734e
chore(deployment): Add local logging driver to service defaults.
junhaoliao Oct 6, 2025
ab64f4a
Improve docs - Apply suggestions from code review
junhaoliao Oct 7, 2025
2d9906f
chore(deployment): Adjust healthcheck defaults (start interval, perio…
junhaoliao Oct 7, 2025
3635883
refactor(deployment): Use long form volume definitions and use bind o…
junhaoliao Oct 7, 2025
5e0d0d2
refactor(deployment): Use long form port definitions; fix CLP_RESULTS…
junhaoliao Oct 7, 2025
e771800
refactor(deployment): Reorder port definitions to long form order.
junhaoliao Oct 7, 2025
6e878cf
refactor(clp-package-utils): Replace deprecated Pydantic (V2) copy() …
junhaoliao Oct 7, 2025
f2b0967
feat(deployment): Add support for configurable logs input directory m…
junhaoliao Oct 7, 2025
01558ba
lint
junhaoliao Oct 7, 2025
8a3985b
refactor(deployment): Remove secrets usage; replace with environment …
junhaoliao Oct 7, 2025
c691735
docs(design): Add health check defaults.
junhaoliao Oct 7, 2025
787c7d7
refactor(clp-package-utils): Improve error handling for config loadin…
junhaoliao Oct 7, 2025
323cb58
Merge branch 'main' into docker-compose
junhaoliao Oct 9, 2025
8b97bc2
refactor(deployment): Remove outdated comment in reducer healthcheck …
junhaoliao Oct 9, 2025
ccb8f66
refactor(deployment): Remove duplicated docstring from docker-compose…
junhaoliao Oct 9, 2025
40b88a0
docs(design): Remove health check defaults section; add comments to h…
junhaoliao Oct 9, 2025
e58512a
docs(quick-start): Add required containerd, Docker CE, CLI, and compo…
junhaoliao Oct 10, 2025
d7b40b0
Merge branch 'main' into docker-compose
junhaoliao Oct 10, 2025
37a0a8a
restore verbose logging option to start_clp script.
junhaoliao Oct 10, 2025
3cdf245
Merge remote-tracking branch 'origin/main' into docker-compose
junhaoliao Oct 14, 2025
95d900a
refactor(clp-py-utils): Use DEFAULT_PORT for WebUi port configuration.
junhaoliao Oct 14, 2025
f6cbc7f
apply docs suggestions
junhaoliao Oct 15, 2025
5bf0887
refactor(controller): Rename clp_config -> _clp_config for consistent…
junhaoliao Oct 15, 2025
26dfaca
refactor(deployment): Rename Redis environment variables for improved…
junhaoliao Oct 15, 2025
99e983f
Apply suggestions - Reformat multiline statements and remove unnecess…
junhaoliao Oct 15, 2025
0b028dc
Apply suggestions - Rename UID/GID environment variables for improved…
junhaoliao Oct 15, 2025
f11963c
Apply suggestions - Use constants for jobs table names.
junhaoliao Oct 15, 2025
91cb9fc
refactor(controller): Remove redundant `stderr` redirection from subp…
junhaoliao Oct 15, 2025
1d57a89
docs: move high-level comments to individual blocks
junhaoliao Oct 15, 2025
f4254fa
docs(controller): reference issue for revisiting worker count logic
junhaoliao Oct 15, 2025
932c334
refactor(controller): add `--wait` flag to `docker compose up` comman…
junhaoliao Oct 15, 2025
a8cee64
refactor(controller): replace `Dict` with a custom `EnvVarsDict` for …
junhaoliao Oct 16, 2025
f0e1740
lint
junhaoliao Oct 16, 2025
6679c2d
use local variable in assignment in _update_settings_object()
junhaoliao Oct 16, 2025
416c31a
refactor(controller): remove redundant try-except blocks around `subp…
junhaoliao Oct 16, 2025
9ab85d2
add warning log for failed Docker dependency check before attempting …
junhaoliao Oct 16, 2025
1345de3
refactor(controller): add explicit return type annotations for all me…
junhaoliao Oct 16, 2025
25c3812
Merge branch 'main' into docker-compose
junhaoliao Oct 16, 2025
3c3e23d
refactor(scripts): move `instance_id` assignment inside `try` block i…
junhaoliao Oct 16, 2025
7c43aad
revert `--config` argument removal in `stop_clp` to allow specifying …
junhaoliao Oct 16, 2025
63c700a
refactor(config): Rename CLP_DEFAULT_ARCHIVE_* -> CLP_DEFAULT_ARCHIVE…
junhaoliao Oct 16, 2025
0f7c78c
apply docs suggestions
junhaoliao Oct 16, 2025
5e07756
docs(clp-package-utils): Update docstrings for clarity and consistency.
junhaoliao Oct 16, 2025
ff4f06c
remove return type from dump_shared_container_config()
junhaoliao Oct 16, 2025
e381c3d
refactor(clp-package-utils): Rename base_config -> component_config i…
junhaoliao Oct 16, 2025
f426c82
Simplify raised error message for missing configuration file: redis
junhaoliao Oct 16, 2025
73ec68f
Apply suggestion - To alphabetize
junhaoliao Oct 16, 2025
4e707ae
Remove individual log file paths and use shared logging volume
junhaoliao Oct 16, 2025
6f53877
fix(controller): revert removal of "ClpQueryEngine" in server_setting…
junhaoliao Oct 16, 2025
e7addd3
refactor(clp-package-utils): Rename CLP_PACKAGE_CONTAINER -> CLP_PACK…
junhaoliao Oct 16, 2025
d4e81f3
fix(docker-compose): use `:?error` syntax to mark required non-empty …
junhaoliao Oct 16, 2025
a5a497d
disallow empty `username` and `password` in `Database` because our db…
junhaoliao Oct 16, 2025
0dd11ce
fix(docker-compose): Update default database image to mariadb:10-jammy.
junhaoliao Oct 16, 2025
d27a0b1
fix(docker-compose): Add stop_grace_period for compression-worker and…
junhaoliao Oct 16, 2025
a6478e6
apply docs suggestions
junhaoliao Oct 16, 2025
7bbd134
refactor(docker-compose): Rename volume_root_logs_readonly -> volume_…
junhaoliao Oct 16, 2025
04e56f0
ensure `healthcheck` goes after `command`
junhaoliao Oct 16, 2025
0fc91d7
fix(docker-compose): Add AWS credentials environment variables in web…
junhaoliao Oct 16, 2025
f307d94
fix(docker-compose): Update environment variables for archive and str…
junhaoliao Oct 16, 2025
e5a90dd
Replace "docker-compose" (usually refer to v1) with "Docker Compose" …
junhaoliao Oct 16, 2025
3015ec4
lint
junhaoliao Oct 16, 2025
ff43c78
refactor(clp-package-utils): Reorder functions and rename is_docker_c…
junhaoliao Oct 16, 2025
699979a
fix(docker-compose): Add mounts for archive and stream output in garb…
junhaoliao Oct 16, 2025
092a86c
refactor(clp-package-utils): Rename _is_docker_compose_running -> _is…
junhaoliao Oct 16, 2025
8d75171
fix(docker-compose): Rename CLP_AWS_ACCESS_KEY_ID -> CLP_STREAM_OUTPU…
junhaoliao Oct 16, 2025
f53339f
fix(docker-compose): Break long volume mount lines for improved reada…
junhaoliao Oct 16, 2025
18d1109
lint
junhaoliao Oct 16, 2025
9d9bd46
refactor(clp-package-utils): Rename should_compose_run -> should_comp…
junhaoliao Oct 18, 2025
225c6c5
Merge branch 'main' into docker-compose
junhaoliao Oct 18, 2025
127aacc
refactor(clp-package-utils): Reorder private functions
junhaoliao Oct 18, 2025
f5ea3e2
Replace EnvironmentError with OSError as per Ruff os-error-alias (UP024)
junhaoliao Oct 18, 2025
22593d8
refactor(general): Improve error handling for Docker and Docker Compose
junhaoliao Oct 18, 2025
bc69b26
refactor(general): Update dump_container_config call to remove return…
junhaoliao Oct 18, 2025
1d2ab61
refactor(docs): Update exception documentation to use 'raise' format
junhaoliao Oct 18, 2025
d4c921a
add missing `f` in f-string
junhaoliao Oct 18, 2025
e6bb002
Add retention periods check for garbage collector and disable the ser…
junhaoliao Oct 20, 2025
90ee7da
Merge branch 'main' into docker-compose
junhaoliao Oct 20, 2025
b0d0bcc
refactor(clp-package-utils): Remove unused `pass` statements from abs…
junhaoliao Oct 21, 2025
955a0ac
refactor(clp-config): Remove unused typing import 'Set'.
junhaoliao Oct 21, 2025
106fdda
apply doc suggestions
junhaoliao Oct 21, 2025
7a1ea6e
refactor(docker-compose): Update environment variable error messages …
junhaoliao Oct 21, 2025
a732939
Adjust volume bindings to fix directory mappings when CLP_STAGED_ARCH…
junhaoliao Oct 21, 2025
c687f78
fix string indents
junhaoliao Oct 21, 2025
19e8a9d
apply doc suggestions
junhaoliao Oct 21, 2025
c53c0f1
apply doc suggestions
junhaoliao Oct 21, 2025
a2fa02c
fix(clp-package-utils): Remove extra space in `.env` file comment.
junhaoliao Oct 21, 2025
05810be
apply doc suggestions
junhaoliao Oct 21, 2025
2a02ecb
apply doc suggestions
junhaoliao Oct 21, 2025
678de80
Update comment for `_get_num_workers` method - apply doc suggestion
junhaoliao Oct 21, 2025
c30adae
Simplify log message for CLP start action - apply suggestion
junhaoliao Oct 21, 2025
07d1e35
Update comments for consistency in `controller.py`.
junhaoliao Oct 21, 2025
0bf6b70
feat(clp-package-utils): Add custom exceptions for Docker dependency …
junhaoliao Oct 21, 2025
2986082
fix(clp-package-utils): Use `docker --version` to check docker availa…
junhaoliao Oct 21, 2025
e499443
Avoid mapping ~/.aws on host when CLP_AWS_CONFIG_DIR_HOST is undefined
junhaoliao Oct 21, 2025
b788c33
alphabetize mounts
junhaoliao Oct 21, 2025
6fc219c
Do not map logs input dir when CLP_LOGS_INPUT_DIR_HOST is undefined
junhaoliao Oct 21, 2025
0926180
lint
junhaoliao Oct 21, 2025
e2daf32
lint
junhaoliao Oct 21, 2025
736c805
Merge remote-tracking branch 'origin/main' into docker-compose
junhaoliao Oct 21, 2025
4956106
fix: Add AWS config directory mount to garbage collector
junhaoliao Oct 21, 2025
91cacb0
fix(clp-package-utils): Add return type annotations to custom excepti…
junhaoliao Oct 21, 2025
3ad7f70
Merge branch 'main' into docker-compose
junhaoliao Oct 21, 2025
e8e4d58
Merge branch 'main' into docker-compose
junhaoliao Oct 22, 2025
c208942
Rename CLP_DB_IMAGE to CLP_DB_CONTAINER_IMAGE_REF.
kirkrodrigues Oct 22, 2025
6cea88b
Simplify package start/stop logs.
kirkrodrigues Oct 22, 2025
8c5a167
Refactor new Docker exceptions.
kirkrodrigues Oct 22, 2025
f02aff4
general.py: Minor refactoring.
kirkrodrigues Oct 22, 2025
667d753
Rename CLP_GC_LOGGING_LEVEL to CLP_GARBAGE_COLLECTOR_LOGGING_LEVEL.
kirkrodrigues Oct 22, 2025
930a2dd
Put operator on next line of multiline statement.
kirkrodrigues Oct 22, 2025
340a5e7
Use term 'config' instead of less common 'params'; Add required newli…
kirkrodrigues Oct 22, 2025
2bc2d2b
Correct AWS profile mount for webui - Apply suggestions from code review
junhaoliao Oct 22, 2025
f0fa5ae
Bump stop grace periods for various services.
junhaoliao Oct 22, 2025
26c3e32
Rewrite design doc.
kirkrodrigues Oct 23, 2025
164e27e
Rename design document file.
kirkrodrigues Oct 23, 2025
7759c5b
Lint.
kirkrodrigues Oct 23, 2025
0fdf31c
Split notes.
kirkrodrigues Oct 23, 2025
7447fef
Tell readers how to find the instance ID.
kirkrodrigues Oct 23, 2025
5227292
Rename Databases to Shared data stores.
kirkrodrigues Oct 23, 2025
a8ca533
Merge branch 'main' into docker-compose
junhaoliao Oct 23, 2025
ed25fc3
Update docs/src/dev-docs/design-deployment-orchestration.md
kirkrodrigues Oct 23, 2025
4926450
building-package.md: Minor fixes.
kirkrodrigues Oct 23, 2025
127cb79
Remove other details from multi-node doc until we add multi-node supp…
kirkrodrigues Oct 23, 2025
649276f
Add comment to explain the `+/tmp` hack.
kirkrodrigues Oct 23, 2025
7139740
docs(design-deployment): Rename "Shared data stores" subgraph to "Dat…
junhaoliao Oct 23, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions tools/deployment/package/docker-compose.base.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -264,6 +264,23 @@ services:
- *volume_clp_logs
- "${CLP_AWS_CONFIG_DIR_HOST:-empty}:/.aws:ro"
- "${CLP_LOGS_INPUT_DIR_HOST:-empty}:${CLP_LOGS_INPUT_DIR_CONTAINER:-/mnt/logs}"

# NOTE: Only one of `CLP_ARCHIVE_OUTPUT_DIR_HOST` and `CLP_STAGED_ARCHIVE_OUTPUT_DIR_HOST` are
# set at a time, but since `./var/data` on the host is mounted into the container and both
# `CLP_ARCHIVE_OUTPUT_DIR_HOST` and `CLP_STAGED_ARCHIVE_OUTPUT_DIR_HOST` default to
# directories under `./var/data`, we need to use a hack to avoid having Docker create the
# unset directory on the host (as root).
#
# For example, let's say we use the following as the mount for staged archives:
# "${CLP_STAGED_ARCHIVE_OUTPUT_DIR_HOST:-empty}:/var/data/staged-archives". If
# `CLP_STAGED_ARCHIVE_OUTPUT_DIR_HOST` is unset, Docker will create
# `/var/data/staged-archives` in the container, but it will also create
# `./var/data/staged-archives` on the host as root; this is because `/var/data` in the
# container is bind mounted to `./var/data` on the host.
#
# The hack to avoid this is if one of `CLP_ARCHIVE_OUTPUT_DIR_HOST` or
# `CLP_STAGED_ARCHIVE_OUTPUT_DIR_HOST` is unset, we set the target for the corresponding mount
# to a path that's not under `/var/data` in the container.
Comment on lines +268 to +283
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderabbitai please create an issue to track removal of this hack

an idea is to accept a var/tmp directory config from the user, and we generate all temporary runtime data in that directory instead of the current var/data directory. then we don't have to add the /var/data bind mount below

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Create GitHub issue to track removal of Docker Compose mount hack

gh issue create \
  --title "Remove Docker Compose mount hack for archive/staged-archive directories" \
  --body "## Context

Currently, the Docker Compose configuration contains a hack to avoid Docker creating directories as root on the host when mounting archive and staged-archive directories.

The issue arises because only one of \`CLP_ARCHIVE_OUTPUT_DIR_HOST\` and \`CLP_STAGED_ARCHIVE_OUTPUT_DIR_HOST\` is set at a time, but since \`./var/data\` on the host is mounted into the container and both variables default to directories under \`./var/data\`, Docker would create the unset directory on the host as root if we naively mount with a default path under \`/var/data\`.

## Current Hack

When one of the variables is unset, we set the target for the corresponding mount to a path that's not under \`/var/data\` in the container to prevent Docker from creating it on the host.

See: https://github.com/y-scope/clp/blob/docker-compose/tools/deployment/package/docker-compose.base.yaml#L267-L283

## Proposed Solution

Accept a \`var/tmp\` directory config from the user and generate all temporary runtime data in that directory instead of the current \`var/data\` directory. This would eliminate the need for the \`/var/data\` bind mount and remove the hack.

## References

- PR #1178
- Comment: https://github.com/y-scope/clp/pull/1178#discussion_r2455414543
- Requested by: @junhaoliao" \
  --assignee junhaoliao

Length of output: 42


@junhaoliao I've created issue #1478 to track the removal of this Docker Compose mount hack. The issue describes the current workaround and includes your proposed solution of using a separate var/tmp directory for temporary runtime data.

- "${CLP_ARCHIVE_OUTPUT_DIR_HOST:-empty}\
:${CLP_STAGED_ARCHIVE_OUTPUT_DIR_HOST:+/tmp}/var/data/archives"
- "${CLP_STAGED_ARCHIVE_OUTPUT_DIR_HOST:-empty}\
Expand Down
Loading