Archive deletion fails in dataset manager due to path mapping mismatch

[coderabbitai]

Description

The native dataset manager script (native/dataset_manager.py) is currently unable to remove archives because it hasn't been updated to use the new in-container path (/var/data) introduced in the Docker Compose migration.

Root Cause

The script still relies on the previous direct host-to-container path mapping instead of the new container path structure.

Related

• PR: feat(deployment)!: Migrate package orchestration to Docker Compose (resolves #1177); Temporarily remove support for multi-node deployments. #1178
• Comment: feat(deployment)!: Migrate package orchestration to Docker Compose (resolves #1177); Temporarily remove support for multi-node deployments. #1178 (comment)
• Reported by: @junhaoliao

Expected Fix

Update native/dataset_manager.py and related native scripts to use the correct in-container paths that align with the Docker Compose deployment model.

[junhaoliao]

@quinntaylormitchell was the first reporter of the problem. Many thanks for the detailed reporting and thoughtful investigations into this!

attaching the offline discussion from @quinntaylormitchell for the record ->

I’ve discovered a problem with dataset_manager. I’ve compressed some data into default dataset; I confirm it’s there with
./sbin/admin-tools/dataset-manager.sh
list
then I run
./sbin/admin-tools/dataset-manager.sh
del default
I get the following error message:
2025-10-29T17:14:00.982 ERROR [dataset_manager] Failed to delete archives of dataset default.
Traceback (most recent call last):
File "/opt/clp/lib/python3/site-packages/clp_package_utils/scripts/native/dataset_manager.py", line 94, in _delete_dataset
_try_deleting_archives(clp_config.archive_output, dataset_archive_storage_dir)
File "/opt/clp/lib/python3/site-packages/clp_package_utils/scripts/native/dataset_manager.py", line 116, in _try_deleting_archives
_try_deleting_archives_from_fs(archive_output_config, dataset_archive_storage_dir)
File "/opt/clp/lib/python3/site-packages/clp_package_utils/scripts/native/dataset_manager.py", line 132, in _try_deleting_archives_from_fs
raise ValueError(
ValueError: '/var/data/archives/default' is not within top-level archive storage directory '/home/quinnmitchell/clp/build/clp-package/var/data/archives'
2025-10-29T17:14:02.510 ERROR [dataset_manager] Dataset manager failed.
I’ve looked around a bit and the error message itself comes from here, but I am pretty confident that that code worked pre-Docker Compose (cause I tested it when Haiqi was writing it). I looked at the mysql logs in the package and found that the archive_storage_directory stored in the database for default is '/var/data/archives/default':
7 Prepare INSERT INTO clp_datasets
(name, archive_storage_directory)
VALUES (?, ?)
7 Execute INSERT INTO clp_datasets
(name, archive_storage_directory)
VALUES ('default', '/var/data/archives/default')
so that means the directory is getting read from the database correctly. beyond that, I can only think that it’s something related to containers; maybe /var/data/archives is being used somewhere and some other part thinks that the /var is referring to the one in the root, not the container? tried to parse through some of the docker compose files but wasn’t really able to make any headway.