[AWS] RAPIDS 24.10/24.12 and the sagemaker library can't be installed together #443
Description
Description
AWS distributes a Python client for SageMaker called sagemaker-sdk-python (aws/sagemaker-python-sdk).
That library's latest versions have a pin of cloudpickle==2.2.1
pyproject.toml: https://github.com/aws/sagemaker-python-sdk/blob/782fbfb27ba6696fda0bfd750ccb601ce511c1e5/pyproject.toml#L36- conda recipe: https://github.com/conda-forge/sagemaker-feedstock/blob/3877391c93926db2d4b17a0f131e8e13da6cf80e/recipe/meta.yaml#L27
RAPIDS 24.10 depends on distributed==2024.9.0 (code link) which pins to cloudpickle>=3.0.0.
Because of that, it's not possible to install sagemaker together with RAPIDS 24.10+ libraries that depend on Dask.
Reproducible Example
docker run \
--rm \
-it rapidsai/base:24.10a-cuda12.5-py3.11 \
bash
conda install -y -n base \
'sagemaker>=2.198'Results in this solver error:
Could not solve for environment specs
The following packages are incompatible
├─ pin-1 is installable and it requires
│ └─ python 3.11.* , which can be installed;
├─ rapids 24.10** is installable with the potential options
│ ├─ rapids 24.10.00a would require
│ │ └─ dask-cuda 24.10.* with the potential options
│ │ ├─ dask-cuda 24.10.00a23 would require
│ │ │ └─ rapids-dask-dependency 24.10.*,>=0.0.0a0 , which requires
│ │ │ └─ distributed 2024.9.0 , which requires
│ │ │ └─ cloudpickle >=3.0.0 , which can be installed;
│ │ ├─ dask-cuda 24.10.00a23 would require
│ │ │ └─ python >=3.10,<3.11.0a0 , which conflicts with any installable versions previously reported;
│ │ └─ dask-cuda 24.10.00a23 would require
│ │ └─ python >=3.12,<3.13.0a0 , which conflicts with any installable versions previously reported;
│ ├─ rapids 24.10.00a would require
│ │ └─ python >=3.10,<3.11.0a0 , which conflicts with any installable versions previously reported;
│ └─ rapids 24.10.00a would require
│ └─ python >=3.12,<3.13.0a0 , which conflicts with any installable versions previously reported;
└─ sagemaker >=2.198 is not installable because it requires
└─ cloudpickle 2.2.1 , which conflicts with any installable versions previously reported.
Notes
Upstream issue about relaxing that cloudpickle pin: aws/sagemaker-python-sdk#4871