Unusual docker image size when custom requirements are installed

[arunaru-te]

Hello,

i am creating a custom docker image from sagemaker-distribution GPU 3.2.0 image, i am not doing anything fancy, just trying to install few python packages and image size is around 28GB.

Need help to confirm if this is expected, if not please tell me what i am doing wrong.

conda-requirements.txt

Dockerfile.txt

[claytonparnell]

Hi, thank you for using SageMaker Distribution! Unexpected size increases happen often when building Docker images and can be hard to diagnose. One thing which can help is combining RUN commands when touching the same file paths, to avoid duplicate size increase in subsequent layers.

One tool which is useful for figuring which layer/file path is responsible for the size increase is dive: https://github.com/wagoodman/dive.

I can try to repro on my end when I have time, but in the meantime you can try dive out to figure out the layer(s) causing increase, and then determine the file path(s) where increase occurs. The key here is to try not to install the same package over again or touch the same file path twice in different layers.