-
Notifications
You must be signed in to change notification settings - Fork 23
Added Dockerfile for CI images #195
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: dev
Are you sure you want to change the base?
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| # CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'} | ||
| ARG BASE_DOCKER=rocm/pytorch:rocm6.4_ubuntu22.04_py3.10_pytorch_release_2.5.1 | ||
| FROM $BASE_DOCKER | ||
| WORKDIR /var/lib/jenkins | ||
|
|
||
| RUN apt update \ | ||
| && apt install -y nano wget ninja-build \ | ||
| && apt install -y python3 python3-pip git \ | ||
| && apt install -y sqlite3 libsqlite3-dev libfmt-dev libmsgpack-dev libsuitesparse-dev | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What are those packages (sqlite3 and further) for?
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I recall aotriton need sqlite3 long time ago, not sure about now
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I have derived this dockerfile from this https://github.com/ROCm/DeepLearningModels/blob/main/docker/pyt_semianalysis_models.ubuntu.amd.Dockerfile, Will remove these, if these are not required.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @VeeraRajasekhar Those are needed for the semianalysis models. We don't need them for the CI images. We only need to keep those packages necessary for building TE and running CI tests. |
||
|
|
||
| RUN python3 -m pip install --upgrade pip | ||
| RUN pip install ninja cmake setuptools wheel | ||
VeeraRajasekhar marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| RUN pip install uv tabulate | ||
|
||
| RUN pip install ipython pytest fire pydantic pybind11 | ||
|
|
||
| RUN pip uninstall -y torch | ||
|
|
||
| RUN apt --fix-broken install -y | ||
VeeraRajasekhar marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| RUN apt install -y libzstd-dev | ||
| RUN apt install -y libibverbs-dev | ||
|
|
||
| ENV LLVM_SYMBOLIZER_PATH=/opt/rocm/llvm/bin/llvm-symbolizer | ||
| ENV PATH=$PATH:/opt/rocm/bin: | ||
| ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/rocm/lib/: | ||
|
|
||
| # Install pytorch | ||
| ARG PYTORCH_COMMIT="f929e0d602a71aa393ca2e6097674b210bdf321c" | ||
| ENV PYTORCH_ROCM_ARCH=gfx942 | ||
VeeraRajasekhar marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| RUN rm -fr pytorch \ | ||
| && git clone https://github.com/pytorch/pytorch \ | ||
| && cd pytorch \ | ||
| && git fetch origin ${PYTORCH_COMMIT} \ | ||
| && git checkout -q ${PYTORCH_COMMIT} \ | ||
| && git submodule update --recursive --init \ | ||
| && ./tools/amd_build/build_amd.py \ | ||
| && BUILD_TEST=0 python3 setup.py install | ||
|
|
||
| WORKDIR /var/lib/jenkins | ||
VeeraRajasekhar marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| # Install flash-attention | ||
| ENV GPU_ARCHS=${PYTORCH_ROCM_ARCH} | ||
| RUN git clone https://github.com/ROCm/flash-attention.git \ | ||
VeeraRajasekhar marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| && cd flash-attention \ | ||
| && git checkout v2.7.3-cktile \ | ||
| && pip install . | ||
|
|
||
| WORKDIR /var/lib/jenkins | ||
|
|
||
| # Install jax | ||
| RUN git clone -b rocm-jaxlib-v0.4.35-qa https://github.com/ROCm/jax.git \ | ||
VeeraRajasekhar marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| && git clone -b rocm-jaxlib-v0.4.35-qa https://github.com/ROCm/xla.git \ | ||
| && cd jax \ | ||
| && echo 'run:rocm_plugin --copt=-DLEGACY_HIPBLAS_DIRECT' > .bazelrc.user \ | ||
VeeraRajasekhar marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| && python3 ./build/build.py --enable_rocm \ | ||
| --build_gpu_plugin \ | ||
| --use_clang=true \ | ||
| --clang_path=/opt/rocm-6.4.0/lib/llvm/bin/clang \ | ||
| --gpu_plugin_rocm_version=60 \ | ||
| --rocm_path=/opt/rocm-6.4.0/ \ | ||
| --rocm_amdgpu_targets=gfx942 \ | ||
VeeraRajasekhar marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| --bazel_options=--override_repository=xla=/var/lib/jenkins/xla \ | ||
| && pip install jax==0.4.35 \ | ||
| && python3 setup.py develop --user && python3 -m pip install dist/*.whl \ | ||
| && pip install jax==0.4.35 | ||
|
|
||
| WORKDIR /workspace/ | ||
| CMD ["/bin/bash"] | ||
Uh oh!
There was an error while loading. Please reload this page.