[TRTLLM-9227][infra] Add Dockerfile step to copy sources

In this change we add a new Dockerfile step which copies third-party
source files into the container image. This is necessary for compliance
with certain open source licenses which require source code
availability. Distributing the sources as-used in the container image
is a simple and direct way of ensuring the sources are used and mapped
to the version that was built in the container.

The script to perform this step relies on the C++ third-party process
which requires third-party libraries to be integrated with CMake's
FetchContent modules. The script scans the _deps/ directory and collects
all the *-src directories which are created by FetchContent. It
archives these into same-named tarballs and places them in
`/third_party_sources/`.

Signed-off-by: Josh Bialkowski <1309820+cheshirekow@users.noreply.github.com>

Author: cheshirekow

docker/Dockerfile.multi

@@ -128,7 +128,10 @@ ARG GITHUB_MIRROR=""
 ARG BUILD_WHEEL_ARGS="--clean --benchmarks"
 ARG BUILD_WHEEL_SCRIPT="scripts/build_wheel.py"
 RUN --mount=type=cache,target=/root/.cache/pip --mount=type=cache,target=${CCACHE_DIR} \
-    GITHUB_MIRROR=$GITHUB_MIRROR python3 ${BUILD_WHEEL_SCRIPT} ${BUILD_WHEEL_ARGS}
+    GITHUB_MIRROR=$GITHUB_MIRROR python3 ${BUILD_WHEEL_SCRIPT} ${BUILD_WHEEL_ARGS} \
+    && python3 scripts/copy_third_party_sources.py \
+      --deps-dir ./build/_deps \
+      --output-dir /third-party-sources
 
 FROM ${DEVEL_IMAGE} AS release
 
@@ -155,6 +158,8 @@ ARG SRC_DIR=/src/tensorrt_llm
 COPY --from=wheel ${SRC_DIR}/benchmarks benchmarks
 ARG CPP_BUILD_DIR=${SRC_DIR}/cpp/build
 COPY --from=wheel \
+     /third-party-sources \
+     ${CPP_BUILD_DIR}/_deps \
      ${CPP_BUILD_DIR}/benchmarks/bertBenchmark \
      ${CPP_BUILD_DIR}/benchmarks/gptManagerBenchmark \
      ${CPP_BUILD_DIR}/benchmarks/disaggServerBenchmark \

scripts/copy_third_party_sources.py

@@ -0,0 +1,51 @@
+"""Copy third-party sources used in the cmake build to a container directory.
+
+The purpose of this script is to simplify the process of producing third party
+sources "as used" in the build. We package up all of the sources we use and
+stash them in a location in the container so that they are automatically
+distributed alongside the build artifacts ensuring that we comply with the
+license requirements in an obvious and transparent manner.
+"""
+
+import argparse
+import logging
+import pathlib
+import subprocess
+
+logger = logging.getLogger(__name__)
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--deps-dir",
+        type=pathlib.Path,
+        required=True,
+        help="Path to the third party dependencies directory, e.g. ${CMAKE_BINARY_DIR}/_deps",
+    )
+    parser.add_argument(
+        "--output-dir",
+        type=pathlib.Path,
+        required=True,
+        help="Path to the output directory where third party sources will be copied",
+    )
+
+    args = parser.parse_args()
+
+    src_dirs = list(sorted(args.deps_dir.glob("*-src")))
+    if not src_dirs:
+        raise ValueError(f"No source directories found in {args.deps_dir}")
+
+    for src_dir in src_dirs:
+        tarball_name = src_dir.name[:-4] + ".tar.gz"
+        output_path = args.output_dir / tarball_name
+        logger.info(f"Creating tarball {output_path} from {src_dir}")
+        args.output_dir.mkdir(parents=True, exist_ok=True)
+        subprocess.run(
+            ["tar", "-czf", str(output_path), "-C", str(src_dir.parent), src_dir.name], check=True
+        )
+
+
+if __name__ == "__main__":
+    logging.basicConfig(level=logging.INFO)
+    main()