feat: Enhance Proxy Support and Build Infrastructure for Secure Boot … #120
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…Images
This commit introduces significant enhancements to the custom image building framework, primarily focused on supporting environments requiring HTTP/HTTPS proxy egress. This is critical for enterprise use cases with strict network policies. Additionally, it improves the robustness and reusability of the Secure Boot image generation process.
Key Changes:
1. **Integrated Proxy Setup:**
- Introduced `startup_script/gce-proxy-setup.sh` to handle system-wide proxy configuration on the builder VM based on instance metadata (`http-proxy`, `no-proxy`, `http-proxy-pem-uri`). This includes settings for apt/dnf, GPG, Java, and Conda.
- `custom_image_utils/shell_script_generator.py` now conditionally uploads `gce-proxy-setup.sh` only if `http-proxy` metadata is present.
- `startup_script/run.sh` executes `gce-proxy-setup.sh` before the user's customization script if proxy metadata is provided.
- `gce-proxy-setup.sh` is designed to be idempotent.
2. **Refactored Host vs. Container Setup:**
- `examples/secure-boot/build-and-run-podman.sh` now handles all host-side operations: sourcing environment, service account creation/validation, IAM bindings, and SA key generation (`key.json`).
- `examples/secure-boot/build-current-images.sh` now runs entirely within the container, consuming the mounted `key.json` via `GOOGLE_APPLICATION_CREDENTIALS`.
- Removed `gcloud config set` calls from scripts run inside the container, relying on the activated SA and per-command `--project` flags where needed.
3. **Improved Build Script Logic:
- `examples/secure-boot/pre-init.sh` now uses a unique temporary directory per image version (e.g., `/tmp/2.1-debian11`) to prevent conflicts during concurrent builds in the screen session.
- Added `--project-id` to all `generate_custom_image.py` calls in `pre-init.sh`.
- `custom_image_utils/shell_script_generator.py` now includes `--project={project_id}` in more `gcloud` calls within the generated workflow script.
- Enhanced `examples/secure-boot/create-key-pair.sh` for more robust Secure Boot key handling and secret management.
- Added `VmDnsSetting=ZonalOnly` to instance metadata to address DNS warnings.
4. **New Base Test Script:**
- Added `examples/secure-boot/no-customization.sh` to test the creation of base secure boot images without further customizations, including disk usage logging.
These changes provide a more reliable and flexible framework for building Dataproc custom images, especially for users in environments with network proxies and Secure Boot requirements.
This commit refactors how GPG keys for external repositories are imported within the `install_gpu_driver.sh` script. A new function, `import_gpg_keys`, is introduced in `install_gpu_driver.sh` to provide a consistent method for fetching keys from URLs or keyservers, handling potential proxy configurations, and importing them into specified keyring files. Key Changes: - **New `import_gpg_keys` Function:** Added a robust function to download and import GPG keys, supporting both `--key-url` and `--key-id` arguments, with keyserver fallback and basic proxy support awareness for `curl`. - **Updated Repository Setup:** All functions responsible for adding package repositories (e.g., `add_repo_nvidia_container_toolkit`, `add_repo_cuda`, `clean_up_sources_lists`) have been updated to use the new `import_gpg_keys` function, simplifying and standardizing key management. - **Conda Package Order:** Minor reordering of packages in the `conda_pkg_list` for Debian 10 in `install_pytorch` function. - **Indentation Cleanup:** Fixed minor indentation in `set_proxy` default_no_proxy_list. This refactoring improves the clarity, maintainability, and robustness of GPG key handling during the GPU driver and related software installation process.
This commit refactors the `set_proxy` function to provide more granular and flexible control over HTTP and HTTPS proxy configurations based on instance metadata.
**Key Enhancements:**
1. **Attribute Prioritization:** The function now reads and respects the following metadata attributes in order of specificity:
* `http-proxy`: For setting HTTP_PROXY.
* `https-proxy`: For setting HTTPS_PROXY.
* `proxy-uri`: As a fallback for either HTTP_PROXY or HTTPS_PROXY if the specific attributes are not set.
2. **Independent Configuration:** HTTP and HTTPS proxies can now be configured to different endpoints if both `http-proxy` and `https-proxy` are provided.
3. **Conditional Environment Variables:** The `HTTP_PROXY` and `HTTPS_PROXY` environment variables (and their lowercase counterparts) are only set if a corresponding value is derived from the metadata. They are unset otherwise.
4. **Clean `/etc/environment` Updates:** Existing proxy-related lines in `/etc/environment` are now removed before new ones are added, preventing duplicates.
5. **Tool Configuration:**
* Package manager (apt/dnf) proxy settings are based on the first available value from `http-proxy` or `https-proxy` metadata.
* GnuPG's `dirmngr.conf` is only configured with an `http-proxy` if the `http-proxy` or `proxy-uri` metadata is provided.
6. **Dynamic Scheme Change:** When `http-proxy-pem-uri` is provided and the certificate is processed, the function updates the relevant configurations (environment variables, package manager settings, dirmngr) to use the `https://` scheme for the proxy connections.
This refined logic allows users to precisely define their proxy setup, accommodating environments with distinct proxies for HTTP and HTTPS traffic, while maintaining backward compatibility with the single `proxy-uri` attribute.
tested as init action with proxy egress
cjac
force-pushed
the
proxy-exercise-2025-11
branch
from
7827ba4 to
c439451
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…Images
This commit introduces significant enhancements to the custom image building framework, primarily focused on supporting environments requiring HTTP/HTTPS proxy egress. This is critical for enterprise use cases with strict network policies. Additionally, it improves the robustness and reusability of the Secure Boot image generation process.
Key Changes:
Integrated Proxy Setup:
startup_script/gce-proxy-setup.shto handle system-wide proxy configuration on the builder VM based on instance metadata (http-proxy,no-proxy,http-proxy-pem-uri). This includes settings for apt/dnf, GPG, Java, and Conda.custom_image_utils/shell_script_generator.pynow conditionally uploadsgce-proxy-setup.shonly ifhttp-proxymetadata is present.startup_script/run.shexecutesgce-proxy-setup.shbefore the user's customization script if proxy metadata is provided.gce-proxy-setup.shis designed to be idempotent.Refactored Host vs. Container Setup:
examples/secure-boot/build-and-run-podman.shnow handles all host-side operations: sourcing environment, service account creation/validation, IAM bindings, and SA key generation (key.json).examples/secure-boot/build-current-images.shnow runs entirely within the container, consuming the mountedkey.jsonviaGOOGLE_APPLICATION_CREDENTIALS.gcloud config setcalls from scripts run inside the container, relying on the activated SA and per-command--projectflags where needed.**Improved Build Script Logic:
examples/secure-boot/pre-init.shnow uses a unique temporary directory per image version (e.g.,/tmp/2.1-debian11) to prevent conflicts during concurrent builds in the screen session.--project-idto allgenerate_custom_image.pycalls inpre-init.sh.custom_image_utils/shell_script_generator.pynow includes--project={project_id}in moregcloudcalls within the generated workflow script.examples/secure-boot/create-key-pair.shfor more robust Secure Boot key handling and secret management.VmDnsSetting=ZonalOnlyto instance metadata to address DNS warnings.New Base Test Script:
examples/secure-boot/no-customization.shto test the creation of base secure boot images without further customizations, including disk usage logging.These changes provide a more reliable and flexible framework for building Dataproc custom images, especially for users in environments with network proxies and Secure Boot requirements.