Skip to content

[Integ] Add integration test for Pcluster Build Image in proxied environment #7324

Merged
himani2411 merged 2 commits intoaws:developfrom
himani2411:proxy-integ-test
Apr 9, 2026
Merged

[Integ] Add integration test for Pcluster Build Image in proxied environment #7324
himani2411 merged 2 commits intoaws:developfrom
himani2411:proxy-integ-test

Conversation

@himani2411
Copy link
Copy Markdown
Contributor

@himani2411 himani2411 commented Apr 1, 2026

Description of changes

Adds integration test which creates a proxied environment for testing ParallelCluster Build Image.
Updated proxy.yaml by introducing EnableBuildImageProxy to distinguish build-time from cluster-time proxy configurations. Key changes include:

  • Expands proxy allowlist for AWS services (CloudFront, S3), FSx repos, EFA installer, EFS-utils dependencies (Rust and Index) and OS repos.
  • Adds snap store allowlist to resolve dpkg lock issues during DCV installation
  • Grants AmazonS3ReadOnlyAccess for Node package presigned URL generation in cookbook recipe and use install_proxy_url Devsetting added as part of cookbook PR [DevSetting] Add install_proxy_url which will allow ParallelCluster to set Proxy environment for Build Image installation aws-parallelcluster-cookbook#3157
  • We skip awsbatch CLI installation for this test as we will be deprecating this dependency.
  • Enables IP forwarding to prevent kernel packet drops in ConfigureSystem step
  • Switches from GitHub to S3 URLs for aws-parallelcluster-cookbook downloads to avoid SSL connection failures through proxy

Tests

  • Successful Build image with ALINUX 2023, Rhel9/8, Rocky9/8 and Ubuntu22/24
  • Integ test
   proxy:
    test_proxy.py::test_proxy:
      dimensions:
        - regions: ["us-east-1"] # c5 instance type is not available in ap-southeast-5
          instances: ["c5.xlarge"]
          oss: [{{ OS_X86_7 }}]
          schedulers: ["slurm"]

References

  • Link to impacted open issues.
  • Link to related PRs in other packages (i.e. cookbook, node).
  • Link to documentation useful to understand the changes.

Checklist

  • Make sure you are pointing to the right branch.
  • If you're creating a patch for a branch other than develop add the branch name as prefix in the PR title (e.g. [release-3.6]).
  • Check all commits' messages are clear, describing what and why vs how.
  • Make sure to have added unit tests or integration tests to cover the new/modified code.
  • Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@himani2411 himani2411 requested review from a team as code owners April 1, 2026 20:45
@himani2411 himani2411 added skip-changelog-update Disables the check that enforces changelog updates in PRs 3.x labels Apr 1, 2026
@himani2411 himani2411 changed the title Proxy integ test [Integ] Add integration test for Pcluster Build Image in proxied environment Apr 1, 2026
@himani2411 himani2411 force-pushed the proxy-integ-test branch 9 times, most recently from 6d28a85 to 04b5a8c Compare April 2, 2026 20:30
stack_parameters = [
{"ParameterKey": "Keypair", "ParameterValue": request.config.getoption("key_name")},
{"ParameterKey": "VpcCidr", "ParameterValue": "10.0.0.0/16"},
{"ParameterKey": "SSHCidr", "ParameterValue": "0.0.0.0/0"},
Copy link
Copy Markdown
Contributor

@gmarciani gmarciani Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not introduced by this PR, so we can address in a different PR, but this is allowing global access to SSH port and could cause alarms. What about restricting the access to the VPC CIDR instead?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One of the checks we do as part of the test_proxy test to to be able to login into the HeadNode using a Bastion instance from another VPC, which is why I will keep the current SSHCidr. Since we have had no issues so far and this is an ephemeral stack we can decrease the priority for this specific change.

@himani2411 himani2411 force-pushed the proxy-integ-test branch 9 times, most recently from a2a02fa to 5abd7f3 Compare April 6, 2026 19:43
Switch proxy to al2023

Fix allowlist
@himani2411 himani2411 force-pushed the proxy-integ-test branch 2 times, most recently from a9dbb22 to 5a35651 Compare April 7, 2026 19:35
gmarciani
gmarciani previously approved these changes Apr 8, 2026
* Use EnableBuildImageProxy to distinguish between Build Image Proxy and Cluster Proxy (default)
* [DevSetting] Use install_http_proxy_address for setting up Proxy environment during build image
* Add snap store allowlist to overcome dpkg lock issue when installing DCV
* Enable IP forwarding to avoid Kernel dropping packets in ConfigureSystem Step of parallelcluster.yaml
* allowlist for awscli cloudfront url and global s3 endpoint
* Add access to FSX repos, efa installer, Rust and Index which are pre-req for Efs-utils using tiny proxy allowlist
* Skip awsbatch cli
* Add AmazonS3ReadOnlyAccess so that Presigned URL for S3 Node package is created
* using s3 bucket url to avoid proxy connection issue for github endpoint

```
url -sS -L -w '%{http_code}' -o /etc/chef/aws-parallelcluster-cookbook.tgz https://github.com/aws/aws-parallelcluster-cookbook/tarball/refs/heads/develop
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to github.com:443
```
* Upgrade Proxy instance OS to ubuntu22
@himani2411 himani2411 added the skip-bad-url-suffix-check Skip the checks regarding the bad URL suffix label Apr 9, 2026
@himani2411 himani2411 merged commit 6870bc8 into aws:develop Apr 9, 2026
26 of 27 checks passed
@gmarciani
Copy link
Copy Markdown
Contributor

We agreed on addressing the compatibility with other partitions in a separate PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3.x skip-bad-url-suffix-check Skip the checks regarding the bad URL suffix skip-changelog-update Disables the check that enforces changelog updates in PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants