(3.x.x) Instance Bootstrap Failures and Protected Mode when Launching Large Number of Nodes

The issue

When launching a large number of instances, bootstrap failures may occur due to a delay in EC2 API eventual consistency. When this happens, the private IP addresses might not be available in the response of DescribeInstances API, even though the ips have been assigned at the response of RunInstances API.

This can be seen in the clustermgtd logs:

2026-03-04 17:50:20,107 - [slurm_plugin.instance_manager:get_cluster_instances] - WARNING - Ignoring instance i-1234abcd because not all EC2 info are available, exception: KeyError, message: 'PrivateIpAddress'

The cluster treats this as a bootstrap failure and can enter protected mode and fail cluster creation. For information on protected mode and how to recover from it, refer to this documentation.

Affected ParallelCluster versions, OSes and schedulers

• All ParallelCluster versions and OSes
• Slurm scheduler

Mitigation

Warning: This mitigation has the side effect that nodes with actual problems take longer to be marked as failed and replaced. Additionally, jobs scheduled to broken nodes wait longer before Slurm realizes the node is unavailable and reschedules them elsewhere.

The following steps should be followed to resolve this issue:

1. Create the following script that sets ec2_instance_missing_max_count=10. This adds a tolerance of incomplete information from EC2 instances(a value of 10 means clustermgtd wouldn't treat the nodes unhealthy and terminate the instances in the first 10 iterations of instance launch).

#!/bin/bash
set -e

CONFIG_FILE="/etc/parallelcluster/slurm_plugin/parallelcluster_clustermgtd.conf"

# Remove existing parameter if present, then add it
sed -i '/^ec2_instance_missing_max_count/d' "$CONFIG_FILE"
echo "ec2_instance_missing_max_count = 10" >> "$CONFIG_FILE"

1. Add the script as a HeadNode CustomAction for OnNodeConfigured in the cluster configuration file:

HeadNode:
  CustomActions:
    OnNodeConfigured:
      Script: s3://<your-bucket>/script.sh
  Iam:
    S3Access:
      - BucketName: <your-bucket>