feat: add support for datastore clusters

[tenthirtyam]

Description

Adds support for specifying a datastore_cluster (in addition to the existing datastore option) for virtual machine builds and post-processing. It ensures that datastore and datastore_cluster are mutually exclusive, updates documentation, and adjusts the build logic to handle Storage DRS recommendations for multi-disk VMs. Several steps in the build and upload process are updated to respect the resolved datastore selection.

Configuration and Validation Updates:

• Added datastore_cluster as a configuration option in builder and post-processor config structs, with validation to ensure only one of datastore or datastore_cluster is specified. [1] [2] [3] [4] [5] [6]

Documentation Improvements:

• Updated documentation for vsphere-clone, vsphere-iso, and vsphere post-processor components to describe the new datastore_cluster option, its mutual exclusivity with datastore, and Storage DRS behavior. [1] [2] [3] [4]

Build and Clone Logic Enhancements:

• Modified the VM cloning step to support disk placement recommendations from Storage DRS when a datastore_cluster is specified, including handling for multiple disks and fallback logic. [1] [2] [3] [4]
• Added a step to resolve the datastore or datastore cluster before VM creation.

Upload and Cleanup Step Adjustments:

• Updated steps for uploading and deleting files (floppy, ISO, CD, and remote cache) to use the resolved datastore from either datastore or datastore_cluster throughout the build and cleanup process. (F1d22be6L35R35, [1] [2] [3] [4] [5] [6] [7] [8]

Resolved Issues

Adds support for the use of datastore clusters in the applicable builders and post-processors.

Closes #9

Rollback Plan

Revert commit.

Changes to Security Controls

None.

[Gazzonyx]

Can we see this merged? It scratches an itch I've got and I'd prefer to not maintain my own downstream branch.

[tenthirtyam]

Can we see this merged? It scratches an itch I've got and I'd prefer to not maintain my own downstream branch.

@Gazzonyx, As the developer, once I have completed end testing of each builder and post processor, it will be removed from draft status and marked ready for review and merging.

Your patience is greatly appreciated, as the majority of this work was completed my personal capacity.

[Gazzonyx]

No worries! I just wanted to make sure it didn't get dropped on the floor. I noticed after commenting that it's tagged for the next 2.X release, so I know which release I'm waiting for. That greatly resolves my anxiety. Thank you for your effort and time, it's much appreciated.

[tenthirtyam]

No worries! I just wanted to make sure it didn't get dropped on the floor. I noticed after commenting that it's tagged for the next 2.X release, so I know which release I'm waiting for. That greatly resolves my anxiety. Thank you for your effort and time, it's much appreciated.

@Gazzonyx It's defintely not going to be dropped. 😉 In fact, this is a priority feature I developed for a customer that I directly support for on behalf of Broadcom.

[Copilot]

Pull request overview

This pull request adds support for specifying datastore_cluster in addition to the existing datastore option for virtual machine builds in both vsphere-iso and vsphere-clone builders, as well as the vsphere post-processor. The feature integrates Storage DRS (Distributed Resource Scheduler) to automatically select optimal datastores from a cluster, with proper fallback handling. The implementation includes comprehensive validation, documentation updates, and test coverage for the new functionality.

Key Changes:

• Added datastore_cluster configuration field with mutual exclusivity validation against datastore across all relevant components
• Implemented Storage DRS integration with intelligent multi-disk placement recommendations and fallback to first-available datastore when DRS fails
• Updated file upload and cleanup operations to use resolved datastores from either direct specification or cluster selection

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
post-processor/vsphere/post-processor.go	Added DatastoreCluster config field, validation logic, Storage DRS resolution, and vCenter connection for datastore selection
post-processor/vsphere/post-processor.hcl2spec.go	Added HCL2 spec for datastore_cluster field
post-processor/vsphere/post-processor_test.go	Updated test to set resolvedDatastore field
builder/vsphere/common/config_location.go	Added DatastoreCluster field with documentation and mutual exclusivity validation
builder/vsphere/common/config_location.hcl2spec.go	Added HCL2 spec for datastore_cluster field in LocationConfig
builder/vsphere/common/step_resolve_datastore.go	New step to resolve datastore from either direct name or cluster using Storage DRS
builder/vsphere/common/step_resolve_datastore_test.go	Comprehensive tests for datastore resolution step covering DRS, fallback, and error scenarios
builder/vsphere/common/step_add_floppy.go	Updated to use resolved datastore from state bag
builder/vsphere/common/step_remove_floppy.go	Updated to use resolved datastore from state bag
builder/vsphere/common/step_remote_upload.go	Updated to use resolved datastore from state bag
builder/vsphere/common/step_download.go	Updated to use resolved datastore from state bag
builder/vsphere/iso/step_create.go	Added multi-disk Storage DRS placement logic with fallback handling
builder/vsphere/iso/config.hcl2spec.go	Added HCL2 spec for datastore_cluster field
builder/vsphere/iso/builder.go	Added StepResolveDatastore to build pipeline
builder/vsphere/clone/step_clone.go	Added multi-disk Storage DRS placement logic with fallback handling
builder/vsphere/clone/config.hcl2spec.go	Added HCL2 spec for datastore_cluster field
builder/vsphere/clone/builder.go	Added StepResolveDatastore to build pipeline
builder/vsphere/driver/storage_drs.go	New file implementing Storage DRS API calls and datastore selection logic for single and multi-disk scenarios
builder/vsphere/driver/datastore_cluster.go	New file implementing datastore cluster interface and driver with methods to find clusters and list datastores
builder/vsphere/driver/datastore_cluster_mock.go	Mock implementation for testing datastore cluster operations
builder/vsphere/driver/disk.go	Updated to support per-disk datastore references for multi-disk placement
docs-partials/post-processor/vsphere/Config-required.mdx	Removed datastore from required fields documentation
docs-partials/post-processor/vsphere/Config-not-required.mdx	Added datastore and datastore_cluster to optional fields with descriptions
docs-partials/builder/vsphere/common/LocationConfig-not-required.mdx	Added datastore_cluster documentation and updated datastore requirements
.web-docs/components/post-processor/vsphere/README.md	Updated documentation with new datastore cluster option
.web-docs/components/builder/vsphere-iso/README.md	Updated documentation with new datastore cluster option
.web-docs/components/builder/vsphere-clone/README.md	Updated documentation with new datastore cluster option

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[tenthirtyam]

This one is now ready for your review, Karthik (@kp2099)! 🚀

Testing completed for the associated builders (vsphere-iso and sphere-clone) and post-processor (vsphere).

[Gazzonyx]

Sorry I missed the PR! Holiday season had me away from my inboxes. Thanks so much for getting this in so quickly! Cheers and Happy New Year!