Add Support for Custom Instance Profiles on Pools (#5144)

## Changes
<!-- Summary of your changes that are easy to understand -->

Added support for custom instance profiles on instance pools on AWS: add
`instance_profile_arn` field to `aws_attributes`.

## Tests
<!--
How is this tested? Please see the checklist below and also describe any
other relevant tests
-->

- [X] `make test` run locally
- [X] relevant change in `docs/` folder
- [X] has entry in `NEXT_CHANGELOG.md` file
- [X] Manually tested
    - Create instance pool with instance profile arn
    - Create cluster against instance pool with instance profile arn
- Updating instance pool resource to a different instance profile arn
recreates the resource

---------

Co-authored-by: Miles Yucht <[email protected]>
Co-authored-by: Alex Ott <[email protected]>

Author: 3 people

NEXT_CHANGELOG.md

@@ -8,6 +8,9 @@
 
 * Add `provider_config` support for manual plugin framework resources and data sources([#5127](https://github.com/databricks/terraform-provider-databricks/pull/5127))
 
+* Added support for custom instance profiles on instance pools on AWS ([#5144](https://github.com/databricks/terraform-provider-databricks/pull/5144))
+
+
 ### Bug Fixes
 
 * Remove unnecessary `SetSuppressDiff()` for `workload_size` in `databricks_model_serving` ([#5152](https://github.com/databricks/terraform-provider-databricks/pull/5152)).

docs/resources/instance_pool.md

@@ -58,6 +58,7 @@ The following options are [available](https://docs.databricks.com/dev-tools/api/
 * `zone_id` - (Optional) (String) Identifier for the availability zone/datacenter in which the instance pool resides. This string is of the form like `"us-west-2a"`. The provided availability zone must be in the same region as the Databricks deployment. For example, `"us-west-2a"` is not a valid zone ID if the Databricks deployment resides in the `"us-east-1"` region. If not specified, a default zone is used. You can find the list of available zones as well as the default value by using the [List Zones API](https://docs.databricks.com/dev-tools/api/latest/clusters.html#clusterclusterservicelistavailablezones).
 * `spot_bid_price_percent` - (Optional) (Integer) The max price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the instance pool needs a new i3.xlarge spot instance, then the max price is half of the price of on-demand i3.xlarge instances. Similarly, if this field is set to 200, the max price is twice the price of on-demand i3.xlarge instances. If not specified, the *default value is 100*. When spot instances are requested for this instance pool, only spot instances whose max price percentage matches this field are considered. *For safety, this field cannot be greater than 10000.*
 * `availability` - (Optional) (String) Availability type used for all instances in the pool. Only `ON_DEMAND` and `SPOT` are supported.
+* `instance_profile_arn` - (Optional) Nodes belonging to the pool will only be placed on AWS instances with this instance profile. Please see [databricks_instance_profile](instance_profile.md) resource documentation for extended examples on adding a valid instance profile using Terraform.
 
 ### azure_attributes Configuration Block
 

pools/resource_instance_pool.go

@@ -15,6 +15,7 @@ import (
 type InstancePoolAwsAttributes struct {
 	Availability        clusters.Availability `json:"availability,omitempty" tf:"force_new"`
 	ZoneID              string                `json:"zone_id,omitempty" tf:"computed,force_new"`
+	InstanceProfileArn  string                `json:"instance_profile_arn,omitempty" tf:"force_new"`
 	SpotBidPricePercent int32                 `json:"spot_bid_price_percent,omitempty" tf:"force_new"`
 }