Skip to content

Commit 1d834a0

Browse files
kcantrelkcantrel
committed
Initial Version
1 parent a78a071 commit 1d834a0

File tree

2 files changed

+260
-0
lines changed

2 files changed

+260
-0
lines changed

‎Management-Utilities/auto_set_fsxn_auto_grow/README.md‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
# Automatically Set Auto Size mode to Grow on FSx for NetApp ONTAP Volumes
2+
3+
## Introduction
4+
This project helps to mitigate the issue of not being able to set the auto size mode to
5+
'grow' when creating a FSxN volume from the AWS console or API. It does this by providing
6+
a Lambda function that will set the mode for you, and instructions on how to set up a
7+
CloudWatch event to trigger the Lambda function whenever a volume is created. With this
8+
combination it ensures that all volumes are effectively created with the auto size mode
9+
set to 'grow'.
10+
11+
## Set Up
12+
There are just a few things you have to do to set this up:
13+
14+
### Create a role for the Lambda function
15+
The Lambda function doesn't leverage that many AWS services, so only a few permissions are required:
16+
17+
18+
| Permission | Minimal Scope | Notes
19+
|:------------------------|:----------------|:----------------|
20+
| Allow:logs:CreateLogGroup | arn:aws:logs:<LAMBDA_REGION>:<ACCOUNT_ID>:* | This is required so you can get logs from the Lambda function. |
21+
| Allow:logs:CreateLogStream<BR>Allow:logs:PutLogEvents | arn:aws:logs:<LAMBDA_REGION>:<ACCOUNT_ID>:/aws/lambda/<LAMBDA_FUNCTION_NAME>:* | This is required so you can get logs from the Lambda function. |
22+
| Allow:secretsmanager:GetSecretValue | <ARN_OF_SECRET_WITHIN_SECRETS_MANAGER> | This is required so the Lambda function can get the credentials for the FSxN file system. |
23+
| Allow:fsx:DescribeFileSystems<BR>Allow:fsx:DescribeVolumes | * | You can't limit these API. They are required to get information regarding the file system and volumes. |
24+
| Allow:ec2:CreateNetworkInterface<BR>Allow:ec2:DeleteNetworkInterface<BR>Allow:ec2:DescribeNetworkInterfaces | * | Since the Lambda function is going to run within your VPC, it has to be able to create a network interface to communicate with the FSxn file system API. |
25+
26+
### Create AWS Endpoints
27+
Since the Lambda function will be configured to run within the VPC that contains the FSxN
28+
file system, so it can issue API calls against it, there will need to be AWS endpoints so
29+
the Lambda function can access some of the AWS service. If you have a Transit Gateway setup
30+
that allows access to the Internet, you may not have to create these endpoints, otherwise, the
31+
following endpoints will need to be created, and attached to the VPC and subnets that the
32+
FSxN file system is attached to.
33+
34+
- FSx
35+
- SecretsManager
36+
37+
### Create the Lambda Function
38+
Create a Lambda function with the following parameters:
39+
40+
- Authored from scratch
41+
- Uses the Python runtime
42+
- Set the permissions to the role created above.
43+
- Enable VPC. Found under the Advanced Settings
44+
- Attached to the VPC that contains the FSxN file system
45+
- Attached to the Subnets that contain the FSxN file system.
46+
- Attached a security group that allows access from any IP within the two subnets.
47+
48+
After you create the function, you will be able to insert the code included with this
49+
projrectinto the code box. Once you have inserted the code, modify the "secretsTable"
50+
array to provide the secrets name, and the keys for the username as password for each
51+
of the FSxN File Systems that you want to manage with this script. Also, set the
52+
secretsManagerRegion variable to the region where your secrets are stored.
53+
54+
Once you have updated the program, click on the "Deploy" button.
55+
56+
Next, click on the Configuration tab, then General and set the timeout to 10 seconds.
57+
58+
### Create an Event Bridge Rule (a.k.a. Cloud Watch Event) that will trigger when a FSx Volume is created
59+
Once on the "Event Bridge" page, click on Rules on the left add side. From there click
60+
on Create Rule. Give the rule a name, and make sure to put the rule on the "Default" bus.
61+
Finally select "Rule with an event pattern" and click Next.
62+
63+
Select "other" as the event source, skip pass the "Sample Event" section, and click on
64+
"Custom pattern (JSON editor)" under the Creation Method. Paste the following in the
65+
Edit Event Pattern text box:
66+
```json
67+
{
68+
"detail-type": [
69+
"AWS API Call via CloudTrail"
70+
],
71+
"detail": {
72+
"eventSource": [
73+
"fsx.amazonaws.com"
74+
],
75+
"eventName": [
76+
"CreateVolume"
77+
]
78+
}
79+
}
80+
```
81+
82+
Click Next. This next page will allow you to select the Lambda function you created above.
83+
Just take the defaults for the remaining pages and click on "Create Rule."
84+
85+
This point every time a volume is created the Lambda function will be called, and it will
86+
set the auto size mode to "grow".
87+
88+
## Author Information
89+
90+
This repository is maintained by the contributors listed on [GitHub](https://github.com/NetApp/FSx-ONTAP-samples-scripts/graphs/contributors).
91+
92+
## License
93+
94+
Licensed under the Apache License, Version 2.0 (the "License").
95+
96+
You may obtain a copy of the License at [apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0).
97+
98+
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an _"AS IS"_ basis, without WARRANTIES or conditions of any kind, either express or implied.
99+
100+
See the License for the specific language governing permissions and limitations under the License.
101+
102+
© 2024 NetApp, Inc. All Rights Reserved.

‎Management-Utilities/auto_set_fsxn_auto_grow/set_fsxn_volume_auto_grow.py‎

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
################################################################################
2+
# THIS SOFTWARE IS PROVIDED BY NETAPP "AS IS" AND ANY EXPRESS OR IMPLIED
3+
# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
4+
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
5+
# EVENT SHALL NETAPP BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
6+
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
7+
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
8+
# OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
9+
# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR'
10+
# OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
11+
# ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
12+
################################################################################
13+
14+
################################################################################
15+
# This Lambda function is used to set the auto size feature to 'grow' on a
16+
# volume that was created in an AWS FSx for NetApp ONTAP file system. It is
17+
# expected to be triggered by a CloudWatch event that is generated when a
18+
# volume is created. The function uses the ONTAP API to set the auto size
19+
# mode to 'grow' on the volume therefore it most run within the VPC where the
20+
# FSx for ONTAP file system is located.
21+
################################################################################
22+
23+
import json
24+
import time
25+
import urllib3
26+
from urllib3.util import Retry
27+
import logging
28+
import botocore
29+
import boto3
30+
#
31+
# Create a table of secret names and keys for the username and password for each of the FSxIds.
32+
secretsTable = [
33+
{"id": "fs-0e8d9172fa545ef3b", "secretName": "mon-fsxn-credentials", "usernameKey": "mon-fsxn-username", "passwordKey": "mon-fsxn-password"},
34+
{"id": "fs-020de2687bd98ccf7", "secretName": "mon-fsxn-credentials", "usernameKey": "mon-fsxn-username", "passwordKey": "mon-fsxn-password"},
35+
{"id": "fs-07bcb7ad84ac75e43", "secretName": "mon-fsxn-credentials", "usernameKey": "mon-fsxn-username", "passwordKey": "mon-fsxn-password"},
36+
{"id": "fs-077b5ff41951c57b2", "secretName": "mon-fsxn-credentials", "usernameKey": "mon-fsxn-username", "passwordKey": "mon-fsxn-password"}
37+
]
38+
#
39+
# Set the region where the secrets are stored.
40+
secretsManagerRegion="us-west-2"
41+
42+
################################################################################
43+
# This function is used to obtain the username and password from AWS's Secrets
44+
# Manager for the fsxnId passed in. It returns empty strings if it can't
45+
# find the credentials.
46+
################################################################################
47+
def getCredentials(secretsManagerClient, fsxnId):
48+
49+
for secretItem in secretsTable:
50+
if secretItem['id'] == fsxnId:
51+
secretsInfo = secretsManagerClient.get_secret_value(SecretId=secretItem['secretName'])
52+
secrets = json.loads(secretsInfo['SecretString'])
53+
username = secrets[secretItem['usernameKey']]
54+
password = secrets[secretItem['passwordKey']]
55+
return (username, password)
56+
return ("", "")
57+
58+
################################################################################
59+
# This function returns the UUID of the volume that has the ARN passed in. It
60+
# tries a few times, sleeping 1 second between attempts, since the UUID
61+
# doesn't exist until the volume has been created on the ONTAP side.
62+
# It returns an empty string if the ARN is not found.
63+
################################################################################
64+
def getVolumeUUID(fsxClient, volumeId, volumeARN):
65+
66+
global logger
67+
68+
cnt = 0
69+
while cnt < 3:
70+
awsVolume = fsxClient.describe_volumes(VolumeIds=[volumeId])['Volumes'][0]
71+
if awsVolume['ResourceARN'] == volumeARN:
72+
return awsVolume['OntapConfiguration']['UUID']
73+
logger.debug(f'Looping, getting the UUID {cnt}')
74+
cnt += 1
75+
time.sleep(1)
76+
77+
return ""
78+
79+
################################################################################
80+
################################################################################
81+
def lambda_handler(event, context):
82+
83+
global logger
84+
#
85+
# Set up "logging" to appropriately display messages. It can be set it up
86+
# to send messages to a syslog server.
87+
logging.basicConfig(datefmt='%Y-%m-%d_%H:%M:%S', format='%(asctime)s:%(name)s:%(levelname)s:%(message)s', encoding='utf-8')
88+
logger = logging.getLogger("scan_create_sm_relationships")
89+
# logger.setLevel(logging.DEBUG)
90+
logger.setLevel(logging.INFO)
91+
#
92+
# Set the logging level higher for these noisy modules to mute thier messages.
93+
logging.getLogger("botocore").setLevel(logging.WARNING)
94+
logging.getLogger("boto3").setLevel(logging.WARNING)
95+
logging.getLogger("urllib3").setLevel(logging.WARNING)
96+
#
97+
# Create a Secrets Manager client.
98+
session = boto3.session.Session()
99+
secretsManagerClient = session.client(service_name='secretsmanager', region_name=secretsManagerRegion)
100+
#
101+
# Disable warning about connecting to servers with self-signed SSL certificates.
102+
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
103+
#
104+
# Set the https retries to 1.
105+
retries = Retry(total=None, connect=1, read=1, redirect=10, status=0, other=0) # pylint: disable=E1123
106+
http = urllib3.PoolManager(cert_reqs='CERT_NONE', retries=retries)
107+
#
108+
# Get the FSxN ID, region, volume name, volume ID, and volume ARN from the CloudWatch event.
109+
fsxId = event['detail']['responseElements']['volume']['fileSystemId']
110+
regionName = event['detail']['awsRegion']
111+
volumeName = event['detail']['requestParameters']['name']
112+
volumeId = event['detail']['responseElements']['volume']['volumeId']
113+
volumeARN = event['detail']['responseElements']['volume']['resourceARN']
114+
if fsxId == "" or regionName == "" or volumeId == "" or volumeName == "" or volumeARN == "":
115+
message = f"Couldn't obtain the fsxId, region, volume name, volume ID or volume ARN from the CloudWatch evevnt."
116+
logger.critcal(message)
117+
raise Exception(mmessage)
118+
119+
logger.debug(f'Data from CloudWatch event: FSxID={fsxId}, Region={regionName}, VolumeName={volumeName}, volumeId={volumeId}.')
120+
#
121+
# Get the username and password for the FSxN ID.
122+
(username, password) = getCredentials(secretsManagerClient, fsxId)
123+
if username == "" or password == "":
124+
message = f'No credentials for FSxN ID: {fsxId}.'
125+
logger.critical(message)
126+
raise Exception(message)
127+
#
128+
# Build a header that is used for all the ONTAP API requests.
129+
auth = urllib3.make_headers(basic_auth=f'{username}:{password}')
130+
headers = { **auth }
131+
#
132+
# Get the management IP of the FSxN file system.
133+
fsxClient = boto3.client('fsx', region_name = regionName)
134+
fs = fsxClient.describe_file_systems(FileSystemIds = [fsxId])['FileSystems'][0]
135+
fsxnIp = fs['OntapConfiguration']['Endpoints']['Management']['IpAddresses'][0]
136+
if fsxnIp == "":
137+
message = f"Can't find manament IP for FSxN file system with an ID of '{fsxId}'."
138+
logger.critical(message)
139+
raise Exception(message)
140+
141+
volumeUUID = getVolumeUUID(fsxClient, volumeId, volumeARN)
142+
if volumeUUID == "":
143+
message = f"Can't find the volumeUUID based on the volume ARN {volumeARN}."
144+
logger.critical(message)
145+
raise Exception(message)
146+
#
147+
# Set the auto grow feature.
148+
try:
149+
endpoint = f'https://{fsxnIp}/api/storage/volumes/{volumeUUID}'
150+
data = '{"autosize": {"mode": "grow"}}'
151+
logger.debug(f'Trying {endpoint} with {data}.')
152+
response = http.request('PATCH', endpoint, headers=headers, timeout=5.0, body=data)
153+
if response.status >= 200 and response.status <= 299:
154+
logger.info(f"Updated the auto grow flag for volume name {volumeName}.")
155+
else:
156+
logger.error(f'API call to {endpoint} failed. HTTP status code: {response.status}.')
157+
except Exception as err:
158+
logger.critical(f'Failed to issue API against {fsxnIp}. The error messages received: "{err}".')

0 commit comments

Comments
 (0)