Skip to content

Commit 5e1db28

Browse files
nicolai86awaelchli
nicolai86
and
awaelchli
authored
update BYOC documentation with AWS details (#16044)
Co-authored-by: Adrian Wälchli <[email protected]>
1 parent 612d43e commit 5e1db28

File tree

6 files changed

+472
-86
lines changed

6 files changed

+472
-86
lines changed

‎docs/source-app/workflows/byoc/aws_cli.rst‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
:orphan:
2+
3+
.. _aws_cli:
4+
5+
6+
############################
7+
Create AWS role with AWS CLI
8+
############################
9+
10+
1. Install AWS CLI (see instructions `here <https://aws.amazon.com/cli/>`_).
11+
12+
2. Protect your role by creating a hard to guess password that will be used to authenticate Lightning (You will need to pass it to Lightning for authentication). In our example we will use `dummy`.
13+
14+
3. Create a role called `lightning-cloud` using the following command (replace <YOUR-HARD-TO-GUESS-PASSWORD> with your own):
15+
16+
.. code:: bash
17+
18+
aws iam create-role \
19+
--role-name lightning-cloud \
20+
--assume-role-policy-document '{"Statement":[{"Action":"sts:AssumeRole","Effect": "Allow", "Principal": {"AWS": "arn:aws:iam::748115360335:root"}, "Condition": {"StringEquals": {"sts:ExternalId": "<YOUR-HARD-TO-GUESS-PASSWORD>"}}}]}' \
21+
--description " " \
22+
--max-session-duration 43200
23+
24+
4. Create a file `iam-policy.json` with the following permissions required for Lightning to manage cloud infrastructure for you:
25+
26+
.. code:: json
27+
28+
{
29+
"Version": "2012-10-17",
30+
"Statement": [
31+
{
32+
"Action": [
33+
"autoscaling:*",
34+
"cloudwatch:*",
35+
"ec2:*",
36+
"ecr:*",
37+
"eks:*",
38+
"elasticloadbalancing:*",
39+
"events:*",
40+
"guardduty:*",
41+
"iam:*",
42+
"logs:*",
43+
"route53resolver:*",
44+
"s3:*",
45+
"sns:*",
46+
"sqs:*",
47+
"tag:GetResources",
48+
"resource-groups:SearchResources"
49+
],
50+
"Effect": "Allow",
51+
"Resource": "*"
52+
},
53+
{
54+
"Effect": "Allow",
55+
"Action": "iam:CreateServiceLinkedRole",
56+
"Resource": "*",
57+
"Condition": {
58+
"StringLike": {
59+
"iam:AWSServiceName": [
60+
"guardduty.amazonaws.com",
61+
"malware-protection.guardduty.amazonaws.com"
62+
]
63+
}
64+
}
65+
},
66+
{
67+
"Effect": "Allow",
68+
"Action": "iam:CreateServiceLinkedRole",
69+
"Resource": "*",
70+
"Condition": {
71+
"StringEquals": {
72+
"iam:AWSServiceName": [
73+
"autoscaling.amazonaws.com",
74+
"ec2scheduled.amazonaws.com",
75+
"elasticloadbalancing.amazonaws.com",
76+
"spot.amazonaws.com",
77+
"spotfleet.amazonaws.com",
78+
"transitgateway.amazonaws.com"
79+
]
80+
}
81+
}
82+
}
83+
]
84+
}
85+
86+
5. Create a IAM policy and associate it with the role we just created, and pass in the path to your new file:
87+
88+
.. code:: bash
89+
90+
aws iam create-policy \
91+
--policy-name lightning-cloud \
92+
--description "policy granting lightning controlplane permissions" \
93+
--policy-document file:///my_dir/iam-policy.json
94+
95+
6. Fetch the role ARN so you can attach the policy:
96+
97+
.. code:: bash
98+
99+
aws iam get-role --role-name lightning-cloud --output json --query Role.Arn
100+
101+
7. Attach the policy to the IAM role you just created:
102+
103+
.. code:: bash
104+
105+
aws iam attach-role-policy \
106+
--role-name lightning-cloud \
107+
--policy-arn arn:aws:iam::1234567890:policy/lightning-cloud
108+
109+
------
110+
111+
**********************
112+
Next: Create a cluster
113+
**********************
114+
115+
You are now ready to create a Lightning cluster!
116+
117+
.. raw:: html
118+
119+
<div class="display-card-container">
120+
<div class="row">
121+
122+
.. Add callout items below this line
123+
124+
.. displayitem::
125+
:header: Create cluster
126+
:description: Create an AWS cluster for running ligthning apps, skip to step 2
127+
:button_link: create_cluster.html
128+
:col_css: col-md-12
129+
:height: 170
130+
131+
.. raw:: html
132+
133+
</div>
134+
</div>

‎docs/source-app/workflows/byoc/aws_console.rst‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
:orphan:
2+
3+
.. _aws_console:
4+
5+
6+
###########################################
7+
Create AWS role with AWS management console
8+
###########################################
9+
10+
See the following video for instructions:
11+
12+
.. raw:: html
13+
14+
<video width="100%" max-width="800px" controls autoplay muted playsinline
15+
src="https://lightningaidev.wpengine.com/wp-content/uploads/2022/12/byoc.mp4"></video>
16+
17+
|
18+
19+
20+
1.1- Create a role
21+
------------------
22+
23+
The created role will give Lightning AI access (by specifying Ligthning account id), and will
24+
25+
1. Sign in to your AWS account. (If you don’t have an AWS account, create one).
26+
27+
2. On AWS console search for IAM > Roles > Create role.
28+
29+
In your new role set the following:
30+
31+
* Trusted entity type -> AWS account.
32+
33+
* An AWS account -> Another AWS account -> Account id: 748115360335. (This is Lightning's account id).
34+
35+
3. Protect your role by creating a hard to guess password that will be used to authenticate Lightning (You will need to pass it to Lightning for authnetication).
36+
37+
* Set Require external ID: <put-your-hard-to-guess-token>
38+
39+
4. Click "Next"
40+
41+
42+
1.2- Create a policy
43+
--------------------
44+
45+
1. Click "Create policy" -> A new tab opens to the Create policy page.
46+
47+
2. Click the JSON tab and copy-paste the `contents of this JSON file <https://gist.github.com/nicolai86/79b9caa5a3557d9d4f8a6e2227e34f2d>` into the JSON workspace.
48+
49+
.. note: Ignore the warnings generated by AWS.
50+
51+
3. Click "Next:Tags" (you can skip this step)
52+
53+
4. Complete policy creation by adding a meaningful name and description to the policy.
54+
55+
For example:
56+
57+
Name -> "lightning-cloud"
58+
59+
Description -> "permission to manage EC2 instances"
60+
61+
5. Add policy to the role: Return to the browser tab that you were using to create a role.
62+
Refresh the policy list and select the custom policy you just created (in this case “lai-byoc-policy”). The policy should appear at the top of the Policy List.
63+
64+
6. Click "Next".
65+
66+
7. Provide a meaningful name and description for the role, and click “Create Role”.
67+
68+
For example:
69+
70+
Name -> "lightning-cloud"
71+
72+
Description -> "Role for Lightning cloud permissions"
73+
74+
Congrats! You now successfully added the set up permissions needed for lightning to create a cluster.
75+
76+
77+
------
78+
79+
**********************
80+
Next: Create a cluster
81+
**********************
82+
83+
You are now ready to create a Lightning cluster!
84+
85+
.. raw:: html
86+
87+
<div class="display-card-container">
88+
<div class="row">
89+
90+
.. Add callout items below this line
91+
92+
.. displayitem::
93+
:header: Create cluster
94+
:description: Create an AWS cluster for running ligthning apps, skip to step 2
95+
:button_link: create_cluster.html
96+
:col_css: col-md-12
97+
:height: 170
98+
99+
.. raw:: html
100+
101+
</div>
102+
</div>

‎docs/source-app/workflows/byoc/create_cluster.rst‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
:orphan:
2+
3+
.. _create_cluster:
4+
5+
6+
##################
7+
Create AWS cluster
8+
##################
9+
10+
**Audience:** Users looking to create a cluster to run Lightning Apps on their own private cloud infrastructure.
11+
12+
**Prereqs:** basic familiarity with cloud provider infrastructure management.
13+
14+
.. note:: This feature is currently available for early access! To create your own clutser `contact us <mailto:[email protected]?subject=I%20want%20to%20run%20on%20my%20private%20cloud!>`_.
15+
16+
17+
----
18+
19+
*******************************************
20+
Step 1- Create roles and permissions on AWS
21+
*******************************************
22+
23+
In this step you’ll be creating a role on your cloud provider that allows Lightning to manage resources on your behalf (for example, creating EC2 instances for your cluster).
24+
To do this you can use the AWS CLI or the AWS management console.
25+
26+
You will only have to preform this step once, and the same role can be used to create multiple clusters.
27+
28+
----
29+
30+
.. raw:: html
31+
32+
<div class="display-card-container">
33+
<div class="row">
34+
35+
.. displayitem::
36+
:header: Create role with AWS CLI
37+
:description: Create role with AWS CLI
38+
:col_css: col-md-4
39+
:button_link: aws_cli.html
40+
:height: 180
41+
:tag: Basic
42+
43+
.. displayitem::
44+
:header: Create role with AWS console
45+
:description: Create role with AWS console
46+
:col_css: col-md-4
47+
:button_link: aws_console.html
48+
:height: 180
49+
:tag: Basic
50+
51+
.. raw:: html
52+
53+
</div>
54+
</div>
55+
56+
57+
----
58+
59+
60+
****************************
61+
Step 2- Get ARN for new role
62+
****************************
63+
64+
To start a cluster, Lightning will need the external-id you set in step 1, and the ARN role. Click on your new role to find them (note: you need the ARN listed at the top of the page, not the one in “trusted entitles”).
65+
66+
67+
Record the ARN and the external ID. You’ll need them for your next step.
68+
69+
-----
70+
71+
**************************************
72+
Step 3-Create a Lightning BYOC cluster
73+
**************************************
74+
75+
Now that you have created a role and policy on AWS, you can start creating Lightning clusters.
76+
77+
Create a Lightning BYOC cluster using the following command:
78+
79+
.. code:: bash
80+
81+
lightning create cluster <cluster-name> --role-arn <ARN> --external-id <EXTERNAL-ID>
82+
83+
Here's an example:
84+
85+
.. code:: bash
86+
87+
lightning create cluster my-byoc-cluster --role-arn arn:aws:iam::1234567890:role/lai-byoc --external-id dummy
88+
89+
.. note:: Cluster creation is going to take an hour or more after you run this command.
90+
91+
92+
Parameters
93+
==========
94+
95+
+------------------------+----------------------------------------------------------------------------------------------------+
96+
|Parameter | Description |
97+
+========================+====================================================================================================+
98+
| cluster_id | The name of the cluster to be created. |
99+
| | |
100+
| | Cluster names can only contain lowercase letters, numbers, and periodic hyphens ( - ). |
101+
+------------------------+----------------------------------------------------------------------------------------------------+
102+
| role-arn | AWS IAM Role ARN used to provision resources |
103+
+------------------------+----------------------------------------------------------------------------------------------------+
104+
| external-id | AWS IAM Role external ID |
105+
| | |
106+
| | To read more on what the AWS external ID is and why it's useful go |
107+
| | `here <https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html>`_|
108+
+------------------------+----------------------------------------------------------------------------------------------------+
109+
110+
----
111+
112+
*******************************************
113+
View a list of your Lightning BYOC clusters
114+
*******************************************
115+
116+
.. code:: bash
117+
118+
lightning list clusters
119+
120+
---
121+
122+
******************************
123+
Next: Run apps on your cluster
124+
******************************
125+
126+
Once your cluster is running, you can start running Lightning apps on your cluster.
127+
128+
.. raw:: html
129+
130+
<div class="display-card-container">
131+
<div class="row">
132+
133+
.. Add callout items below this line
134+
135+
.. displayitem::
136+
:header: Run apps on your cluster
137+
:description: Learn how to start apps on your Lightning cluster
138+
:button_link: run_on_cluster.html
139+
:col_css: col-md-12
140+
:height: 170
141+
142+
.. raw:: html
143+
144+
</div>
145+
</div>

0 commit comments

Comments
 (0)