Skip to content

Commit 33db2e7

Browse files
quetzalliwritesquetzalliwrites
committed
added written content to all docs pages
1 parent fe19c8b commit 33db2e7

36 files changed

+5523
-6
lines changed

‎src/content/docs/integrations/infrastructure-as-code/cloud-custodian.md‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
---
2+
title: Cloud Custodian
3+
description: Use Cloud Custodian with LocalStack
4+
template:
5+
hero:
6+
tagline:
7+
image:
8+
file:
9+
actions:
10+
nav:
11+
---
12+
13+
## Introduction
14+
15+
Cloud Custodian is an open-source rules engine and cloud management tool designed to help organizations maintain security and compliance across their cloud environments.
16+
Cloud Custodian's YAML DSL allows definition of rules to filter and tag resources, and then apply actions to those resources.
17+
18+
Cloud Custodian can be used to manage local AWS resources in LocalStack, resembling the live AWS environment, allowing you to test and validate your security policies locally.
19+
You can use Cloud Custodian with LocalStack by just specifying the Cloud Custodian package to use the LocalStack profile configured with your AWS CLI.
20+
21+
## Getting started
22+
23+
This guide is designed for users who are new to Cloud Custodian and assumes basic knowledge of the AWS CLI and our [`awslocal`](https://github.com/localstack/awscli-local) wrapper script.
24+
25+
Start your LocalStack container using your preferred method.
26+
We will demonstrate how you can spin up an EC2 instance and tag it with the key `Custodian`, and then use Cloud Custodian to stop the instance.
27+
28+
### Install Cloud Custodian
29+
30+
To install Cloud Custodian, run the following command:
31+
32+
{{< command >}}
33+
$ pip install c7n
34+
{{< / command >}}
35+
36+
After installing Cloud Custodian, you can configure a [custom LocalStack profile](http://docs.localstack.cloud/user-guide/integrations/aws-cli/#configuring-a-custom-profile) in your AWS CLI configuration file.
37+
38+
### Create an EC2 instance
39+
40+
You can create an EC2 instance using the `awslocal` wrapper script.
41+
You can use the [`RunInstances`](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_RunInstances.html) API to create an EC2 instance.
42+
The following example creates an EC2 instance with the tag `Custodian` (any value):
43+
44+
{{< command >}}
45+
$ awslocal ec2 run-instances \
46+
--image-id ami-ff0fea8310f3 \
47+
--count 1 \
48+
--instance-type t3.nano \
49+
--tag-specifications "ResourceType=instance,Tags=[{Key=Custodian,Value=AnyValue}]"
50+
{{< / command >}}
51+
52+
You can navigate to the LocalStack logs to verify that the EC2 instance was created successfully:
53+
54+
```bash
55+
2023-10-16T15:27:35.479 INFO --- [ asgi_gw_0] l.u.container_networking : Determined main container network: bridge
56+
2023-10-16T15:27:35.504 INFO --- [ asgi_gw_0] l.u.container_networking : Determined main container target IP: 172.17.0.2
57+
2023-10-16T15:27:36.154 INFO --- [ asgi_gw_0] l.s.ec2.vmmanager.docker : Instance i-d87f1ab75e95ab0d2 will be accessible via SSH at: 127.0.0.1:22, 172.17.0.3:22
58+
2023-10-16T15:27:36.155 INFO --- [ asgi_gw_0] l.s.ec2.vmmanager.docker : Instance i-d87f1ab75e95ab0d2 port mappings (container -> host): {'22/tcp': 22}
59+
2023-10-16T15:27:36.218 INFO --- [ asgi_gw_0] localstack.request.aws : AWS ec2.RunInstances => 200
60+
```
61+
62+
### Create a Cloud Custodian policy
63+
64+
You can now create a Cloud Custodian policy to stop the EC2 instances with the tag `Custodian`.
65+
Create a file named `custodian.yml` and add the following content:
66+
67+
```yaml
68+
policies:
69+
- name: my-first-policy
70+
resource: aws.ec2
71+
filters:
72+
- "tag:Custodian": present
73+
actions:
74+
- stop
75+
```
76+
77+
The above policy specifies the following:
78+
79+
- `name`: The name of the policy.
80+
- `resource`: The AWS resource to apply the policy to.
81+
- `filters`: The filters to apply to the resource.
82+
In this case, the filter is `tag:Custodian` and the value is `present`.
83+
- `actions`: The actions to apply to the resource.
84+
In this case, the action is `stop`.
85+
86+
### Run the Cloud Custodian policy
87+
88+
You can now run the Cloud Custodian policy using the following command:
89+
90+
{{< command >}}
91+
$ custodian run \
92+
--output-dir=.
93+
custodian.yml \
94+
--profile localstack
95+
{{< / command >}}
96+
97+
{{< callout "tip" >}}
98+
Alternatively, you can also set the `AWS_PROFILE=localstack` environment variable, in which case the `--profile localstack` parameter can be omitted in the commands above.
99+
{{< /callout >}}
100+
101+
You should see the following output:
102+
103+
```sh
104+
2023-10-16 21:06:13,853: custodian.policy:INFO policy:my-first-policy resource:aws.ec2 region:us-east-1 count:1 time:0.20
105+
2023-10-16 21:06:13,961: custodian.policy:INFO policy:my-first-policy action:stop resources:1 execution_time:0.10
106+
```
107+
108+
You can then navigate to the LocalStack logs to verify that the EC2 instance was stopped successfully:
109+
110+
```bash
111+
2023-10-16T15:36:13.583 INFO --- [ asgi_gw_0] localstack.request.aws : AWS sts.GetCallerIdentity => 200
112+
2023-10-16T15:36:13.851 INFO --- [ asgi_gw_1] localstack.request.aws : AWS ec2.DescribeInstances => 200
113+
2023-10-16T15:36:13.960 INFO --- [ asgi_gw_0] localstack.request.aws : AWS ec2.StopInstances => 200
114+
```
115+
116+
### Create CloudWatch metrics for Cloud Custodian
117+
118+
Cloud Custodian creates CloudWatch metrics for each policy.
119+
These metrics show how many resources met the filters, the time taken to gather and filter those resources, and the time required to perform actions.
120+
121+
Certain filters and actions might produce their own metrics.
122+
To activate metric output, you must set the `metrics` flag running Cloud Custodian.
123+
124+
{{< command >}}
125+
$ custodian run -s . \
126+
--metrics aws custodian.yml \
127+
--profile localstack
128+
{{< / command >}}
129+
130+
You can access the CloudWatch metrics in the [LocalStack Web Application](https://app.localstack.cloud/inst/default/resources/cloudwatch).
131+
132+
## Further reading
133+
134+
- [Example Policies](https://cloudcustodian.io/docs/aws/examples/index.html) for practical examples of specific policies for individual AWS modules.
135+
- [Cloud Custodian documentation](https://cloudcustodian.io/docs/quickstart/index.html) to learn more about Cloud Custodian.
136+
- [Integrations](https://cloudcustodian.io/docs/aws/topics/index.html) with Config, Security Hub, Systems Manager, and X-Ray.
137+
- [Monitoring the environment](https://cloudcustodian.io/docs/aws/usage.html) by generating CloudWatch metrics.

‎src/content/docs/integrations/infrastructure-as-code/crossplane.md‎

Lines changed: 237 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,237 @@
1+
---
2+
title: Crossplane
3+
description: Use the Crossplane cloud-native control plane framework with LocalStack.
4+
template:
5+
hero:
6+
tagline:
7+
image:
8+
file:
9+
actions:
10+
nav:
11+
---
12+
13+
## Overview
14+
15+
[Crossplane](https://www.crossplane.io) is a cloud-native control plane framework, which offers an extensible backend that enables orchestrating applications and infrastructure via declarative APIs and resource definitions.
16+
17+
Crossplane offers a native [AWS provider](https://github.com/upbound/provider-aws) which can be used to create and manage AWS cloud resources via the Crossplane platform.
18+
For example, it can be used to create S3 buckets, SQS queues, Lambda functions, among many other resources.
19+
Crossplane AWS provider supports a comprehensive set of some [900+ resource types](https://marketplace.upbound.io/providers/upbound/provider-aws).
20+
21+
## Getting started
22+
23+
In the following, we provide a step-by-step guide for installing Crossplane in a local test environment, and creating AWS resources (S3 bucket, SQS queue) in LocalStack via Crossplane.
24+
25+
### Prerequisites
26+
27+
* LocalStack running in local Docker
28+
* A local Kubernetes cluster:
29+
* We can use the [embedded Kubernetes cluster](https://docs.docker.com/desktop/kubernetes) that ships with modern versions of Docker Desktop (can be easily enabled in the Docker settings)
30+
* Alternatively, you can [create a local EKS cluster](https://docs.localstack.cloud/user-guide/aws/elastic-kubernetes-service/#create-an-embedded-kubernetes-cluster) in LocalStack directly, which will spin up a light-weight embedded `k3d` Kubernetes cluster in your Docker environment
31+
* The [`helm`](https://helm.sh) and [`kubectl`](https://kubernetes.io/docs/tasks/tools/#kubectl) command-line clients installed
32+
33+
## Installing Crossplane in local Kubernetes
34+
35+
Once your `kubectl` is configured to point to the local Kubernetes cluster, we first install Crossplane via `helm`:
36+
{{<command>}}
37+
$ helm repo add crossplane-stable https://charts.crossplane.io/stable
38+
$ helm repo update
39+
$ helm install crossplane crossplane-stable/crossplane --namespace crossplane-system --create-namespace
40+
{{</command>}}
41+
42+
The installation may take a few minutes.
43+
In parallel, we can install the `crossplane` command-line tool.
44+
{{<command>}}
45+
$ curl -sL https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh | bash
46+
<disable-copy>...</disable-copy>
47+
$ sudo mv crossplane /usr/local/bin
48+
{{</command>}}
49+
To confirm that the installation was successful, we can run these commands, which should yield output similar to the following:
50+
{{<command>}}
51+
$ crossplane version
52+
<disable-copy>Client Version: v1.17.0
53+
Server Version: v1.17.0</disable-copy>
54+
55+
$ kubectl get crds | grep crossplane
56+
<disable-copy>compositions.apiextensions.crossplane.io 2023-09-03T11:30:36Z
57+
configurations.pkg.crossplane.io 2023-09-03T11:30:36Z
58+
...</disable-copy>
59+
{{</command>}}
60+
61+
### Installing the Crossplane AWS Provider
62+
63+
Once the basic Crossplane installation is running properly, we can proceed with installing the AWS provider.
64+
Newer versions of Crossplane promote the use of [provider families](https://docs.upbound.io/providers/provider-families), which are collections of providers for different groups of resources.
65+
For example, there is a separate provider for each individual AWS service (like S3, SQS, Lambda, etc), and in addition provider family provides shared resources for common configuration of all services (e.g., credentials, etc).
66+
67+
In the following, we first install the AWS provider for S3.
68+
Note that you can copy/paste the entire multi-line command below into your terminal:
69+
{{<command>}}
70+
$ cat <<EOF | kubectl apply -f -
71+
apiVersion: pkg.crossplane.io/v1
72+
kind: Provider
73+
metadata:
74+
name: provider-aws-s3
75+
spec:
76+
package: xpkg.upbound.io/upbound/provider-aws-s3:v0.40.0
77+
EOF
78+
{{</command>}}
79+
80+
We also install the AWS provider for SQS:
81+
{{<command>}}
82+
$ cat <<EOF | kubectl apply -f -
83+
apiVersion: pkg.crossplane.io/v1
84+
kind: Provider
85+
metadata:
86+
name: provider-aws-sqs
87+
spec:
88+
package: xpkg.upbound.io/upbound/provider-aws-sqs:v0.40.0
89+
EOF
90+
{{</command>}}
91+
92+
After some time, the providers should get into healthy state, which can be confirmed via `kubectl get providers`:
93+
{{<command>}}
94+
$ kubectl get providers<disable-copy>
95+
NAME INSTALLED HEALTHY PACKAGE AGE
96+
upbound-provider-family-aws True True xpkg.upbound.io/upbound/provider-family-aws:v0.40.0 2m
97+
provider-aws-s3 True True xpkg.upbound.io/upbound/provider-aws-s3:v0.40.0 2m
98+
provider-aws-sqs True True xpkg.upbound.io/upbound/provider-aws-sqs:v0.40.0 2m
99+
</disable-copy>
100+
{{</command>}}
101+
102+
Next, we install a secret to define the test credentials for the AWS provider:
103+
{{<command>}}
104+
$ cat <<EOF | kubectl apply -f -
105+
apiVersion: v1
106+
kind: Secret
107+
metadata:
108+
name: localstack-aws-secret
109+
stringData:
110+
creds: |
111+
[default]
112+
aws_access_key_id = test
113+
aws_secret_access_key = test
114+
EOF
115+
{{</command>}}
116+
117+
Finally, we create an AWS `ProviderConfig` that references the secret created above, and defines a static `endpoint` pointing to the LocalStack URL `http://host.docker.internal:4566`:
118+
{{<command>}}
119+
$ cat <<EOF | kubectl apply -f -
120+
apiVersion: aws.upbound.io/v1beta1
121+
kind: ProviderConfig
122+
metadata:
123+
name: default
124+
spec:
125+
credentials:
126+
source: Secret
127+
secretRef:
128+
name: localstack-aws-secret
129+
namespace: default
130+
key: creds
131+
endpoint:
132+
hostnameImmutable: true
133+
# TODO: add more services to this list, as needed
134+
services: [iam, s3, sqs, sts]
135+
url:
136+
type: Static
137+
static: http://host.docker.internal:4566
138+
skip_credentials_validation: true
139+
skip_metadata_api_check: true
140+
skip_requesting_account_id: true
141+
s3_use_path_style: true
142+
EOF
143+
{{</command>}}
144+
145+
{{< callout >}}
146+
The endpoint `http://host.docker.internal:4566` in the listing above assumes that you are running Kubernetes in the local Docker engine, and that LocalStack is up and running and available on default port `4566`.
147+
{{< /callout >}}
148+
149+
{{< callout >}}
150+
The Crossplane AWS provider currently requires us to specify the list of `services` for which the local `endpoint` is used as the target URL.
151+
Please make sure to extend this list accordingly if you're working with additional LocalStack services.
152+
{{< /callout >}}
153+
154+
### Deploying sample resources in LocalStack
155+
156+
After the Crossplane AWS provider is properly installed and configured, we can proceed with creating some local resources.
157+
158+
First, we create an S3 bucket named `crossplane-test-bucket`:
159+
{{<command>}}
160+
$ cat <<EOF | kubectl apply -f -
161+
apiVersion: s3.aws.upbound.io/v1beta1
162+
kind: Bucket
163+
metadata:
164+
name: crossplane-test-bucket
165+
spec:
166+
forProvider:
167+
region: us-east-1
168+
EOF
169+
{{</command>}}
170+
171+
If everything is wired up correctly, you should now see some activity in the LocalStack log outputs, where Crossplane starts deploying the S3 bucket against LocalStack.
172+
After some time, the bucket should be transitioning into `ready` state within Crossplane:
173+
{{<command>}}
174+
$ kubectl get buckets<disable-copy>
175+
NAME READY SYNCED EXTERNAL-NAME AGE
176+
crossplane-test-bucket True True crossplane-test-bucket 30s
177+
</disable-copy>
178+
{{</command>}}
179+
180+
...
181+
and the bucket it should also be visible when querying the local S3 buckets in LocalStack via [`awslocal`](https://github.com/localstack/awscli-local):
182+
{{<command>}}
183+
$ awslocal s3 ls<disable-copy>
184+
2023-09-03 15:18:47 crossplane-test-bucket
185+
</disable-copy>
186+
{{</command>}}
187+
188+
We can repeat the same exercise for creating a local SQS queue named `crossplane-test-queue`:
189+
{{<command>}}
190+
$ cat <<EOF | kubectl apply -f -
191+
apiVersion: sqs.aws.upbound.io/v1beta1
192+
kind: Queue
193+
metadata:
194+
name: crossplane-test-queue
195+
spec:
196+
forProvider:
197+
name: crossplane-test-queue
198+
region: us-east-1
199+
EOF
200+
{{</command>}}
201+
202+
After some time, the queue should transition into `ready` state in Crossplane:
203+
{{<command>}}
204+
$ kubectl get queues<disable-copy>
205+
NAME READY SYNCED EXTERNAL-NAME AGE
206+
crossplane-test-queue True True http://host.docker.internal:4566/000000000000/crossplane-test-queue 40s
207+
</disable-copy>
208+
{{</command>}}
209+
210+
...
211+
and the queue should be visible when listing the SQS queues in LocalStack:
212+
{{<command>}}
213+
$ awslocal sqs list-queues<disable-copy>
214+
{
215+
"QueueUrls": [
216+
"http://localhost:4566/000000000000/crossplane-test-queue"
217+
]
218+
}
219+
</disable-copy>
220+
{{</command>}}
221+
222+
### Summary
223+
224+
The Crossplane AWS provider is a great way to manage AWS resources, and by leveraging the `endpoint` configuration of the provider, we can seamlessly run resource deployments against LocalStack.
225+
226+
In this tutorial, we have provided an end-to-end walkthrough of how to provision two simple resources - an S3 bucket, and an SQS queue.
227+
Crossplane supports a vast range of additional AWS resource types, as well as advanced operations like updating, deleting, or composing resources.
228+
You can refer to the additional reading material to learn and explore more advanced features.
229+
230+
## Further Reading
231+
232+
* Kubernetes on Docker Desktop: https://docs.docker.com/desktop/kubernetes
233+
* Kubernetes getting started guide: https://kubernetes.io/docs/setup
234+
* EKS Kubernetes clusters on LocalStack: https://docs.localstack.cloud/user-guide/aws/elastic-kubernetes-service
235+
* Crossplane user docs: https://docs.crossplane.io
236+
* Crossplane AWS provider family: https://marketplace.upbound.io/providers/upbound/provider-family-aws
237+
* Crossplane AWS provider source code: https://github.com/upbound/provider-aws

0 commit comments

Comments
 (0)