Skip to content

Commit b1d2cb7

Browse files
petercrockerpetercrocker
authored
feat: add vendor-agnostic cloud schema for AWS, GCP, and Azure (#20)
* feat: add vendor-agnostic cloud schema for AWS, GCP, and Azure Add a comprehensive cloud schema to store compute and networking objects across multiple cloud providers in a vendor-agnostic way. Schema includes: - CloudProvider, CloudAccount, CloudRegion, CloudAvailabilityZone - CloudVirtualNetwork, CloudSubnet, CloudSecurityGroup - CloudInstance, CloudNetworkInterface - CloudInternetGateway, CloudNATGateway, CloudRouteTable, CloudElasticIP Also includes: - Demo object files with sample data for all 3 cloud providers - New `invoke demo-cloud` task to load the cloud demo data * ruff * docs: add cloud resource management tutorial Add documentation for the cloud schema and demo data: - New cloud-management.mdx tutorial explaining the vendor-agnostic cloud schema for AWS, GCP, and Azure - Update readme.mdx to reference the new tutorial - Update sidebars.ts to include the new page in navigation * Edits of docs * Spelling
1 parent 770f586 commit b1d2cb7

15 files changed

+2522
-1
lines changed

‎.gitignore‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,8 @@
1515
**/*.tar.xz
1616
docker-compose.yml
1717
generated-configs/
18+
infrahub_backups/
1819
infrahub_bundle_dc.egg-info
1920
infrahub-backup
2021
scripts/debug/
21-
service_catalog/.streamlit/
22+
service_catalog/.streamlit/

‎.vale/styles/spelling-exceptions.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ dns
2323
Docker's
2424
Dockerfile
2525
Docusaurus
26+
ENIs
2627
env
2728
envrc
2829
evolvability
@@ -47,6 +48,8 @@ mermaid
4748
namespace
4849
namespaces
4950
Netbox
51+
NICs
52+
NSGs
5053
netmiko
5154
npm
5255
OSPF
@@ -72,5 +75,7 @@ uv
7275
VIPs
7376
VLAN
7477
VLANs
78+
VNets
79+
VPCs
7580
VXLAN
7681
walkthrough

‎docs/docs/cloud-management.mdx‎

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
---
2+
title: Cloud resource management
3+
---
4+
5+
import Tabs from '@theme/Tabs';
6+
import TabItem from '@theme/TabItem';
7+
8+
This tutorial shows an example of cloud resource management schema, which provides a vendor-agnostic way to model cloud infrastructure across AWS, GCP, and Azure. You'll load sample cloud data and explore how Infrahub can serve as a unified inventory for multi-cloud environments.
9+
10+
## Overview
11+
12+
The cloud schema enables you to track:
13+
14+
- **Cloud providers** - AWS, GCP, Azure, or other cloud platforms
15+
- **Cloud accounts** - AWS accounts, GCP projects, Azure subscriptions
16+
- **Regions and availability zones** - Geographic locations and fault domains
17+
- **Virtual networks** - VPCs (AWS), VPC networks (GCP), VNets (Azure)
18+
- **Subnets** - Network segments within virtual networks
19+
- **Security groups** - Network access control rules (Security Groups, Firewall Rules, NSGs)
20+
- **Compute instances** - Virtual machines across all providers
21+
- **Network infrastructure** - Internet gateways, NAT gateways, route tables, elastic IPs
22+
- **Network interfaces** - ENIs, NICs attached to instances
23+
24+
This vendor-agnostic model allows you to manage multi-cloud infrastructure from a single source of truth, with consistent naming and relationships regardless of the underlying cloud provider.
25+
26+
## Prerequisites
27+
28+
Before starting this tutorial, ensure you have:
29+
30+
- Completed the [installation guide](./install.mdx) and have Infrahub running
31+
- Loaded the bootstrap data and schemas
32+
- Access to the Infrahub web interface at `http://localhost:8000`
33+
34+
## Loading cloud demo data
35+
36+
The demo includes sample data for all three major cloud providers with realistic infrastructure examples.
37+
38+
The easiest way to load the cloud demo is using the provided invoke task:
39+
40+
```bash
41+
uv run invoke demo-cloud
42+
```
43+
44+
This command:
45+
46+
1. Creates a new branch named `demo-cloud`
47+
2. Loads all schemas (including the cloud schema)
48+
3. Loads cloud object files with sample data for AWS, GCP, and Azure
49+
4. Displays a URL to view the cloud resources
50+
51+
## Exploring cloud resources
52+
53+
After loading the demo data, navigate to the cloud resources in the Infrahub web interface.
54+
55+
### Viewing all cloud resources
56+
57+
1. Ensure you're on the correct branch (for example, `demo-cloud`)
58+
2. Navigate to **Cloud Resource** in the left sidebar menu
59+
3. You'll see a list of all cloud resource types
60+
61+
Or access the cloud resources directly:
62+
63+
```text
64+
http://localhost:8000/objects/CloudResource?branch=demo-cloud
65+
```
66+
67+
### Sample data structure
68+
69+
The demo includes a comprehensive multi-cloud environment:
70+
71+
#### Cloud providers (3)
72+
73+
- Amazon Web Services (AWS)
74+
- Google Cloud Platform (GCP)
75+
- Microsoft Azure
76+
77+
#### Cloud accounts (12)
78+
79+
Each provider has production, staging, and development accounts:
80+
81+
- `opsmill-aws-production`, `opsmill-aws-staging`, `opsmill-aws-dev`
82+
- `opsmill-gcp-production`, `opsmill-gcp-staging`, `opsmill-gcp-dev`
83+
- `opsmill-azure-production`, `opsmill-azure-staging`, `opsmill-azure-dev`
84+
85+
#### Regions and availability zones
86+
87+
- **AWS**: US East (N. Virginia), US West (Oregon), EU West (Ireland)
88+
- **GCP**: US Central (Iowa), US East (South Carolina), Europe West (Belgium)
89+
- **Azure**: East US, West US 2, West Europe
90+
91+
Each region includes 3 availability zones.
92+
93+
#### Virtual networks (12)
94+
95+
VPCs and VNets across all accounts with various configurations:
96+
97+
- Production VPCs with public and private subnets
98+
- Staging and development networks
99+
- DNS support and hostname configuration
100+
101+
#### Compute instances (19)
102+
103+
Various instance types across all providers:
104+
105+
- Web servers, application servers, database servers
106+
- Linux and Windows instances
107+
- Different instance sizes (t3.large, m5.xlarge, n1-standard-2, Standard_D2s_v3, etc.)
108+
109+
#### Network infrastructure
110+
111+
- Internet gateways for public connectivity
112+
- NAT gateways for private subnet outbound access
113+
- Route tables for traffic routing
114+
- Elastic/static IP addresses
115+
- Network interfaces with security group associations
116+
117+
## Schema architecture
118+
119+
The cloud schema uses a hierarchical structure with clear relationships:
120+
121+
```text
122+
CloudProvider
123+
└── CloudAccount
124+
└── CloudVirtualNetwork
125+
├── CloudSubnet
126+
├── CloudSecurityGroup
127+
├── CloudInternetGateway
128+
└── CloudRouteTable
129+
130+
CloudRegion
131+
└── CloudAvailabilityZone
132+
└── CloudInstance
133+
└── CloudNetworkInterface
134+
```
135+
136+
### Key relationships
137+
138+
- **CloudAccount** belongs to a **CloudProvider** (parent relationship)
139+
- **CloudRegion** is associated with a **CloudProvider**
140+
- **CloudAvailabilityZone** belongs to a **CloudRegion** (parent relationship)
141+
- **CloudVirtualNetwork** is associated with a **CloudAccount** and **CloudRegion**
142+
- **CloudSubnet** belongs to a **CloudVirtualNetwork** (parent relationship)
143+
- **CloudInstance** is associated with a **CloudAccount**, **CloudAvailabilityZone**, and **CloudSubnet**
144+
- **CloudSecurityGroup** can be attached to **CloudInstance** and **CloudNetworkInterface**
145+
146+
### Common attributes
147+
148+
All cloud resources inherit from the `CloudResource` generic, providing:
149+
150+
- `name` - Resource name
151+
- `description` - Optional description
152+
- `cloud_id` - Provider-specific resource identifier (ARN, resource ID, etc.)
153+
- `status` - Operational status (active, stopped, provisioning, terminating, error)
154+
- `tags` - Optional tags for categorization
155+
156+
## Use cases
157+
158+
### Multi-cloud inventory
159+
160+
Use Infrahub as a single source of truth for all cloud resources:
161+
162+
- Track resources across AWS, GCP, and Azure in one place
163+
- Maintain consistent naming conventions
164+
- Link cloud resources to on-premises infrastructure
165+
166+
### Security auditing
167+
168+
Query security groups and their associations:
169+
170+
- Identify instances with specific security group configurations
171+
- Audit network access rules across all clouds
172+
- Track public IP assignments
173+
174+
### Capacity planning
175+
176+
Analyze compute resources across your cloud footprint:
177+
178+
- Count instances by type, region, or provider
179+
- Track resource utilization patterns
180+
- Plan for growth and optimization
181+
182+
### Network documentation
183+
184+
Document your cloud network architecture:
185+
186+
- Map virtual networks, subnets, and routing
187+
- Track NAT and internet gateway configurations
188+
- Document network interface assignments
189+
190+
## Next steps
191+
192+
For more information on Infrahub concepts, see:
193+
194+
- **[Understanding the concepts](./concepts.mdx)** - Core Infrahub patterns
195+
- **[Developer guide](./developer-guide.mdx)** - Extending schemas and creating transforms

‎docs/docs/readme.mdx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ This documentation is organized following the [Diataxis framework](https://diata
3535
| **[User walkthrough](./user-walkthrough.mdx)** | A complete hands-on tutorial that guides you through the end-user experience: creating topologies, managing branches, running generators, creating proposed changes, and validating configurations. Perfect for learning the workflow from start to finish. |
3636
| **[Deploy a virtual lab with Containerlab](./containerlab-deployment.mdx)** | Learn how to deploy generated configurations to a virtual network lab using Containerlab. Extract device configurations and topology files from Infrahub, spin up virtual Arista cEOS switches, and test your data center fabric before production deployment. |
3737
| **[Working with security management](./security-management.mdx)** | Explore Infrahub's security management capabilities by examining firewall policies, security zones, and address objects. Learn how structured security data transforms into vendor-specific firewall configurations (Juniper JunOS) and how to modify policies safely using branches. |
38+
| **[Cloud resource management](./cloud-management.mdx)** | Manage multi-cloud infrastructure (AWS, GCP, Azure) with a vendor-agnostic schema. Load demo cloud data including accounts, regions, virtual networks, instances, and security groups. Learn how Infrahub serves as a unified inventory for cloud resources. |
3839
| **[Using the service catalog](./service-catalog.mdx)** | Learn how to use the Service Catalog web interface for simplified infrastructure provisioning. Enable the Streamlit application, navigate between branches, view existing infrastructure, and create new data centers through a guided form-based workflow that automates branch creation and generator execution. |
3940

4041
### Guides
@@ -82,6 +83,7 @@ The demo implements realistic network topologies including:
8283
- **Point of presence (POP) networks** with edge routers and peering connections
8384
- **Network segments** with load balancers and service endpoints
8485
- **Security zones and policies** with firewall rules and access control
86+
- **Cloud infrastructure** with vendor-agnostic modeling for AWS, GCP, and Azure
8587
- **Resource pools** for IP address, VLAN, and ASN allocation
8688
- **Multi-vendor support** (Arista, Juniper, Cisco, and SONiC templates)
8789
- **Automated topology generation** from abstract design definitions

‎docs/sidebars.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ const sidebars: SidebarsConfig = {
1717
'user-walkthrough',
1818
'containerlab-deployment',
1919
'security-management',
20+
'cloud-management',
2021
'service-catalog',
2122
],
2223
},

‎objects/cloud/01_cloud_providers.yml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Cloud Providers - AWS, GCP, and Azure
2+
---
3+
apiVersion: infrahub.app/v1
4+
kind: Object
5+
spec:
6+
kind: CloudProvider
7+
data:
8+
- name: "Amazon Web Services"
9+
provider_type: aws
10+
console_url: "https://console.aws.amazon.com"
11+
organization: "AWS"
12+
13+
- name: "Google Cloud Platform"
14+
provider_type: gcp
15+
console_url: "https://console.cloud.google.com"
16+
organization: "Google Cloud"
17+
18+
- name: "Microsoft Azure"
19+
provider_type: azure
20+
console_url: "https://portal.azure.com"
21+
organization: "Azure"

‎objects/cloud/02_cloud_accounts.yml‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Cloud Accounts/Projects/Subscriptions for all three providers
2+
---
3+
apiVersion: infrahub.app/v1
4+
kind: Object
5+
spec:
6+
kind: CloudAccount
7+
data:
8+
# AWS Accounts
9+
- name: "opsmill-aws-production"
10+
account_id: "123456789012"
11+
description: "OpsMill Production AWS Account"
12+
status: active
13+
environment: production
14+
provider: "Amazon Web Services"
15+
16+
- name: "opsmill-aws-staging"
17+
account_id: "123456789013"
18+
description: "OpsMill Staging AWS Account"
19+
status: active
20+
environment: staging
21+
provider: "Amazon Web Services"
22+
23+
- name: "opsmill-aws-dev"
24+
account_id: "123456789014"
25+
description: "OpsMill Development AWS Account"
26+
status: active
27+
environment: development
28+
provider: "Amazon Web Services"
29+
30+
# GCP Projects
31+
- name: "opsmill-gcp-production"
32+
account_id: "opsmill-prod-12345"
33+
description: "OpsMill Production GCP Project"
34+
status: active
35+
environment: production
36+
provider: "Google Cloud Platform"
37+
38+
- name: "opsmill-gcp-staging"
39+
account_id: "opsmill-stg-12345"
40+
description: "OpsMill Staging GCP Project"
41+
status: active
42+
environment: staging
43+
provider: "Google Cloud Platform"
44+
45+
- name: "opsmill-gcp-dev"
46+
account_id: "opsmill-dev-12345"
47+
description: "OpsMill Development GCP Project"
48+
status: active
49+
environment: development
50+
provider: "Google Cloud Platform"
51+
52+
# Azure Subscriptions
53+
- name: "opsmill-azure-production"
54+
account_id: "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
55+
description: "OpsMill Production Azure Subscription"
56+
status: active
57+
environment: production
58+
provider: "Microsoft Azure"
59+
60+
- name: "opsmill-azure-staging"
61+
account_id: "b2c3d4e5-f6a7-8901-bcde-f12345678901"
62+
description: "OpsMill Staging Azure Subscription"
63+
status: active
64+
environment: staging
65+
provider: "Microsoft Azure"
66+
67+
- name: "opsmill-azure-dev"
68+
account_id: "c3d4e5f6-a7b8-9012-cdef-123456789012"
69+
description: "OpsMill Development Azure Subscription"
70+
status: active
71+
environment: development
72+
provider: "Microsoft Azure"

0 commit comments

Comments
 (0)