Skip to content

Commit 593c97e

Browse files
dogancanbakirdogancanbakir
authored
Merge pull request #723 from projectdiscovery/dev
cloudlist v1.3.0
2 parents 95db0ec + ff1a952 commit 593c97e

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

66 files changed

+12056
-848
lines changed

‎.gitignore‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,14 @@ cmd/cloudlist/cloudlist
33
dist
44
.env
55
provider-config.yaml
6-
cloudlist
6+
cloudlist
7+
.claude/settings.local.json
8+
9+
# Test files and credentials
10+
TEST_RESULTS.md
11+
test-*.yaml
12+
*-sa.json
13+
*.key.json
14+
/cloudlist
15+
/terraform/
16+
azure-readonly-config.yaml

‎CLAUDE.md‎

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## Project Overview
6+
7+
**Cloudlist** is a multi-cloud asset discovery tool written in Go that enumerates resources (IPs, DNS names) from various cloud providers using their APIs. It's designed for blue team security operations to maintain centralized cloud asset inventories.
8+
9+
## Build & Development Commands
10+
11+
### Building
12+
```bash
13+
# Build the binary
14+
make build
15+
# or
16+
go build -v -o cloudlist cmd/cloudlist/main.go
17+
```
18+
19+
### Testing
20+
```bash
21+
# Run all tests
22+
make test
23+
# or
24+
go test -v ./...
25+
26+
# Run tests for specific package
27+
go test -v ./pkg/providers/aws/
28+
```
29+
30+
### Linting
31+
```bash
32+
# Run golangci-lint (same as CI)
33+
golangci-lint run --timeout 5m
34+
35+
# Auto-fix issues
36+
golangci-lint run --fix --timeout 5m
37+
```
38+
39+
### Running
40+
```bash
41+
# Basic usage
42+
./cloudlist -pc ~/.config/cloudlist/provider-config.yaml
43+
44+
# Filter by provider
45+
./cloudlist -pc provider-config.yaml -p aws,gcp
46+
47+
# Filter by service
48+
./cloudlist -pc provider-config.yaml -s compute,dns
49+
50+
# Output formats
51+
./cloudlist -pc provider-config.yaml -json > output.json
52+
./cloudlist -pc provider-config.yaml -host # DNS names only
53+
./cloudlist -pc provider-config.yaml -ip # IPs only
54+
```
55+
56+
## Architecture
57+
58+
### Core Components
59+
60+
1. **Provider Interface** (`pkg/schema/schema.go:18-28`)
61+
- All providers implement the `schema.Provider` interface
62+
- Key methods: `Name()`, `ID()`, `Resources(ctx)`, `Services()`
63+
- Returns `*schema.Resources` containing `[]*schema.Resource`
64+
65+
2. **Provider Registration** (`pkg/inventory/inventory.go:92-137`)
66+
- `nameToProvider()` function maps provider names to implementations
67+
- Add new providers here as a case statement
68+
69+
3. **Resource Model** (`pkg/schema/schema.go:142-163`)
70+
- Each resource represents a cloud asset with IP/DNS information
71+
- Fields: `Public`, `Provider`, `Service`, `ID`, `PublicIPv4/v6`, `PrivateIpv4/v6`, `DNSName`, `Metadata`
72+
- Resources are automatically deduplicated by IP/DNS values
73+
74+
4. **Provider Structure Pattern**
75+
- Providers in `pkg/providers/<provider>/`
76+
- Main file: `<provider>.go` with `New()` constructor and `Resources()` implementation
77+
- Service-specific files: `instances.go`, `dns.go`, etc. for complex providers (see AWS)
78+
- Each provider has a `Services` slice listing supported service types
79+
80+
### Provider Implementation Pattern
81+
82+
When implementing providers, follow this structure:
83+
84+
1. **Main provider file** (`<provider>.go`):
85+
- Struct with provider-specific client and configuration
86+
- `New(block schema.OptionBlock)` constructor that validates config
87+
- `Resources(ctx context.Context) (*schema.Resources, error)` that orchestrates resource gathering
88+
- Service-specific helper methods
89+
90+
2. **Service-specific files** (for complex providers):
91+
- AWS example: `instances.go`, `route53.go`, `s3.go`, `eks.go`, etc.
92+
- Each file handles one service type's resource enumeration
93+
- Returns `*schema.Resources` that main `Resources()` method merges
94+
95+
3. **Configuration Parsing**:
96+
- Use `block.GetMetadata(key)` to read config values
97+
- Environment variables supported via `$VAR_NAME` syntax (auto-resolved)
98+
- Return `&schema.ErrNoSuchKey{Name: key}` for missing required config
99+
100+
### Important Patterns
101+
102+
- **Resource Deduplication**: Resources are automatically deduplicated by the `ResourceDeduplicator` in schema
103+
- **Service Filtering**: Users can filter by service via `-s` flag; providers check this in `Services()` method
104+
- **Metadata**: Extended metadata can be added to resources via the `Metadata` map[string]string field
105+
- **Context Handling**: All provider `Resources()` methods receive context for cancellation support
106+
107+
## Adding a New Provider
108+
109+
Required steps:
110+
111+
1. **Create provider directory**: `pkg/providers/<provider-name>/`
112+
113+
2. **Implement provider struct** with these methods:
114+
```go
115+
func New(block schema.OptionBlock) (schema.Provider, error)
116+
func (p *Provider) Name() string
117+
func (p *Provider) ID() string
118+
func (p *Provider) Resources(ctx context.Context) (*schema.Resources, error)
119+
func (p *Provider) Services() []string
120+
```
121+
122+
3. **Register in inventory**: Add case to `nameToProvider()` in `pkg/inventory/inventory.go`
123+
124+
4. **Add to services map**: Add provider and its services to `Providers` map in `pkg/inventory/inventory.go`
125+
126+
5. **Document configuration**: Add provider config documentation to `PROVIDERS.md`
127+
128+
See reference implementations:
129+
- Simple: `pkg/providers/digitalocean/` (single service)
130+
- Complex: `pkg/providers/aws/` (multiple services, sub-files)
131+
- With special auth: `pkg/providers/gcp/` (JSON service account keys)
132+
133+
## Key Files
134+
135+
- `cmd/cloudlist/main.go` - CLI entry point
136+
- `internal/runner/runner.go` - Main enumeration orchestrator
137+
- `internal/runner/options.go` - CLI flag definitions
138+
- `pkg/schema/schema.go` - Core interfaces and data structures
139+
- `pkg/inventory/inventory.go` - Provider registration and factory
140+
- `pkg/providers/*/` - Individual cloud provider implementations
141+
142+
## GCP-Specific Notes
143+
144+
GCP provider supports **two discovery modes**:
145+
146+
1. **Individual Service APIs** (default): Project-level discovery using service-specific APIs
147+
2. **Organization-Level Asset API**: Org-wide discovery when `organization_id` is specified
148+
149+
Key distinction in code:
150+
- Check for `organization_id` in config to determine mode
151+
- Asset API mode uses `cloud.google.com/go/asset` package
152+
- Individual APIs mode uses service-specific clients (compute, dns, etc.)
153+
154+
See `docs/GCP_ASSET_API.md` for detailed implementation guide.
155+
156+
## Testing Guidelines
157+
158+
- Integration tests require cloud provider credentials (usually skipped in CI)
159+
- Unit tests should mock cloud provider clients
160+
- Use table-driven tests for resource parsing logic
161+
- Test error handling for missing/invalid credentials
162+
163+
## GitHub Actions CI
164+
165+
- **lint-test.yml**: Runs `golangci-lint` on Go code changes
166+
- **build-test.yml**: Tests builds on Ubuntu, Windows, macOS with Go 1.22.x
167+
- **release-binary.yml**: Creates releases with GoReleaser
168+
169+
## Common Gotchas
170+
171+
1. **Resource fields**: Either IP or DNS must be populated; empty resources are invalid
172+
2. **Provider IDs**: The `id` field in config is user-defined for filtering, not a cloud resource ID
173+
3. **Service names**: Must match entries in provider's `Services` slice and inventory's `Providers` map
174+
4. **Credential handling**: Use environment variables (`$VAR`) in config rather than hardcoding
175+
5. **Context cancellation**: Always respect context in long-running API calls
176+
6. **Deduplication**: Don't manually deduplicate; use `Resources.Append()` which handles it automatically
177+
178+
## Module Information
179+
180+
- Go version: 1.24+ (see `go.mod`)
181+
- Module path: `github.com/projectdiscovery/cloudlist`
182+
- Key dependencies:
183+
- Provider SDKs: `aws-sdk-go`, `azure-sdk-for-go`, `google.golang.org/api`, etc.
184+
- ProjectDiscovery libs: `goflags`, `gologger`, `utils`
185+
- Concurrency: `github.com/alitto/pond/v2` for worker pools

‎PROVIDERS.md‎

Lines changed: 154 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -47,25 +47,170 @@ References -
4747

4848
### Google Cloud Platform (GCP)
4949

50-
Google Cloud Platform can be integrated by using the following configuration block.
50+
Google Cloud Platform supports **two discovery approaches** and **two authentication modes**:
51+
52+
#### Authentication Modes
53+
54+
**A. Traditional Authentication (Static Credentials)**
55+
- Service account keys (JSON files) - long-lived
56+
- Application Default Credentials (ADC) - often long-lived
57+
58+
**B. Short-lived Credentials (Recommended for Enhanced Security)**
59+
- Generate temporary access tokens (up to 1 hour)
60+
- Eliminates reliance on static service account keys
61+
- Uses Service Account Credentials API for token generation
62+
- Supports service account impersonation
63+
64+
---
65+
66+
#### 1. Individual Service APIs (Project-Level Discovery)
67+
68+
**Option 1: Traditional Static Credentials**
5169

5270
```yaml
5371
- # provider is the name of the provider
5472
provider: gcp
5573
# id is the name defined by user for filtering (optional)
56-
id: staging
74+
id: project-discovery
5775
# gcp_service_account_key is the key token of service account.
58-
gcp_service_account_key: '{}'
76+
gcp_service_account_key: '{
77+
"type": "service_account",
78+
"project_id": "your-project-id",
79+
"private_key_id": "...",
80+
"private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
81+
"client_email": "cloudlist-sa@your-project-id.iam.gserviceaccount.com",
82+
"client_id": "...",
83+
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
84+
"token_uri": "https://oauth2.googleapis.com/token",
85+
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
86+
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/cloudlist-sa%40your-project-id.iam.gserviceaccount.com",
87+
"universe_domain": "googleapis.com"
88+
}'
5989
```
6090

61-
`gcp_service_account_key` can be retrieved by creating a new service account. To do so, create service account with Read Only access to `cloudresourcemanager` and `dns` scopes in IAM. Next, generate a new account key for the Service Account by following steps in Reference 2. This should give you a json which can be pasted in a single line in the `gcp_service_account_key`.
91+
**Option 2: Short-lived Credentials (Developer Workflow - Zero Keys)**
6292

63-
Scopes Required -
64-
1. Cloud DNS
93+
```yaml
94+
- provider: gcp
95+
id: dev-discovery
96+
use_short_lived_credentials: true
97+
service_account_email: "cloudlist@project.iam.gserviceaccount.com"
98+
# Uses: gcloud auth login → ADC → short-lived token
99+
# No service account key file needed!
100+
```
65101

66-
References -
67-
1. https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_iam_read-only-console.html
68-
2. https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
102+
**Option 3: Short-lived Credentials (CI/CD with Minimal Permissions)**
103+
104+
```yaml
105+
- provider: gcp
106+
id: ci-discovery
107+
use_short_lived_credentials: true
108+
service_account_email: "powerful-sa@project.iam.gserviceaccount.com"
109+
source_credentials: "minimal-ci-sa.json" # Only has impersonation permission
110+
token_lifetime: "7200s" # 2 hours
111+
```
112+
113+
**Option 4: Short-lived Credentials (GKE/Compute Engine - Zero Secrets)**
114+
115+
```yaml
116+
- provider: gcp
117+
id: workload-discovery
118+
use_short_lived_credentials: true
119+
service_account_email: "cloudlist@project.iam.gserviceaccount.com"
120+
# Uses workload identity automatically
121+
```
122+
123+
**Option 5: Short-lived Credentials (Migration from Existing Keys)**
124+
125+
```yaml
126+
- provider: gcp
127+
id: migrating-discovery
128+
use_short_lived_credentials: true
129+
service_account_email: "cloudlist@project.iam.gserviceaccount.com"
130+
gcp_service_account_key: "existing-key.json" # Generates short-lived from static key
131+
token_lifetime: "3600s" # 1 hour (default)
132+
```
133+
134+
**Required Scopes (Traditional Authentication):**
135+
1. `roles/compute.viewer` - Compute instances and forwarding rules
136+
2. `roles/dns.reader` - DNS records
137+
3. `roles/storage.objectViewer` - Storage buckets
138+
4. `roles/run.viewer` - Cloud Run services
139+
5. `roles/cloudfunctions.viewer` - Cloud Functions
140+
6. `roles/container.viewer` - GKE clusters
141+
7. `roles/tpu.viewer` - TPU nodes
142+
8. `roles/file.viewer` - Filestore instances
143+
9. `roles/resourcemanager.viewer` - List projects
144+
145+
**Additional Requirements for Short-lived Credentials:**
146+
- **Source credentials** need: `roles/iam.serviceAccountTokenCreator` or `iam.serviceAccounts.generateAccessToken` permission on the target service account
147+
- **Target service account** needs: Same viewer roles listed above
148+
- **Service Account Credentials API** must be enabled in the project
149+
150+
**Configuration Parameters:**
151+
- `use_short_lived_credentials` (bool): Enable short-lived token generation (default: false)
152+
- `service_account_email` (string, required if short-lived): Target service account to impersonate
153+
- `source_credentials` (string, optional): Path to source credentials file (uses ADC if not provided)
154+
- `token_lifetime` (string, optional): Token lifetime in seconds (e.g., "3600s") or Go duration format (e.g., "1h"). Range: 1s to 3600s (1 hour). Default: "3600s"
155+
156+
---
157+
158+
#### 2. Organization-Level Asset API (Organization-Wide Discovery)
159+
160+
**Traditional Authentication:**
161+
162+
```yaml
163+
- # provider is the name of the provider
164+
provider: gcp
165+
# id is the name defined by user for filtering (optional)
166+
id: org-discovery
167+
# organization_id enables Asset API for organization-wide discovery
168+
organization_id: "123456789012"
169+
# gcp_service_account_key with organization-level permissions
170+
gcp_service_account_key: '{
171+
"type": "service_account",
172+
"project_id": "your-project-id",
173+
"private_key_id": "...",
174+
"private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
175+
"client_email": "asset-viewer-sa@your-project-id.iam.gserviceaccount.com",
176+
"client_id": "...",
177+
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
178+
"token_uri": "https://oauth2.googleapis.com/token",
179+
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
180+
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/asset-viewer-sa%40your-project-id.iam.gserviceaccount.com",
181+
"universe_domain": "googleapis.com"
182+
}'
183+
```
184+
185+
**Short-lived Credentials (Organization-Level):**
186+
187+
```yaml
188+
- provider: gcp
189+
id: org-discovery-secure
190+
organization_id: "123456789012"
191+
use_short_lived_credentials: true
192+
service_account_email: "asset-viewer-sa@project.iam.gserviceaccount.com"
193+
token_lifetime: "7200s" # 2 hours
194+
```
195+
196+
**Required Organization-Level Roles:**
197+
1. `roles/cloudasset.viewer` - Core Asset API access
198+
2. `roles/resourcemanager.viewer` - List projects in organization
199+
3. (For short-lived) `roles/iam.serviceAccountTokenCreator` - On source credentials
200+
201+
**Key Differences:**
202+
- **Individual APIs**: Fast, project-specific, detailed results
203+
- **Asset API**: Comprehensive, organization-wide, higher resource count
204+
- **Short-lived Credentials**: Enhanced security, tokens auto-expire (up to 1 hour)
205+
206+
📚 **For detailed setup instructions, see: [docs/GCP_ASSET_API.md](docs/GCP_ASSET_API.md)**
207+
208+
References -
209+
1. https://cloud.google.com/asset-inventory/docs/overview
210+
2. https://cloud.google.com/iam/docs/creating-managing-service-accounts
211+
3. https://cloud.google.com/iam/docs/understanding-roles
212+
4. https://cloud.google.com/iam/docs/service-account-creds (Short-lived credentials)
213+
5. https://cloud.google.com/docs/authentication/provide-credentials-adc (Application Default Credentials)
69214

70215

71216
### Microsoft Azure

0 commit comments

Comments
 (0)