Skip to content

Commit 8499a03

Browse files
kimsauceJV0812jpipkin1
authored
CloudQuery-GCP C2C doc (#4069)
* CloudQuery GCP C2C * fix examples * more edits * more edits * minor formatting * Added more information as per the template * Update cid-redirects.json * Update cloudquery-gcp-source.md * Update docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-gcp-source.md Co-authored-by: John Pipkin (Sumo Logic) <[email protected]> * Update docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-gcp-source.md Co-authored-by: John Pipkin (Sumo Logic) <[email protected]> * Update docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-gcp-source.md Co-authored-by: John Pipkin (Sumo Logic) <[email protected]> * Update docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-gcp-source.md Co-authored-by: John Pipkin (Sumo Logic) <[email protected]> * Update docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-gcp-source.md Co-authored-by: John Pipkin (Sumo Logic) <[email protected]> * Update docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-gcp-source.md Co-authored-by: John Pipkin (Sumo Logic) <[email protected]> * Update docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-gcp-source.md Co-authored-by: John Pipkin (Sumo Logic) <[email protected]> * Update docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-gcp-source.md Co-authored-by: John Pipkin (Sumo Logic) <[email protected]> * Update docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-gcp-source.md Co-authored-by: John Pipkin (Sumo Logic) <[email protected]> * Update docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-gcp-source.md Co-authored-by: Kim (Sumo Logic) <[email protected]> * Update docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-gcp-source.md Co-authored-by: Kim (Sumo Logic) <[email protected]> --------- Co-authored-by: Jagadisha V <[email protected]> Co-authored-by: John Pipkin (Sumo Logic) <[email protected]>
1 parent fbbcfa1 commit 8499a03

File tree

4 files changed

+176
-0
lines changed

4 files changed

+176
-0
lines changed

‎cid-redirects.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2559,6 +2559,7 @@
25592559
"/cid/19901": "/docs/metrics/metrics-operators/topk",
25602560
"/cid/19902": "/docs/metrics/metrics-operators/where",
25612561
"/cid/15631": "/docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/citrix-cloud-source",
2562+
"/cid/15634": "/docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-gcp-source",
25622563
"/cid/15632": "/docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-azure-plugin-source",
25632564
"/cid/15633": "/docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-source",
25642565
"/cid/14323": "/docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/docusign-source",

‎docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-gcp-source.md‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
---
2+
id: cloudquery-gcp-source
3+
title: CloudQuery GCP (Beta)
4+
sidebar_label: CloudQuery GCP
5+
tags:
6+
- cloud-to-cloud
7+
- cloudquery
8+
- gcp
9+
- google cloud platform
10+
description: Learn how to collect inventory from GCP APIs using the CloudQuery Plugin SDK and send it to Sumo Logic.
11+
---
12+
13+
<head>
14+
<meta name="robots" content="noindex" />
15+
</head>
16+
17+
<p><a href="/docs/beta"><span className="beta">Beta</span></a></p>
18+
19+
import CodeBlock from '@theme/CodeBlock';
20+
import useBaseUrl from '@docusaurus/useBaseUrl';
21+
import ExampleJSON from '/files/c2c/cloudquery-gcp/example.json';
22+
import MyComponentSource from '!!raw-loader!/files/c2c/cloudquery-gcp/example.json';
23+
import TerraformExample from '!!raw-loader!/files/c2c/cloudquery-gcp/example.tf';
24+
25+
<img src={useBaseUrl('img/send-data/cloudquery-logo.png')} alt="cloudquery-icon" width="50" />
26+
27+
The CloudQuery GCP integration pulls inventory from various Google Cloud Platform (GCP) APIs via the CloudQuery GCP plugin, transforms it into the CloudQuery schema, and then sends it to Sumo Logic.
28+
29+
import FedDeploymentNote from '../../../reuse/fed-deployment-note.md';
30+
31+
<FedDeploymentNote/>
32+
33+
## Data collected
34+
35+
| Polling Interval | Data |
36+
| :--- | :--- |
37+
| 12 hours | [Data service table data](https://github.com/cloudquery/cloudquery/blob/plugins-source-gcp-v10.0.0/plugins/source/gcp/resources/plugin/tables.go) |
38+
39+
## Setup
40+
41+
### Prerequisites
42+
43+
[Enable](https://cloud.google.com/endpoints/docs/openapi/enable-api) all the service APIs in the authenticated GCP account related with the services from which you want to fetch data via CloudQuery GCP plugin.
44+
45+
### Vendor configuration
46+
47+
To download the user credentials JSON file by using the gcloud CLI, refer to the [Google Cloud documentation](https://cloud.google.com/docs/authentication/application-default-credentials#personal).
48+
49+
### Source configuration
50+
51+
When you create a CloudQuery GCP source, you add it to a Hosted Collector. Before creating the source, identify the Hosted Collector you want to use or create a new Hosted Collector. For instructions, see [Configure a Hosted Collector and Source](/docs/send-data/hosted-collectors/configure-hosted-collector).
52+
53+
To configure a CloudQuery GCP Source:
54+
1. In Sumo Logic, select **Manage Data** > **Collection** > **Collection**. 
55+
1. On the Collection page, click **Add Source** next to a Hosted Collector.
56+
1. Search for and select **CloudQuery GCP**.
57+
1. Enter a **Name** for the source. The description is optional.
58+
1. **GCP Credentials**. Upload the credentials JSON file downloaded by running the gcloud CLI command.
59+
1. **(Optional) Project ID(s)**. Enter the unique identifier number for your project. You can find this from the Google Cloud Console. If not specified, all the available active projects will be considered by default.
60+
1. **(Optional) Organization ID(s)**. Enter the unique identifier number for your organization. You can find this from the Google Cloud Console. If not specified, all the available active projects will be considered by default.
61+
1. **Services**. Enter the type of service from which the GCP data should be fetched via CloudQuery GCP plugin.
62+
1. By default, **Polling Interval** is set to 12 hours.
63+
1. **(Optional) Processing Rules for Logs**. Configure any desired filters, such as allowlist, denylist, hash, or mask, as described in [Create a Processing Rule](/docs/send-data/collection/processing-rules/create-processing-rule).
64+
1. When you are finished configuring the Source, click **Save**.
65+
66+
## JSON schema
67+
68+
Sources can be configured using UTF-8 encoded JSON files with the Collector Management API. See [Use JSON to Configure Sources](/docs/send-data/use-json-configure-sources) for details. 
69+
70+
| Parameter | Type | Required | Description |
71+
|:---|:---|:---|:---|
72+
| config | JSON Object | Yes | Contains the configuration parameters for the source. |
73+
| schemaRef | JSON Object | Yes | Use `{“type”: “CloudQuery GCP Inventory”}`. |
74+
| sourceType | String | Yes | Use “Universal” for CloudQuery GCP source Inventory. |
75+
76+
### Configuration object
77+
78+
| Parameter | Type | Required | Default | Description | Access |
79+
|:---|:---|:---|:---|:---|:---|
80+
| name | string | Yes | `null` | Type the desired name of the Source. The name must be unique per Collector. This value is assigned to the metadata field `_source`. | modifiable |
81+
| description | string | No | `null` | Type a description of the Source. | modifiable |
82+
| credentialsJson | JSON object | Yes | `null` | Service account key JSON file. | modifiable |
83+
| projectIds | array | No | `null` | List of Project IDs from which the data needs to be pulled. | modifiable |
84+
| orgIds | array | No | `null` | List of Organization IDs from which the data needs to be pulled. | modifiable |
85+
| limitToServices | array | Yes | `null` | List of services from which the data should be pulled. | modifiable |
86+
| pollingInterval | number | Yes | 12 Hours | This sets how often the Source checks for data. | modifiable |
87+
88+
### JSON example
89+
90+
<CodeBlock language="json">{MyComponentSource}</CodeBlock>
91+
92+
[Download example](/files/c2c/cloudquery-gcp/example.json)
93+
94+
### Terraform example
95+
96+
<CodeBlock language="json">{TerraformExample}</CodeBlock>
97+
98+
[Download example](/files/c2c/cloudquery-gcp/example.tf)
99+
100+
## Troubleshooting
101+
102+
### Warning message
103+
104+
#### Message
105+
106+
```
107+
transport: received unexpected content-type \"text/html; charset=UTF-8
108+
```
109+
110+
#### Solution
111+
112+
If you come across this warning message while fetching service data for the configured source, then contact the CloudQuery or GCP team.
113+
114+
### Error handling
115+
116+
For the SDK, if the response code is `401` or `403`, it indicates a config error. You'll need to quit and restart the integration.
117+
118+
For any other error returned by CloudQuery, we will set ErrorGeneric as an error in the source. The integration will not stop its execution.
119+
120+
## FAQ
121+
122+
:::info
123+
Click [here](/docs/c2c/info) for more information about Cloud-to-Cloud sources.
124+
:::

‎static/files/c2c/cloudquery-gcp/example.json‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
{
2+
"api.version": "v1",
3+
"source": {
4+
"config": {
5+
"name": "CloudQuery GCP",
6+
"credentialsJson": "aa",
7+
"projectIds": [
8+
"test-gcp-435276"
9+
],
10+
"organizationIds": [
11+
"24584326357",
12+
"736586988"
13+
],
14+
"limitToServices": [
15+
"compute",
16+
"storage"
17+
],
18+
"pollingInterval": 12
19+
},
20+
"schemaRef": {
21+
"type": "CloudQuery GCP Inventory"
22+
}
23+
},
24+
"sourceType": "Universal"
25+
}

‎static/files/c2c/cloudquery-gcp/example.tf‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
resource "sumologic_cloud_to_cloud_source" "asana_source" {
2+
collector_id = sumologic_collector.collector.id
3+
schema_ref = {
4+
type = "CloudQuery GCP"
5+
}
6+
config = jsonencode({
7+
"name": "CloudQuery GCP",
8+
"credentialsJson": "aa",
9+
"projectIds": {
10+
"test-gcp-435276"
11+
},
12+
"organizationIds": {
13+
"24584326357",
14+
"736586988"
15+
},
16+
"limitToServices": {
17+
"compute",
18+
"storage"
19+
},
20+
"pollingInterval": 12
21+
})
22+
}
23+
resource "sumologic_collector" "collector" {
24+
name = "my-collector"
25+
description = "Just testing this"
26+
}

0 commit comments

Comments
 (0)