Skip to content

Commit f3c0ae7

Browse files
JV0812jpipkin1kimsauce
authored
cloudquery-source doc added (#3400)
* cloudquery-source doc added * Update docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-source.md * link updated * Update docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-source.md Co-authored-by: John Pipkin (Sumo Logic) <[email protected]> * Vendor configuration added for the CloudQuery source * updated the product list * CQ link - index * Updated the docs for beta release --------- Co-authored-by: John Pipkin (Sumo Logic) <[email protected]> Co-authored-by: Kim <[email protected]>
1 parent c54ded2 commit f3c0ae7

File tree

6 files changed

+157
-1
lines changed

6 files changed

+157
-1
lines changed

‎docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-source.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
---
2+
id: cloudquery-source
3+
title: CloudQuery Source (Beta)
4+
sidebar_label: CloudQuery
5+
tags:
6+
- cloud-to-cloud
7+
- cloudquery
8+
description: Learn how to collect inventory from the AWS APIs and transform them into the CloudQuery schema and send it to Sumo Logic.
9+
---
10+
11+
import CodeBlock from '@theme/CodeBlock';
12+
import ExampleJSON from '/files/c2c/cloudquery/example.json';
13+
import MyComponentSource from '!!raw-loader!/files/c2c/cloudquery/example.json';
14+
import TerraformExample from '!!raw-loader!/files/c2c/cloudquery/example.tf';
15+
import useBaseUrl from '@docusaurus/useBaseUrl';
16+
17+
<head>
18+
<meta name="robots" content="noindex" />
19+
</head>
20+
21+
<p><a href="/docs/beta"><span className="beta">Beta</span></a></p>
22+
23+
<img src={useBaseUrl('img/send-data/cloudquery-logo.png')} alt="cloudquery-icon" width="70" />
24+
25+
The CloudQuery integration is used to pull inventory from the AWS APIs and transform them into the CloudQuery schema and send it to Sumo Logic.
26+
27+
:::note
28+
This source is available in the [Fed deployment](https://hub.cloudquery.io/plugins/source/cloudquery/aws/v22.19.2/docs).
29+
:::
30+
31+
## Data collected
32+
33+
| Polling Interval | Data |
34+
| :--- | :--- |
35+
| 24 hours | [Data service table data](https://hub.cloudquery.io/plugins/source/cloudquery/aws/v22.19.2/docs) |
36+
37+
## Setup
38+
39+
### Vendor configuration
40+
41+
The integration must be configured with the Access Key ID and Secret Access Key. Refer to the [AWS documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) for guidance to create the Access Key ID and Secret Access Key.
42+
43+
### Source configuration
44+
45+
When you create an CloudQuery source, you add it to a Hosted Collector. Before creating the source, identify the Hosted Collector you want to use or create a new Hosted Collector. For instructions, see [Configure a Hosted Collector](/docs/send-data/hosted-collectors/configure-hosted-collector).
46+
47+
To configure a CloudQueryr Source:
48+
1. In Sumo Logic, select **Manage Data** > **Collection** > **Collection**. 
49+
1. On the Collection page, click **Add Source** next to a Hosted Collector.
50+
1. Search for and select **CloudQuery**.
51+
1. Enter a **Name** for the source. The description is optional.
52+
1. (Optional) For **Source Category**, enter any string to tag the output collected from the Source. Category metadata is stored in a searchable field called `_sourceCategory`.
53+
1. (Optional) **Fields**. Click the **+Add** button to define the fields you want to associate. Each field needs a name (key) and value.
54+
* ![green check circle.png](/img/reuse/green-check-circle.png) A green circle with a check mark is shown when the field exists in the Fields table schema.
55+
* ![orange exclamation point.png](/img/reuse/orange-exclamation-point.png) An orange triangle with an exclamation point is shown when the field doesn't exist in the Fields table schema. In this case, an option to automatically add the nonexistent fields to the Fields table schema is provided. If a field is sent to Sumo Logic that does not exist in the Fields schema it is ignored, known as dropped.
56+
1. **AWS Access Key ID**. Enter the Access Key ID collected from the [AWS Management Console](#vendor-configuration).
57+
1. **AWS Secret Access Key**. Enter the Secret Access Key collected from the [AWS Management Console](#vendor-configuration).
58+
1. **Regions**. Identify and enter your Region based on your Base URL.
59+
1. **Services**. Enter the type of service from which the data needs to be collected.
60+
1. By default, **Polling Interval** is set to 24 hours.
61+
1. When you are finished configuring the Source, click **Save**.
62+
63+
## JSON schema
64+
65+
Sources can be configured using UTF-8 encoded JSON files with the Collector Management API. See [Use JSON to configure Sources](/docs/send-data/use-json-configure-sources) for details. 
66+
67+
| Parameter | Type | Value | Required | Description |
68+
|:--|:--|:--|:--|:--|
69+
| schemaRef | JSON Object | `{"type":"CloudQuery"}` | Yes | Define the specific schema type. |
70+
| sourceType | String | `"Universal"` | Yes | Type of source. |
71+
| config | JSON Object | [Configuration object](#configuration-object) | Yes | Source type specific values. |
72+
73+
### Configuration Object
74+
75+
| Parameter | Type | Required | Default | Description | Example |
76+
|:--|:--|:--|:--|:--|:--|
77+
| name | String | Yes | `null` | Type a desired name of the source. The name must be unique per Collector. This value is assigned to the [metadata](/docs/search/get-started-with-search/search-basics/built-in-metadata) field `_source`. | `"mySource"` |
78+
| description | String | No | `null` | Type a description of the source. | `"Testing source"`
79+
| category | String | No | `null` | Type a category of the source. This value is assigned to the [metadata](/docs/search/get-started-with-search/search-basics/built-in-metadata) field `_sourceCategory`. See [best practices](/docs/send-data/best-practices) for details. | `"mySource/test"`
80+
| fields | JSON Object | No | `null` | JSON map of key-value fields (metadata) to apply to the Collector or Source. Use the boolean field `_siemForward` to enable forwarding to SIEM.|`{"_siemForward": false, "fieldA": "valueA"}` |
81+
| awsId | String | Yes | `null` | The AWS access ID that you want to use to authenticate collection requests. | |
82+
| awsKey | String | Yes | `null` | The AWS secret key that you want to use to authenticate collection requests. | |
83+
| limitToRegions | Array | Yes | | List of regions for which the data has to be fetched. | |
84+
| limitToServices | Array | Yes | | List of services for which the data has to be fetched. | |
85+
| pollingInterval | Integer | No | 24 | This sets how often the Source checks for data. | |
86+
87+
### JSON example
88+
89+
<CodeBlock language="json">{MyComponentSource}</CodeBlock>
90+
91+
[Download example](/files/c2c/cloudquery/example.json)
92+
93+
### Terraform example
94+
95+
<CodeBlock language="json">{TerraformExample}</CodeBlock>
96+
97+
[Download example](/files/c2c/cloudquery/example.tf)
98+
99+
## Limitations
100+
101+
- The integration lacks support for fetching service table data from multiple AWS accounts.
102+
- The integration does not support to retrieve data from services like Classic load balancer, Network load balancer, Aurora, and VPC.
103+
104+
## FAQ
105+
106+
:::info
107+
Click [here](/docs/c2c/info) for more information about Cloud-to-Cloud sources.
108+
:::

‎docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ For the Federal environments, a different set of Static IPs is available for eac
4343

4444
## Integrations
4545

46-
The topics below are the available integrations. In Sumo Logic these are called Sources. Check out the Sources we have available in beta.
46+
The topics below are the available integrations. In Sumo Logic these are called Sources. Check out the Sources we have available in beta.
4747

4848
## Versions
4949

‎sidebars.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -271,6 +271,7 @@ module.exports = {
271271
'send-data/hosted-collectors/cloud-to-cloud-integration-framework/citrix-cloud-source',
272272
'send-data/hosted-collectors/cloud-to-cloud-integration-framework/cse-aws-ec-inventory-source',
273273
'send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloud-to-cloud-source-versions',
274+
//'send-data/hosted-collectors/cloud-to-cloud-integration-framework/cloudquery-source',
274275
'send-data/hosted-collectors/cloud-to-cloud-integration-framework/code42-incydr-source',
275276
'send-data/hosted-collectors/cloud-to-cloud-integration-framework/crowdstrike-source',
276277
'send-data/hosted-collectors/cloud-to-cloud-integration-framework/crowdstrike-fdr-source',

‎static/files/c2c/cloudquery/example.json‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
{
2+
"api.version": "v1",
3+
"source": {
4+
"config": {
5+
"name": "CloudQuery AWS Inventory",
6+
"accessKeyId": "************",
7+
"secretAccessKey": "***************************",
8+
"limitToRegions": [
9+
"us-east-1",
10+
"us-east-2"
11+
],
12+
"awsServices": [
13+
"ECS",
14+
"EC2"
15+
],
16+
"pollInterval": 24
17+
},
18+
"schemaRef": {
19+
"type": "CloudQuery AWS Inventory"
20+
},
21+
"sourceType": "Universal"
22+
}
23+
}

‎static/files/c2c/cloudquery/example.tf‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
resource "sumologic_cloud_to_cloud_source" "cloudquery_source" {
2+
collector_id = sumologic_collector.collector.id
3+
schema_ref = {
4+
type = "CloudQuery"
5+
}
6+
config = jsonencode({
7+
"name": "CloudQuery AWS Inventory",
8+
"accessKeyId": "************",
9+
"secretAccessKey": "***************************",
10+
"limitToRegions": [
11+
"us-east-1",
12+
"us-east-2"
13+
],
14+
"awsServices": [
15+
"ECS",
16+
"EC2"
17+
],
18+
"pollInterval": 24
19+
})
20+
}
21+
resource "sumologic_collector" "collector" {
22+
name = "my-collector"
23+
description = "Just testing this"
24+
}

‎static/img/send-data/cloudquery-logo.png‎

12.4 KB
Loading

0 commit comments

Comments
 (0)