diff --git a/blog-service/2021/12-31.md b/blog-service/2021/12-31.md index 3743ba82a6..3a5f0b5385 100644 --- a/blog-service/2021/12-31.md +++ b/blog-service/2021/12-31.md @@ -137,7 +137,7 @@ New - Our [Cloud-to-Cloud Integration Framework](/docs/send-data/hosted-collec --- ## September 20, 2021 (Manage) -New - You can now [forward aggregate data from a Scheduled View to AWS S3](/docs/manage/data-forwarding/amazon-s3-bucket). Previously, aggregate data was dropped and not included in forwarded file objects. Now, aggregate fields are automatically appended when your Scheduled View conducts aggregation. +New - You can now [forward aggregate data from a Scheduled View to AWS S3](/docs/manage/data-forwarding/forward-data-from-sumologic). Previously, aggregate data was dropped and not included in forwarded file objects. Now, aggregate fields are automatically appended when your Scheduled View conducts aggregation. --- ## September 15, 2021 (Collection) diff --git a/blog-service/2022/12-31.md b/blog-service/2022/12-31.md index ed7b6009c2..d24b5523e6 100644 --- a/blog-service/2022/12-31.md +++ b/blog-service/2022/12-31.md @@ -167,7 +167,7 @@ Update - We’ve released an improved, re-organized UI for Data Forwarding. Ther * Destinations that receive data forwarded from Sumo Logic partitions or scheduled views are still managed on the [**Data Forwarding**](/docs/manage/data-forwarding/view-list-data-forwarding/) page. * Destinations that receive data from Installed Collectors are managed on a new page [**Archive**](/docs/manage/data-archiving/archive/#archive-page) page. -For more information, see [Forward Data from Sumo Logic to S3](/docs/manage/data-forwarding/amazon-s3-bucket). +For more information, see [Forward Data from Sumo Logic to S3](/docs/manage/data-forwarding/forward-data-from-sumologic). --- ## October 3, 2022 (Search) diff --git a/blog-service/2024/12-31.md b/blog-service/2024/12-31.md index 3cfedd519a..ec4503e6ce 100644 --- a/blog-service/2024/12-31.md +++ b/blog-service/2024/12-31.md @@ -441,7 +441,7 @@ We are happy to announce that you can now configure the schema and format of log Options to forward raw data -To learn more, see the *Forward data to an S3 forwarding destination* section in our article [Forward Data from Sumo Logic to S3](/docs/manage/data-forwarding/amazon-s3-bucket). +To learn more, see the *Forward data to an S3 forwarding destination* section in our article [Forward Data from Sumo Logic to S3](/docs/manage/data-forwarding/forward-data-from-sumologic). ### October 02, 2024 (Apps) diff --git a/blog-service/2025-09-15-manage.md b/blog-service/2025-09-15-manage.md new file mode 100644 index 0000000000..56aab7600e --- /dev/null +++ b/blog-service/2025-09-15-manage.md @@ -0,0 +1,17 @@ +--- +title: Forward Data from Sumo Logic to GCS (Manage) +image: https://assets-www.sumologic.com/company-logos/_800x418_crop_center-center_82_none/SumoLogic_Preview_600x600.jpg?mtime=1617040082 +keywords: + - data-forwarding + - manage + - google-cloud-storage +hide_table_of_contents: true +--- + +import useBaseUrl from '@docusaurus/useBaseUrl'; + +We are happy to announce that you can now configure and forward the log data from Sumo Logic to a Google Cloud Storage (GCS) destination. Previously, data forwarding was limited only to AWS S3. Now you can forward the log data, log data with metadata, or log data with metadata and enriched fields, in either CSV or JSON format to GCS. [Learn more](/docs/manage/data-forwarding/forward-data-from-sumologic). + +:::info +To request access, contact your Sumo Logic account representative or Support. +::: diff --git a/cid-redirects.json b/cid-redirects.json index eb312456b7..2999d184da 100644 --- a/cid-redirects.json +++ b/cid-redirects.json @@ -2405,10 +2405,10 @@ "/cid/51621": "/docs/manage/security/service-accounts", "/cid/5163": "/docs/search/search-query-language/search-operators/geo-lookup-map", "/cid/5164": "/", - "/cid/5165": "/docs/manage/data-forwarding/amazon-s3-bucket", - "/cid/5166": "/docs/manage/data-forwarding/amazon-s3-bucket", + "/cid/5165": "/docs/manage/data-forwarding/forward-data-from-sumologic", + "/cid/5166": "/docs/manage/data-forwarding/forward-data-from-sumologic", "/cid/5167": "/docs/search/get-started-with-search/search-basics/chart-search-results", - "/cid/5168": "/docs/manage/data-forwarding/amazon-s3-bucket", + "/cid/5168": "/docs/manage/data-forwarding/forward-data-from-sumologic", "/cid/5169": "/docs/manage/data-forwarding", "/cid/5170": "/docs/send-data/collection/processing-rules/hash-rules", "/cid/5172": "/docs/send-data/collection/processing-rules", @@ -2592,7 +2592,7 @@ "/cid/5458": "/docs/integrations/microsoft-azure/windows-legacy", "/cid/5460": "/docs/search/get-started-with-search/build-search", "/cid/5500": "/docs/dashboards", - "/cid/5615": "/docs/manage/data-forwarding/amazon-s3-bucket", + "/cid/5615": "/docs/manage/data-forwarding/forward-data-from-sumologic", "/cid/56234": "/docs/integrations/microsoft-azure/sql", "/cid/5624": "/", "/cid/5901": "/docs/send-data/hosted-collectors/amazon-aws/collection-aws-govcloud", @@ -3270,9 +3270,9 @@ "/Manage/Data_Forwarding": "/docs/manage/data-forwarding", "/Manage/Data-Forwarding/Configure-Data-Forwarding-for-Installed-Collectors": "/docs/manage/data-forwarding/installed-collectors", "/docs/manage/data-archiving/installed-collectors": "/docs/manage/data-forwarding/installed-collectors", - "/Manage/Data-Forwarding/Configure-Data-Forwarding-from-Sumo-Logic-to-S3/02File-Format-for-Data-Forwarding-to-an-Amazon-S3-Bucket": "/docs/manage/data-forwarding/amazon-s3-bucket", - "/Manage/Data-Forwarding/Configure-Data-Forwarding-from-Sumo-Logic-to-S3": "/docs/manage/data-forwarding/amazon-s3-bucket", - "/docs/manage/partitions/data-tiers/infrequent-tier-data-forwarding": "/docs/manage/data-forwarding/amazon-s3-bucket", + "/Manage/Data-Forwarding/Configure-Data-Forwarding-from-Sumo-Logic-to-S3/02File-Format-for-Data-Forwarding-to-an-Amazon-S3-Bucket": "/docs/manage/data-forwarding/forward-data-from-sumologic", + "/Manage/Data-Forwarding/Configure-Data-Forwarding-from-Sumo-Logic-to-S3": "/docs/manage/data-forwarding/forward-data-from-sumologic", + "/docs/manage/partitions/data-tiers/infrequent-tier-data-forwarding": "/docs/manage/data-forwarding/forward-data-from-sumologic", "/Manage/Data-Forwarding/Manage_Data_Forwarding": "/docs/manage/data-forwarding/manage", "/Manage/Field-Extractions": "/docs/manage/field-extractions", "/Manage/Field-Extractions/01-Field-Naming-Convention": "/docs/manage/field-extractions/field-naming-convention", @@ -4484,5 +4484,6 @@ "/docs/integrations/microsoft-azure/microsoft-defender-for-cloud": "/docs/integrations/microsoft-azure/azure-security-defender-for-cloud", "/docs/integrations/azure": "/docs/integrations/microsoft-azure", "/docs/search/copilot": "/docs/search/mobot", - "/docs/search/copilot-unstructured-logs-beta": "/docs/search/mobot-unstructured-logs-beta" + "/docs/search/copilot-unstructured-logs-beta": "/docs/search/mobot-unstructured-logs-beta", + "/docs/manage/data-forwarding/amazon-s3-bucket": "/docs/manage/data-forwarding/forward-data-from-sumologic" } diff --git a/docs/api/logs-data-forwarding.md b/docs/api/logs-data-forwarding.md index 75caca3b55..d70540774a 100644 --- a/docs/api/logs-data-forwarding.md +++ b/docs/api/logs-data-forwarding.md @@ -11,7 +11,7 @@ import ApiRoles from '../reuse/api-roles.md'; icon -The Logs Data Forwarding Management API allows you to forward log data from a Partition or Scheduled View to an S3 bucket. For more information, see [Forwarding Data to S3](/docs/manage/data-forwarding/amazon-s3-bucket). +The Logs Data Forwarding Management API allows you to forward log data from a Partition or Scheduled View to an S3 bucket. For more information, see [Forwarding Data to S3](/docs/manage/data-forwarding/forward-data-from-sumologic). ## Documentation diff --git a/docs/manage/data-forwarding/amazon-s3-bucket.md b/docs/manage/data-forwarding/forward-data-from-sumologic.md similarity index 51% rename from docs/manage/data-forwarding/amazon-s3-bucket.md rename to docs/manage/data-forwarding/forward-data-from-sumologic.md index 508a58618a..399aed5886 100644 --- a/docs/manage/data-forwarding/amazon-s3-bucket.md +++ b/docs/manage/data-forwarding/forward-data-from-sumologic.md @@ -1,16 +1,16 @@ --- -id: amazon-s3-bucket -title: Forward Data from Sumo Logic to S3 -description: Learn about how to forward data from Sumo Logic to S3. +id: forward-data-from-sumologic +title: Forward Data from Sumo Logic to S3 or GCS +description: Learn about how to forward data from Sumo Logic to S3 or GCS. --- import useBaseUrl from '@docusaurus/useBaseUrl'; -You can forward log data from a [partition](/docs/manage/partitions) or [Scheduled View](/docs/manage/scheduled-views) to an S3 bucket. Only new data is forwarded from a partition or Scheduled View once it is set to forward data.  +This document outlines the instructions that needs to be followed to forward log data from a [partition](/docs/manage/partitions) or [Scheduled View](/docs/manage/scheduled-views) to an S3 or Google Cloud Storage (GCS) bucket. Only new data is forwarded from a partition or Scheduled View once it is set to forward data.  -To forward data to an S3 bucket: -1. [Configure an S3 forwarding destination](#configure-an-s3-data-forwarding-destination). -1. [Forward data to the S3 forwarding destination](#forward-datato-an-s3-forwarding-destination) from a partition or Schedule View. +To forward data to a storage bucket: +1. [Configure forwarding destination](#configure-data-forwarding-destination). +1. [Forward data to destination](#forward-datato-forwarding-destination) from a partition or schedule view. After data forwarding is configured, you should start to see file objects posted within your configured bucket. If your Scheduled View conducts aggregation, which is a best practice, your aggregate fields are automatically appended to the forwarded objects. @@ -22,23 +22,23 @@ Data forwarding is not currently supported for data assigned to the Infrequent T * An administrator role on the partition where you want to set up forwarding. * Follow the instructions on [Grant Access to an AWS Product](/docs/send-data/hosted-collectors/amazon-aws/grant-access-aws-product) to grant Sumo Logic permission to send data to the destination S3 bucket. -* A partition or Scheduled View to push to Amazon S3. +* A partition or Scheduled View to push to Amazon S3 or Google Cloud Storage (GCS). ## Forwarding interval  Messages are buffered during data ingest for either **approximately** five minutes or until 100MB of data is received, whichever is first. Then the buffered data is written to a new CSV file and forwarded after compression.  -The limits mentioned here are upper limits. Actual file size may vary depending on the ingestion volume in Scheduled Views or partitions of an account.  +The limits mentioned here are upper limits. Actual file size may vary depending on the ingestion volume in scheduled views or partitions of an account.  :::note -It takes approximately five minutes to propagate a new or changed S3 data forwarding rule or bucket across the Sumo Logic service. So, it is possible after you create or modify a rule, the first five minutes of data forwarded might not be written to S3. +It takes approximately five minutes to propagate a new or changed data forwarding rule or bucket across the Sumo Logic service. So, it is possible after you create or modify a rule, the first five minutes of data forwarded might not be written to S3 or GCS. ::: ## File format of forwarded data -After you start forwarding data to S3, you should start to see file objects posted in your configured bucket. The log messages are accumulated and returned after being ingested by Sumo Logic. You can choose to forward only log data, log data and metadata, or log data with metadata and enriched fields, in either CSV or JSON format. +After you start forwarding data, you should start to see file objects posted in your configured bucket. The log messages are accumulated and returned after being ingested by Sumo Logic. You can choose to forward only log data, log data and metadata, or log data with metadata and enriched fields, in either CSV or JSON format. -The log messages are saved in CSV or JSON files in compressed gzip files and named according to the convention you specified when you configured Sumo Logic to start data forwarding, as described in [Forward data to an an S3 forwarding destination](#forward-datato-an-s3-forwarding-destination). The file naming convention for legacy data forwarding is described below in [Legacy file naming format](#legacy-file-naming-format).  +The log messages are saved in CSV or JSON files in compressed gzip files and named according to the convention you specified when you configured Sumo Logic to start data forwarding. The file naming convention for legacy data forwarding is described below in [Legacy file naming format](#legacy-file-naming-format).  Messages are buffered during data ingest for either approximately five minutes or until 100MB of data is received, whichever is first. Then the buffered data is written to a new CSV or JSON file and forwarded.  @@ -58,7 +58,7 @@ These file objects will contain the messages received as well as the system met * **encoding**: The encoding of the original file contents. * **message**: The raw log message as read from the original Source. * **field**: Aggregate fields are added based on your query. - + ### Ordering of fields in forwarded file * The order of the system fields is fixed, and the order is: `messageId, sourceName, sourceHost, sourceCategory, messageTime, receiptTime, sourceId, collectorId, count, format, view, encoding, message`. @@ -74,13 +74,10 @@ When forwarding data from Sumo Logic, the system will write structured logs that `messageId,sourceName,sourceHost,sourceCategory,messageTime,receiptTime,sourceId,collectorId,count,format,view,encoding,message,aggregatefield1,aggregatefield2` - **Sample object** - `"-9223371513354977010","","","","1472590091453","1472590094034","101688020","100607825","979","plain:atp:o:0:l:29:p:yyyy-MM-dd HH:mm:ss,SSSZZZZ","JchenTest2","UTF8","2016-08-30 13:48:11,453 -0700 WARN [hostId=nite-cqsplitter-1] [module=cqsplitter] [localUserName=cqsplitter] [logger=cqsplitter.engine.CQsMultiMatchersManager] [thread=DTP-cqsplitter.receiver.consumer.v2.threadpool-6] MultiMatcher queue for customer 0000000000000131 is at capacity, adding element will block.","25","0000000000000131"` - ### Legacy file naming format The file naming convention for legacy data forwarding (prior to January 2017) is: `---.csv.gz` @@ -91,46 +88,62 @@ Where: * `end_epoch` is the epoch time representing the parsed message time of the last message contained within the file. * `objectid` is a unique ID for the file object, which is generated by Sumo Logic at creation time. - -## Configure an S3 data forwarding destination - -Before you can [forward data](#forward-datato-an-s3-forwarding-destination) from a partition or Scheduled View, you must create a destination that indicates the S3 bucket where you want to send the forwarded data. - -1. [**New UI**](/docs/get-started/sumo-logic-ui/). In the main Sumo Logic menu select **Data Management**, and then under **Logs** select **Data Forwarding**. You can also click the **Go To...** menu at the top of the screen and select **Data Forwarding**.
[**Classic UI**](/docs/get-started/sumo-logic-ui-classic/). In the main Sumo Logic menu, select **Manage Data > Logs > Data Forwarding**. -1. Click **+ Destination** to add a new destination. -1. The **Create New Destination** popup appears.
Create S3 Destination popup -1. **Destination Name**. Enter a name to identify the destination. -1. **Bucket Name**. Enter the [exact name of the S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/view-bucket-properties.html). - :::note - You can create only one destination with a particular bucket name.  If you try to create a new destination with the bucket name of an existing destination, the new destination replaces the old one. - ::: -1. **Description**. You can provide a meaningful description of the connection. -1. **Access Method**. Select **Role-based access** or **Key access** based on the AWS authentication you are providing. Role-based access is preferred. This was completed in the prerequisite step [Grant Access to an AWS Product](/docs/send-data/hosted-collectors/amazon-aws/grant-access-aws-product). - * For **Role-based access** enter the Role ARN that was provided by AWS after creating the role. - * For **Key access** enter the **Access Key ID** and **Secret Access Key**. See [Managing access keys for IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) for details. -1. **S3 Region**. Select the S3 region or keep the default value of Others. The S3 region must match the appropriate S3 bucket created in your Amazon account. -1. **Enable S3 server-side encryption**. Select the check box if you want the forwarded data to be encrypted. For more information, see [Using server-side encryption with Amazon S3 managed keys (SSE-S3)](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html) in AWS help. -1. **Active**. Select this check box to enable data forwarding for the entire S3 bucket. To start forwarding data, you will also need to enable forwarding for the desired indexes, as described below. -1. Click **Save**.
If Sumo Logic is able to verify the S3 credentials, the destination will be added to the list of destinations. If the destination is not added successfully, see [Error and alert conditions](#error-and-alert-conditions) for examples of errors that can occur. - -Once the destination is created, you can start data forwarding for specific partitions or Scheduled Views as described in [Forward data to an S3 forwarding destination](#forward-datato-an-s3-forwarding-destination) below. - -## Forward data to an S3 forwarding destination - -Once you [configure an S3 forwarding destination](#configure-an-s3-data-forwarding-destination) that indicates the S3 bucket to receive the data, you can forward data to the destination from partitions and Scheduled Views. +## Configure data forwarding destination + +Before you can [forward data](#forward-datato-forwarding-destination) from a partition or Scheduled View, you must create a destination that indicates the S3 or GCS bucket where you want to send the forwarded data. + +1. [**New UI**](/docs/get-started/sumo-logic-ui/). In the main Sumo Logic menu select **Manage Data**, and then under **Logs** select **Data Forwarding**. You can also click the **Go To...** menu at the top of the screen and select **Data Forwarding**.
[**Classic UI**](/docs/get-started/sumo-logic-ui-classic/). In the main Sumo Logic menu, select **Manage Data > Logs > Data Forwarding**. +1. Click **+ Add Destination** to add a new destination. +1. The **Create New Destination** popup appears. +1. **Destination Type**. You can either select **Amazon S3** or **Google Cloud Storage** as your destination type. + - For **Amazon S3** as the destination type, follow the below steps:
Create S3 Destination popup + 1. **Destination Name**. Enter a name to identify the destination. + 1. **Bucket Name**. Enter the [exact name of the S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/view-bucket-properties.html). + :::note + You can create only one destination with a particular bucket name.  If you try to create a new destination with the bucket name of an existing destination, the new destination replaces the old one. + ::: + 1. (Optional)**Description**. You can provide a meaningful description of the connection. + 1. **Access Method**. Select **Role-based access** or **Key access** based on the AWS authentication you are providing. Role-based access is preferred. This was completed in the prerequisite step [Grant Access to an AWS Product](/docs/send-data/hosted-collectors/amazon-aws/grant-access-aws-product). + * For **Role-based access** enter the Role ARN that was provided by AWS after creating the role. + * For **Key access** enter the **Access Key ID** and **Secret Access Key**. See [Managing access keys for IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) for details. + 1. **S3 Region**. Select the S3 region or keep the default value of Others. The S3 region must match the appropriate S3 bucket created in your Amazon account. + 1. **Enable S3 server-side encryption**. Select the check box if you want the forwarded data to be encrypted. For more information, see [Using server-side encryption with Amazon S3 managed keys (SSE-S3)](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html) in AWS help. + - For **Google Cloud Storage** as the destination type, follow the below steps:
Create S3 Destination popup + 1. **Destination Name**. Enter a name to identify the destination. + 1. **Bucket Name**. Enter the [exact name of the GCS bucket](https://cloud.google.com/storage/docs/buckets). + :::note + You can create only one destination with a particular bucket name.  If you try to create a new destination with the bucket name of an existing destination, the new destination replaces the old one. + ::: + 1. (Optional) **Description**. Provide a meaningful description of the connection. + 1. For **HMAC Access Key** and **HMAC Secret Key** enter the values collected from the Google platform service account. See [Manage HMAC keys for service account](https://cloud.google.com/storage/docs/authentication/managing-hmackeys#console_1) for details. +1. **Active**. Select this check box to enable data forwarding for the entire bucket. To start forwarding data, you will also need to enable forwarding for the desired indexes, as described below. +1. Click **Save**.
If Sumo Logic is able to verify the credentials, the destination will be added to the list of destinations. If the destination is not added successfully, see [Error and alert conditions](#error-and-alert-conditions) for examples of errors that can occur. + +Once the destination is created, you can start data forwarding for specific partitions or Scheduled Views as described in [Forward data to an forwarding destination](#forward-datato-forwarding-destination) below. + +## Forward data to forwarding destination + +Once you [configure date forwarding destination](#configure-data-forwarding-destination) that indicates the bucket to receive the data, you can forward data to the destination from partitions and Scheduled Views. 1. Depending on whether you want to forward data from a partition or a Scheduled View: - * **Partition**:
[**New UI**](/docs/get-started/sumo-logic-ui/). In the main Sumo Logic menu select **Data Management**, and then under **Logs** select **Partitions**. You can also click the **Go To...** menu at the top of the screen and select **Partitions**.
[**Classic UI**](/docs/get-started/sumo-logic-ui-classic/). In the main Sumo Logic menu, select **Manage Data > Logs > Partitions**. - * **Scheduled View**:
[**New UI**](/docs/get-started/sumo-logic-ui/). In the main Sumo Logic menu select **Data Management**, and then under **Logs** select **Scheduled Views**. You can also click the **Go To...** menu at the top of the screen and select **Scheduled Views**.
[**Classic UI**](/docs/get-started/sumo-logic-ui-classic/). In the main Sumo Logic menu, select **Manage Data > Logs > Scheduled Views**. + * **Partition**:
[**New UI**](/docs/get-started/sumo-logic-ui/). In the main Sumo Logic menu select **Manage Data**, and then under **Logs** select **Partitions**. You can also click the **Go To...** menu at the top of the screen and select **Partitions**.
[**Classic UI**](/docs/get-started/sumo-logic-ui-classic/). In the main Sumo Logic menu, select **Manage Data > Logs > Partitions**. + * **Scheduled View**:
[**New UI**](/docs/get-started/sumo-logic-ui/). In the main Sumo Logic menu select **Manage Data**, and then under **Logs** select **Scheduled Views**. You can also click the **Go To...** menu at the top of the screen and select **Scheduled Views**.
[**Classic UI**](/docs/get-started/sumo-logic-ui-classic/). In the main Sumo Logic menu, select **Manage Data > Logs > Scheduled Views**. 1. Select the partition or Scheduled View for which you want to enable data forwarding and click the **Edit** button. The edit dialog for the partition or Scheduled View displays. Following is the edit dialog for a partition.
Enable Data Forwarding checkbox :::tip In addition to forwarding data from existing partitions and Scheduled Views, you can also enable data forwarding by selecting the **Enable Data Forwarding** check box when you first [create a partition](/docs/manage/partitions/flex/create-edit-partition-flex/) or [create a Scheduled View](/docs/manage/scheduled-views/add-scheduled-view/). ::: -1. Click the **Enable Data Forwarding** checkbox. More options appear.
Forwarding destination options -1. **Forwarding Destination**. Choose one of the following: - * **Existing Amazon S3 Destination**. If you select this option, select the destination in the **Amazon S3 Destination** field below. - * **New Amazon S3 Destination**. Follow the instructions in [Configure an S3 data forwarding destination](#configure-an-s3-data-forwarding-destination) above to create a new S3 destination. -1. **Amazon S3 Destination**. If you chose **Existing Amazon S3 Destination** for the forwarding destination, select the destination here. +1. Click the **Enable Data Forwarding** checkbox. More options appear. +1. **Destination Type**. You can either select **Amazon S3** or **Google Cloud Storage** as your destination type. + - For **Amazon S3** as the destination type, follow the below steps:
Forwarding destination options + 1. **Forwarding Destination**. Choose one of the following: + * **Existing Amazon S3 Destination**. If you select this option, select the destination in the **Amazon S3 Destination** field below. + * **New Amazon S3 Destination**. Follow the instructions in [Configure data forwarding destination](#forward-datato-forwarding-destination) above to create a new S3 destination. + 1. **Amazon S3 Destination**. If you chose **Existing Amazon S3 Destination** for the forwarding destination, select the destination here. + - For **Google Cloud Storage** as the destination type, follow the below steps:
Forwarding destination options + 1. **Forwarding Destination**. Choose one of the following: + * **Existing Google Cloud Storage Destination**. If you select this option, select the destination in the **Google Cloud Storage Destination** field below. + * **New Google Cloud Storage Destination**. Follow the instructions in [Configure data forwarding destination](#configure-data-forwarding-destination) above to create a new S3 destination. + 1. **Google Cloud Storage Destination**. If you chose **Existing Google Cloud Storage Destination** for the forwarding destination, select the destination here. 1. Click **Data Forwarding Configuration**. Options appear for forwarding the data.
Options to forward raw data 1. **Included Data**. Select the kind of data to forward: * **Raw**. Raw logs only. @@ -140,7 +153,7 @@ Once you [configure an S3 forwarding destination](#configure-an-s3-data-forwardi * **Text**. Plain text. (Available only if you choose **Raw** above.) * **CSV**. Comma-separated values. (Available if you choose **Raw + Metadata** or **All** above.) * **JSON**. Java Script Object Notation. (Available if you choose **Raw + Metadata** or **All** above.) Select **JSON** if you want to ensure that forwarded data can be re-ingested easily. - 1. **File Prefix**. Enter the path prefix to a directory in the S3 bucket. You can include any of the following variables: + 1. **File Prefix**. Enter the path prefix to a directory in the S3 or GCS bucket. You can include any of the following variables: * `{index}` will be replaced by the name of the partition or scheduled view. * `{day}` will be replaced by the day of the year in the yyyy-MM-dd format. * `{hour}` will be replaced by the hour of the day (0-23). @@ -156,17 +169,18 @@ For information about how the data is forwarded, see [Forwarding interval](#forw ## Data forwarding example -Let's say you want to take data from Sumo Logic and run additional analysis on it in tools separate from Sumo Logic. You can forward the data from Sumo Logic to an S3 bucket where it is available for download and analysis by your tools. +Let's say you want to take data from Sumo Logic and run additional analysis on it in tools separate from Sumo Logic. In this example, you can forward the data from Sumo Logic to an S3 or GCS bucket where it is available for download and analysis by your tools. -Let's suppose you have an S3 bucket named `amzn-s3-demo-bucket1` where you want to forward your Sumo Logic data. Do the following: +Let's suppose you have an S3 or GCS bucket named `demo-bucket1` where you want to forward your Sumo Logic data. Do the following: -1. [Create a destination](/docs/manage/data-forwarding/amazon-s3-bucket/#configure-an-s3-data-forwarding-destination) that points to the `amzn-s3-demo-bucket1` bucket. For example, name it **Test destination**. -1. Open the partition or Scheduled View whose data you want to [forward data to the new destination](/docs/manage/data-forwarding/amazon-s3-bucket/#configure-an-s3-data-forwarding-destination). +1. [Create a destination](/docs/manage/data-forwarding/forward-data-from-sumologic/#configure-data-forwarding-destination) that points to the `demo-bucket1` bucket. For example, name it **Test destination**. +1. Open the partition or Scheduled View whose data you want to [forward data to the new destination](/docs/manage/data-forwarding/forward-data-from-sumologic/#configure-data-forwarding-destination). 1. In the partition or Scheduled View, select **Enable Data Forwarding**, and fill out the fields that appear: - 1. In **Forwarding Destination** select **Existing Amazon S3 Destination**. - 1. In **Amazon S3 Destination** select the name of the destination you created earlier, for example, **Test destination**. + 1. In **Destination Type** select **Amazon S3** or **Google Cloud Storage** depending on your requirement. + 1. In **Forwarding Destination** select any **Existing Destination**. + 1. In **Destination** select the name of the destination you created earlier, for example, **Test destination**. 1. Use the **Data Forwarding Configuration** section to specify whether to forward only log data, log data with metadata, or log data with metadata and enriched fields. -1. Click **Save** on the partition or Scheduled View. The data will start forwarding to the S3 bucket specified in the destination. +1. Click **Save** on the partition or Scheduled View. The data will start forwarding to the selected destination bucket specified in the destination. ## Error and alert conditions @@ -179,3 +193,4 @@ An error or alert condition can occur with an S3 data forwarding destination fo * Hover over the icon to display the message.
Hover message In this example, Sumo Logic has disabled data forwarding due to errors in connecting to the S3 bucket. This occurs if the Amazon account or credentials change so that Sumo Logic is no longer able to authenticate to the bucket.   + diff --git a/docs/manage/data-forwarding/index.md b/docs/manage/data-forwarding/index.md index 0949ba28b8..f72fd82c9d 100644 --- a/docs/manage/data-forwarding/index.md +++ b/docs/manage/data-forwarding/index.md @@ -8,7 +8,7 @@ import useBaseUrl from '@docusaurus/useBaseUrl'; Document with a forward symbol icon -Data Forwarding allows you to forward log data to an external server or supported storage service. You can forward log data to an AWS S3 bucket through [Partitions](/docs/manage/partitions) or [Scheduled Views](/docs/manage/scheduled-views). See [Forwarding Data from Sumo Logic to S3](/docs/manage/data-forwarding/amazon-s3-bucket/) for details. +Data Forwarding allows you to forward log data to an external server or supported storage service. You can forward log data to an AWS S3 bucket or Google Cloud Storage (GCS) through [Partitions](/docs/manage/partitions) or [Scheduled Views](/docs/manage/scheduled-views). ## Guide contents @@ -26,9 +26,9 @@ In this section, we'll introduce the following concepts:
- + Document with a forward symbol  icon -

Forward Data from Sumo Logic to S3

+

Forward Data from Sumo Logic to S3 or GCS

Learn step-by-step instructions of data forwarding from Sumo Logic to S3.

diff --git a/docs/manage/data-forwarding/installed-collectors.md b/docs/manage/data-forwarding/installed-collectors.md index 4c379ebfad..58744edd81 100644 --- a/docs/manage/data-forwarding/installed-collectors.md +++ b/docs/manage/data-forwarding/installed-collectors.md @@ -78,7 +78,7 @@ Follow the instructions for the destination type you chose. :::note -Data forwarding to S3 Archive locations will forward log data from Installed Collectors to AWS S3 buckets to collect at a later time. Data **will not** be forked to both Sumo Logic and AWS S3. In that case, you will want to send the data to Sumo Logic first and then configure [Forwarding Data from Sumo Logic to S3](/docs/manage/data-forwarding/amazon-s3-bucket/). +Data forwarding to S3 Archive locations will forward log data from Installed Collectors to AWS S3 buckets to collect at a later time. Data **will not** be forked to both Sumo Logic and AWS S3. In that case, you will want to send the data to Sumo Logic first and then configure [Forwarding Data from Sumo Logic to S3](/docs/manage/data-forwarding/forward-data-from-sumologic/). ::: * **Bucket Name**. Enter the exact name of the S3 bucket.You can create only one destination with a particular bucket name. If you try to create a new destination with the bucket name of an existing destination, the new destination replaces the old one. diff --git a/docs/manage/partitions/edit-data-forwarding-destinations-partition.md b/docs/manage/partitions/edit-data-forwarding-destinations-partition.md index 361250a932..76c08cd4a3 100644 --- a/docs/manage/partitions/edit-data-forwarding-destinations-partition.md +++ b/docs/manage/partitions/edit-data-forwarding-destinations-partition.md @@ -16,4 +16,4 @@ You can specify data forwarding settings for a partition so that the messages th partitions-page 1. The partition details are displayed on the right side of the page.
edit-partition-pane-search-icon 1. Click **Edit** to open the pane for editing.
edit-partition-pane.png -1. You can configure Data Forwarding, or if Data Forwarding is already configured, modify the configuration. For more information, see [Data Forwarding](../data-forwarding/amazon-s3-bucket.md). +1. You can configure Data Forwarding, or if Data Forwarding is already configured, modify the configuration. For more information, see [Forward Data from Sumo Logic to S3 or GCS](../data-forwarding/forward-data-from-sumologic.md). diff --git a/docs/manage/partitions/index.md b/docs/manage/partitions/index.md index 874d0fcff1..4fd675fe98 100644 --- a/docs/manage/partitions/index.md +++ b/docs/manage/partitions/index.md @@ -13,7 +13,7 @@ Creating a partition enhances search performance by narrowing down the search s [**Classic UI**](/docs/get-started/sumo-logic-ui-classic). To access the Partitions page, in the main Sumo Logic menu select **Manage Data > Logs > Partitions**. -A partition stores your data in an index separate from the rest of your account's data so you can [optimize searches](../../search/optimize-search-performance.md), [manage variable retention](manage-indexes-variable-retention.md), and specify certain [data to forward to S3](../data-forwarding/amazon-s3-bucket.md). +A partition stores your data in an index separate from the rest of your account's data so you can [optimize searches](../../search/optimize-search-performance.md), [manage variable retention](manage-indexes-variable-retention.md), and specify certain [data to forward to S3 or GCS](../data-forwarding/forward-data-from-sumologic.md). :::note Data stored in a partition is not stored anywhere else.  diff --git a/docs/manage/scheduled-views/add-scheduled-view.md b/docs/manage/scheduled-views/add-scheduled-view.md index dfb083fc3c..ec310c4b2a 100644 --- a/docs/manage/scheduled-views/add-scheduled-view.md +++ b/docs/manage/scheduled-views/add-scheduled-view.md @@ -22,7 +22,7 @@ For Scheduled View query requirements, see [Scheduled Views Best Practices and E ::: 1. **Timezone**. Select the timezone for the scheduled view of your choice from the drop-down. If you do not make a selection, the Scheduled View will default to the timezone preference in Sumo Logic. But if the timezone is not set in Sumo Logic *User Preferences* page, then this will default to the timezone from your browser. 1. **Retention Period.** Either enter a retention period for the data in the index, in days, or click **Apply the retention period of Default Partition**. For more information, see [Manage Indexes with Variable Retention](../partitions/manage-indexes-variable-retention.md). -1. **Data Forwarding.** (Optional). Choose **Enable Data Forwarding** to [forward data from Sumo to Amazon S3](../data-forwarding/amazon-s3-bucket.md). The results from the Scheduled View are forwarded to S3. Raw logs are sent if the view query does not use an aggregate operator. If the view query performs an aggregation, aggregate results are sent. See [File Format](../data-forwarding/amazon-s3-bucket.md) for details on how the file objects are structured. +1. **Data Forwarding.** (Optional). Choose **Enable Data Forwarding** to [forward data from Sumo Logic to Amazon S3 or Google Cloud Storage](../data-forwarding/forward-data-from-sumologic.md). The results from the Scheduled View are forwarded to S3 or GCS. Raw logs are sent if the view query does not use an aggregate operator. If the view query performs an aggregation, aggregate results are sent. See [File Format](../data-forwarding/forward-data-from-sumologic.md) for details on how the file objects are structured. 1. Click **Save**. The view begins to index data as soon as you create it. Allow a few hours for the indexing to complete. If you've chosen to index a large amount of data and/or have chosen a long date range for the view, it could take a bit longer. diff --git a/docs/manage/scheduled-views/view-list-scheduled-views.md b/docs/manage/scheduled-views/view-list-scheduled-views.md index ad9a19f2a5..0800f8f3b2 100644 --- a/docs/manage/scheduled-views/view-list-scheduled-views.md +++ b/docs/manage/scheduled-views/view-list-scheduled-views.md @@ -21,7 +21,7 @@ You must have a role that grants you the View Scheduled Views [role capability * **Name**. The name assigned to the scheduled view. * **Storage Consumed**. The total volume of uncompressed data ingested across the duration of the retention period. * **Retention Period**. The number of days configured as the retention period. - * **Data Forwarding**. Indicates the name of the [data forwarding](../data-forwarding/amazon-s3-bucket.md) destination if the scheduled view is configured to forward data to the S3 bucket. + * **Data Forwarding**. Indicates the name of the [data forwarding](../data-forwarding/forward-data-from-sumologic.md) destination if the scheduled view is configured to forward data to an S3 or GCS bucket. 1. To view details of a scheduled view configuration, click the row containing the view. A pane will appear on the right side of the page with the following information. * **Name**. Displays the name of the scheduled view. * **Query**. The query that returns the data for the scheduled view. @@ -34,7 +34,7 @@ You must have a role that grants you the View Scheduled Views [role capability * **Lag Time**. If the scheduled view is not up-to-date, **Lag Time** contains the actual lag time. For more information, see [Scheduled View Lag Time](scheduled-view-lag-time.md). * **Timezone**. Displays the selected time zone or the default timezone of your browser while creating the scheduled view. * **Query**. The query that returns that data to be written to the scheduled view. - * **Data Forwarding**. If the scheduled view is configured to forward data to an S3 bucket, the name of the [data forwarding](../data-forwarding/amazon-s3-bucket.md) destination.   + * **Data Forwarding**. If the scheduled view is configured to forward data to an S3 or GCS bucket, the name of the [data forwarding](../data-forwarding/forward-data-from-sumologic.md) destination.   * **Created by** and **Modified by**. The user that created the view, and the user that most recently modified the view.
sched-view-details diff --git a/docs/manage/users-roles/roles/role-capabilities.md b/docs/manage/users-roles/roles/role-capabilities.md index 0aaf84077f..5db2a64d65 100644 --- a/docs/manage/users-roles/roles/role-capabilities.md +++ b/docs/manage/users-roles/roles/role-capabilities.md @@ -19,7 +19,7 @@ Following are the capabilities you can assign when you [create roles](create-ma | View Fields | View [fields](/docs/manage/fields), which are custom metadata fields you can assign to logs.| | Manage Fields | Manage fields. Note that if you grant a role the Manage Fields capability, users with that role will also have the View Fields and View Field Extraction Rules capabilities.| | Manage Field Extraction Rules | Manage [field extractions](/docs/manage/field-extractions), which speed the search process by automatically parsing fields as log messages are ingested. Note that if you grant a role the Manage Field Extraction Rules capability, users with that role will also have the Manage Fields, View Fields, and View Field Extraction Rules capabilities.| -| Manage S3 Data Forwarding | Manage [S3 data forwarding](/docs/manage/data-forwarding/amazon-s3-bucket) from Sumo Logic to an S3 bucket.| +| Manage S3 and GCS Data Forwarding | Manage [S3 and GCS data forwarding](/docs/manage/data-forwarding/forward-data-from-sumologic) from Sumo Logic to an S3 or GCS bucket.| | Manage Content | Manage the content for your organization. This provides access to [Admin Mode](/docs/manage/content-sharing/admin-mode) in the Library.| | Manage Apps | Install and manage [apps](/docs/integrations). | | Manage Connections | Manage the [connections](/docs/alerts/webhook-connections/) that allow you to send alerts to other tools. | diff --git a/docs/search/index.md b/docs/search/index.md index 108ef5b43c..06655c574c 100644 --- a/docs/search/index.md +++ b/docs/search/index.md @@ -127,7 +127,7 @@ In this micro lesson, learn about the ingestion pipeline and the journey that a Logs collected by Sumo Logic are indexed in Partitions and Scheduled Views. In addition, there are internal indexes such as Health Events, Archive, Audit, and Volume indexes. -* A Partition stores your data in an index separate from the rest of your account data so you can [optimize searches](optimize-search-performance.md), [manage variable retention](/docs/manage/partitions/manage-indexes-variable-retention), and specify certain [data to forward to S3](/docs/manage/data-forwarding/amazon-s3-bucket). See how to [Run a Search Against a Partition](/docs/search/optimize-search-partitions). +* A Partition stores your data in an index separate from the rest of your account data so you can [optimize searches](optimize-search-performance.md), [manage variable retention](/docs/manage/partitions/manage-indexes-variable-retention), and specify certain [data to forward to S3 or GCS](/docs/manage/data-forwarding/forward-data-from-sumologic). See how to [Run a Search Against a Partition](/docs/search/optimize-search-partitions). * Scheduled Views speed the search process subsets of your data by functioning as a pre-aggregated index. See how to [Run a Search Against a Scheduled View](/docs/manage/scheduled-views/run-search-against-scheduled-view). * Health Events monitor the health of your Collectors and Sources. See how to [Search Health Events](/docs/manage/health-events). * Archive allows you to forward log data from Installed Collectors to Amazon S3 buckets to collect at a later time. See how to [Search ingested Archive data](/docs/manage/data-archiving/archive). diff --git a/docs/search/optimize-search-partitions.md b/docs/search/optimize-search-partitions.md index cd664e9a36..c674e5296a 100644 --- a/docs/search/optimize-search-partitions.md +++ b/docs/search/optimize-search-partitions.md @@ -6,7 +6,7 @@ sidebar_label: Optimize Search with Partitions ## What is a Partition? -A partition stores your data in an index separate from the rest of your account's data so you can optimize searches, [manage variable retention](/docs/manage/partitions/manage-indexes-variable-retention), and specify certain [data to forward to S3](/docs/manage/data-forwarding/amazon-s3-bucket). +A partition stores your data in an index separate from the rest of your account's data so you can optimize searches, [manage variable retention](/docs/manage/partitions/manage-indexes-variable-retention), and specify certain [data to forward to S3 or GCS](/docs/manage/data-forwarding/forward-data-from-sumologic). Partitions route your data to an index becoming a separate subset of data in your account. Creating smaller and separate subsets of data is central to search optimization. When you run a search against an index, results are returned more quickly and efficiently because the search runs against a smaller data set. diff --git a/sidebars.ts b/sidebars.ts index fd6a65beb4..64a4dc27d2 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -1019,7 +1019,7 @@ module.exports = { link: {type: 'doc', id: 'manage/data-forwarding/index'}, items: [ 'manage/data-forwarding/installed-collectors', - 'manage/data-forwarding/amazon-s3-bucket', + 'manage/data-forwarding/forward-data-from-sumologic', 'manage/data-forwarding/manage', 'manage/data-forwarding/view-list-data-forwarding', ] diff --git a/static/img/manage/data-forwarding/create-GCS-destination.png b/static/img/manage/data-forwarding/create-GCS-destination.png new file mode 100644 index 0000000000..e00333f4c8 Binary files /dev/null and b/static/img/manage/data-forwarding/create-GCS-destination.png differ diff --git a/static/img/manage/data-forwarding/create-S3-destination.png b/static/img/manage/data-forwarding/create-S3-destination.png index 84411ce93d..36d65e8400 100644 Binary files a/static/img/manage/data-forwarding/create-S3-destination.png and b/static/img/manage/data-forwarding/create-S3-destination.png differ diff --git a/static/img/manage/data-forwarding/specify-destination-gcs.png b/static/img/manage/data-forwarding/specify-destination-gcs.png new file mode 100644 index 0000000000..72c89d2537 Binary files /dev/null and b/static/img/manage/data-forwarding/specify-destination-gcs.png differ diff --git a/static/img/manage/data-forwarding/specify-destination.png b/static/img/manage/data-forwarding/specify-destination.png index a9dd1421bb..eced2a82cc 100644 Binary files a/static/img/manage/data-forwarding/specify-destination.png and b/static/img/manage/data-forwarding/specify-destination.png differ