From e64f061c5ee5839e53bf0397b393bf5c585686b7 Mon Sep 17 00:00:00 2001 From: Angela Costa Date: Wed, 12 Feb 2025 19:46:14 +0000 Subject: [PATCH 1/2] Adds datasets to Log Explorer --- src/content/docs/logs/log-explorer.mdx | 89 ++++++++++++++++++++++---- 1 file changed, 78 insertions(+), 11 deletions(-) diff --git a/src/content/docs/logs/log-explorer.mdx b/src/content/docs/logs/log-explorer.mdx index 8fde4c64fc1a8a2..fc780a29fd622c4 100644 --- a/src/content/docs/logs/log-explorer.mdx +++ b/src/content/docs/logs/log-explorer.mdx @@ -19,29 +19,49 @@ Log Explorer is currently in beta. To request access, complete the [sign-up form ## Supported datasets -The following zone-level datasets are currently available with Log Explorer: +Log Explorer is available at the account and zone level. At the zone level, datasets currently available are: - [HTTP requests](/logs/reference/log-fields/zone/http_requests/) (`FROM http_requests`) - [Firewall events](/logs/reference/log-fields/zone/firewall_events/) (`FROM firewall_events`) +At the account level, the datasets available are: + +- [Access requests](/logs/reference/log-fields/account/access_requests/) (`FROM access_requests`) +- [CASB Findings](/logs/reference/log-fields/account/casb_findings/) (`FROM casb_findings`) +- [Device posture results](/logs/reference/log-fields/account/device_posture_results/) (`FROM device_posture_results`) +- [Gateway DNS](/logs/reference/log-fields/account/gateway_dns/) (`FROM gateway_dns`) +- [Gateway HTTP](/logs/reference/log-fields/account/gateway_http/) (`FROM gateway_http`) +- [Gateway Network](/logs/reference/log-fields/account/gateway_network/) (`FROM gateway_network`) + ## Authentication -In order to communicate with the API, you will need to configure the appropriate authentication headers. +Log Explorer is available to users with the following permissions: + +- **Logs Edit**: users with Logs Edit permissions can enable datasets. +- **Logs Read**: users with Logs Read permissions can run queries via the UI or API. + +Note that these permissions exist at the account and zone level and you need the appropriate permission level for the datasets you wish to query. + +Authentication with the API can be done via auth header or API token. Append your API call with either of the following additional parameters. -- `X-Auth-Email` - the Cloudflare account email address associated with the domain -- `X-Auth-Key` - the Cloudflare API key +- **Auth header** -Alternatively, API tokens with Account and Zone level Logs Edit permissions can also be used for authentication: + - `X-Auth-Email` - the Cloudflare account email address associated with the domain + - `X-Auth-Key` - the Cloudflare API key -- `Authorization: Bearer ` +- **API token** + + - `Authorization: Bearer ` To create an appropriately scoped API token, refer to [Create API token](/fundamentals/api/get-started/create-token/) documentation. Copy and paste the token into the Authorization parameter for your API call. + + ## Enable Log Explorer -You can use the dashboard or the API to enable the datasets you want to query with Log Explorer. +In order for Log Explorer to begin storing logs, you need to enable the desired datasets. You can do this via the dashboard or the API. -1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/login) and select your account and domain. +1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/login) and and select your account or domain (also known as zone). 2. Go to **Analytics & Logs** > **Log Explorer**. 3. Select **Enable a dataset** to select the datasets you want to query. You can enable more datasets later. @@ -52,9 +72,9 @@ It may take a few minutes for the logs to become available for querying. -Use the Log Explorer API to enable Log Explorer for each dataset you wish to store. It may take up to 30 minutes after a logstream is enabled before you can view the logs. +Use the Log Explorer API to enable Log Explorer for each dataset you wish to store. It may take a few minutes after a log stream is enabled before you can view the logs. -The following curl command is an example for enabling `http_requests`, as well as the expected response when the command succeeds. +The following curl command is an example for enabling the zone-level dataset `http_requests`, as well as the expected response when the command succeeds. ```bash curl https://api.cloudflare.com/client/v4/zones/{zone_id}/logs/explorer/datasets \ @@ -79,6 +99,17 @@ curl https://api.cloudflare.com/client/v4/zones/{zone_id}/logs/explorer/datasets } ``` +If you would like to enable an account-level dataset, substitute `zones/{zone_id}` with `accounts/{account_id}` in the curl command. For example: + +```bash +curl https://api.cloudflare.com/client/v4/accounts/{account_id}/logs/explorer/datasets \ +--header "Authorization: Bearer " \ +--header "Content-Type: application/json" \ +--data '{ + "dataset": "access_requests" +}' +``` + ## Use Log Explorer @@ -87,7 +118,7 @@ Filtering and viewing your logs is available via the Cloudflare Dashboard or via -1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/login) and select your account and domain. +1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/login) and select your account or domain (also known as zone). 2. Go to **Analytics & Logs** > **Log Explorer**. 3. From the dropdown, select the **Dataset** you want to use. 4. Select a **Limit**. That is the maximum number of results to return, for example, 50. @@ -133,6 +164,38 @@ Which returns the following HTTP request details: } ``` +For another example using an account-level dataset, to find Cloudflare Access requests with selected columns from a specific timeframe, you can perform the following SQL query. + +```bash +curl https://api.cloudflare.com/client/v4/account/{account_id}/logs/explorer/query/sql \ +--header "Authorization: Bearer " \ +--url-query query="SELECT CreatedAt, AppDomain, AppUUID, Action, Allowed, Country, RayID, Email, IPAddress, UserUID FROM access_requests WHERE Date >= '2025-02-06' AND Date <= '2025-02-06' AND CreatedAt >= '2025-02-06T12:28:39Z' AND CreatedAt <= '2025-02-06T12:58:39Z'" +``` + +Which returns the following request details: + +```json +{ + "result": [ + { + "createdat": "2025-01-14T18:17:55Z", + "appdomain": "example.com", + "appuuid": "a66b4ab0-ccdf-4d60-a6d0-54a59a827d92", + "action": "login", + "allowed": true, + "country": "us", + "rayid": "90fbb07c0b316957", + "email": "user@example.com", + "ipaddress": "1.2.3.4", + "useruid": "52859e81-711e-4de0-8b31-283336060e79" + } + ], + "success": true, + "errors": [], + "messages": [] +} +``` + ## Output formats @@ -214,6 +277,10 @@ Log Explorer performs best when query parameters focus on narrower ranges of tim If your query times out with an HTTP status of 524 (Gateway Timeout), consider using one of the [streaming output formats](/logs/log-explorer/#output-formats), such as `application/x-ndjson`. +### Why don't I see any logs in my queries after enabling the dataset? + +Log Explorer starts ingesting logs from the moment you enable the dataset. It will not display logs for events that occurred before the dataset was enabled. Make sure that new events have been generated since enabling the dataset, and check again. + ### My query returned an error. How do I figure out what went wrong? We are actively working on improving error codes. If you receive a generic error, check your SQL syntax (if you are using the custom SQL feature), make sure you have included a date and a limit, and that the field you are filtering is not a key-value pair. If the query still fails it is likely timing out. Try refining your filters. From ff1260ec57fb7195c55d316c9b7375a369c3562e Mon Sep 17 00:00:00 2001 From: angelampcosta <92738954+angelampcosta@users.noreply.github.com> Date: Thu, 13 Feb 2025 09:20:23 +0000 Subject: [PATCH 2/2] Apply suggestions from code review Co-authored-by: marciocloudflare <83226960+marciocloudflare@users.noreply.github.com> --- src/content/docs/logs/log-explorer.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/content/docs/logs/log-explorer.mdx b/src/content/docs/logs/log-explorer.mdx index fc780a29fd622c4..9ad4e569b1486d8 100644 --- a/src/content/docs/logs/log-explorer.mdx +++ b/src/content/docs/logs/log-explorer.mdx @@ -42,16 +42,16 @@ Log Explorer is available to users with the following permissions: Note that these permissions exist at the account and zone level and you need the appropriate permission level for the datasets you wish to query. -Authentication with the API can be done via auth header or API token. Append your API call with either of the following additional parameters. +Authentication with the API can be done via an authentication header or API token. Append your API call with either of the following additional parameters. -- **Auth header** +- **Authentication header** - `X-Auth-Email` - the Cloudflare account email address associated with the domain - `X-Auth-Key` - the Cloudflare API key - **API token** - - `Authorization: Bearer ` To create an appropriately scoped API token, refer to [Create API token](/fundamentals/api/get-started/create-token/) documentation. Copy and paste the token into the Authorization parameter for your API call. + - `Authorization: Bearer ` To create an appropriately scoped API token, refer to [Create API token](/fundamentals/api/get-started/create-token/) documentation. Copy and paste the token into the authorization parameter for your API call. @@ -61,7 +61,7 @@ In order for Log Explorer to begin storing logs, you need to enable the desired -1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/login) and and select your account or domain (also known as zone). +1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/login) and select your account or domain (also known as zone). 2. Go to **Analytics & Logs** > **Log Explorer**. 3. Select **Enable a dataset** to select the datasets you want to query. You can enable more datasets later. @@ -99,7 +99,7 @@ curl https://api.cloudflare.com/client/v4/zones/{zone_id}/logs/explorer/datasets } ``` -If you would like to enable an account-level dataset, substitute `zones/{zone_id}` with `accounts/{account_id}` in the curl command. For example: +If you would like to enable an account-level dataset, replace `zones/{zone_id}` with `accounts/{account_id}` in the curl command. For example: ```bash curl https://api.cloudflare.com/client/v4/accounts/{account_id}/logs/explorer/datasets \