Skip to content

Commit d8c1a11

Browse files
Maddy-CloudflareMaddy-Cloudflare
committed
Adding manage datasets and api page + updates
1 parent a9258a6 commit d8c1a11

File tree

5 files changed

+222
-165
lines changed

5 files changed

+222
-165
lines changed

‎src/content/docs/log-explorer/api.mdx‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
---
2+
pcx_content_type: reference
3+
title: Log Explorer API
4+
sidebar:
5+
order: 5
6+
---
7+
8+
Log searches are available via the [API](https://developers.cloudflare.com/log-explorer/).
9+
10+
Log Explorer exposes a query endpoint that uses familiar SQL syntax for querying your logs generated with Cloudflare's network.
11+
12+
For example, to find an HTTP request with a specific [Ray ID](/fundamentals/reference/cloudflare-ray-id/), you can perform the following SQL query:
13+
14+
```bash
15+
curl https://api.cloudflare.com/client/v4/zones/{zone_id}/logs/explorer/query/sql \
16+
--header "Authorization: Bearer <API_TOKEN>" \
17+
--url-query query="SELECT clientRequestScheme, clientRequestHost, clientRequestMethod, edgeResponseStatus, clientRequestUserAgent FROM http_requests WHERE RayID = '806c30a3cec56817' LIMIT 1"
18+
```
19+
20+
Which returns the following HTTP request details:
21+
22+
```json
23+
{
24+
"result": [
25+
{
26+
"clientrequestscheme": "https",
27+
"clientrequesthost": "example.com",
28+
"clientrequestmethod": "GET",
29+
"clientrequestuseragent": "curl/7.88.1",
30+
"edgeresponsestatus": 200
31+
}
32+
],
33+
"success": true,
34+
"errors": [],
35+
"messages": []
36+
}
37+
```
38+
39+
A example to find Cloudflare Access requests with selected columns from a specific timeframe, you can perform the following SQL query:
40+
41+
```bash
42+
curl https://api.cloudflare.com/client/v4/account/{account_id}/logs/explorer/query/sql \
43+
--header "Authorization: Bearer <API_TOKEN>" \
44+
--url-query query="SELECT CreatedAt, AppDomain, AppUUID, Action, Allowed, Country, RayID, Email, IPAddress, UserUID FROM access_requests WHERE Date >= '2025-02-06' AND Date <= '2025-02-06' AND CreatedAt >= '2025-02-06T12:28:39Z' AND CreatedAt <= '2025-02-06T12:58:39Z'"
45+
```
46+
47+
Which returns the following request details:
48+
49+
```json
50+
{
51+
"result": [
52+
{
53+
"createdat": "2025-01-14T18:17:55Z",
54+
"appdomain": "example.com",
55+
"appuuid": "a66b4ab0-ccdf-4d60-a6d0-54a59a827d92",
56+
"action": "login",
57+
"allowed": true,
58+
"country": "us",
59+
"rayid": "90fbb07c0b316957",
60+
"email": "[email protected]",
61+
"ipaddress": "1.2.3.4",
62+
"useruid": "52859e81-711e-4de0-8b31-283336060e79"
63+
}
64+
],
65+
"success": true,
66+
"errors": [],
67+
"messages": []
68+
}
69+
```
70+
71+
## Authentication
72+
73+
Log Search is available to users with the following permissions:
74+
75+
- **Logs Edit**: users with Logs Edit permissions can enable datasets.
76+
- **Logs Read**: users with Logs Read permissions can run queries via the UI or API.
77+
78+
Note that these permissions exist at the account and zone level and you need the appropriate permission level for the datasets you wish to query.
79+
80+
Authentication with the API can be done via an authentication header or API token. Append your API call with either of the following additional parameters.
81+
82+
- **Authentication header**
83+
84+
- `X-Auth-Email` - the Cloudflare account email address associated with the domain
85+
- `X-Auth-Key` - the Cloudflare API key
86+
87+
- **API token**
88+
89+
- `Authorization: Bearer <API_TOKEN>` To create an appropriately scoped API token, refer to [Create API token](/fundamentals/api/get-started/create-token/) documentation. Copy and paste the token into the authorization parameter for your API call.

‎src/content/docs/log-explorer/custom-dashboards.mdx‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,14 +7,15 @@ sidebar:
77

88
Custom dashboards allow you to create tailored dashboards to monitor application security, performance, and usage. You can create monitors for ongoing monitoring of a previous incident, use them to identify indicators of suspicious activity, and access templates to help you get started.
99

10-
Dashboards provide a visual interface that displays key metrics and analytics, helping you monitor and analyze data efficiently. Different dashboards serve different purposes. For example, a security dashboard tracks attack attempts and threats, a performance dashboard monitors API latency and uptime, and a usage dashboard analyzes traffic patterns and user behavior.
11-
12-
Different metrics serve distinct roles in providing insights into your application's performance. For example, total HTTP requests offers an overview of traffic volume, while average response time helps assess application speed. Additionally, usage metrics such as traffic patterns and user behavior provide insight into how users interact with your application. These metrics together enable you to spot trends, identify problems, and make informed, data-driven decisions.
13-
1410
:::note
15-
Custom dashboards is currently available to customers participating in the Log Explorer beta. To begin using custom dashboards, you will first need to request access to [Log Explorer](/logs/log-explorer/).
11+
Enterprise customers can create up to 100 dashboards.
12+
Customers on Pro and Business plans can create up to 5 dashboards.
1613
:::
1714

15+
Dashboards provide a visual interface that displays key metrics and analytics, helping you monitor and analyze data efficiently. Different dashboards serve different purposes. For example, a security dashboard tracks attack attempts and threats, a performance dashboard monitors API latency and uptime, and a usage dashboard analyzes traffic patterns and user behavior.
16+
17+
Different metrics serve distinct roles in providing insights into your application's performance. For example, total HTTP requests offer an overview of traffic volume, while average response time helps assess application speed. Additionally, usage metrics such as traffic patterns and user behavior provide insight into how users interact with your application. These metrics together enable you to spot trends, identify problems, and make informed, data-driven decisions.
18+
1819
## Create a new dashboard
1920

2021
To create a new dashboard:

‎src/content/docs/log-explorer/index.mdx‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,17 +17,25 @@ With Log Explorer, you can monitor security and performance issues with custom d
1717

1818
## Features
1919

20-
<Feature header="Log Search">
20+
<Feature header="Log Search" href="/log-explorer/log-search/">
2121
Search logs enable you to store and explore your Cloudflare logs directly within the Cloudflare dashboard or API.
2222
</Feature>
2323

24-
<Feature header="Custom dashboards">
24+
<Feature header="Custom dashboards" href="/log-explorer/custom-dashboards/">
2525
Custom dashboards enable you to design customized views for tracking application security, performance, and usage metrics.
2626
</Feature>
2727

28+
<Feature header="Manage datasets" href="/log-explorer/manage-datasets/">
29+
Manage the data you want to store within Log Explorer.
30+
</Feature>
31+
32+
<Feature header="API" href= "/log-explorer/api/">
33+
Manage configuration and perform queries via the API.
34+
</Feature>
35+
2836
## Related products
2937

30-
<RelatedProduct header="Logs" href="/logs/" product="logs">
38+
<RelatedProduct header="Logpush" href="/logs/" product="logs">
3139
Detailed logs that contain metadata generated by Cloudflare products helpful for debugging, identifying configuration adjustments, and creating analytics.
3240
</RelatedProduct>
3341

‎src/content/docs/log-explorer/log-search.mdx‎

Lines changed: 52 additions & 157 deletions
Original file line numberDiff line numberDiff line change
@@ -11,101 +11,14 @@ Log Explorer enables you to store and explore your Cloudflare logs directly with
1111

1212
## Supported datasets
1313

14-
Log Explorer is available at the account and zone level. At the zone level, datasets currently available are:
14+
Log Explorer currently supports:
1515

1616
- [HTTP requests](/logs/reference/log-fields/zone/http_requests/) (`FROM http_requests`)
1717
- [Firewall events](/logs/reference/log-fields/zone/firewall_events/) (`FROM firewall_events`)
1818

19-
At the account level, the datasets available are:
20-
21-
<Render file="log-explorer-account-datasets" product="logs" />
22-
23-
## Authentication
24-
25-
Log Search is available to users with the following permissions:
26-
27-
- **Logs Edit**: users with Logs Edit permissions can enable datasets.
28-
- **Logs Read**: users with Logs Read permissions can run queries via the UI or API.
29-
30-
Note that these permissions exist at the account and zone level and you need the appropriate permission level for the datasets you wish to query.
31-
32-
Authentication with the API can be done via an authentication header or API token. Append your API call with either of the following additional parameters.
33-
34-
- **Authentication header**
35-
36-
- `X-Auth-Email` - the Cloudflare account email address associated with the domain
37-
- `X-Auth-Key` - the Cloudflare API key
38-
39-
- **API token**
40-
41-
- `Authorization: Bearer <API_TOKEN>` To create an appropriately scoped API token, refer to [Create API token](/fundamentals/api/get-started/create-token/) documentation. Copy and paste the token into the authorization parameter for your API call.
42-
43-
## Enable Log Explorer
44-
45-
In order for Log Explorer to begin storing logs, you need to enable the desired datasets. You can do this via the dashboard or the API.
46-
47-
<Tabs syncKey="dashPlusAPI"> <TabItem label="Dashboard">
48-
49-
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/login) and select your account or domain (also known).
50-
2. Go to **Log Explorer** > **Manage datasets**.
51-
3. Select **Add dataset** to select the datasets you want to query.
52-
4. Choose a dataset and then a zone. Then, select **Add**. You can always return to this page to enable more datasets or manage your existing ones.
53-
54-
:::note
55-
It may take a few minutes for the logs to become available for querying.
56-
:::
57-
58-
</TabItem> <TabItem label="API">
59-
60-
Use the Log Explorer API to enable Log Explorer for each dataset you wish to store. It may take a few minutes after a log stream is enabled before you can view the logs.
61-
62-
The following curl command is an example for enabling the zone-level dataset `http_requests`, as well as the expected response when the command succeeds.
63-
64-
```bash
65-
curl https://api.cloudflare.com/client/v4/zones/{zone_id}/logs/explorer/datasets \
66-
--header "Authorization: Bearer <API_TOKEN>" \
67-
--header "Content-Type: application/json" \
68-
--data '{
69-
"dataset": "http_requests"
70-
}'
71-
```
72-
73-
```json
74-
{
75-
"result": {
76-
"dataset": "http_requests",
77-
"object_type": "zone",
78-
"object_id": "<ZONE ID>",
79-
"created_at": "2025-06-03T14:33:16Z",
80-
"updated_at": "2025-06-03T14:33:16Z",
81-
"dataset_id": "01973635f7e273a1964a02f4d4502499",
82-
"enabled": true
83-
},
84-
"success": true,
85-
"errors": [],
86-
"messages": []
87-
}
88-
```
89-
90-
If you would like to enable an account-level dataset, replace `zones/{zone_id}` with `accounts/{account_id}` in the curl command. For example:
91-
92-
```bash
93-
curl https://api.cloudflare.com/client/v4/accounts/{account_id}/logs/explorer/datasets \
94-
--header "Authorization: Bearer <API_TOKEN>" \
95-
--header "Content-Type: application/json" \
96-
--data '{
97-
"dataset": "access_requests"
98-
}'
99-
```
100-
101-
</TabItem> </Tabs>
102-
10319
## Use Log Explorer
10420

105-
Filtering and viewing your logs is available via the Cloudflare Dashboard or via query API.
106-
107-
<Tabs syncKey="dashPlusAPI"> <TabItem label="Dashboard">
108-
21+
You can filter and view your logs via the Cloudflare dashboard or the API.
10922

11023
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/login) and select your account.
11124
2. Go to **Log Explorer** > **Log Search**.
@@ -117,88 +30,70 @@ Filtering and viewing your logs is available via the Cloudflare Dashboard or via
11730
8. Select **Run query** when you are done. The results are displayed below within the **Query results** section.
11831

11932
:::note
120-
12133
You can also access the Log Explorer dashboard directly from the [Security Analytics dashboard](/waf/analytics/security-analytics/#logs). When doing so, the filters you applied in Security Analytics will automatically carry over to your query in Log Explorer.
122-
12334
:::
12435

125-
</TabItem> <TabItem label="API">
126-
127-
Log Explorer exposes a query endpoint that uses a familiar SQL syntax for querying your logs generated with Cloudflare's network.
128-
129-
For example, to find an HTTP request with a specific [Ray ID](/fundamentals/reference/cloudflare-ray-id/), you can perform the following SQL query:
130-
131-
```bash
132-
curl https://api.cloudflare.com/client/v4/zones/{zone_id}/logs/explorer/query/sql \
133-
--header "Authorization: Bearer <API_TOKEN>" \
134-
--url-query query="SELECT clientRequestScheme, clientRequestHost, clientRequestMethod, edgeResponseStatus, clientRequestUserAgent FROM http_requests WHERE RayID = '806c30a3cec56817' LIMIT 1"
135-
```
136-
137-
Which returns the following HTTP request details:
138-
139-
```json
140-
{
141-
"result": [
142-
{
143-
"clientrequestscheme": "https",
144-
"clientrequesthost": "example.com",
145-
"clientrequestmethod": "GET",
146-
"clientrequestuseragent": "curl/7.88.1",
147-
"edgeresponsestatus": 200
148-
}
149-
],
150-
"success": true,
151-
"errors": [],
152-
"messages": []
153-
}
36+
For example, to find an HTTP request with a specific [Ray ID](/fundamentals/reference/cloudflare-ray-id/), go to **Custom SQL**, and enter the following SQL query:
37+
38+
```sql
39+
SELECT
40+
clientRequestScheme,
41+
clientRequestHost,
42+
clientRequestMethod,
43+
edgeResponseStatus,
44+
clientRequestUserAgent
45+
FROM http_requests
46+
WHERE RayID = '806c30a3cec56817'
47+
LIMIT 1
15448
```
15549

156-
For another example using an account-level dataset, to find Cloudflare Access requests with selected columns from a specific timeframe, you can perform the following SQL query:
157-
158-
```bash
159-
curl https://api.cloudflare.com/client/v4/account/{account_id}/logs/explorer/query/sql \
160-
--header "Authorization: Bearer <API_TOKEN>" \
161-
--url-query query="SELECT CreatedAt, AppDomain, AppUUID, Action, Allowed, Country, RayID, Email, IPAddress, UserUID FROM access_requests WHERE Date >= '2025-02-06' AND Date <= '2025-02-06' AND CreatedAt >= '2025-02-06T12:28:39Z' AND CreatedAt <= '2025-02-06T12:58:39Z'"
162-
```
163-
164-
Which returns the following request details:
165-
166-
```json
167-
{
168-
"result": [
169-
{
170-
"createdat": "2025-01-14T18:17:55Z",
171-
"appdomain": "example.com",
172-
"appuuid": "a66b4ab0-ccdf-4d60-a6d0-54a59a827d92",
173-
"action": "login",
174-
"allowed": true,
175-
"country": "us",
176-
"rayid": "90fbb07c0b316957",
177-
"email": "[email protected]",
178-
"ipaddress": "1.2.3.4",
179-
"useruid": "52859e81-711e-4de0-8b31-283336060e79"
180-
}
181-
],
182-
"success": true,
183-
"errors": [],
184-
"messages": []
185-
}
50+
Another example to find Cloudflare Access requests with selected columns from a specific timeframe, you can perform the following SQL query:
51+
52+
```sql
53+
SELECT
54+
CreatedAt,
55+
AppDomain,
56+
AppUUID,
57+
Action,
58+
Allowed,
59+
Country,
60+
RayID,
61+
Email,
62+
IPAddress,
63+
UserUID
64+
FROM access_requests
65+
WHERE Date >= '2025-02-06' AND Date <= '2025-02-06' AND CreatedAt >= '2025-02-06T12:28:39Z' AND CreatedAt <= '2025-02-06T12:58:39Z'
18666
```
18767

188-
</TabItem> </Tabs>
189-
19068
### Save queries
19169

19270
After selecting all the fields for your query, you can save it by selecting **Save query**. Provide a name and description to help identify it later. To view your saved and recent queries, select **Queries** — they will appear in a side panel where you can insert a new query, or delete any query.
19371

19472
## Optimize your queries
19573

196-
All the tables supported by Log Explorer contain a special column called `date`, which helps to narrow down the amount of data that is scanned to respond to your query, resulting in faster query response times. The value of `date` must be in the form of `YYYY-MM-DD`. For example, to query logs that occurred on October 12, 2023, add the following to your `WHERE` clause: `date = '2023-10-12'`. The column supports the standard operators of `<`, `>`, and `=`:
74+
All the tables supported by Log Explorer contain a special column called `date`, which helps to narrow down the amount of data that is scanned to respond to your query, resulting in faster query response times. The value of `date` must be in the form of `YYYY-MM-DD`. For example, to query logs that occurred on October 12, 2023, add the following to your `WHERE` clause: `date = '2023-10-12'`. The column supports the standard operators of `<`, `>`, and `=`.
19775

198-
```bash
199-
curl https://api.cloudflare.com/client/v4/zones/{zone_id}/logs/explorer/query/sql \
200-
--header "Authorization: Bearer <API_TOKEN>" \
201-
--url-query query="SELECT clientRequestMethod, clientRequestPath, clientRequestProtocol FROM http_requests WHERE date = '2023-10-12' LIMIT 500"
76+
1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/login) and select your account.
77+
2. Go to **Log Explorer** > **Log Search** > **Custom SQL**.
78+
3. Enter the following SQL query:
79+
80+
```sql
81+
SELECT
82+
clientip,
83+
clientrequesthost,
84+
clientrequestmethod,
85+
clientrequesturi,
86+
edgeendtimestamp,
87+
edgeresponsestatus,
88+
originresponsestatus,
89+
edgestarttimestamp,
90+
rayid,
91+
clientcountry,
92+
clientrequestpath, date
93+
FROM
94+
http_requests
95+
WHERE
96+
date = '2023-10-12' LIMIT 500
20297
```
20398

20499
### Additional query optimization tips

0 commit comments

Comments
 (0)