You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/content/docs/log-explorer/api.mdx
+68-31Lines changed: 68 additions & 31 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,15 +1,72 @@
1
-
---
2
-
pcx_content_type: reference
3
-
title: Log Explorer API
4
-
sidebar:
5
-
order: 5
6
-
---
1
+
Configuration and Log searches are also available via a public API.
2
+
## Authentication
3
+
4
+
Log Explorer is available to users with the following permissions:
5
+
6
+
-**Logs Edit**: users with Logs Edit permissions can enable datasets.
7
+
-**Logs Read**: users with Logs Read permissions can run queries via the UI or API.
8
+
9
+
Note that these permissions exist at the account and zone level and you need the appropriate permission level for the datasets you wish to query.
7
10
8
-
Log searches are available via the [API](https://developers.cloudflare.com/log-explorer/).
11
+
Authentication with the API can be done via an authentication header or API token. Append your API call with either of the following additional parameters.
12
+
13
+
-**Authentication header**
9
14
10
-
Log Explorer exposes a query endpoint that uses familiar SQL syntax for querying your logs generated with Cloudflare's network.
15
+
-`X-Auth-Email` - the Cloudflare account email address associated with the domain
16
+
-`X-Auth-Key` - the Cloudflare API key
17
+
18
+
-**API token**
19
+
20
+
-`Authorization: Bearer <API_TOKEN>` To create an appropriately scoped API token, refer to [Create API token](/fundamentals/api/get-started/create-token/) documentation. Copy and paste the token into the authorization parameter for your API call.
21
+
22
+
## Manage Datasets
23
+
24
+
Use the Log Explorer API to enable Log Explorer for each dataset you wish to store. It may take a few minutes after a log stream is enabled before you can view the logs.
25
+
26
+
The following curl command is an example for enabling the zone-level dataset `http_requests`, as well as the expected response when the command succeeds.
@@ -66,24 +123,4 @@ Which returns the following request details:
66
123
"errors": [],
67
124
"messages": []
68
125
}
69
-
```
70
-
71
-
## Authentication
72
-
73
-
Log Search is available to users with the following permissions:
74
-
75
-
-**Logs Edit**: users with Logs Edit permissions can enable datasets.
76
-
-**Logs Read**: users with Logs Read permissions can run queries via the UI or API.
77
-
78
-
Note that these permissions exist at the account and zone level and you need the appropriate permission level for the datasets you wish to query.
79
-
80
-
Authentication with the API can be done via an authentication header or API token. Append your API call with either of the following additional parameters.
81
-
82
-
-**Authentication header**
83
-
84
-
-`X-Auth-Email` - the Cloudflare account email address associated with the domain
85
-
-`X-Auth-Key` - the Cloudflare API key
86
-
87
-
-**API token**
88
-
89
-
-`Authorization: Bearer <API_TOKEN>` To create an appropriately scoped API token, refer to [Create API token](/fundamentals/api/get-started/create-token/) documentation. Copy and paste the token into the authorization parameter for your API call.
Store and explore your Cloudflare logs directly within the Cloudflare dashboard or API.
12
12
</Description>
13
13
14
-
Log Explorer is a log storage platform that allows security teams and developers to analyze, investigate, and monitor attacks natively within the Cloudflare dashboard. Log Explorer is built on [Cloudflare R2](/r2/) which allows you to search your logs using SQL queries.
15
14
16
-
With Log Explorer, you can monitor security and performance issues with custom dashboards, investigate and troubleshoot issues with Log Search, save time and collaborate with saved queries.
15
+
Log Explorer is Cloudflare's native observability and forensics product that enables security teams and developers to analyze, investigate, and monitor issues directly from the Cloudflare dashboard, without the expense and complexity of forwarding logs to third party tools.
16
+
17
+
Log Explorer provides access to Cloudflare logs with all the context available within the Cloudflare platform. You can monitor security and performance issues with custom dashboards or investigate and troubleshoot issues with log search. Benefits include:
18
+
19
+
-**Reduced cost and complexity**: Drastically reduce the expense and operational overhead associated with forwarding, storing, and analyzing terabytes of log data in external tools.
20
+
-**Faster detection and triage**: Access Cloudflare-native logs directly, eliminating cumbersome data pipelines and the ingest lags that delay critical security insights.
21
+
-**Accelerated investigations with full context**: Investigate incidents with Cloudflare's unparalleled contextual data, accelerating your analysis and understanding of "What exactly happened?" and "How did it happen?"
22
+
-**Minimal recovery time**: Seamlessly transition from investigation to action with direct mitigation capabilities via the Cloudflare platform.
Detailed logs that contain metadata generated by Cloudflare products helpful for debugging, identifying configuration adjustments, and creating analytics.
45
+
Forward Cloudflare logs to third party tools for debugging, identifying configuration adjustments, and creating analytics.
Log Explorer enables you to store and explore your Cloudflare logs directly within the Cloudflare dashboard or API. Giving you visibility into your logs without the need to forward them to third parties. Logs are stored on Cloudflare's global network using the R2 object storage platform and can be queried via the Dashboard or SQL API.
11
11
12
-
## Supported datasets
12
+
## SQL queries supported
13
+
14
+
The diagram below displays the example sql grammar for SELECT statements as a railroad syntax diagram:
Any path from left to right forms a valid query. There is a limit of 25 predicates in the `WHERE` clause. Predicates can be grouped using parenthesis. If the `LIMIT` clause is not specified, then the default limit of 10000 is applied. The maximum number for the `LIMIT` clause is 10000. Results are returned in descending order by time.
19
+
20
+
Examples of queries include:
21
+
22
+
-`SELECT * FROM table WHERE (a = 1 OR b = "hello") AND c < 25.89`
23
+
-`SELECT a, b, c FROM table WHERE d >= "GB" LIMIT 10`
24
+
25
+
### SELECT
26
+
27
+
The `SELECT` clause specifies the columns that you want to retrieve from the database tables. It can include individual column names, expressions, or even wildcard characters to select all columns.
28
+
29
+
### FROM
30
+
31
+
The `FROM` clause specifies the tables from which to retrieve data. It indicates the source of the data for the `SELECT` statement.
32
+
33
+
### WHERE
34
+
35
+
The `WHERE` clause filters the rows returned by a query based on specified conditions. It allows you to specify conditions that must be met for a row to be included in the result set.
36
+
37
+
### GROUP BY
38
+
39
+
The `GROUP BY` clause is used to group rows that have the same values into summary rows.
40
+
41
+
### ORDER BY
13
42
14
-
Log Explorer currently supports:
43
+
The `ORDER BY` clause is used to sort the result set by one or more columns in ascending or descending order.
The `LIMIT` clause is used to constrain the number of rows returned by a query. It is often used in conjunction with the `ORDER BY` clause to retrieve the top N rows or to implement pagination.
48
+
49
+
:::note
50
+
51
+
Log Explorer does not support `JOINs`, `DDL`, `DML`, or `EXPLAIN` queries.
52
+
53
+
:::
18
54
19
55
## Use Log Explorer
20
56
@@ -30,8 +66,9 @@ You can filter and view your logs via the Cloudflare dashboard or the API.
30
66
8. Select **Run query** when you are done. The results are displayed below within the **Query results** section.
31
67
32
68
:::note
33
-
You can also access the Log Explorer dashboard directly from the [Security Analytics dashboard](/waf/analytics/security-analytics/#logs). When doing so, the filters you applied in Security Analytics will automatically carry over to your query in Log Explorer.
34
-
:::
69
+
70
+
71
+
35
72
36
73
For example, to find an HTTP request with a specific [Ray ID](/fundamentals/reference/cloudflare-ray-id/), go to **Custom SQL**, and enter the following SQL query:
37
74
@@ -47,6 +84,7 @@ For example, to find an HTTP request with a specific [Ray ID](/fundamentals/refe
47
84
LIMIT1
48
85
```
49
86
87
+
50
88
Another example to find Cloudflare Access requests with selected columns from a specific timeframe, you can perform the following SQL query:
51
89
52
90
```sql
@@ -65,10 +103,22 @@ Another example to find Cloudflare Access requests with selected columns from a
After selecting all the fields for your query, you can save it by selecting **Save query**. Provide a name and description to help identify it later. To view your saved and recent queries, select **Queries** — they will appear in a side panel where you can insert a new query, or delete any query.
71
115
116
+
## Integrated with Security Analytics
117
+
118
+
You can also access the Log Explorer dashboard directly from the [Security Analytics dashboard](/waf/analytics/security-analytics/#logs). When doing so, the filters you applied in Security Analytics will automatically carry over to your query in Log Explorer.
119
+
120
+
:::
121
+
72
122
## Optimize your queries
73
123
74
124
All the tables supported by Log Explorer contain a special column called `date`, which helps to narrow down the amount of data that is scanned to respond to your query, resulting in faster query response times. The value of `date` must be in the form of `YYYY-MM-DD`. For example, to query logs that occurred on October 12, 2023, add the following to your `WHERE` clause: `date = '2023-10-12'`. The column supports the standard operators of `<`, `>`, and `=`.
@@ -102,40 +152,6 @@ All the tables supported by Log Explorer contain a special column called `date`,
102
152
- Omit `ORDER BY` and `LIMIT` clauses. These clauses can slow down queries, especially when dealing with large datasets. For queries that return a large number of records, reduce the time frame instead of limiting to the newest `N` records from a broader time frame.
103
153
- Select only necessary columns. For example, replace `SELECT *` with the list of specific columns you need. You can also use `SELECT RayId` as a first iteration and follow up with a query that filters by the Ray IDs to retrieve additional columns. Additionally, you can use `SELECT COUNT(*)` to probe for time frames with matching records without retrieving the full dataset.
104
154
105
-
## SQL queries supported
106
-
107
-
These are the SQL query clauses supported by Log Explorer.
108
-
109
-
### SELECT
110
-
111
-
The `SELECT` clause specifies the columns that you want to retrieve from the database tables. It can include individual column names, expressions, or even wildcard characters to select all columns.
112
-
113
-
### FROM
114
-
115
-
The `FROM` clause specifies the tables from which to retrieve data. It indicates the source of the data for the `SELECT` statement.
116
-
117
-
### WHERE
118
-
119
-
The `WHERE` clause filters the rows returned by a query based on specified conditions. It allows you to specify conditions that must be met for a row to be included in the result set.
120
-
121
-
### GROUP BY
122
-
123
-
The `GROUP BY` clause is used to group rows that have the same values into summary rows.
124
-
125
-
### ORDER BY
126
-
127
-
The `ORDER BY` clause is used to sort the result set by one or more columns in ascending or descending order.
128
-
129
-
### LIMIT
130
-
131
-
The `LIMIT` clause is used to constrain the number of rows returned by a query. It is often used in conjunction with the `ORDER BY` clause to retrieve the top N rows or to implement pagination.
132
-
133
-
:::note
134
-
135
-
Log Explorer does not support `JOINs`, `DDL`, `DML`, or `EXPLAIN` queries.
136
-
137
-
:::
138
-
139
155
## FAQs
140
156
141
157
### Which fields (or columns) are available for querying?
Copy file name to clipboardExpand all lines: src/content/docs/logs/log-explorer.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -207,7 +207,7 @@ You can choose the output format with an HTTP `Accept` header, as shown in the t
207
207
| CSV |`text/csv`| Yes |
208
208
| Plain text |`text/plain`| Yes |
209
209
210
-
## Optimizing your queries
210
+
## Optimize your queries
211
211
212
212
All the tables supported by Log Explorer contain a special column called `date`, which helps to narrow down the amount of data that is scanned to respond to your query, resulting in faster query response times. The value of `date` must be in the form of `YYYY-MM-DD`. For example, to query logs that occurred on October 12, 2023, add the following to your `WHERE` clause: `date = '2023-10-12'`. The column supports the standard operators of `<`, `>`, and `=`.
0 commit comments