Skip to content

Commit ae93709

Browse files
BohuTANGBohuTANG
authored
aduit trail (#2486)
1 parent dc3006f commit ae93709

File tree

8 files changed

+235
-33
lines changed

8 files changed

+235
-33
lines changed

‎docs/en/guides/56-security/audit-trail.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
---
2+
title: Audit Trail
3+
---
4+
5+
import EEFeature from '@site/src/components/EEFeature';
6+
7+
<EEFeature featureName='AUDIT TRAIL'/>
8+
9+
Databend system history tables automatically capture detailed records of database activities, providing a complete audit trail for compliance and security monitoring.
10+
11+
Allows the auditing of the user:
12+
- **Query execution** - Complete SQL execution audit trail (`query_history`)
13+
- **Data access** - Database object access and modifications (`access_history`)
14+
- **Authentication** - Login attempts and session tracking (`login_history`)
15+
16+
## Available Audit Tables
17+
18+
Databend provides five system history tables that capture different aspects of database activity:
19+
20+
| Table | Purpose | Key Use Cases |
21+
|-------|---------|---------------|
22+
| [query_history](/sql/sql-reference/system-history-tables/query-history) | Complete SQL execution audit trail | Performance monitoring, security auditing, compliance reporting |
23+
| [access_history](/sql/sql-reference/system-history-tables/access-history) | Database object access and modifications | Data lineage tracking, compliance auditing, change management |
24+
| [login_history](/sql/sql-reference/system-history-tables/login-history) | Authentication attempts and sessions | Security monitoring, failed login detection, access pattern analysis |
25+
26+
## Audit Use Cases & Examples
27+
28+
### Security Monitoring
29+
30+
**Monitor Failed Login Attempts**
31+
32+
Track authentication failures to identify potential security threats and unauthorized access attempts.
33+
34+
```sql
35+
-- Check for failed login attempts (security audit)
36+
SELECT event_time, user_name, client_ip, error_message
37+
FROM system_history.login_history
38+
WHERE event_type = 'LoginFailed'
39+
ORDER BY event_time DESC;
40+
```
41+
42+
Example output:
43+
```
44+
event_time: 2025-06-03 06:07:32.512021
45+
user_name: root1
46+
client_ip: 127.0.0.1:62050
47+
error_message: UnknownUser. Code: 2201, Text = User 'root1'@'%' does not exist.
48+
```
49+
50+
### Compliance Reporting
51+
52+
**Track Database Schema Changes**
53+
54+
Monitor DDL operations for compliance and change management requirements.
55+
56+
```sql
57+
-- Audit DDL operations (compliance tracking)
58+
SELECT query_id, query_start, user_name, object_modified_by_ddl
59+
FROM system_history.access_history
60+
WHERE object_modified_by_ddl != '[]'
61+
ORDER BY query_start DESC;
62+
```
63+
64+
Example for `CREATE TABLE` operation:
65+
```
66+
query_id: c2c1c7be-cee4-4868-a28e-8862b122c365
67+
query_start: 2025-06-12 03:31:19.042128
68+
user_name: root
69+
object_modified_by_ddl: [{"object_domain":"Table","object_name":"default.default.t","operation_type":"Create"}]
70+
```
71+
72+
**Audit Data Access Patterns**
73+
74+
Track who accessed what data and when for compliance and data governance.
75+
76+
```sql
77+
-- Track data access for compliance
78+
SELECT query_id, query_start, user_name, base_objects_accessed
79+
FROM system_history.access_history
80+
WHERE base_objects_accessed != '[]'
81+
ORDER BY query_start DESC;
82+
```
83+
84+
### Operational Monitoring
85+
86+
**Complete Query Execution Audit**
87+
88+
Maintain comprehensive records of all SQL operations with user and timing information.
89+
90+
```sql
91+
-- Complete query audit with user and timing information
92+
SELECT query_id, sql_user, query_text, query_start_time, query_duration_ms, client_address
93+
FROM system_history.query_history
94+
WHERE event_date >= TODAY() - INTERVAL 7 DAY
95+
ORDER BY query_start_time DESC;
96+
```
97+
98+
Example output:
99+
```
100+
query_id: 4e1f50a9-bce2-45cc-86e4-c7a36b9b8d43
101+
sql_user: root
102+
query_text: SELECT * FROM t
103+
query_start_time: 2025-06-12 03:31:35.041725
104+
query_duration_ms: 94
105+
client_address: 127.0.0.1
106+
```
107+
108+
For detailed information about each audit table and their specific fields, see the [System History Tables](/sql/sql-reference/system-history-tables/) reference documentation.

‎docs/en/guides/56-security/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ Databend offers **enterprise-grade security and reliability features** that safe
77
| Security Feature | Purpose | When to Use |
88
|-----------------|---------|------------|
99
| [**Access Control**](/guides/security/access-control) | Manage user permissions | When you need to control data access with role-based security and object ownership |
10+
| [**Audit Trail**](audit-trail.md) | Track database activities | When you need comprehensive audit trails for security monitoring, compliance, and performance analysis |
1011
| [**Network Policy**](/guides/security/network-policy) | Restrict network access | When you want to limit connections to specific IP ranges even with valid credentials |
1112
| [**Password Policy**](/guides/security/password-policy) | Set password requirements | When you need to enforce password complexity, rotation, and account lockout rules |
1213
| [**Masking Policy**](/guides/security/masking-policy) | Hide sensitive data | When you need to protect confidential data while still allowing authorized access |

‎docs/en/sql-reference/00-sql-reference/32-system-history-tables/access-history.md‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,20 @@
22
title: system_history.access_history
33
---
44

5-
This table provides detailed logging of objects accessed and modified by each query, including tables, columns, and stages, as part of the query metadata. It provides structured information about DDL and DML operations to enhance auditing.
5+
import FunctionDescription from '@site/src/components/FunctionDescription';
6+
7+
<FunctionDescription description="Introduced or updated: v1.2.764"/>
8+
9+
import EEFeature from '@site/src/components/EEFeature';
10+
11+
<EEFeature featureName='ACCESS HISTORY'/>
12+
13+
**Data lineage and access control audit** - Tracks all database objects (tables, columns, stages) accessed or modified by queries. Essential for:
14+
15+
- **Data Lineage**: Understand data flow and dependencies across your database
16+
- **Compliance Reporting**: Track who accessed sensitive data and when
17+
- **Change Management**: Monitor DDL operations and schema modifications
18+
- **Security Analysis**: Identify unusual access patterns or unauthorized data access
619

720

821
## Fields

‎docs/en/sql-reference/00-sql-reference/32-system-history-tables/index.md‎

Lines changed: 54 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -2,57 +2,84 @@
22
title: System History Tables
33
---
44

5-
import FunctionDescription from '@site/src/components/FunctionDescription';
5+
import EEFeature from '@site/src/components/EEFeature';
66

7-
<FunctionDescription description="Introduced or updated: v1.2.752"/>
7+
<EEFeature featureName='SYSTEM HISTORY'/>
88

99
# System History Tables
1010

11-
System history tables store persistent data in the `system_history` schema for auditing, troubleshooting, and compliance purposes. They track query execution, user logins, and system logs that can be queried using standard SQL.
11+
Databend's system history tables provide **Data Governance** capabilities by automatically tracking database activities for compliance, security monitoring, and performance analysis.
1212

13-
## Available System History Tables
13+
## Available Tables
1414

15-
| Table | Description |
16-
|-----------------------------------------------------|-----------------------------------------------------------------|
17-
| [system_history.log_history](log-history.md) | Stores raw log entries from various system components. |
18-
| [system_history.query_history](query-history.md) | Stores structured details of query execution. |
19-
| [system_history.profile_history](profile-history.md)| Stores detailed query execution profiles and statistics. |
20-
| [system_history.login_history](login-history.md) | Records information about user login events. |
21-
| [system_history.access_history](access-history.md) | Stores information about query access events. |
15+
| Table | Purpose | Key Use Cases |
16+
|-------|---------|---------------|
17+
| [query_history](query-history.md) | Complete SQL execution audit trail | Performance analysis, compliance tracking, usage monitoring |
18+
| [access_history](access-history.md) | Data access and modification logs | Data lineage, compliance reporting, change management |
19+
| [login_history](login-history.md) | User authentication tracking | Security auditing, failed login monitoring, access pattern analysis |
20+
| [profile_history](profile-history.md) | Detailed query execution profiles | Performance optimization, resource planning, bottleneck identification |
21+
| [log_history](log-history.md) | Raw system logs and events | System troubleshooting, error analysis, operational monitoring |
2222

23-
## Enabling System History Tables
23+
## Configuration
2424

25-
> **Note:** In **Databend Cloud**, system history tables are automatically enabled and ready to use without any configuration needed. The following section applies only to **self-hosted Databend**.
25+
### Databend Cloud
26+
✅ **Automatically enabled** - All system history tables are ready to use without any configuration.
2627

27-
In self-hosted Databend, system history tables are disabled by default. To enable them, configure the `[log.history]` section in your `databend-query.toml` file.
28+
### Self-Hosted Databend
2829

29-
Configuration Example:
30+
<details>
31+
<summary>📝 **Manual configuration required** - Click to expand configuration details</summary>
32+
33+
#### Minimal Configuration
34+
To enable system history tables, you must configure all 5 tables in your `databend-query.toml`:
3035

3136
```toml
3237
[log.history]
33-
# Enable history tables
3438
on = true
35-
level = "INFO"
3639

37-
# Configure retention policies for each table
40+
# All 5 tables must be configured to enable history logging
41+
# retention is optional (default: 168 hours = 7 days)
3842
[[log.history.tables]]
39-
table_name = "log_history"
40-
retention = 168 # 7 days (in hours)
43+
table_name = "query_history"
44+
retention = 168 # Optional: 7 days (default)
4145

4246
[[log.history.tables]]
43-
table_name = "query_history"
44-
retention = 168
47+
table_name = "login_history"
48+
retention = 168 # Optional: 7 days (default)
49+
50+
[[log.history.tables]]
51+
table_name = "access_history"
52+
retention = 168 # Optional: 7 days (default)
4553

4654
[[log.history.tables]]
4755
table_name = "profile_history"
48-
retention = 168
56+
retention = 168 # Optional: 7 days (default)
4957

5058
[[log.history.tables]]
51-
table_name = "login_history"
52-
retention = 168
59+
table_name = "log_history"
60+
retention = 168 # Optional: 7 days (default)
61+
```
62+
63+
#### Custom Storage (Optional)
64+
By default, history tables use your main database storage. To use separate S3 storage:
65+
66+
```toml
67+
[log.history]
68+
on = true
69+
70+
[log.history.storage]
71+
type = "s3"
72+
73+
[log.history.storage.s3]
74+
bucket = "your-history-bucket"
75+
root = "history_tables"
76+
endpoint_url = "https://s3.amazonaws.com"
77+
access_key_id = "your-access-key"
78+
secret_access_key = "your-secret-key"
5379
```
5480

55-
> **Note:** The `log_history` table is enabled by default when history logging is turned on. The `level` configuration determines the number of log entries stored in the log_history table. A more detailed level will result in more entries.
81+
> ⚠️ **Note:** When changing storage configuration, existing history tables will be dropped and recreated.
5682
83+
</details>
5784

58-
For more details about configuration options, see [Query Configuration: [log.history] Section](/guides/deploy/references/node-config/query-config#loghistory-section).
85+
For complete configuration options, see [Query Configuration: [log.history] Section](/guides/deploy/references/node-config/query-config#loghistory-section).

‎docs/en/sql-reference/00-sql-reference/32-system-history-tables/log-history.md‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,22 @@
22
title: system_history.log_history
33
---
44

5-
Stores raw log entries ingested from various nodes. This table acts as the primary source for subsequent log transformations.
5+
import FunctionDescription from '@site/src/components/FunctionDescription';
66

7-
All the other log tables are derived from this table, the difference is that other log tables will do some transformations to make the data more structured.
7+
<FunctionDescription description="Introduced or updated: v1.2.764"/>
8+
9+
import EEFeature from '@site/src/components/EEFeature';
10+
11+
<EEFeature featureName='LOG HISTORY'/>
12+
13+
**System operations audit trail** - Raw log repository from all Databend nodes and components. Foundation for operational intelligence:
14+
15+
- **System Monitoring**: Track system health, performance, and resource usage
16+
- **Troubleshooting**: Debug issues with detailed error logs and system events
17+
- **Operational Analytics**: Analyze system behavior patterns and trends
18+
- **Root Cause Analysis**: Investigate system failures and performance bottlenecks
19+
20+
> **Note:** This table contains raw log data that feeds into other specialized history tables. Other tables provide structured, query-specific views of this data.
821
922
## Fields
1023

‎docs/en/sql-reference/00-sql-reference/32-system-history-tables/login-history.md‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,20 @@
22
title: system_history.login_history
33
---
44

5-
Records all login attempts in the system, including successful and failed login attempts. This table is useful for auditing user access and troubleshooting authentication issues.
5+
import FunctionDescription from '@site/src/components/FunctionDescription';
6+
7+
<FunctionDescription description="Introduced or updated: v1.2.764"/>
8+
9+
import EEFeature from '@site/src/components/EEFeature';
10+
11+
<EEFeature featureName='LOGIN HISTORY'/>
12+
13+
**Authentication security audit** - Comprehensive logging of all user login attempts (successful and failed). Critical for:
14+
15+
- **Security Monitoring**: Detect brute force attacks and unauthorized access attempts
16+
- **Compliance Auditing**: Track user authentication for regulatory requirements
17+
- **Access Pattern Analysis**: Monitor when and how users access the system
18+
- **Incident Investigation**: Investigate security incidents and authentication issues
619

720

821
## Fields

‎docs/en/sql-reference/00-sql-reference/32-system-history-tables/profile-history.md‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,21 @@
11
---
22
title: system_history.profile_history
33
---
4-
Stores detailed execution profiles for SQL queries in Databend. Each entry provides performance metrics and execution statistics, allowing users to analyze and optimize query performance.
4+
5+
import FunctionDescription from '@site/src/components/FunctionDescription';
6+
7+
<FunctionDescription description="Introduced or updated: v1.2.764"/>
8+
9+
import EEFeature from '@site/src/components/EEFeature';
10+
11+
<EEFeature featureName='PROFILE HISTORY'/>
12+
13+
**Query performance deep-dive analytics** - Detailed execution profiles and statistics for every SQL query. Essential for:
14+
15+
- **Performance Optimization**: Identify bottlenecks and optimize slow queries
16+
- **Resource Planning**: Understand memory, CPU, and I/O usage patterns
17+
- **Execution Analysis**: Analyze query plans and execution statistics
18+
- **Capacity Management**: Monitor resource consumption trends over time
519

620
## Fields
721

‎docs/en/sql-reference/00-sql-reference/32-system-history-tables/query-history.md‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,20 @@
22
title: system_history.query_history
33
---
44

5-
Records detailed logs of all SQL query executions in Databend. For each query, two entries are generated: one when the query starts and another when it finishes. This table is valuable for monitoring query activity, auditing user actions, and analyzing performance metrics.
5+
import FunctionDescription from '@site/src/components/FunctionDescription';
6+
7+
<FunctionDescription description="Introduced or updated: v1.2.764"/>
8+
9+
import EEFeature from '@site/src/components/EEFeature';
10+
11+
<EEFeature featureName='QUERY HISTORY'/>
12+
13+
**Complete SQL execution audit trail** - Records comprehensive details of all SQL queries executed in Databend. Each query generates two entries (start and finish), providing complete visibility into:
14+
15+
- **Performance Analysis**: Query duration, resource usage, and optimization opportunities
16+
- **Security Auditing**: Who executed what queries, when, and from where
17+
- **Compliance Tracking**: Complete audit trail for regulatory requirements
18+
- **Usage Monitoring**: Database activity patterns and user behavior analysis
619

720
## Fields
821

0 commit comments

Comments
 (0)