Skip to content

Commit 43616f0

Browse files
huskar-thuskar-t
authored
docs: add support for recording STMT to CSV files (#34276) (#34536)
resolve: https://project.feishu.cn/taosdata_td/feature/detail/6622887611 * docs: add support for recording STMT to CSV files * docs: update version for STMT recording feature in CSV files
1 parent 7adf54e commit 43616f0

File tree

2 files changed

+625
-5
lines changed

2 files changed

+625
-5
lines changed

‎docs/en/14-reference/01-components/03-taosadapter.md‎

Lines changed: 319 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -734,24 +734,32 @@ The log can be configured with the following parameters:
734734
735735
Whether to record SQL to CSV files(Default: `false`).For details, see [Recording SQL to CSV Files](#recording-sql-to-csv-files).
736736
737+
- **`log.enableStmtToCsvLogging`**
738+
739+
Whether to record STMT to CSV files(Default: `false`).For details, see [Recording STMT to CSV Files](#recording-stmt-to-csv-files).
740+
737741
- **`log.enableRecordHttpSql`**
738742
739-
**It is not recommended to continue using this parameter. We suggest using [Recording SQL to CSV Files](#recording-sql-to-csv-files) as the alternative solution.**
743+
**This parameter is deprecated; use [Recording SQL to CSV Files](#recording-sql-to-csv-files) instead.**
744+
740745
Whether to record HTTP SQL requests (Default: `false`).
741746
742747
- **`log.sqlRotationCount`**
743748
744-
**It is not recommended to continue using this parameter. We suggest using [Recording SQL to CSV Files](#recording-sql-to-csv-files) as the alternative solution.**
749+
**This parameter is deprecated; use [Recording SQL to CSV Files](#recording-sql-to-csv-files) instead.**
750+
745751
Number of SQL log files to rotate (Default: `2`).
746752
747753
- **`log.sqlRotationSize`**
748754
749-
**It is not recommended to continue using this parameter. We suggest using [Recording SQL to CSV Files](#recording-sql-to-csv-files) as the alternative solution.**
755+
**This parameter is deprecated; use [Recording SQL to CSV Files](#recording-sql-to-csv-files) instead.**
756+
750757
Maximum size of a single SQL log file (Supports KB/MB/GB units, Default: `"1GB"`).
751758
752759
- **`log.sqlRotationTime`**
753760
754-
**It is not recommended to continue using this parameter. We suggest using [Recording SQL to CSV Files](#recording-sql-to-csv-files) as the alternative solution.**
761+
**This parameter is deprecated; use [Recording SQL to CSV Files](#recording-sql-to-csv-files) instead.**
762+
755763
SQL log rotation interval (Default: `24h`).
756764
757765
You can set the taosAdapter log output detail level by setting the --log.level parameter or the environment variable TAOS_ADAPTER_LOG_LEVEL. Valid values include: panic, fatal, error, warn, warning, info, debug, and trace.
@@ -1222,6 +1230,7 @@ Configuration Parameters and their corresponding environment variables:
12221230
| `log.compress` | `TAOS_ADAPTER_LOG_COMPRESS` |
12231231
| `log.enableRecordHttpSql` | `TAOS_ADAPTER_LOG_ENABLE_RECORD_HTTP_SQL` |
12241232
| `log.enableSqlToCsvLogging` | `TAOS_ADAPTER_LOG_ENABLE_SQL_TO_CSV_LOGGING` |
1233+
| `log.enableStmtToCsvLogging` | `TAOS_ADAPTER_LOG_ENABLE_STMT_TO_CSV_LOGGING` |
12251234
| `log.keepDays` | `TAOS_ADAPTER_LOG_KEEP_DAYS` |
12261235
| `log.level` | `TAOS_ADAPTER_LOG_LEVEL` |
12271236
| `log.path` | `TAOS_ADAPTER_LOG_PATH` |
@@ -1403,7 +1412,7 @@ Send an HTTP POST request to the `/record_sql` endpoint to dynamically enable re
14031412
```bash
14041413
curl --location --request POST 'http://127.0.0.1:6041/record_sql' \
14051414
-u root:taosdata \
1406-
--data '{"start_time":"2025-07-15 17:00:00","end_time":"2025-07-15 18:00:00","location":"Asia/Shanghai"}'
1415+
--data '{"start_time":"2026-01-13 17:00:00","end_time":"2026-01-13 18:00:00","location":"Asia/Shanghai"}'
14071416
```
14081417
14091418
Supported parameters:
@@ -1538,6 +1547,311 @@ Example:
15381547
2025-07-23 17:10:08.724775,show databases,127.0.0.1,root,http,0x2000000000000008,2025-07-23 17:10:08.707741,2025-07-23 17:10:08.724775,14191,965,1706,17034,53600,jdbc_test_app
15391548
```
15401549
1550+
## Recording STMT to CSV Files
1551+
1552+
taosAdapter supports recording STMT requests to CSV files starting from version v3.4.0.1. Users can enable this feature through the configuration parameter `log.enableStmtToCsvLogging` or dynamically enable/disable it via HTTP requests.
1553+
1554+
:::warning
1555+
Enabling this feature leads to severe performance degradation.
1556+
:::
1557+
1558+
### Configuration Parameters
1559+
1560+
1. New configuration item `log.enableStmtToCsvLogging` (boolean, default: false) determines whether STMT logging is enabled.
1561+
When set to true, STMT records will be saved to CSV files.
1562+
The recording start time is the service startup time, and the end time is `2300-01-01 00:00:00`.
1563+
1564+
2. File naming follows the same rules as logs: `taosadapterStmt_{instanceId}_{yyyyMMdd}.csv[.index]`
1565+
- `instanceId`: taosAdapter instance ID, configurable via the instanceId parameter.
1566+
- `yyyyMMdd`: Date in year-month-day format.
1567+
- `index`: If multiple files exist, a numeric suffix will be appended to the filename.
1568+
1569+
3. Existing log parameters are used for space retention, file splitting, and storage path:
1570+
- `log.path`: Storage path
1571+
- `log.keepDays`: Retention period in days
1572+
- `log.rotationCount`: Maximum number of retained files
1573+
- `log.rotationSize`: Maximum size per file
1574+
- `log.compress`: Whether compression is enabled
1575+
- `log.reservedDiskSize`: Reserved disk space size
1576+
1577+
### Dynamic Enablement
1578+
1579+
Send an HTTP POST request to the `/record_stmt` endpoint to dynamically enable recording. Authentication is the same as for `/rest/sql`. Example:
1580+
1581+
```bash
1582+
curl --location --request POST 'http://127.0.0.1:6041/record_stmt' \
1583+
-u root:taosdata \
1584+
--data '{"start_time":"2026-01-13 17:00:00","end_time":"2026-01-13 18:00:00","location":"Asia/Shanghai"}'
1585+
```
1586+
1587+
Supported parameters:
1588+
1589+
- `start_time`: [Optional] Start time for recording, formatted as `yyyy-MM-dd HH:mm:ss`. Defaults to the current time if not specified.
1590+
- `end_time`: [Optional] End time for recording, formatted as `yyyy-MM-dd HH:mm:ss`. Defaults to the current time plus one hour if not specified.
1591+
- `location`: [Optional] Timezone for parsing start and end times, using IANA format (e.g., `Asia/Shanghai`). Defaults to the server's timezone.
1592+
1593+
If all parameters use default values, the data field can be omitted. Example:
1594+
1595+
```bash
1596+
curl --location --request POST 'http://127.0.0.1:6041/record_stmt' \
1597+
-u root:taosdata
1598+
```
1599+
1600+
Successful response: HTTP code 200 with the following structure:
1601+
1602+
```json
1603+
{"code":0,"desc":""}
1604+
```
1605+
1606+
Failed response: Non-200 HTTP code with the following JSON structure (non-zero code and error description in desc):
1607+
1608+
```json
1609+
{"code":65535,"desc":"unmarshal json error"}
1610+
```
1611+
1612+
### Dynamic Disablement
1613+
1614+
Send an HTTP DELETE request to the `/record_stmt` endpoint to disable recording. Authentication is the same as for `/rest/sql`. Example:
1615+
1616+
```bash
1617+
curl --location --request DELETE 'http://127.0.0.1:6041/record_stmt' \
1618+
-u root:taosdata
1619+
```
1620+
1621+
Successful response: HTTP code 200.
1622+
1623+
1. If a task exists, the response is:
1624+
1625+
```json
1626+
{
1627+
"code": 0,
1628+
"message": "",
1629+
"start_time": "2026-01-13 17:00:00",
1630+
"end_time": "2026-01-13 18:00:00"
1631+
}
1632+
```
1633+
1634+
- `start_time`: Configured start time of the canceled task (timezone: server's timezone).
1635+
- `end_time`: Configured end time of the canceled task (timezone: server's timezone).
1636+
1637+
1. If no task exists, the response is:
1638+
1639+
```json
1640+
{
1641+
"code": 0,
1642+
"message": ""
1643+
}
1644+
```
1645+
1646+
### Query Status
1647+
1648+
Send an HTTP GET request to the `/record_stmt` endpoint to query the task status. Authentication is the same as for `/rest/sql`. Example:
1649+
1650+
```bash
1651+
curl --location 'http://127.0.0.1:6041/record_stmt' \
1652+
-u root:taosdata
1653+
```
1654+
1655+
Successful response: HTTP code 200 with the following structure:
1656+
1657+
```json
1658+
{
1659+
"code": 0,
1660+
"desc": "",
1661+
"exists": true,
1662+
"running": true,
1663+
"start_time": "2026-01-13 17:00:00",
1664+
"end_time": "2026-01-13 18:00:00",
1665+
"current_concurrent": 100
1666+
}
1667+
```
1668+
1669+
- `code`: Error code (0 for success).
1670+
- `desc`: Error message (empty string for success).
1671+
- `exists`: Whether the task exists.
1672+
- `running`: Whether the task is active.
1673+
- `start_time`: Start time (timezone: server's timezone).
1674+
- `end_time`: End time (timezone: server's timezone).
1675+
- `current_concurrent`: Current STMT recording concurrency.
1676+
1677+
### Recording Format
1678+
1679+
Records are written after `prepare`, `bind`, and `exec` operations, or when the task ends (reaching the end time or being manually stopped).
1680+
Records are stored in CSV format without headers. Each line includes the following fields:
1681+
1682+
1. `TS`: Log timestamp (format: yyyy-MM-dd HH:mm:ss.SSSSSS, timezone: server's timezone).
1683+
2. `IP`: Client IP.
1684+
3. `SourcePort`: Client port.
1685+
4. `AppName`: Client application name.
1686+
5. `User`: Username of the current connection.
1687+
6. `ConnType`: Connection type (ws).
1688+
7. `QID`: Request ID, saved as hexadecimal.
1689+
8. `StartTime`: Start processing time, formatted as yyyy-MM-dd HH:mm:ss.SSSSSS, timezone is the server timezone where taosAdapter is located.
1690+
9. `STMT2`: stmt2 memory address, saved as hexadecimal.
1691+
10. `Action`: Operation (prepare, bind, exec).
1692+
11. `Code`: Operation result, 0 represents success, other values represent error codes.
1693+
12. `Duration(us)`: Execution time in microseconds, -1 if not completed.
1694+
13. Data: Request or result data
1695+
1. prepare: Prepared SQL statement.
1696+
2. exec: Affected row count.
1697+
3. bind: Bound data, parsed successfully as JSON data, parsing failure will be raw binary data.
1698+
1699+
#### Bind Data JSON Format
1700+
1701+
The Data field content for a bind operation is in JSON format and contains the following fields:
1702+
1703+
| Field Name | Type | Description |
1704+
|:------------|:---------------------------|:---------------------------------------------------------|
1705+
| count | int | Number of tables with bound parameters |
1706+
| table_names | [count]string | List of table names with bound parameters |
1707+
| tags | [count][tag_count]col_info | List of tag column information for bound parameters |
1708+
| cols | [count][col_count]col_info | List of ordinary column information for bound parameters |
1709+
1710+
Where the col_info structure contains the following fields:
1711+
1712+
| Field Name | Type | Description |
1713+
|:-----------|:------|:-------------------------------------------------------------|
1714+
| type | int | Column type |
1715+
| data | []any | One-dimensional array, each element represents a row of data |
1716+
1717+
The parsing rules for each type of data in the data field are as follows:
1718+
1719+
1. varbinary, blob, geometry are parsed as hexadecimal strings
1720+
2. binary, nchar, json, decimal are parsed as utf-8 strings
1721+
3. Numeric types are converted to numbers
1722+
4. Datetime types are converted to numbers
1723+
5. Bool types are converted to json bool (true/false)
1724+
6. null data is converted to json null (null)
1725+
1726+
Sample example:
1727+
1728+
```json
1729+
{
1730+
"count": 1,
1731+
"table_names": ["test1"],
1732+
"tags": [
1733+
[{
1734+
"type": 9,
1735+
"data": [1726803356466]
1736+
}, {
1737+
"type": 1,
1738+
"data": [true]
1739+
}, {
1740+
"type": 2,
1741+
"data": [1]
1742+
}, {
1743+
"type": 3,
1744+
"data": [2]
1745+
}, {
1746+
"type": 4,
1747+
"data": [3]
1748+
}, {
1749+
"type": 5,
1750+
"data": [4]
1751+
}, {
1752+
"type": 6,
1753+
"data": [5.5]
1754+
}, {
1755+
"type": 7,
1756+
"data": [6.6]
1757+
}, {
1758+
"type": 11,
1759+
"data": [7]
1760+
}, {
1761+
"type": 12,
1762+
"data": [8]
1763+
}, {
1764+
"type": 13,
1765+
"data": [9]
1766+
}, {
1767+
"type": 14,
1768+
"data": [10]
1769+
}, {
1770+
"type": 8,
1771+
"data": ["binary"]
1772+
}, {
1773+
"type": 10,
1774+
"data": ["nchar"]
1775+
}, {
1776+
"type": 20,
1777+
"data": ["010100000000000000000059400000000000005940"]
1778+
}, {
1779+
"type": 16,
1780+
"data": ["76617262696e617279"]
1781+
}, {
1782+
"type": 17,
1783+
"data": ["12345.6789"]
1784+
}, {
1785+
"type": 21,
1786+
"data": ["98765.4321"]
1787+
}, {
1788+
"type": 18,
1789+
"data": ["7468697320697320626c6f622064617461"]
1790+
}]
1791+
],
1792+
"cols": [
1793+
[{
1794+
"type": 9,
1795+
"data": [1726803356466, 1726803357466, 1726803358466]
1796+
}, {
1797+
"type": 1,
1798+
"data": [true, null, false]
1799+
}, {
1800+
"type": 2,
1801+
"data": [11, null, 12]
1802+
}, {
1803+
"type": 3,
1804+
"data": [11, null, 12]
1805+
}, {
1806+
"type": 4,
1807+
"data": [11, null, 12]
1808+
}, {
1809+
"type": 5,
1810+
"data": [11, null, 12]
1811+
}, {
1812+
"type": 6,
1813+
"data": [11.2, null, 12.2]
1814+
}, {
1815+
"type": 7,
1816+
"data": [11.2, null, 12.2]
1817+
}, {
1818+
"type": 11,
1819+
"data": [11, null, 12]
1820+
}, {
1821+
"type": 12,
1822+
"data": [11, null, 12]
1823+
}, {
1824+
"type": 13,
1825+
"data": [11, null, 12]
1826+
}, {
1827+
"type": 14,
1828+
"data": [11, null, 12]
1829+
}, {
1830+
"type": 8,
1831+
"data": ["binary1", null, "binary2"]
1832+
}, {
1833+
"type": 10,
1834+
"data": ["nchar1", null, "nchar2"]
1835+
}, {
1836+
"type": 20,
1837+
"data": ["010100000000000000000059400000000000005940", null, "010100000000000000000059400000000000005940"]
1838+
}, {
1839+
"type": 16,
1840+
"data": ["76617262696e61727931", null, "76617262696e61727932"]
1841+
}, {
1842+
"type": 17,
1843+
"data": ["12345.6789", null, "22345.6789"]
1844+
}, {
1845+
"type": 21,
1846+
"data": ["98765.4321", null, "88765.4321"]
1847+
}, {
1848+
"type": 18,
1849+
"data": ["7468697320697320626c6f622064617461", null, "7468697320697320616e6f7468657220626c6f622064617461"]
1850+
}]
1851+
]
1852+
}
1853+
```
1854+
15411855
## Monitoring Metrics
15421856
15431857
Currently, taosAdapter only collects monitoring indicators for RESTful/WebSocket related requests. There are no monitoring indicators for other interfaces.

0 commit comments

Comments
 (0)