Skip to content

Conversation

@TDroyal
Copy link
Contributor

@TDroyal TDroyal commented Sep 3, 2025

What problem does this PR solve?

1.Issue Number: fix #10 #14 .

What is changed and how it works?

支持数据导出到本地和远程数据库

>ts-cli export -h
2025-09-03 11:04:56.8120923 +0800 CST m=+0.025423501 init logger, conf: {app:single Format:auto Level:info MaxSize:67108864 MaxNum:16 MaxAge:7 CompressEnabled:true Path:/home/royal_111/.openGemini/logs}
(EXPERIMENTAL) Export data from openGemini to file or remote
Usage:
ts-cli export [flags]
Examples:
        $ ts-cli export --format txt --out /tmp/openGemini/export/export.txt --data /tmp/openGemini/data --wal /tmp/openGemini/data
        --dbfilter NOAA_water_database
        $ ts-cli export --format csv --out /tmp/openGemini/export/export.csv --data /tmp/openGemini/data --wal /tmp/openGemini/data
        --dbfilter NOAA_water_database --mstfilter h2o_pH --timefilter "2019-08-25T09:18:00Z~2019-08-26T07:48:00Z"
        $ ts-cli export --format remote --remote ${host}:8086 --data /tmp/openGemini/data --wal /tmp/openGemini/data
        --dbfilter NOAA_water_database --mstfilter h2o_feet
Flags:
--compress                 Optional. Compress the export output.
--data string              Data storage path to export.
--dbfilter string          Database to export
--format string            Export data format, support csv, txt, remote. (default "txt")
-h, --help                     help for export
--mstfilter string         Optional.Measurement to export.
--out string               Destination file to export to.
--remote string            Remote address to export data.
-p, --remotepassword string    Remote export Optional.Password to connect to remote openGemini.
--remotessl                Remote export Optional.Use https for connecting to remote openGemini.
-u, --remoteusername string    Remote export Optional.Username to connect to remote openGemini.
--resume                   Resume the export progress from the last point.
--retentionfilter string   Optional. Retention policy to export.
--timefilter string        Optional.Export time range, support 'start~end'
--wal string               WAL storage path to export.

用法

1.支持将数据导出txt/csv格式,需要指定数据存储的data目录、wal目录以及数据库名称,可选参数:表名、保留策略、时间戳范围、压缩数据等

使用示例如下:

# 导出txt格式
ts-cli export --format txt --out /tmp/openGemini/export/export.txt --data /tmp/openGemini/data --wal /tmp/openGemini/data --dbfilter db31
# 导出csv格式
ts-cli export --format csv --out /tmp/openGemini/export/export.csv --data /tmp/openGemini/data --wal /tmp/openGemini/data --dbfilter db31
# 指定表名和时间戳范围
ts-cli export --format csv --out /tmp/openGemini/export/export.csv --data /tmp/openGemini/data --wal /tmp/openGemini/data --dbfilter db31 --mstfilter cpu --timefilter "2019-08-25T09:18:00Z~2019-08-26T07:48:00Z"
# 指定数据压缩
ts-cli export --format csv --out /tmp/openGemini/export/export.csv --data /tmp/openGemini/data --wal /tmp/openGemini/data --dbfilter db31 --mstfilter cpu --timefilter "2019-08-25T09:18:00Z~2019-08-26T07:48:00Z" --compress true

2.支持断点续传:导出任务中断后再次启动时,自动跳过已完成的 TSSP 文件(例如已导出 4/9 个文件),并从后续文件(第 5 个)继续导出,确保数据导出的幂等性和资源高效利用。

ts-cli export --format csv --out /tmp/openGemini/export/export.csv --data /tmp/openGemini/data --wal /tmp/openGemini/data --dbfilter db31 --resume true

3.支持将a节点的数据导出到b节点(跨数据库)

ts-cli export --format remote --remote ${host}:8086 --data /tmp/openGemini/data --wal /tmp/openGemini/data --dbfilter db31

Checklist:

✅ My code follows the style guidelines of this project
✅ I have performed a self-review of my own code
✅ I have commented my code, particularly in hard-to-understand areas
❌ I have made corresponding changes to the documentation
✅ My changes generate no new warnings
❌ I have added tests that prove my fix is effective or that my feature works
✅ New and existing unit tests pass locally with my changes
✅ Any dependent changes have been merged and published in downstream modules

@dosubot
Copy link

dosubot bot commented Sep 3, 2025

Related Documentation

1 document(s) may need updating based on files changed in this PR

How did I do? Any feedback?  Join Discord

@xuthus5
Copy link
Member

xuthus5 commented Sep 3, 2025

我们能否不依赖内核仓库,将依赖的公共代码抽离出来,减小构建体积。

@TDroyal
Copy link
Contributor Author

TDroyal commented Sep 3, 2025

@xuthus5 直接依赖的代码有的还会依赖其它代码,抽离出来可能也是一大堆文件,后续评估后可以考虑,目前考虑先以这样的方式。

@TDroyal TDroyal force-pushed the feat-support-export-data branch 3 times, most recently from ea36991 to 5c59a38 Compare September 3, 2025 07:21
Signed-off-by: royal_111 <royal_111@qq.com>
@TDroyal TDroyal force-pushed the feat-support-export-data branch from 5c59a38 to d7dbc48 Compare September 3, 2025 07:32
@xuthus5 xuthus5 merged commit ef4e06c into openGemini:main Sep 3, 2025
6 checks passed
@dosubot
Copy link

dosubot bot commented Sep 3, 2025

Documentation updates

Checked 2 published document(s). No updates required.

How did I do? Any feedback?  Join Discord

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test the ts-cli data export function and merge it into the main branch

2 participants