Skip to content

Commit ea09459

Browse files
jt2594838Copilot
jt2594838
and
Copilot
authored
Support TsFile properties and optional table point counting (#882)
* add property read-write interface * May record table point count * Add perf test * Add table point count repair tool * add point-count tool * add point-count tool * supplement usage for other os Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * supplement usage for other os Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * use byte[] for properties * use byte[8] for table point count --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
1 parent ac496ba commit ea09459

21 files changed

Lines changed: 1142 additions & 36 deletions

‎java/common/src/main/resources/org/apache/tsfile/i18n/messages.properties‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1548,3 +1548,10 @@ log.tools.csv_read_error = Error reading CSV file: %1$s
15481548

15491549
# CsvSourceReader — error closing CSV reader
15501550
log.tools.csv_close_reader_error = Error closing CSV reader
1551+
1552+
# TsFileTablePointCountTool
1553+
error.utils.table_point_count_tool_usage = Usage: TsFileTablePointCountTool <file.tsfile>
1554+
error.utils.table_point_count_tool_file_not_found = TsFile does not exist: %1$s
1555+
error.utils.table_point_count_tool_incomplete_file = TsFile is incomplete: %1$s
1556+
error.utils.table_point_count_tool_copy_failed = Failed to copy the TsFile data and metadata prefix
1557+
info.utils.table_point_count_tool_result = Processed %1$s: %2$s

‎java/common/src/main/resources/org/apache/tsfile/i18n/messages_zh.properties‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1548,3 +1548,10 @@ log.tools.csv_read_error = 读取 CSV 文件出错: %1$s
15481548

15491549
# CsvSourceReader — error closing CSV reader
15501550
log.tools.csv_close_reader_error = 关闭 CSV reader 出错
1551+
1552+
# TsFileTablePointCountTool
1553+
error.utils.table_point_count_tool_usage = 用法：TsFileTablePointCountTool <file.tsfile>
1554+
error.utils.table_point_count_tool_file_not_found = TsFile 不存在：%1$s
1555+
error.utils.table_point_count_tool_incomplete_file = TsFile 不完整：%1$s
1556+
error.utils.table_point_count_tool_copy_failed = 复制 TsFile 数据和元数据前缀失败
1557+
info.utils.table_point_count_tool_result = 已处理 %1$s：%2$s

‎java/tools/README-zh.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -229,3 +229,24 @@ arrow2tsfile.bat --source .\data\arrow --target .\output --fail_dir .\failed
229229
- 多批次：`{源文件名}_1.tsfile`、`{源文件名}_2.tsfile`、...
230230
- 表名与输出文件名相互独立——表名来自 schema 或 `--table_name`，文件名来自源文件。
231231

232+
## 表点数统计工具
233+
234+
`tools/tsfile-table-point-count.sh`（Linux/macOS）和 `tools\tsfile-table-point-count.bat`（Windows）用于检查完整 TsFile 是否包含表级点数统计属性，并在属性缺失时补写。每张表的点数是其所有 FIELD 列中非空值的总数，不统计 TAG 列和时间列。
235+
236+
设置 `JAVA_HOME` 后，传入且仅传入一个 TsFile 路径：
237+
238+
# Linux/macOS
239+
tools/tsfile-table-point-count.sh /data/example.tsfile
240+
241+
:: Windows
242+
tools\tsfile-table-point-count.bat C:\data\example.tsfile
243+
244+
工具会输出以下状态之一：
245+
246+
| 状态 | 说明 |
247+
|------|------|
248+
| `UPDATED` | 已计算缺失的点数统计属性并写回文件。 |
249+
| `ALREADY_PRESENT` | 每张表都已有有效的点数统计属性，未修改文件。 |
250+
| `NO_TABLE` | 文件不包含表 Schema，例如仅含树模型数据的 TsFile，未修改文件。 |
251+
252+
仅当状态为 `UPDATED` 时，工具才会原地修改输入文件。工具先在源文件所在目录生成完整的临时文件，确保重写后的元数据落盘后再替换源文件。运行前请确保目录可写、磁盘空间足以存放临时副本，并备份重要文件。不存在的文件和未写完整的 TsFile 会被拒绝处理。

‎java/tools/README.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -228,3 +228,29 @@ arrow2tsfile.bat --source .\data\arrow --target .\output --fail_dir .\failed
228228
- Multiple batches: `{source_basename}_1.tsfile`, `{source_basename}_2.tsfile`, ...
229229
- Table name and output filename are independent — table name comes from schema or `--table_name`, filename comes from source file.
230230

231+
## Table Point Count Tool
232+
233+
`tools/tsfile-table-point-count.sh` (Linux/macOS) and `tools\tsfile-table-point-count.bat` (Windows) check whether a complete TsFile contains table-level
234+
point-count properties and backfills them when they are missing. For each table, the point count is
235+
the total number of non-null values in its FIELD columns; TAG and time columns are not counted.
236+
237+
Set `JAVA_HOME`, then run the tool with exactly one TsFile path:
238+
239+
# Linux/macOS
240+
tools/tsfile-table-point-count.sh /data/example.tsfile
241+
242+
:: Windows
243+
tools\tsfile-table-point-count.bat C:\data\example.tsfile
244+
245+
The tool prints one of the following statuses:
246+
247+
| Status | Description |
248+
|--------|-------------|
249+
| `UPDATED` | Missing point-count properties were calculated and written to the file. |
250+
| `ALREADY_PRESENT` | Every table already had a valid point-count property; the file was not modified. |
251+
| `NO_TABLE` | The file contains no table schema, for example a tree-model-only TsFile; the file was not modified. |
252+
253+
The tool modifies the input file in place only for `UPDATED`. It writes a complete temporary copy
254+
next to the source file and replaces the source after the rewritten metadata has been flushed. Make
255+
sure the directory is writable, leave enough free space for a temporary copy, and back up important
256+
files before running the tool. Missing files and incomplete TsFiles are rejected.

‎java/tools/src/assembly/resources/tools/tsfile-table-point-count.bat‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
@REM
2+
@REM Licensed to the Apache Software Foundation (ASF) under one
3+
@REM or more contributor license agreements. See the NOTICE file
4+
@REM distributed with this work for additional information
5+
@REM regarding copyright ownership. The ASF licenses this file
6+
@REM to you under the Apache License, Version 2.0 (the
7+
@REM "License"); you may not use this file except in compliance
8+
@REM with the License. You may obtain a copy of the License at
9+
@REM
10+
@REM http://www.apache.org/licenses/LICENSE-2.0
11+
@REM
12+
@REM Unless required by applicable law or agreed to in writing,
13+
@REM software distributed under the License is distributed on an
14+
@REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
@REM KIND, either express or implied. See the License for the
16+
@REM specific language governing permissions and limitations
17+
@REM under the License.
18+
@REM
19+
20+
@echo off
21+
setlocal enabledelayedexpansion
22+
23+
if "%OS%" == "Windows_NT" setlocal
24+
25+
pushd %~dp0..
26+
if NOT DEFINED TSFILE_HOME set TSFILE_HOME=%CD%
27+
popd
28+
29+
set JAVA_OPTS=-ea^
30+
-DTSFILE_HOME="%TSFILE_HOME%"
31+
32+
if NOT DEFINED JAVA_HOME goto :err
33+
34+
echo ------------------------------------------
35+
echo Starting TsFile Table Point Count Tool
36+
echo ------------------------------------------
37+
38+
set CLASSPATH="%TSFILE_HOME%\lib\*"
39+
set MAIN_CLASS=org.apache.tsfile.utils.TsFileTablePointCountTool
40+
41+
@REM The argument is the TsFile to inspect. The tool rewrites its properties only when table-level
42+
@REM point-count statistics are missing.
43+
"%JAVA_HOME%\bin\java" -DTSFILE_HOME=!TSFILE_HOME! !JAVA_OPTS! -cp !CLASSPATH! !MAIN_CLASS! %*
44+
exit /b %ERRORLEVEL%
45+
46+
:err
47+
echo JAVA_HOME environment variable must be set!
48+
set ret_code=1
49+
exit /b %ret_code%

‎java/tools/src/assembly/resources/tools/tsfile-table-point-count.sh‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
#!/bin/bash
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
#
20+
21+
echo ------------------------------------------
22+
echo Starting TsFile Table Point Count Tool
23+
echo ------------------------------------------
24+
25+
if [ -z "${TSFILE_HOME}" ]; then
26+
export TSFILE_HOME="$(cd "`dirname "$0"`"/..; pwd)"
27+
fi
28+
29+
if [ -n "$JAVA_HOME" ]; then
30+
for java in "$JAVA_HOME"/bin/amd64/java "$JAVA_HOME"/bin/java; do
31+
if [ -x "$java" ]; then
32+
JAVA="$java"
33+
break
34+
fi
35+
done
36+
else
37+
JAVA=java
38+
fi
39+
40+
if [ -z $JAVA ] ; then
41+
echo Unable to find java executable. Check JAVA_HOME and PATH environment variables. > /dev/stderr
42+
exit 1;
43+
fi
44+
45+
CLASSPATH=${TSFILE_HOME}/lib/*
46+
MAIN_CLASS=org.apache.tsfile.utils.TsFileTablePointCountTool
47+
48+
# The argument is the TsFile to inspect. The tool rewrites its properties only when table-level
49+
# point-count statistics are missing.
50+
exec "$JAVA" -DTSFILE_HOME=${TSFILE_HOME} -cp "$CLASSPATH" "$MAIN_CLASS" "$@"

‎java/tools/src/assembly/tools.xml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,5 +69,14 @@
6969
<source>${maven.multiModuleProjectDirectory}/java/tools/src/assembly/resources/tools/arrow2tsfile.bat</source>
7070
<destName>tools/arrow2tsfile.bat</destName>
7171
</file>
72+
<file>
73+
<source>${maven.multiModuleProjectDirectory}/java/tools/src/assembly/resources/tools/tsfile-table-point-count.sh</source>
74+
<destName>tools/tsfile-table-point-count.sh</destName>
75+
<fileMode>0755</fileMode>
76+
</file>
77+
<file>
78+
<source>${maven.multiModuleProjectDirectory}/java/tools/src/assembly/resources/tools/tsfile-table-point-count.bat</source>
79+
<destName>tools/tsfile-table-point-count.bat</destName>
80+
</file>
7281
</files>
7382
</assembly>

‎java/tsfile/src/main/java/org/apache/tsfile/file/metadata/TsFileMetadata.java‎

Lines changed: 47 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919

2020
package org.apache.tsfile.file.metadata;
2121

22+
import org.apache.tsfile.common.conf.TSFileConfig;
2223
import org.apache.tsfile.compatibility.DeserializeConfig;
2324
import org.apache.tsfile.encrypt.EncryptUtils;
2425
import org.apache.tsfile.exception.encrypt.EncryptException;
@@ -45,7 +46,7 @@ public class TsFileMetadata {
4546
private Map<String, MetadataIndexNode> tableMetadataIndexNodeMap;
4647
private Map<String, TableSchema> tableSchemaMap;
4748
private boolean hasTableSchemaMapCache;
48-
private Map<String, String> tsFileProperties;
49+
private Map<String, byte[]> tsFileProperties;
4950

5051
// offset of MetaMarker.SEPARATOR
5152
private long metaOffset;
@@ -117,21 +118,27 @@ public static TsFileMetadata deserializeFrom(
117118

118119
if (buffer.hasRemaining()) {
119120
int propertiesSize = ReadWriteForEncodingUtils.readVarInt(buffer);
120-
Map<String, String> propertiesMap = new HashMap<>();
121+
Map<String, byte[]> propertiesMap = new HashMap<>();
121122
for (int i = 0; i < propertiesSize; i++) {
122123
String key = ReadWriteIOUtils.readVarIntString(buffer);
123-
String value = ReadWriteIOUtils.readVarIntString(buffer);
124+
int valueSize = ReadWriteForEncodingUtils.readVarInt(buffer);
125+
byte[] value = null;
126+
if (valueSize >= 0) {
127+
value = new byte[valueSize];
128+
buffer.get(value);
129+
}
124130
propertiesMap.put(key, value);
125131
}
132+
String encryptLevel = getPropertyAsString(propertiesMap, "encryptLevel");
126133
// if the file is not encrypted, set the default value(for compatible reason)
127-
if (!propertiesMap.containsKey("encryptLevel") || propertiesMap.get("encryptLevel") == null) {
128-
propertiesMap.put("encryptLevel", "0");
129-
propertiesMap.put("encryptType", "org.apache.tsfile.encrypt.UNENCRYPTED");
130-
propertiesMap.put("encryptKey", "");
131-
} else if (propertiesMap.get("encryptLevel").equals("0")) {
132-
propertiesMap.put("encryptType", "org.apache.tsfile.encrypt.UNENCRYPTED");
133-
propertiesMap.put("encryptKey", "");
134-
} else if (propertiesMap.get("encryptLevel").equals("1")) {
134+
if (!propertiesMap.containsKey("encryptLevel") || encryptLevel == null) {
135+
propertiesMap.put("encryptLevel", stringToBytes("0"));
136+
propertiesMap.put("encryptType", stringToBytes("org.apache.tsfile.encrypt.UNENCRYPTED"));
137+
propertiesMap.put("encryptKey", stringToBytes(""));
138+
} else if (encryptLevel.equals("0")) {
139+
propertiesMap.put("encryptType", stringToBytes("org.apache.tsfile.encrypt.UNENCRYPTED"));
140+
propertiesMap.put("encryptKey", stringToBytes(""));
141+
} else if (encryptLevel.equals("1")) {
135142
if (!propertiesMap.containsKey("encryptType")) {
136143
throw new EncryptException(
137144
Messages.format("error.file.tsfile_metadata_no_encrypt_type", 1));
@@ -140,15 +147,15 @@ public static TsFileMetadata deserializeFrom(
140147
throw new EncryptException(
141148
Messages.format("error.file.tsfile_metadata_no_encrypt_key", 1));
142149
}
143-
if (propertiesMap.get("encryptKey") == null || propertiesMap.get("encryptKey").isEmpty()) {
150+
String encryptKey = getPropertyAsString(propertiesMap, "encryptKey");
151+
if (encryptKey == null || encryptKey.isEmpty()) {
144152
throw new EncryptException(
145153
Messages.format("error.file.tsfile_metadata_null_encrypt_key", 1));
146154
}
147-
String str = propertiesMap.get("encryptKey");
148155
fileMetaData.encryptLevel = 1;
149-
fileMetaData.secondKey = EncryptUtils.getSecondKeyFromStr(str);
150-
fileMetaData.encryptType = propertiesMap.get("encryptType");
151-
} else if (propertiesMap.get("encryptLevel").equals("2")) {
156+
fileMetaData.secondKey = EncryptUtils.getSecondKeyFromStr(encryptKey);
157+
fileMetaData.encryptType = getPropertyAsString(propertiesMap, "encryptType");
158+
} else if (encryptLevel.equals("2")) {
152159
if (!propertiesMap.containsKey("encryptType")) {
153160
throw new EncryptException(
154161
Messages.format("error.file.tsfile_metadata_no_encrypt_type", 2));
@@ -157,27 +164,25 @@ public static TsFileMetadata deserializeFrom(
157164
throw new EncryptException(
158165
Messages.format("error.file.tsfile_metadata_no_encrypt_key", 2));
159166
}
160-
if (propertiesMap.get("encryptKey") == null || propertiesMap.get("encryptKey").isEmpty()) {
167+
String encryptKey = getPropertyAsString(propertiesMap, "encryptKey");
168+
if (encryptKey == null || encryptKey.isEmpty()) {
161169
throw new EncryptException(
162170
Messages.format("error.file.tsfile_metadata_null_encrypt_key", 2));
163171
}
164172
fileMetaData.encryptLevel = 2;
165-
String str = propertiesMap.get("encryptKey");
166-
fileMetaData.secondKey = EncryptUtils.getSecondKeyFromStr(str);
167-
fileMetaData.encryptType = propertiesMap.get("encryptType");
173+
fileMetaData.secondKey = EncryptUtils.getSecondKeyFromStr(encryptKey);
174+
fileMetaData.encryptType = getPropertyAsString(propertiesMap, "encryptType");
168175
} else {
169176
throw new EncryptException(
170-
Messages.format(
171-
"error.file.tsfile_metadata_unsupported_encrypt_level",
172-
propertiesMap.get("encryptLevel")));
177+
Messages.format("error.file.tsfile_metadata_unsupported_encrypt_level", encryptLevel));
173178
}
174179
fileMetaData.tsFileProperties = propertiesMap;
175180
}
176181

177182
return fileMetaData;
178183
}
179184

180-
public void addProperty(String key, String value) {
185+
public void addProperty(String key, byte[] value) {
181186
if (tsFileProperties == null) {
182187
tsFileProperties = new HashMap<>();
183188
}
@@ -248,9 +253,15 @@ public int serializeTo(OutputStream outputStream) throws IOException {
248253
ReadWriteForEncodingUtils.writeVarInt(
249254
tsFileProperties != null ? tsFileProperties.size() : 0, outputStream);
250255
if (tsFileProperties != null) {
251-
for (Entry<String, String> entry : tsFileProperties.entrySet()) {
256+
for (Entry<String, byte[]> entry : tsFileProperties.entrySet()) {
252257
byteLen += ReadWriteIOUtils.writeVar(entry.getKey(), outputStream);
253-
byteLen += ReadWriteIOUtils.writeVar(entry.getValue(), outputStream);
258+
byte[] value = entry.getValue();
259+
byteLen +=
260+
ReadWriteForEncodingUtils.writeVarInt(value == null ? -1 : value.length, outputStream);
261+
if (value != null) {
262+
outputStream.write(value);
263+
byteLen += value.length;
264+
}
254265
}
255266
}
256267

@@ -295,7 +306,16 @@ public Map<String, TableSchema> getTableSchemaMap() {
295306
return tableSchemaMap;
296307
}
297308

298-
public Map<String, String> getTsFileProperties() {
309+
public Map<String, byte[]> getTsFileProperties() {
299310
return tsFileProperties;
300311
}
312+
313+
private static String getPropertyAsString(Map<String, byte[]> properties, String key) {
314+
byte[] value = properties.get(key);
315+
return value == null ? null : new String(value, TSFileConfig.STRING_CHARSET);
316+
}
317+
318+
private static byte[] stringToBytes(String value) {
319+
return value.getBytes(TSFileConfig.STRING_CHARSET);
320+
}
301321
}

‎java/tsfile/src/main/java/org/apache/tsfile/read/TsFileSequenceReader.java‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -607,6 +607,11 @@ public Map<String, TableSchema> getTableSchemaMap() throws IOException {
607607
return getTableSchemaMap(null);
608608
}
609609

610+
/** Get the properties stored in the TsFile metadata. */
611+
public Map<String, byte[]> getTsFileProperties() throws IOException {
612+
return readFileMetadata().getTsFileProperties();
613+
}
614+
610615
public Map<String, TableSchema> getTableSchemaMap(LongConsumer ioSizeRecorder)
611616
throws IOException {
612617
if (tsFileMetaData != null && tsFileMetaData.hasTableSchemaMapCache()) {

‎java/tsfile/src/main/java/org/apache/tsfile/read/v4/DeviceTableModelReader.java‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,12 @@ public Optional<TableSchema> getTableSchemas(String tableName) throws IOExceptio
7878
return Optional.ofNullable(tableSchemaMap.get(tableName.toLowerCase()));
7979
}
8080

81+
@Override
82+
@TsFileApi
83+
public Map<String, byte[]> getTsFileProperties() throws IOException {
84+
return fileReader.getTsFileProperties();
85+
}
86+
8187
@TsFileApi
8288
public ResultSet query(String tableName, List<String> columnNames, long startTime, long endTime)
8389
throws IOException, NoTableException, NoMeasurementException, ReadProcessException {

0 commit comments

Comments
 (0)