Alter column and timeseries datatype with ANSI SQL statement (apache#16027)

* init

* adapt write and add tests

* add test case

* disable altering from TEXT to STRING & add aggregation support

* supplement test

* add more test

* add concurrent test

* fix check data type and add load IT

* Add license

* fix ut

* fix ut

* fix test

* move update mod file existence inside

(cherry picked from commit ea9f726)

* add print for test failure

* fix it

* fix it

* fix deleteColumn in TVList

* fix delete memoryBinaryChunkSize

* fix partial insert

* fix test

* add debug print

* fix partial insert

* fix insertion retry after detecting incompatible type

* spotless

* fix data type check in memtable

* compact compatibility with alter column datatype (apache#15010)

* compact_alter_column

* modify chunk metadata and delete related debug output

* code review

* add UT test for column type alter in compaction

* update tsfile version

---------

Co-authored-by: Jiang Tian <[email protected]>

* fix missing check in procedure manager

* fix tests

* fix tests

* fix a problem about data type of field was displayed wrong.

* delete unuseful code.

* The logic that the data type of alignedChunkMetadata is different of the data type of measurement after changed which it can be considered is an defect in the FastAlignedSeriesCompactionExecutor.

* To Aligned Series:
Resolve the problem that only the first data type of chunkMetadata is adjusted.

Add the logic that the statistics of chunkMetadata need be analyzed with new data type. In addition, the data type of valueChunkMetadata need be updated.

To NonAligned Series:
Avoid the problem that chunk in the memory need be still rewrited when data type don't be changed.

* Adjust the chunk object need be merged statistics.
Don't need rewrite chunk data when data type have not been changed.

* Because this problem is not exist as below:
 "The logic that the data type of alignedChunkMetadata is different of the data type of measurement after changed which it can be considered is an defect in the FastAlignedSeriesCompactionExecutor."

So delete unuseful code.

* Fix some problems after merge code from master branch.

* Resolve the problem can not find symbol that variable ColumnCategory.

* Resolve the problem the query result is null from MemTable when don't execute the flush operation.

* Resolve the problem CI can not be passed sucessfully.
This is reason that the problem cause:
When type of timeseries is the align, the chunkMetadata of chunkMetadataElement contain two inner ChunkMetadata struct, only change their value with reference, don't need change value with the "set" function.

* Format code

* fix a problem the schema variable is wrong.

* rollback to original value

* format code

* Don't allow alter the data type of column that view table.
Add an IT function for it.

* Adjust some accumulator type of classes for tree model.
Supplement an integration test that belong tree model.

* Fix unexpected problem in the testLoadAndAccumulator function.

* Verify the data type of every column in the tsBlock, if occur their types compare with real data type are not the same, transfer directly value to correct type.

* Adjust some problems.

* Add some query test case.

* Format code.

* Correct the type convert for tsBlock.

* Add anthor test case.

* Merge branch 'master' into alter_column_datatype_draft

* Support other type transfer to TEXT/STRING type.

* fix bug.

* Verify datatype is contained the same origin so that adapt low version.

* Used Set<Pair<TSDataType,TSDataType>> structure to optimize code.

* Change tsfile version

* Add precision settings

* For sync metadata.

* Fix

* Test two sqls idempotent in the execute process

* Fix

* Set the 'modified' value is true so that resolve the problem that the statistics is not valid when any type transfer to STRING/TEXT type.
For example, when try to get max() value in the sql, will occur '512' > '1024', max() value is 512;

* Fix some problems for "Alter column from other type to String/Text type"

* format code.

* Resolve the problem that the value of String Data type is transfer to "String", but not "Binary".

* Delete unuseful code.

* Fix IT test.

* Replace tsfile version.

* Resolve compatible problems.

* Fix IT problems, because due to use wrong type of variable to validate value two sides of assert validation ago.

* cause by don't delete completely, verify the value of count() early.

* Clear some unuseful codes.

* Remove unnecessary input parameters.

* Resolve the problem merge statistics will occur "Statistics classes mismatched: class StringStatistics vs BinaryStatistics" when tsfile is unclosed and current type and previous type is not compatible.

* format the code of IT.

* Carry "testAlterColumnDataTypeIdempotent" IT test to correct position;
Fix the problem that valueChunkMetadata get wrong datatype so that can't generate valid statistics object in the switch code block cause by newChunkMetadata set the property value of TsDataType

* Support alter sql and procedure so that time series data type can be altered.

* Add the license description.

* Repair all problems for IoTDBAlterTimeSeriesTypeIT

* Resolve the problem that time series value cannot be standard date format.

* Resolve the problem that cannot format standard date string for the accumulator query among tree and table model.

* Add an overlapped data case on the scene of query.

* Fix issue and adjust some bugs on the IT, remove some debug code.

* update tsfile version.

* plan

* optimize codes.

* Support OBJECT data type

* Fix some issues and make OBJECT data type be compatible on some logics.

* AlterTimeSeriesTask only support "alter time series set data type" statement.

* Use the AlterTimeSeriesDataTypeStatement on the AlterTimeSeriesTask instead of AlterTimeSeriesStatement, can resolve other alter timeseries statement can run normally.

* Use reference object instead of create a new object.

* pipe-relatecd

* fix

* Fix the problem that query result is fault after alter timeseries data type cause by AlterTimeSeriesDataTypeProcedure won't run.

* Also adjust the encoding when changing column data type on a table;
Repair all accumulator query logics about TEXT、STRING、BLOB on IoTDBAlterColumnTypeIT;
Remove unuseful and incorrect code

* Adjust the code according to the revision suggestions.

* Remove spare codes.

* fix

* fix

---------

Co-authored-by: Tian Jiang <[email protected]>
Co-authored-by: jintao zhu <[email protected]>
Co-authored-by: Caideyipi <[email protected]>

Author: 4 people

integration-test/src/test/java/org/apache/iotdb/db/it/query/IoTDBResultSetIT.java

@@ -53,11 +53,11 @@ public class IoTDBResultSetIT {
         "CREATE DATABASE root.t1",
         "CREATE TIMESERIES root.t1.wf01.wt01.status WITH DATATYPE=BOOLEAN, ENCODING=PLAIN",
         "CREATE TIMESERIES root.t1.wf01.wt01.temperature WITH DATATYPE=FLOAT, ENCODING=RLE",
-        "CREATE TIMESERIES root.t1.wf01.wt01.type WITH DATATYPE=INT32, ENCODING=RLE",
+        "CREATE TIMESERIES root.t1.wf01.wt01.`type` WITH DATATYPE=INT32, ENCODING=RLE",
         "CREATE TIMESERIES root.t1.wf01.wt01.grade WITH DATATYPE=INT64, ENCODING=RLE",
         "CREATE TIMESERIES root.t1.wf01.wt02.status WITH DATATYPE=BOOLEAN, ENCODING=PLAIN",
         "CREATE TIMESERIES root.t1.wf01.wt02.temperature WITH DATATYPE=FLOAT, ENCODING=RLE",
-        "CREATE TIMESERIES root.t1.wf01.wt02.type WITH DATATYPE=INT32, ENCODING=RLE",
+        "CREATE TIMESERIES root.t1.wf01.wt02.`type` WITH DATATYPE=INT32, ENCODING=RLE",
         "CREATE TIMESERIES root.t1.wf01.wt02.grade WITH DATATYPE=INT64, ENCODING=RLE",
         "CREATE TIMESERIES root.sg.dev.status WITH DATATYPE=text,ENCODING=PLAIN",
         "insert into root.sg.dev(time,status) values(1,3.14)"
@@ -81,9 +81,9 @@ public void intAndLongConversionTest() {
     try (Connection connection = EnvFactory.getEnv().getConnection();
         Statement statement = connection.createStatement()) {
       statement.execute(
-          "insert into root.t1.wf01.wt01(timestamp, status, type, grade) values (1000, true, 1, 1000)");
+          "insert into root.t1.wf01.wt01(timestamp, status, `type`, grade) values (1000, true, 1, 1000)");
       statement.execute(
-          "insert into root.t1.wf01.wt01(timestamp, status, type, grade) values (2000, false, 2, 2000)");
+          "insert into root.t1.wf01.wt01(timestamp, status, `type`, grade) values (2000, false, 2, 2000)");
 
       try (ResultSet resultSet1 =
           statement.executeQuery("select count(status) from root.t1.wf01.wt01")) {
@@ -94,7 +94,8 @@ public void intAndLongConversionTest() {
       }
 
       try (ResultSet resultSet2 =
-          statement.executeQuery("select type from root.t1.wf01.wt01 where time = 1000 limit 1")) {
+          statement.executeQuery(
+              "select `type` from root.t1.wf01.wt01 where time = 1000 limit 1")) {
         resultSet2.next();
         // type of r2 is INT32(int), test int convert to long
         long type = resultSet2.getLong(2);