Skip to content
Closed
Show file tree
Hide file tree
Changes from 125 commits
Commits
Show all changes
126 commits
Select commit Hold shift + click to select a range
ee8c8ab
Add related connector classes for clp_s
wraymo Jun 5, 2024
f38f962
add a constructor for ClpPlugin
wraymo Jun 5, 2024
8b71203
add ClpClient; add methods to get all tables
wraymo Jun 5, 2024
3b49f9a
add schema tree implementation
wraymo Jun 7, 2024
57dab45
add basic metadata operations
wraymo Jun 7, 2024
9c94bde
style fix
wraymo Jun 7, 2024
97bc690
a relatively complete metadata implementation
wraymo Jun 7, 2024
cdb8c27
fix bugs and add test cases
wraymo Jun 9, 2024
253cf7a
modify logs and add a test case
wraymo Jun 9, 2024
6f5ddba
add decompression
wraymo Jun 10, 2024
4866a10
add record cursor impl
wraymo Jun 10, 2024
409c3e8
store decompressed logs to tmpdir
wraymo Jun 10, 2024
6450e87
and tablelayouthandle and add a basic record cursor test
wraymo Jun 10, 2024
bfe9b7e
make iteration predictable
wraymo Jun 10, 2024
383efe9
clean the code
wraymo Jun 10, 2024
8c2574e
fix a test case
wraymo Jun 11, 2024
93570fe
remove unused dependency
wraymo Jun 11, 2024
856a53f
fix bugs to make it work
wraymo Jun 11, 2024
4165e1b
modify tests
wraymo Jun 11, 2024
a2fbc13
fix bugs
wraymo Jun 14, 2024
22dbb83
add basic query pushdown
wraymo Jun 26, 2024
8c48969
add test cases
wraymo Jun 26, 2024
a408e54
fix build error
wraymo Jun 26, 2024
00e3a62
fix a bug
wraymo Jun 26, 2024
c88055c
remove unused import
wraymo Jun 26, 2024
fbd02c7
get a functional pushdown impl
wraymo Jun 26, 2024
b7a094f
correctly handle string fields and nested fields
wraymo Jun 26, 2024
af883cf
remove unused import
wraymo Jun 26, 2024
2ab5892
add query optimizer and fix a bug
wraymo Jun 26, 2024
3ceeca0
add support for cardinality split and like predicates
wraymo Jun 28, 2024
c2b997e
temporary workaround
wraymo Jun 28, 2024
3e510f5
update clp connector version
wraymo Jul 19, 2024
476ed3c
add new files
wraymo Jul 25, 2024
22845ee
avoid generating weird LIKE predicates
wraymo Jul 25, 2024
e9050b9
a more robust ClpPlanOptimizer
wraymo Jul 26, 2024
fc7a497
fix a bug and add LIKE predicate handler
wraymo Jul 26, 2024
5346beb
fix a bug
wraymo Jul 26, 2024
3231507
add array support
wraymo Jul 26, 2024
a1511f1
Add a unit test
wraymo Jul 27, 2024
36bddec
Fix bugs
wraymo Jul 28, 2024
9bede0d
make it more robust for logical binaries
wraymo Jul 28, 2024
519fa7f
add projection support
wraymo Aug 16, 2024
eeedf2d
add clp module
wraymo Aug 16, 2024
2122ec5
update submodules
wraymo Aug 16, 2024
6e0c386
update submodules
wraymo Aug 16, 2024
9788a01
add clp worker support
wraymo Aug 19, 2024
cd076f2
add more methods
wraymo Aug 20, 2024
2342ed5
add more methods
wraymo Aug 21, 2024
da67953
modify CMakeLists.txt
wraymo Aug 21, 2024
fde90b3
add clp data source support
wraymo Aug 24, 2024
04e8afd
remove duplicate values and add null support
wraymo Aug 28, 2024
4419bb1
add some configuration examples
wraymo Sep 4, 2024
38a4049
add CLP Split support
wraymo Sep 12, 2024
d99aefe
fix a bug
wraymo Sep 14, 2024
9dad342
remove duplicate columns for projection
wraymo Sep 16, 2024
49aed85
destroy clp-s process when closing the record cursor
wraymo Sep 17, 2024
6bdec9d
remove unused import
wraymo Sep 19, 2024
9813613
add concurrent split support
wraymo Sep 30, 2024
79ac281
close the buffer when closing teh CLP Record curor
wraymo Sep 28, 2024
6d059cf
remove unused file
wraymo Sep 30, 2024
440e7a0
resolve several issues
wraymo Oct 3, 2024
c089cdb
add support for reading metadata from mysql
wraymo Jan 2, 2025
dddc29b
add support for searching from S3 and reading schema from MySQL
wraymo Jan 20, 2025
a81dabb
rename tableMetadataPrefix
wraymo Jan 22, 2025
e4c7584
minor fixes
wraymo Jan 22, 2025
c02b71f
rename file source to input source
wraymo Jan 22, 2025
1c199da
minor fix
wraymo Jan 22, 2025
f7cd4d7
fix mysql runtime error
wraymo Jan 24, 2025
5a227c5
update git submodule path
wraymo Jan 24, 2025
387a3ed
only compile presto_server target
wraymo Jan 27, 2025
bd8defe
update worker configurations
wraymo Jan 27, 2025
cbe0786
do not treat warnings as errors
wraymo Jan 28, 2025
ad9f515
add an s3 coordinator configuration example
wraymo Feb 5, 2025
91b36f0
update velox dependency and modify example configuration files
wraymo Feb 5, 2025
985adb5
remove connectorFactory variable from ClpPlugin
wraymo Feb 26, 2025
c77e33b
remove parseSchemaFile from ClpClient
wraymo Feb 26, 2025
9bbbe5c
refactor metadata api, delete uncessary files and refactor the test
wraymo Feb 28, 2025
34bfc3b
apply prestissimo changes
wraymo Feb 28, 2025
d431a7a
add test cases for PlanOptimizer
wraymo Mar 4, 2025
bcf4096
remove SchemaTree
wraymo Mar 5, 2025
73e6a06
Finished ClP plan optimizer and metadata unit tests, fixed related bug
wraymo Mar 6, 2025
e4db337
update protocol code for clp connector
wraymo Mar 11, 2025
20e7613
update presto-clp pom.xml
wraymo Mar 11, 2025
39fe294
fix build issues
Mar 12, 2025
2af6106
refactor metadata, add ClpMySQLSplitProvider implementation and unit …
Mar 13, 2025
3195c9f
apply presto-native-execution changes
wraymo Mar 17, 2025
cf6faad
update ClpMySQLSplitProvider and ClpMySQLMetadataProvider
wraymo Mar 17, 2025
c9415c6
apply changes in PrestoToVeloxConnector.cpp
wraymo Mar 17, 2025
0a5178c
update presto-clp description and update split APIs
wraymo Mar 17, 2025
ac0e315
advance velox
wraymo Mar 17, 2025
2938392
update presto-native-execution and velox
Mar 18, 2025
cf96632
revert presto-native-execution/Makefile back and clean up ClpConfig
wraymo Mar 20, 2025
6fabaa1
remove dependency-reduced-pom.xml
wraymo Mar 20, 2025
19fdd85
revert gitignore back
wraymo Mar 20, 2025
27a3749
remove dependency-reduced-pom.xml
wraymo Mar 20, 2025
e173b0a
change set to list for metadata
wraymo Mar 20, 2025
7e0dbf5
apply coderabbit suggestions
wraymo Mar 20, 2025
65ea0a8
remove unused dependencies in pom.xml
wraymo Mar 20, 2025
b764f9c
allow like predicate to be transformed to substr match and optimize i…
wraymo Mar 20, 2025
4a6b4ff
clean up the code
wraymo Mar 21, 2025
e4b558e
add clp to presto-dpcs and clean up dependencies
wraymo Mar 25, 2025
a45341b
remove config examples
wraymo Mar 25, 2025
ed0397d
revert some files back
wraymo Mar 25, 2025
9945517
revert presto-native-execution back
wraymo Mar 25, 2025
57a8aa9
remove clp files
wraymo Mar 25, 2025
31816a2
add row type in clp
wraymo Mar 26, 2025
2747ec6
add optimizer logic for row type
wraymo Mar 28, 2025
95a0db1
update unit test cases for row type
wraymo Mar 28, 2025
3dbedd2
revert lowercase change back
wraymo Mar 28, 2025
4b6fd82
support row type for more expressions
wraymo Mar 28, 2025
9a22e2a
test nested types and array case
wraymo Mar 29, 2025
6253650
improve clp docs
wraymo Mar 31, 2025
0201632
fix sql injection issue
wraymo Mar 31, 2025
baa1097
add row type to the doc
wraymo Mar 31, 2025
a80ad57
improve the docs
wraymo Mar 31, 2025
977dead
remove handlePolymorphicType in ClpClient and support original names …
wraymo Apr 2, 2025
ea73a1a
improve comments
wraymo Apr 2, 2025
7bd75ee
rename splitsource and metadatasource
wraymo Apr 2, 2025
f5ca7f5
address review comments
wraymo Apr 3, 2025
1b63cbc
use archive_id
wraymo Apr 3, 2025
37e1b4c
add comments in ClpSchemaTree
wraymo Apr 3, 2025
8428517
rename package name and class name
wraymo Apr 4, 2025
5771f1d
add comments in ClpFilterToKqlConverter.java
wraymo Apr 4, 2025
4f042a2
fix typo
wraymo Apr 4, 2025
943d502
add comments
wraymo Apr 4, 2025
f36b5b8
remove the empty line
wraymo Apr 4, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,7 @@
<module>presto-native-sidecar-plugin</module>
<module>presto-base-arrow-flight</module>
<module>presto-function-server</module>
<module>presto-clp</module>
</modules>

<dependencyManagement>
Expand Down Expand Up @@ -761,6 +762,12 @@
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-clp</artifactId>
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-expressions</artifactId>
Expand Down
134 changes: 134 additions & 0 deletions presto-clp/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
<?xml version="1.0"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.292-SNAPSHOT</version>
</parent>

<artifactId>presto-clp</artifactId>
<description>Presto - CLP Connector</description>
<packaging>presto-plugin</packaging>

<properties>
<air.main.basedir>${project.parent.basedir}</air.main.basedir>
</properties>

<dependencies>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<scope>runtime</scope>
</dependency>

<dependency>
<groupId>com.facebook.airlift</groupId>
<artifactId>bootstrap</artifactId>
</dependency>

<dependency>
<groupId>com.facebook.airlift</groupId>
<artifactId>json</artifactId>
</dependency>

<dependency>
<groupId>com.facebook.airlift</groupId>
<artifactId>log</artifactId>
</dependency>

<dependency>
<groupId>com.facebook.airlift</groupId>
<artifactId>configuration</artifactId>
</dependency>

<dependency>
<groupId>com.google.inject</groupId>
<artifactId>guice</artifactId>
</dependency>

<dependency>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
<optional>true</optional>
</dependency>

<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>

<dependency>
<groupId>javax.inject</groupId>
<artifactId>javax.inject</artifactId>
</dependency>

<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-spi</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-common</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>io.airlift</groupId>
<artifactId>units</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>io.airlift</groupId>
<artifactId>slice</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-main</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-analyzer</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-parser</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-math3</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.facebook.presto.plugin.clp;

import com.facebook.presto.common.type.Type;
import com.facebook.presto.spi.ColumnHandle;
import com.facebook.presto.spi.ColumnMetadata;
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;

import java.util.Objects;

import static com.google.common.base.MoreObjects.toStringHelper;

public class ClpColumnHandle
implements ColumnHandle
{
private final String columnName;
private final String originalColumnName;
private final Type columnType;
private final boolean nullable;

@JsonCreator
public ClpColumnHandle(
@JsonProperty("columnName") String columnName,
@JsonProperty("originalColumnName") String originalColumnName,
@JsonProperty("columnType") Type columnType,
@JsonProperty("nullable") boolean nullable)
{
this.columnName = columnName;
this.originalColumnName = originalColumnName;
this.columnType = columnType;
this.nullable = nullable;
}

public ClpColumnHandle(String columnName, Type columnType, boolean nullable)
{
this(columnName, columnName, columnType, nullable);
}

@JsonProperty
public String getColumnName()
{
return columnName;
}

@JsonProperty
public String getOriginalColumnName()
{
return originalColumnName;
}

@JsonProperty
public Type getColumnType()
{
return columnType;
}

@JsonProperty
public boolean isNullable()
{
return nullable;
}

public ColumnMetadata getColumnMetadata()
{
ColumnMetadata.Builder builder = ColumnMetadata.builder()
.setName(columnName)
.setType(columnType)
.setNullable(nullable);
return builder.build();
}

@Override
public int hashCode()
{
return Objects.hash(columnName, columnType);
}

@Override
public boolean equals(Object obj)
{
if (this == obj) {
return true;
}
if (obj == null || getClass() != obj.getClass()) {
return false;
}
ClpColumnHandle other = (ClpColumnHandle) obj;
return Objects.equals(this.columnName, other.columnName) &&
Objects.equals(this.columnType, other.columnType);
}
Comment on lines +91 to +103
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Revisit equality checks to include all relevant fields.

Only columnName and columnType are considered in equality, but originalColumnName and nullable are omitted. This can lead to inconsistent behaviour.

 return Objects.equals(this.columnName, other.columnName)
-       && Objects.equals(this.columnType, other.columnType);
+       && Objects.equals(this.originalColumnName, other.originalColumnName)
+       && Objects.equals(this.columnType, other.columnType)
+       && this.nullable == other.nullable;
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
@Override
public boolean equals(Object obj)
{
if (this == obj) {
return true;
}
if (obj == null || getClass() != obj.getClass()) {
return false;
}
ClpColumnHandle other = (ClpColumnHandle) obj;
return Objects.equals(this.columnName, other.columnName) &&
Objects.equals(this.columnType, other.columnType);
}
@Override
public boolean equals(Object obj)
{
if (this == obj) {
return true;
}
if (obj == null || getClass() != obj.getClass()) {
return false;
}
ClpColumnHandle other = (ClpColumnHandle) obj;
return Objects.equals(this.columnName, other.columnName)
&& Objects.equals(this.originalColumnName, other.originalColumnName)
&& Objects.equals(this.columnType, other.columnType)
&& this.nullable == other.nullable;
}


@Override
public String toString()
{
return toStringHelper(this)
.add("columnName", columnName)
.add("columnType", columnType)
.add("nullable", nullable)
.toString();
}
}
Loading
Loading