forked from prestodb/presto
-
Notifications
You must be signed in to change notification settings - Fork 3
feat: Add CLP connector #3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 121 commits
Commits
Show all changes
126 commits
Select commit
Hold shift + click to select a range
ee8c8ab
Add related connector classes for clp_s
wraymo f38f962
add a constructor for ClpPlugin
wraymo 8b71203
add ClpClient; add methods to get all tables
wraymo 3b49f9a
add schema tree implementation
wraymo 57dab45
add basic metadata operations
wraymo 9c94bde
style fix
wraymo 97bc690
a relatively complete metadata implementation
wraymo cdb8c27
fix bugs and add test cases
wraymo 253cf7a
modify logs and add a test case
wraymo 6f5ddba
add decompression
wraymo 4866a10
add record cursor impl
wraymo 409c3e8
store decompressed logs to tmpdir
wraymo 6450e87
and tablelayouthandle and add a basic record cursor test
wraymo bfe9b7e
make iteration predictable
wraymo 383efe9
clean the code
wraymo 8c2574e
fix a test case
wraymo 93570fe
remove unused dependency
wraymo 856a53f
fix bugs to make it work
wraymo 4165e1b
modify tests
wraymo a2fbc13
fix bugs
wraymo 22dbb83
add basic query pushdown
wraymo 8c48969
add test cases
wraymo a408e54
fix build error
wraymo 00e3a62
fix a bug
wraymo c88055c
remove unused import
wraymo fbd02c7
get a functional pushdown impl
wraymo b7a094f
correctly handle string fields and nested fields
wraymo af883cf
remove unused import
wraymo 2ab5892
add query optimizer and fix a bug
wraymo 3ceeca0
add support for cardinality split and like predicates
wraymo c2b997e
temporary workaround
wraymo 3e510f5
update clp connector version
wraymo 476ed3c
add new files
wraymo 22845ee
avoid generating weird LIKE predicates
wraymo e9050b9
a more robust ClpPlanOptimizer
wraymo fc7a497
fix a bug and add LIKE predicate handler
wraymo 5346beb
fix a bug
wraymo 3231507
add array support
wraymo a1511f1
Add a unit test
wraymo 36bddec
Fix bugs
wraymo 9bede0d
make it more robust for logical binaries
wraymo 519fa7f
add projection support
wraymo eeedf2d
add clp module
wraymo 2122ec5
update submodules
wraymo 6e0c386
update submodules
wraymo 9788a01
add clp worker support
wraymo cd076f2
add more methods
wraymo 2342ed5
add more methods
wraymo da67953
modify CMakeLists.txt
wraymo fde90b3
add clp data source support
wraymo 04e8afd
remove duplicate values and add null support
wraymo 4419bb1
add some configuration examples
wraymo 38a4049
add CLP Split support
wraymo d99aefe
fix a bug
wraymo 9dad342
remove duplicate columns for projection
wraymo 49aed85
destroy clp-s process when closing the record cursor
wraymo 6bdec9d
remove unused import
wraymo 9813613
add concurrent split support
wraymo 79ac281
close the buffer when closing teh CLP Record curor
wraymo 6d059cf
remove unused file
wraymo 440e7a0
resolve several issues
wraymo c089cdb
add support for reading metadata from mysql
wraymo dddc29b
add support for searching from S3 and reading schema from MySQL
wraymo a81dabb
rename tableMetadataPrefix
wraymo e4c7584
minor fixes
wraymo c02b71f
rename file source to input source
wraymo 1c199da
minor fix
wraymo f7cd4d7
fix mysql runtime error
wraymo 5a227c5
update git submodule path
wraymo 387a3ed
only compile presto_server target
wraymo bd8defe
update worker configurations
wraymo cbe0786
do not treat warnings as errors
wraymo ad9f515
add an s3 coordinator configuration example
wraymo 91b36f0
update velox dependency and modify example configuration files
wraymo 985adb5
remove connectorFactory variable from ClpPlugin
wraymo c77e33b
remove parseSchemaFile from ClpClient
wraymo 9bbbe5c
refactor metadata api, delete uncessary files and refactor the test
wraymo 34bfc3b
apply prestissimo changes
wraymo d431a7a
add test cases for PlanOptimizer
wraymo bcf4096
remove SchemaTree
wraymo 73e6a06
Finished ClP plan optimizer and metadata unit tests, fixed related bug
wraymo e4db337
update protocol code for clp connector
wraymo 20e7613
update presto-clp pom.xml
wraymo 39fe294
fix build issues
2af6106
refactor metadata, add ClpMySQLSplitProvider implementation and unit …
3195c9f
apply presto-native-execution changes
wraymo cf6faad
update ClpMySQLSplitProvider and ClpMySQLMetadataProvider
wraymo c9415c6
apply changes in PrestoToVeloxConnector.cpp
wraymo 0a5178c
update presto-clp description and update split APIs
wraymo ac0e315
advance velox
wraymo 2938392
update presto-native-execution and velox
cf96632
revert presto-native-execution/Makefile back and clean up ClpConfig
wraymo 6fabaa1
remove dependency-reduced-pom.xml
wraymo 19fdd85
revert gitignore back
wraymo 27a3749
remove dependency-reduced-pom.xml
wraymo e173b0a
change set to list for metadata
wraymo 7e0dbf5
apply coderabbit suggestions
wraymo 65ea0a8
remove unused dependencies in pom.xml
wraymo b764f9c
allow like predicate to be transformed to substr match and optimize i…
wraymo 4a6b4ff
clean up the code
wraymo e4b558e
add clp to presto-dpcs and clean up dependencies
wraymo a45341b
remove config examples
wraymo ed0397d
revert some files back
wraymo 9945517
revert presto-native-execution back
wraymo 57a8aa9
remove clp files
wraymo 31816a2
add row type in clp
wraymo 2747ec6
add optimizer logic for row type
wraymo 95a0db1
update unit test cases for row type
wraymo 3dbedd2
revert lowercase change back
wraymo 4b6fd82
support row type for more expressions
wraymo 9a22e2a
test nested types and array case
wraymo 6253650
improve clp docs
wraymo 0201632
fix sql injection issue
wraymo baa1097
add row type to the doc
wraymo a80ad57
improve the docs
wraymo 977dead
remove handlePolymorphicType in ClpClient and support original names …
wraymo ea73a1a
improve comments
wraymo 7bd75ee
rename splitsource and metadatasource
wraymo f5ca7f5
address review comments
wraymo 1b63cbc
use archive_id
wraymo 37e1b4c
add comments in ClpSchemaTree
wraymo 8428517
rename package name and class name
wraymo 5771f1d
add comments in ClpFilterToKqlConverter.java
wraymo 4f042a2
fix typo
wraymo 943d502
add comments
wraymo f36b5b8
remove the empty line
wraymo File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| <?xml version="1.0"?> | ||
| <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
| <modelVersion>4.0.0</modelVersion> | ||
|
|
||
| <parent> | ||
| <groupId>com.facebook.presto</groupId> | ||
| <artifactId>presto-root</artifactId> | ||
| <version>0.292-SNAPSHOT</version> | ||
| </parent> | ||
|
|
||
| <groupId>com.yscope.presto</groupId> | ||
| <artifactId>presto-clp</artifactId> | ||
| <description>Presto - CLP Connector</description> | ||
| <packaging>presto-plugin</packaging> | ||
|
|
||
| <properties> | ||
| <air.main.basedir>${project.parent.basedir}</air.main.basedir> | ||
| </properties> | ||
|
|
||
| <dependencies> | ||
| <dependency> | ||
| <groupId>mysql</groupId> | ||
| <artifactId>mysql-connector-java</artifactId> | ||
| <scope>runtime</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.facebook.airlift</groupId> | ||
| <artifactId>bootstrap</artifactId> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.facebook.airlift</groupId> | ||
| <artifactId>json</artifactId> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.facebook.airlift</groupId> | ||
| <artifactId>log</artifactId> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.facebook.airlift</groupId> | ||
| <artifactId>configuration</artifactId> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.google.inject</groupId> | ||
| <artifactId>guice</artifactId> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.google.code.findbugs</groupId> | ||
| <artifactId>jsr305</artifactId> | ||
| <optional>true</optional> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.google.guava</groupId> | ||
| <artifactId>guava</artifactId> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>javax.inject</groupId> | ||
| <artifactId>javax.inject</artifactId> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.fasterxml.jackson.core</groupId> | ||
| <artifactId>jackson-annotations</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.facebook.presto</groupId> | ||
| <artifactId>presto-spi</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.facebook.presto</groupId> | ||
| <artifactId>presto-common</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>io.airlift</groupId> | ||
| <artifactId>units</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>io.airlift</groupId> | ||
| <artifactId>slice</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.testng</groupId> | ||
| <artifactId>testng</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.facebook.presto</groupId> | ||
| <artifactId>presto-main</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.facebook.presto</groupId> | ||
| <artifactId>presto-analyzer</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.h2database</groupId> | ||
| <artifactId>h2</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.facebook.presto</groupId> | ||
| <artifactId>presto-parser</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.commons</groupId> | ||
| <artifactId>commons-math3</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| </dependencies> | ||
| </project> |
114 changes: 114 additions & 0 deletions
114
presto-clp/src/main/java/com/yscope/presto/ClpColumnHandle.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| /* | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package com.yscope.presto; | ||
|
|
||
| import com.facebook.presto.common.type.Type; | ||
| import com.facebook.presto.spi.ColumnHandle; | ||
| import com.facebook.presto.spi.ColumnMetadata; | ||
| import com.fasterxml.jackson.annotation.JsonCreator; | ||
| import com.fasterxml.jackson.annotation.JsonProperty; | ||
|
|
||
| import java.util.Objects; | ||
|
|
||
| import static com.google.common.base.MoreObjects.toStringHelper; | ||
|
|
||
| public class ClpColumnHandle | ||
| implements ColumnHandle | ||
| { | ||
| private final String columnName; | ||
| private final String originalColumnName; | ||
| private final Type columnType; | ||
| private final boolean nullable; | ||
|
|
||
| @JsonCreator | ||
| public ClpColumnHandle( | ||
| @JsonProperty("columnName") String columnName, | ||
| @JsonProperty("originalColumnName") String originalColumnName, | ||
| @JsonProperty("columnType") Type columnType, | ||
| @JsonProperty("nullable") boolean nullable) | ||
| { | ||
| this.columnName = columnName; | ||
| this.originalColumnName = originalColumnName; | ||
| this.columnType = columnType; | ||
| this.nullable = nullable; | ||
| } | ||
|
|
||
| public ClpColumnHandle(String columnName, Type columnType, boolean nullable) | ||
| { | ||
| this(columnName, columnName, columnType, nullable); | ||
| } | ||
|
|
||
| @JsonProperty | ||
| public String getColumnName() | ||
| { | ||
| return columnName; | ||
| } | ||
|
|
||
| @JsonProperty | ||
| public String getOriginalColumnName() | ||
| { | ||
| return originalColumnName; | ||
| } | ||
|
|
||
| @JsonProperty | ||
| public Type getColumnType() | ||
| { | ||
| return columnType; | ||
| } | ||
|
|
||
| @JsonProperty | ||
| public boolean isNullable() | ||
| { | ||
| return nullable; | ||
| } | ||
|
|
||
| public ColumnMetadata getColumnMetadata() | ||
| { | ||
| ColumnMetadata.Builder builder = ColumnMetadata.builder() | ||
| .setName(columnName) | ||
| .setType(columnType) | ||
| .setNullable(nullable); | ||
| return builder.build(); | ||
| } | ||
|
|
||
| @Override | ||
| public int hashCode() | ||
| { | ||
| return Objects.hash(columnName, columnType); | ||
| } | ||
|
|
||
| @Override | ||
| public boolean equals(Object obj) | ||
| { | ||
| if (this == obj) { | ||
| return true; | ||
| } | ||
| if (obj == null || getClass() != obj.getClass()) { | ||
| return false; | ||
| } | ||
| ClpColumnHandle other = (ClpColumnHandle) obj; | ||
| return Objects.equals(this.columnName, other.columnName) && | ||
| Objects.equals(this.columnType, other.columnType); | ||
| } | ||
|
|
||
| @Override | ||
| public String toString() | ||
| { | ||
| return toStringHelper(this) | ||
| .add("columnName", columnName) | ||
| .add("columnType", columnType) | ||
| .add("nullable", nullable) | ||
| .toString(); | ||
| } | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.