Skip to content

HIVE-29377: Fix inconsistency for partition filter LIKE clause #6240

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wecharyu wants to merge 1 commit into
base: master
Choose a base branch
from
+97 −5
Open

HIVE-29377: Fix inconsistency for partition filter LIKE clause #6240

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions ql/src/test/queries/clientpositive/ppd_like_filter.q
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,8 @@ select * from test_tbl where b like '___';
select * from test_tbl where b like '%%%';
select * from test_tbl where b like '%\\\\%';



select * from test_tbl where b like 'a.*';
select * from test_tbl where b like 'd.\%.*';
select * from test_tbl where b like 'd.\\%.*';
select * from test_tbl where b like 'd.\\\\%.*';
select * from test_tbl where b like '.\\_.*';
48 changes: 48 additions & 0 deletions ql/src/test/results/clientpositive/llap/ppd_like_filter.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,3 +256,51 @@ POSTHOOK: type: QUERY
POSTHOOK: Input: default@test_tbl
POSTHOOK: Input: default@test_tbl@b=d_%5C%25ae
#### A masked pattern was here ####
PREHOOK: query: select * from test_tbl where b like 'a.*'
Copy link
Contributor

@zhangbutao zhangbutao Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The conversion rule for this PR is: if the character .* is encountered in the partition predicate of Hive SQL, it is mapped to % in MySQL (matching zero or more characters), then the partition predicate a.* should be converted to a% in MySQL, and the direct SQL would query two partition entries, namely abc and af%.

However, why is the query result from q.out empty here?

Copy link
Contributor Author

@wecharyu wecharyu Dec 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The POSTHOOK Input shows that it got matching partitions from HMS, but hive optimizer PartitionConditionRemover will filter partitions again by original pattern like a.*, and does not convert to a% so it will filter out the matching partitions.

POSTHOOK: Input: default@test_tbl
POSTHOOK: Input: default@test_tbl@b=abc
POSTHOOK: Input: default@test_tbl@b=af%25

Copy link
Contributor

@zhangbutao zhangbutao Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So there is an issue here. The HMS can return partition information based on like .*, but the PartitionConditionRemover considers these partitions invalid. This seems inconsistent—we need to maintain consistency. cc @dengzhhu653 @deniskuzZ

Additionally, I am not sure if .* is a commonly used expression in SQL LIKE filtering. Could you give examples of similar LIKE matching usages in other sql engines?

Copy link
Contributor Author

@wecharyu wecharyu Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SQL engines usually not use .* as a filter, but HMS thrift APIs support it as a pattern, I think it's because HMS uses JDO matches() for like filter.

And Spark is using the .* pattern for partition filter like here:
https://github.com/apache/spark/blob/8cc2e4677d596bada67d05792f6fd896e5fc640f/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala#L827-L834

Copy link
Member

@dengzhhu653 dengzhhu653 Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

per my understanding, we don't have a single pattern for both jdo and the direct sql, jdo accepts .* (Java-type pattern) for any sequences and the sql allows _% for the same game. When comes to HMS API, the user needs to take care of the pattern, whether it should be a Java-type pattern or sql pattern.

Not sure why PartitionConditionRemover would remove the partition condition, given the pruned partitions from Metastore is not empty.

Copy link
Contributor Author

@wecharyu wecharyu Jan 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why PartitionConditionRemover would remove the partition condition, given the pruned partitions from Metastore is not empty.

We can make a new topic on the difference in hive ql, maybe we can first fix the HMS inconsistency in JDO and Direct SQL here.

PREHOOK: type: QUERY
PREHOOK: Input: default@test_tbl
PREHOOK: Input: default@test_tbl@b=abc
PREHOOK: Input: default@test_tbl@b=af%25
#### A masked pattern was here ####
POSTHOOK: query: select * from test_tbl where b like 'a.*'
POSTHOOK: type: QUERY
POSTHOOK: Input: default@test_tbl
POSTHOOK: Input: default@test_tbl@b=abc
POSTHOOK: Input: default@test_tbl@b=af%25
#### A masked pattern was here ####
PREHOOK: query: select * from test_tbl where b like 'd.\%.*'
PREHOOK: type: QUERY
PREHOOK: Input: default@test_tbl
#### A masked pattern was here ####
POSTHOOK: query: select * from test_tbl where b like 'd.\%.*'
POSTHOOK: type: QUERY
POSTHOOK: Input: default@test_tbl
#### A masked pattern was here ####
PREHOOK: query: select * from test_tbl where b like 'd.\\%.*'
PREHOOK: type: QUERY
PREHOOK: Input: default@test_tbl
#### A masked pattern was here ####
POSTHOOK: query: select * from test_tbl where b like 'd.\\%.*'
POSTHOOK: type: QUERY
POSTHOOK: Input: default@test_tbl
#### A masked pattern was here ####
PREHOOK: query: select * from test_tbl where b like 'd.\\\\%.*'
PREHOOK: type: QUERY
PREHOOK: Input: default@test_tbl
PREHOOK: Input: default@test_tbl@b=d_%5C%25ae
#### A masked pattern was here ####
POSTHOOK: query: select * from test_tbl where b like 'd.\\\\%.*'
POSTHOOK: type: QUERY
POSTHOOK: Input: default@test_tbl
POSTHOOK: Input: default@test_tbl@b=d_%5C%25ae
#### A masked pattern was here ####
PREHOOK: query: select * from test_tbl where b like '.\\_.*'
PREHOOK: type: QUERY
PREHOOK: Input: default@test_tbl
PREHOOK: Input: default@test_tbl@b=d_%5C%25ae
#### A masked pattern was here ####
POSTHOOK: query: select * from test_tbl where b like '.\\_.*'
POSTHOOK: type: QUERY
POSTHOOK: Input: default@test_tbl
POSTHOOK: Input: default@test_tbl@b=d_%5C%25ae
#### A masked pattern was here ####
9 changes: 9 additions & 0 deletions ...tore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DatabaseProduct.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -856,6 +856,15 @@ public int getMaxRows(int batch, int paramSize) {
return batch;
}

public String escapeClause() {
if (isMYSQL()) {
// MySQL does not support ESCAPE '\'
return " ESCAPE '\\\\' ";
} else {
return " ESCAPE '\\' ";
}
}

// This class implements the Configurable interface for the benefit
// of "plugin" instances created via reflection (see invocation of
// ReflectionUtils.newInstance in method determineDatabaseProduct)
Expand Down
38 changes: 35 additions & 3 deletions ...e/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1528,7 +1528,11 @@ public void visit(LeafNode node) throws MetaException {
String tableValue = "\"FILTER" + partColIndex + "\".\"PART_KEY_VAL\"";

if (node.isReverseOrder && nodeValue != null) {
params.add(nodeValue);
if (node.operator == Operator.LIKE) {
params.add(makeDirectFilterForLike(nodeValue.toString()));
} else {
params.add(nodeValue);
}
}
String tableColumn = tableValue;
if ((colType != FilterType.String) && (!isDefaultPartition)) {
Expand Down Expand Up @@ -1560,12 +1564,16 @@ public void visit(LeafNode node) throws MetaException {
}

if (!node.isReverseOrder && nodeValue != null) {
params.add(nodeValue);
if (node.operator == Operator.LIKE) {
params.add(makeDirectFilterForLike(nodeValue.toString()));
} else {
params.add(nodeValue);
}
}

// The following syntax is required for using LIKE clause wildcards '_' and '%' as literals.
if (node.operator == Operator.LIKE) {
nodeValue0 = nodeValue0 + " ESCAPE '\\' ";
nodeValue0 = nodeValue0 + dbType.escapeClause();
}
String filter = node.isReverseOrder
? nodeValue0 + " " + node.operator.getSqlOp() + " " + tableValue
Expand Down Expand Up @@ -1607,6 +1615,30 @@ public void visit(LeafNode node) throws MetaException {

filterBuffer.append("(" + filter + ")");
}

private String makeDirectFilterForLike(String likePattern) {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < likePattern.length(); i++) {
char n = likePattern.charAt(i);
// MySQL default ESCAPE is "\\\\", but prepared statement parameters
// are transmitted as raw values, so only "\\" is needed.
if (dbType.isMYSQL() && n == '\\' && i + 1 < likePattern.length()
&& likePattern.charAt(i + 1) == '\\') {
i++;
}
if (n == '.') {
if (i + 1 < likePattern.length() && likePattern.charAt(i + 1) == '*') {
sb.append("%");
i++;
} else {
sb.append("_");
}
continue;
}
sb.append(n);
}
return sb.toString();
}
}

/**
Expand Down