Skip to content

Commit eadeca2

Browse files
yuancuxinyual
yuancu
and
xinyual
authored
Migrate existing UDFs to PPLFuncImpTable (#3576)
* refactor using timestmap Signed-off-by: xinyual <xinyual@amazon.com> * refactor for timestmap Signed-off-by: xinyual <xinyual@amazon.com> * transfer basic functions Signed-off-by: xinyual <xinyual@amazon.com> * add functions Signed-off-by: xinyual <xinyual@amazon.com> * Refactor math UDFs Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor text UDFs Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix function name resolving when system using Turkic languages - Problem: FunctionName.of(UPPER_CASE_I) could not be properly resolved since I is converted to dotless ı Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor date part, date/time manipulation UDFs - date part: hour, hour_of_day, month_of_year, month, dayofweek, day_of_week, day, dayofyear, day_of_year, dayofmonth, day_of_month, minute_of_day, minute_of_hour, minute, quarter, second, second_of_minute, microsecond - date/time manipulation: addtime, adddate, date_sub, date_add, subtime, subdate, extract Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor current, name, format UDFs Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor convert tz, date/timestamp diff UDFs Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor date / time functions with non-date/time arguments by creating an adapt function to convert an expr function to a UDF Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor datetime, sec_to_time, time_to_sec, sysdate, timestampadd, timediff UDFs Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor week, nullif, ifnull, if UDFs Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Remove BuiltinFunctionUtils and unnecessary UDFs Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Patch: throw proper semantic errors for date_part-related udfs Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix bugs in datetime UDFs Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor: move all udf to org.opensearch.sql.expression.function.udf Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * use timestamp to replace post process Signed-off-by: xinyual <xinyual@amazon.com> * Add more ITs for last day & restore Prometheus settings Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix euler with UDF instead of directly returning a literal to match its logical plan Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Reimplment last_day UDF with expr methods Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Ignore tests related to parameter validaton temporarily since it's in another PR Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Deprecate timestamp string as data context for function properties Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor: unify datetime udf structures Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Apply spotless & fix zone id acquirement Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix unixtimestamp from double when locale uses non-arabic numbers Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Box operands so that relection calls match signatures containing non-primitive types Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Use system default time zone when restoring function properties Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor return types Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Use builtin case and coalesce to implement if, ifnull, nullif UDFs Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Replace replace UDF with builtin implementations & add docs for replace, locate Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Correct locate function documents Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Separate adddate and date_add implementations for readability Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Add class documentations for UDFs Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Create PPLReturnTypes class to store common return types that it not covered in calcite ReturnTypes class Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Remove udf operators with the same implementations but different names Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor: Move type comparison earlier in Add/Sub Date function implementation Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor: update add/sub time's implementation with adaptExprMethodWithPropertiesToUDF Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor: Move type comparison earlier in last day function implementation Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor: implement date function with adaptExprMethodToUDF Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor: Move type comparison earlier in extract function implementation Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor: Move type comparison earlier in format function implementation Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor: Move type comparison earlier in timestampdiff function implementation Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor: Move type comparison earlier in weekday function implementation Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor: split add and sub date to 2 static functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Eliminate convertRelDataTypeToSqlTypeName API, replacing it with ExprType comparision Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor: pass expr values to udf implementations instead of passing java object and expr type separately Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor: unify types in udf to use expr types (deprecating SqlTypeName) Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor: change return types of UDF implementations to their actual return types (instead of Object) Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Add a test to unix_timestamp with microseconds Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Minor: fix calcite datetime tests Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Replace sqrt udf with a call to calcite's power(x, 0.5) Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Replace locate udf with SqlStdOperatorTable.POSITION Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix sqrt logical plan test Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix expectedSparkSql in sqrt test Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: xinyual <xinyual@amazon.com> Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Co-authored-by: xinyual <xinyual@amazon.com>
1 parent 1c046a2 commit eadeca2

105 files changed

Lines changed: 3584 additions & 4465 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎core/src/main/java/org/opensearch/sql/calcite/CalciteRexNodeVisitor.java‎

Lines changed: 4 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -9,14 +9,11 @@
99
import static org.apache.calcite.sql.SqlKind.AS;
1010
import static org.opensearch.sql.ast.expression.SpanUnit.NONE;
1111
import static org.opensearch.sql.ast.expression.SpanUnit.UNKNOWN;
12-
import static org.opensearch.sql.calcite.utils.BuiltinFunctionUtils.VARCHAR_FORCE_NULLABLE;
13-
import static org.opensearch.sql.calcite.utils.UserDefinedFunctionUtils.TransferUserDefinedFunction;
1412

1513
import java.math.BigDecimal;
1614
import java.util.ArrayList;
1715
import java.util.Collections;
1816
import java.util.List;
19-
import java.util.stream.Collectors;
2017
import lombok.RequiredArgsConstructor;
2118
import org.apache.calcite.rel.RelNode;
2219
import org.apache.calcite.rel.type.RelDataType;
@@ -25,7 +22,6 @@
2522
import org.apache.calcite.rex.RexCall;
2623
import org.apache.calcite.rex.RexNode;
2724
import org.apache.calcite.sql.SqlIntervalQualifier;
28-
import org.apache.calcite.sql.SqlOperator;
2925
import org.apache.calcite.sql.fun.SqlStdOperatorTable;
3026
import org.apache.calcite.sql.type.SqlTypeName;
3127
import org.apache.calcite.util.DateString;
@@ -60,15 +56,14 @@
6056
import org.opensearch.sql.ast.expression.subquery.ScalarSubquery;
6157
import org.opensearch.sql.ast.tree.UnresolvedPlan;
6258
import org.opensearch.sql.calcite.type.ExprSqlType;
63-
import org.opensearch.sql.calcite.udf.datetimeUDF.PostprocessDateToStringFunction;
64-
import org.opensearch.sql.calcite.utils.BuiltinFunctionUtils;
6559
import org.opensearch.sql.calcite.utils.OpenSearchTypeFactory;
6660
import org.opensearch.sql.calcite.utils.PlanUtils;
6761
import org.opensearch.sql.common.utils.StringUtils;
6862
import org.opensearch.sql.data.type.ExprType;
6963
import org.opensearch.sql.exception.CalciteUnsupportedException;
7064
import org.opensearch.sql.exception.SemanticCheckException;
7165
import org.opensearch.sql.expression.function.BuiltinFunctionName;
66+
import org.opensearch.sql.expression.function.PPLBuiltinOperators;
7267
import org.opensearch.sql.expression.function.PPLFuncImpTable;
7368

7469
@RequiredArgsConstructor
@@ -220,18 +215,8 @@ public RexNode visitCompare(Compare node, CalcitePlanContext context) {
220215
private RexNode transferCompareForDateRelated(
221216
RexNode candidate, CalcitePlanContext context, boolean whetherCompareByTime) {
222217
if (whetherCompareByTime) {
223-
SqlOperator postToStringNode =
224-
TransferUserDefinedFunction(
225-
PostprocessDateToStringFunction.class,
226-
"PostprocessDateToString",
227-
VARCHAR_FORCE_NULLABLE);
228218
RexNode transferredStringNode =
229-
context.rexBuilder.makeCall(
230-
postToStringNode,
231-
List.of(
232-
candidate,
233-
context.rexBuilder.makeLiteral(
234-
context.functionProperties.getQueryStartClock().instant().toString())));
219+
context.rexBuilder.makeCall(PPLBuiltinOperators.TIMESTAMP, candidate);
235220
return transferredStringNode;
236221
} else {
237222
return candidate;
@@ -351,26 +336,14 @@ public RexNode visitLet(Let node, CalcitePlanContext context) {
351336
@Override
352337
public RexNode visitFunction(Function node, CalcitePlanContext context) {
353338
List<RexNode> arguments =
354-
node.getFuncArgs().stream().map(arg -> analyze(arg, context)).collect(Collectors.toList());
339+
node.getFuncArgs().stream().map(arg -> analyze(arg, context)).toList();
355340
RexNode resolvedNode =
356341
PPLFuncImpTable.INSTANCE.resolveSafe(
357342
context.rexBuilder, node.getFuncName(), arguments.toArray(new RexNode[0]));
358343
if (resolvedNode != null) {
359344
return resolvedNode;
360345
}
361-
// TODO: Remove below code after migrating all functions to PPLFuncImpTable,
362-
// https://github.com/opensearch-project/sql/issues/3524
363-
SqlOperator operator = BuiltinFunctionUtils.translate(node.getFuncName());
364-
List<RexNode> translatedArguments =
365-
BuiltinFunctionUtils.translateArgument(
366-
node.getFuncName(),
367-
arguments,
368-
context,
369-
context.functionProperties.getQueryStartClock().instant().toString());
370-
RelDataType returnType =
371-
BuiltinFunctionUtils.deriveReturnType(
372-
node.getFuncName(), context.rexBuilder, operator, translatedArguments);
373-
return context.rexBuilder.makeCall(returnType, operator, translatedArguments);
346+
throw new IllegalArgumentException("Unsupported operator: " + node.getFuncName());
374347
}
375348

376349
@Override

‎core/src/main/java/org/opensearch/sql/calcite/udf/conditionUDF/IfFunction.java‎

Lines changed: 0 additions & 25 deletions
This file was deleted.

‎core/src/main/java/org/opensearch/sql/calcite/udf/conditionUDF/IfNullFunction.java‎

Lines changed: 0 additions & 24 deletions
This file was deleted.

‎core/src/main/java/org/opensearch/sql/calcite/udf/conditionUDF/NullIfFunction.java‎

Lines changed: 0 additions & 28 deletions
This file was deleted.

‎core/src/main/java/org/opensearch/sql/calcite/udf/datetimeUDF/ConvertTZFunction.java‎

Lines changed: 0 additions & 33 deletions
This file was deleted.

‎core/src/main/java/org/opensearch/sql/calcite/udf/datetimeUDF/DateAddSubFunction.java‎

Lines changed: 0 additions & 61 deletions
This file was deleted.

‎core/src/main/java/org/opensearch/sql/calcite/udf/datetimeUDF/DateDiffFunction.java‎

Lines changed: 0 additions & 41 deletions
This file was deleted.

‎core/src/main/java/org/opensearch/sql/calcite/udf/datetimeUDF/DateFormatFunction.java‎

Lines changed: 0 additions & 41 deletions
This file was deleted.

‎core/src/main/java/org/opensearch/sql/calcite/udf/datetimeUDF/DateFunction.java‎

Lines changed: 0 additions & 34 deletions
This file was deleted.

0 commit comments

Comments
 (0)