Skip to content
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
88de8b7
first classes
drewdaemon Sep 20, 2024
9588aca
syntax error fix
drewdaemon Sep 20, 2024
5a5c383
update some tests
drewdaemon Sep 20, 2024
590e0e4
register the writeable
drewdaemon Sep 20, 2024
2d0b623
implement reverse and add tests
drewdaemon Sep 20, 2024
5a588fe
add unit tests
drewdaemon Sep 23, 2024
19c60ee
update test
drewdaemon Sep 23, 2024
6a7bf6b
add some docs
drewdaemon Sep 23, 2024
23bc6ee
more docs
drewdaemon Sep 23, 2024
03e9c12
fix some docs
drewdaemon Sep 23, 2024
67b228f
add capability test gating
drewdaemon Sep 23, 2024
647902a
add serialization test
drewdaemon Sep 23, 2024
bceb604
update instruction manual
drewdaemon Sep 23, 2024
27716dd
Update docs/changelog/113297.yaml
drewdaemon Sep 23, 2024
234b795
update labeling instructions
drewdaemon Sep 23, 2024
f1c4694
Merge branch '98545/reverse-function' of https://github.com/drewdaemo…
drewdaemon Sep 23, 2024
d900561
add an example
drewdaemon Sep 23, 2024
2c1bbf4
docs update
drewdaemon Sep 23, 2024
39f2e1b
formatting fix
drewdaemon Sep 23, 2024
ead4a09
Merge branch 'main' of https://github.com/elastic/elasticsearch into …
drewdaemon Sep 23, 2024
0cfde3d
Delete docs/changelog/113297.yaml
drewdaemon Sep 24, 2024
ab49f66
Update docs/changelog/113297.yaml
drewdaemon Sep 24, 2024
4e0d422
remove local, sort registry
drewdaemon Sep 24, 2024
60d670e
Merge branch '98545/reverse-function' of https://github.com/drewdaemo…
drewdaemon Sep 24, 2024
364d357
Reverse inherits from EsqlScalarFunction
drewdaemon Sep 25, 2024
1c0268b
fix style
drewdaemon Sep 25, 2024
60b36bb
add yml test
drewdaemon Sep 25, 2024
627144c
add extensive CSV test
drewdaemon Sep 25, 2024
8f11f09
add nested function test case
drewdaemon Sep 25, 2024
f5007e7
a couple more CSV tests
drewdaemon Sep 25, 2024
5bfcd0a
Update capabilities for rest test
drewdaemon Sep 25, 2024
257c9f9
Apparently it's like this
nik9000 Sep 25, 2024
7c0b1aa
baseline grapheme cluster
drewdaemon Sep 26, 2024
fa2fc0b
add grapheme cluster support
drewdaemon Sep 26, 2024
791d767
add the super-fast path for utf8
drewdaemon Sep 26, 2024
11046e4
spotless
drewdaemon Sep 26, 2024
f644990
add some crazy grapheme clusters
drewdaemon Sep 26, 2024
a891a08
generate docs
drewdaemon Sep 26, 2024
4b558c9
use UnaryScalarFunction
drewdaemon Sep 27, 2024
26d5a3c
reverse tests in a loop
drewdaemon Sep 27, 2024
de7446f
update unit test
drewdaemon Sep 27, 2024
3160be0
rework reverse tests
drewdaemon Sep 27, 2024
d8e78db
update docs
drewdaemon Sep 27, 2024
d7b5a28
Merge branch 'main' of https://github.com/elastic/elasticsearch into …
drewdaemon Sep 27, 2024
8ce4406
update order
drewdaemon Sep 27, 2024
8cad604
update test
drewdaemon Sep 27, 2024
28c8d8b
Merge branch 'main' of https://github.com/elastic/elasticsearch into …
drewdaemon Sep 30, 2024
e21f87f
narrow tests
nik9000 Sep 30, 2024
ad7c68f
utilize UnaryScalarFunction methods where possible
drewdaemon Oct 3, 2024
6c09607
move reverseArray to the utils class
drewdaemon Oct 3, 2024
b512026
spotless
drewdaemon Oct 3, 2024
247495a
Merge branch 'main' of https://github.com/elastic/elasticsearch into …
drewdaemon Oct 3, 2024
7763155
update serialization tests
drewdaemon Oct 3, 2024
9041899
Merge branch 'main' of https://github.com/elastic/elasticsearch into …
drewdaemon Oct 4, 2024
43a63e7
update evaluator call signature
drewdaemon Oct 4, 2024
9c11030
spotless
drewdaemon Oct 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/changelog/113297.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
pr: 113297
summary: "[ES|QL] add reverse function"
area: ES|QL
type: enhancement
issues: []
5 changes: 5 additions & 0 deletions docs/reference/esql/functions/description/reverse.asciidoc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

22 changes: 22 additions & 0 deletions docs/reference/esql/functions/examples/reverse.asciidoc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

37 changes: 37 additions & 0 deletions docs/reference/esql/functions/kibana/definition/reverse.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion docs/reference/esql/functions/kibana/docs/mv_sum.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 10 additions & 0 deletions docs/reference/esql/functions/kibana/docs/reverse.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

15 changes: 15 additions & 0 deletions docs/reference/esql/functions/layout/reverse.asciidoc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions docs/reference/esql/functions/parameters/reverse.asciidoc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions docs/reference/esql/functions/signature/reverse.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions docs/reference/esql/functions/string-functions.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
* <<esql-ltrim>>
* <<esql-repeat>>
* <<esql-replace>>
* <<esql-reverse>>
* <<esql-right>>
* <<esql-rtrim>>
* <<esql-space>>
Expand All @@ -38,6 +39,7 @@ include::layout/locate.asciidoc[]
include::layout/ltrim.asciidoc[]
include::layout/repeat.asciidoc[]
include::layout/replace.asciidoc[]
include::layout/reverse.asciidoc[]
include::layout/right.asciidoc[]
include::layout/rtrim.asciidoc[]
include::layout/space.asciidoc[]
Expand Down
10 changes: 10 additions & 0 deletions docs/reference/esql/functions/types/reverse.asciidoc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ double pi()
"double pow(base:double|integer|long|unsigned_long, exponent:double|integer|long|unsigned_long)"
"keyword repeat(string:keyword|text, number:integer)"
"keyword replace(string:keyword|text, regex:keyword|text, newString:keyword|text)"
"keyword|text reverse(str:keyword|text)"
"keyword right(string:keyword|text, length:integer)"
"double|integer|long|unsigned_long round(number:double|integer|long|unsigned_long, ?decimals:integer)"
"keyword|text rtrim(string:keyword|text)"
Expand Down Expand Up @@ -199,6 +200,7 @@ pi |null |null
pow |[base, exponent] |["double|integer|long|unsigned_long", "double|integer|long|unsigned_long"] |["Numeric expression for the base. If `null`\, the function returns `null`.", "Numeric expression for the exponent. If `null`\, the function returns `null`."]
repeat |[string, number] |["keyword|text", integer] |[String expression., Number times to repeat.]
replace |[string, regex, newString] |["keyword|text", "keyword|text", "keyword|text"] |[String expression., Regular expression., Replacement string.]
reverse |str |"keyword|text" |String expression. If `null`, the function returns `null`.
right |[string, length] |["keyword|text", integer] |[The string from which to returns a substring., The number of characters to return.]
round |[number, decimals] |["double|integer|long|unsigned_long", integer] |["The numeric value to round. If `null`\, the function returns `null`.", "The number of decimal places to round to. Defaults to 0. If `null`\, the function returns `null`."]
rtrim |string |"keyword|text" |String expression. If `null`, the function returns `null`.
Expand Down Expand Up @@ -329,6 +331,7 @@ pi |Returns {wikipedia}/Pi[Pi], the ratio of a circle's circumference
pow |Returns the value of `base` raised to the power of `exponent`.
repeat |Returns a string constructed by concatenating `string` with itself the specified `number` of times.
replace |The function substitutes in the string `str` any match of the regular expression `regex` with the replacement string `newStr`.
reverse |Returns a new string representing the input string in reverse order.
right |Return the substring that extracts 'length' chars from 'str' starting from the right.
round |Rounds a number to the specified number of decimal places. Defaults to 0, which returns the nearest integer. If the precision is a negative number, rounds to the number of digits left of the decimal point.
rtrim |Removes trailing whitespaces from a string.
Expand Down Expand Up @@ -461,6 +464,7 @@ pi |double
pow |double |[false, false] |false |false
repeat |keyword |[false, false] |false |false
replace |keyword |[false, false, false] |false |false
reverse |"keyword|text" |false |false |false
right |keyword |[false, false] |false |false
round |"double|integer|long|unsigned_long" |[false, true] |false |false
rtrim |"keyword|text" |false |false |false
Expand Down Expand Up @@ -536,5 +540,5 @@ required_capability: meta
meta functions | stats a = count(*), b = count(*), c = count(*) | mv_expand c;

a:long | b:long | c:long
119 | 119 | 119
120 | 120 | 120
;
Original file line number Diff line number Diff line change
Expand Up @@ -1194,6 +1194,101 @@ a:keyword | upper:keyword | lower:keyword
π/2 + a + B + Λ ºC | Π/2 + A + B + Λ ºC | π/2 + a + b + λ ºc
;

reverse
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add more complex csv tests. For example:

  • from employees | sort emp_no | eval jobs_reversed = reverse(job_positions) | keep job*, emp_no | limit 5 which deals with multi-values and nulls
  • from employees | sort emp_no | eval name_reversed = reverse(reverse(first_name)), eq = name_reversed == first_name | keep emp_no, first_name, name_reversed, eq | limit 5 that uses nested function calls.
  • row x = null | eval y = reverse(x)
  • row x = 1 | eval y = reverse((null + 1)::string)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For csv tests, I forgot we now have books index that also has text field type (employees and row only have keyword field types). Please, look into adding couple of tests with books index as well. Thanks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in the following commits.

627144c
8f11f09
f5007e7

required_capability: fn_reverse
from employees | sort emp_no | eval name_reversed = REVERSE(first_name) | keep emp_no, first_name, name_reversed | limit 1;

emp_no:integer | first_name:keyword | name_reversed:keyword
10001 | Georgi | igroeG
;

reverseRow
required_capability: fn_reverse
// tag::reverse[]
ROW message = "Some Text" | EVAL message_reversed = REVERSE(message);
// end::reverse[]

// tag::reverse-result[]
message:keyword | message_reversed:keyword
Some Text | txeT emoS
// end::reverse-result[]
;

reverseEmoji
required_capability: fn_reverse
// tag::reverseEmoji[]
ROW message = "😂😁😅😐🥺😢😭" | EVAL message_reversed = REVERSE(message);
// end::reverseEmoji[]

// tag::reverseEmoji-result[]
message:keyword | message_reversed:keyword
😂😁😅😐🥺😢😭 | 😭😢🥺😐😅😁😂
// end::reverseEmoji-result[]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice - small comment, I would use different emoticons to better signal visually the reversal: 🏡🚌✈ becomes ✈ 🚌🏡

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally add also a string that combines and intertwines regular chars + emoticons + extended ones (こんにちは) and check they are properly reversed (namely that the encoding is preserved).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice - small comment, I would use different emoticons to better signal visually the reversal

I think of the current ones like a sort of easter egg in the docs... you have to squint a bit and then you're rewarded with "getting it."

That said, I defer to you — LMK

cc @leemthompo in case he has input

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think Costin's right that it would be easier with a bit more visual variance in the emojis but we're knee deep in nit territory here 😆

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually Liam this is the most important conversation going on in this PR right now 😆

Okay, I'll take another stab at emoji art 🖼️

Copy link
Contributor Author

@drewdaemon drewdaemon Sep 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My battle for these emojis is met with opposition at every turn... first it was our handling of unicode graphemes and now it's a custom font on the docs website... will I prevail?....

Screen.Recording.2024-09-26.at.4.01.39.PM.mov

;

reverseMultiValue
required_capability: fn_reverse
FROM employees | SORT emp_no | EVAL jobs_reversed = REVERSE(job_positions) | KEEP job*, emp_no | LIMIT 5;

warning:Line 1:53: evaluation of [REVERSE(job_positions)] failed, treating result as null. Only first 20 failures recorded.
warning:Line 1:53: java.lang.IllegalArgumentException: single-value function encountered multi-value

job_positions:keyword | jobs_reversed:keyword | emp_no:integer
["Accountant", "Senior Python Developer"] | null | 10001
Senior Team Lead | daeL maeT roineS | 10002
null | null | 10003
[Head Human Resources, Reporting Analyst, Support Engineer, Tech Lead] | null | 10004
null | null | 10005
;

reverseNested
required_capability: fn_reverse
FROM employees | SORT emp_no | EVAL name_reversed = REVERSE(REVERSE(first_name)), eq = name_reversed == first_name | KEEP first_name, name_reversed, eq, emp_no | LIMIT 5;

first_name:keyword | name_reversed:keyword | eq:boolean | emp_no:integer
Georgi | Georgi | true | 10001
Bezalel | Bezalel | true | 10002
Parto | Parto | true | 10003
Chirstian | Chirstian | true | 10004
Kyoichi | Kyoichi | true | 10005
;

reverseRowNull
required_capability: fn_reverse
ROW x = null | EVAL y = REVERSE(x);

x:null | y:null
null | null
;


reverseRowInlineCastWithNull
required_capability: fn_reverse
ROW x = 1 | EVAL y = REVERSE((null + 1)::string);

x:integer | y:string
1 | null
;

reverseWithTextFields
required_capability: fn_reverse
FROM books
| EVAL title_reversed = REVERSE(title), author_reversed_twice = REVERSE(REVERSE(author)), eq = author_reversed_twice == author
| KEEP title, title_reversed, author, author_reversed_twice, eq, book_no
| SORT book_no
| LIMIT 2;

warning:Line 2:25: evaluation of [REVERSE(title)] failed, treating result as null. Only first 20 failures recorded.
warning:Line 2:25: java.lang.IllegalArgumentException: single-value function encountered multi-value
warning:Line 2:73: evaluation of [REVERSE(author)] failed, treating result as null. Only first 20 failures recorded.
warning:Line 2:73: java.lang.IllegalArgumentException: single-value function encountered multi-value

title:text | title_reversed:text | author:text | author_reversed_twice:text | eq:boolean | book_no:keyword
The brothers Karamazov | vozamaraK srehtorb ehT | Fyodor Dostoevsky | Fyodor Dostoevsky | true | 1211
Realms of Tolkien: Images of Middle-earth | htrae-elddiM fo segamI :neikloT fo smlaeR | J. R. R. Tolkien | J. R. R. Tolkien | true | 1463
;


values
required_capability: agg_values

Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading