You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[SPARK-46070][SQL] Compile regex pattern in SparkDateTimeUtils.getZoneId outside the hot loop
### What changes were proposed in this pull request?
Compile the regex patterns used in `SparkDateTimeUtils.getZoneId` outside of the method, that can be called for each dataset row..
### Why are the changes needed?
`String.replaceFirst` internally does `Pattern.compile(regex).matcher(this).replaceFirst(replacement)`. `Pattern.compile` is very expensive method, that should not be called in a loop.
When using method like `from_utc_timestamp` with non-literal timezone, the `SparkDateTimeUtils.getZoneId` is called for each loop. In one of my usecases adding `from_utc_timestamp` increased the runtime from 15min to 6h.
### Does this PR introduce _any_ user-facing change?
Performance improvement.
### How was this patch tested?
Existing UTs
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#43976 from tanelk/SPARK-46070_precompile_regex.
Authored-by: Tanel Kiis <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
0 commit comments