Skip to content

Commit 8f66aef

Browse files
jingz-dbHyukjinKwon
authored andcommitted
[SPARK-50908][SS][PYTHON] Disable flaky TTL test in test_pandas_transform_with_state.py
### What changes were proposed in this pull request? Disable the flaky TTL suite for TransformWithStateInPandas. ### Why are the changes needed? The suite is flaky on CI because for this test case, we want to test on scenarios of one key expires and the other key is still in the state store. The tricky thing is that we don't have equivalent of `StreamManualClock` in Scala and we have to manually set a Thread.sleep() time duration such that the TTL expires after a certain time duration while the other key does not. As the batch duration varies a lot on CI, this suite is flaky. We decided to disable the suite until we find a holistic fix. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Tested on CI. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #49634 from jingz-db/fix-ttl-suite. Authored-by: jingz-db <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
1 parent 2b657fe commit 8f66aef

File tree

1 file changed

+2
-3
lines changed

1 file changed

+2
-3
lines changed

python/pyspark/sql/tests/pandas/test_pandas_transform_with_state.py

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -330,9 +330,8 @@ def check_results(batch_df, batch_id):
330330
SimpleTTLStatefulProcessor(), check_results, False, "processingTime"
331331
)
332332

333-
@unittest.skipIf(
334-
"COVERAGE_PROCESS_START" in os.environ, "Flaky with coverage enabled, skipping for now."
335-
)
333+
# TODO SPARK-50908 holistic fix for TTL suite
334+
@unittest.skip("test is flaky and it is only a timing issue, skipping until we can resolve")
336335
def test_value_state_ttl_expiration(self):
337336
def check_results(batch_df, batch_id):
338337
if batch_id == 0:

0 commit comments

Comments
 (0)