Skip to content

Commit 8fe006b

Browse files
committed
[SPARK-54882][PYTHON] Remove legacy PYARROW_IGNORE_TIMEZONE
### What changes were proposed in this pull request? Remove legacy PYARROW_IGNORE_TIMEZONE ### Why are the changes needed? it was added in spark 3.0 for integration with pyarrow 2.0 5e33155 seems it is no longer needed for pyspark tests ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? ci ### Was this patch authored or co-authored using generative AI tooling? no Closes #53660 from zhengruifeng/test_without_PYARROW_IGNORE_TIMEZONE. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>
1 parent 33769b8 commit 8fe006b

File tree

3 files changed

+0
-15
lines changed

3 files changed

+0
-15
lines changed

binder/postBuild

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -39,9 +39,6 @@ fi
3939

4040
pip install plotly "pandas<2.0.0" "pyspark[sql,ml,mllib,pandas_on_spark,connect]$SPECIFIER$VERSION"
4141

42-
# Set 'PYARROW_IGNORE_TIMEZONE' to suppress warnings from PyArrow.
43-
echo "export PYARROW_IGNORE_TIMEZONE=1" >> ~/.profile
44-
4542
# Add sbin to PATH to run `start-connect-server.sh`.
4643
SPARK_HOME=$(python -c "from pyspark.find_spark_home import _find_spark_home; print(_find_spark_home())")
4744
echo "export PATH=${PATH}:${SPARK_HOME}/sbin" >> ~/.profile

python/pyspark/pandas/__init__.py

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -39,16 +39,6 @@
3939
else:
4040
raise
4141

42-
if "PYARROW_IGNORE_TIMEZONE" not in os.environ:
43-
warnings.warn(
44-
"'PYARROW_IGNORE_TIMEZONE' environment variable was not set. It is required to "
45-
"set this environment variable to '1' in both driver and executor sides if you use "
46-
"pyarrow>=2.0.0. "
47-
"pandas-on-Spark will set it for you but it does not work if there is a Spark context "
48-
"already launched."
49-
)
50-
os.environ["PYARROW_IGNORE_TIMEZONE"] = "1"
51-
5242
from pyspark.pandas.frame import DataFrame
5343
from pyspark.pandas.indexes.base import Index
5444
from pyspark.pandas.indexes.category import CategoricalIndex

python/run-tests.py

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -196,8 +196,6 @@ def run_individual_python_test(target_dir, test_name, pyspark_python, keep_test_
196196
'SPARK_PREPEND_CLASSES': '1',
197197
'PYSPARK_PYTHON': which(pyspark_python),
198198
'PYSPARK_DRIVER_PYTHON': which(pyspark_python),
199-
# Preserve legacy nested timezone behavior for pyarrow>=2, remove after SPARK-32285
200-
'PYARROW_IGNORE_TIMEZONE': '1',
201199
})
202200

203201
if "SPARK_CONNECT_TESTING_REMOTE" in os.environ:

0 commit comments

Comments
 (0)