[2.15.0] - 2025-12-13

[2.15.0] - 2025-12-13

Added

• Support encrypted parquet files (#324)

Changed

• Remove support for Spark 3.0 and Spark 3.1 (#332)
• Make all undocumented unintended public API parts private (#331)
• Reading Parquet metadata can use Parquet Hadoop version different to version coming with Spark (#330)

[2.14.2] - 2025-07-21

[2.14.2] - 2025-07-21

Changed

• Fixed release process (#320)

[2.14.1] - 2025-07-17

[2.14.1] - 2025-07-17

Changed

• Fixed release process (#319)

[2.14.0] - 2025-07-17

[2.14.0] - 2025-07-17

Added

• Support for Spark 4.0 (#269, #272, #293)

Changed

• Improve backticks (#265)

  New: This escapes backticks that already exist in column names.

  Change: This does not quote columns that only contain letters, numbers
  and underscores, which were quoted before.

• Move Python dependencies into setup.py, build jar from setup.py (#301)

[2.13.0] - 2024-11-04

Fixes

• Support diff for Spark Connect implemened via PySpark Dataset API (#251)

Added

• Add ignore columns to diff in Python API (#252)
• Check that the Java / Scala package is installed when needed by Python (#250)

[2.12.0] - 2024-04-26

Fixes

• Diff change column should respect comparators (#238)

Changed

• Make create_temporary_dir work with pyspark-extension only (#222). This allows installing PIP packages and Poetry projects via pure Python spark-extension package (Maven package not required any more).
• Add map diff comparator to Python API (#226)

[2.11.0] - 2024-01-04

Added

• Add count_null aggregate function (#206)
• Support reading parquet schema (#208)
• Add more columns to reading parquet metadata (#209, #211)
• Provide groupByKey shortcuts for groupBy.as (#213)
• Allow to install PIP packages into PySpark job (#215)
• Allow to install Poetry projects into PySpark job (#216)

[2.10.0] - 2023-09-27

Fixed

• Update setup.py to include parquet methods in python package (#191)

Added

• Add --statistics option to diff app (#189)
• Add --filter option to diff app (#190)

[2.9.0] - 2023-08-23

Added

• Add key order sensitive map comparator (#187)

Changed

• Use dataset encoder rather than implicit value encoder for implicit dataset extension class (#183)

Fixed

• Fix key-sensitivity in map comparator (#186)

[2.8.0] - 2023-05-24

Added

• Add method to set and automatically unset Spark job description. (#172)
• Add column function that converts between .Net (C#, F#, Visual Basic) DateTime.Ticks and Spark timestamp / Unix epoch timestamps. (#153)