Add support pyspark.sql.classic.dataframe.DataFrame transformer

[arbaobao]

Tracking issue

#6478

Why are the changes needed?

There is a new type of dataframe when pyspark>=4.0.0. It's not recognized by the current structured dataset. Therefore, the type transformer will fail to serialize and deserialize it.

What changes were proposed in this pull request?

Add a new type of spark dataframe and register it.

How was this patch tested?

Github CI will fail if we don't add the new type of dataframe and pyspark>=4.0.0.

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

Summary by Bito

This pull request adds support for the new `pyspark.sql.classic.dataframe.DataFrame` type for compatibility with `pyspark` version 4.0.0 and above. It includes new classes for reading and writing this DataFrame type, along with serialization and deserialization handlers. Plugin requirements have also been updated accordingly.

[flyte-bot]

Bito Automatic Review Skipped - Draft PR

Bito didn't auto-review because this pull request is in draft status.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change draft PR review settings here, or contact your Bito workspace admin at [email protected].

[flyte-bot]

Bito Automatic Review Skipped - Draft PR

Bito didn't auto-review because this pull request is in draft status.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change draft PR review settings here, or contact your Bito workspace admin at [email protected].

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.63%. Comparing base (3c6b61d) to head (f7ae136).
⚠️ Report is 21 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (3c6b61d) and HEAD (f7ae136). Click for more details.

HEAD has 38 uploads less than BASE

Flag	BASE (3c6b61d)	HEAD (f7ae136)
	41	3

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3272      +/-   ##
==========================================
- Coverage   83.35%   75.63%   -7.73%     
==========================================
  Files         347      215     -132     
  Lines       28791    22520    -6271     
  Branches     2960     2961       +1     
==========================================
- Hits        23999    17033    -6966     
- Misses       3956     4615     +659     
- Partials      836      872      +36     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kumare3]

In 4.0.0 do we have a new dataframe type?

[arbaobao]

In 4.0.0 do we have a new dataframe type?

@kumare3 Yes, I add a new type of dataframe to support pyspark.sql.classic.dataframe.DataFrame for pyspark versions 4.0.0 and above.