Build: Bump pyspark from 3.5.6 to 4.0.0

Bumps [pyspark](https://github.com/apache/spark) from 3.5.6 to 4.0.0.
- [Commits](apache/spark@v3.5.6...v4.0.0)

---
updated-dependencies:
- dependency-name: pyspark
  dependency-version: 4.0.0
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

Author: dependabot[bot]

poetry.lock

@@ -98,7 +98,7 @@ requests-mock = "1.12.1"
 moto = { version = "^5.0.2", extras = ["server"] }
 typing-extensions = "4.14.1"
 pytest-mock = "3.14.1"
-pyspark = "3.5.6"
+pyspark = "4.0.0"
 cython = "3.1.2"
 deptry = ">=0.14,<0.24"
 docutils = "!=0.21.post1"   # https://github.com/python-poetry/poetry/issues/9248#issuecomment-2026240520