Poetry 1.2 fails installing Spark 2.4.4

[jvanlier]

Hi,

I'd like to upgrade a bunch of our pyspark==2.4.4 projects to poetry 1.2. However, installation of pyspark 2.4.4 fails with the following error:

AttributeError: module 'pypandoc' has no attribute 'convert'

The culprit appears to be this line in setup.py: https://github.com/apache/spark/blob/7955b3962ac46b89564e0613db7bea98a1478bf2/python/setup.py#L156

The function was removed in pypandoc 1.8, which was released in May 2022.

It is not so strange to me that Poetry 1.2 uses a recent version of pypandoc that doesn't have this function, since the version of pypandoc in pyspark 2.4.4's setup.py is unbounded.

However, what is strange is that this just works in poetry 1.1.15 and also with pip.

Any ideas why that is the case? I don't see anything in the release notes for Poetry 1.2 that could explain this.

Is there any way to solve this, besides repackaging pyspark==2.4.4 with a version constraint pypandoc<1.8 on setup_requires? (I've done that actually, and it works, but would rather use vanilla pyspark.)

Minimal reproducible pyproject.toml:

[tool.poetry]
name = "x"
version = "0.1.0"
description = ""
authors = ["me"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.7"
pyspark = "2.4.4"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

(fyi, pinning pypandoc to < 1.8 in dependencies does not work).