Py4JJavaError: An error occurred while calling o226.save. : java.lang.NoClassDefFoundError: scala/Product$class

[mednourconsulting]

i am getting this error
Py4JJavaError: An error occurred while calling o226.save.
: java.lang.NoClassDefFoundError: scala/Product$class
in this step

write ratings data

ratings.write.format("es").save("ratings")
num_ratings_es = es.count(index="ratings")['count']
num_ratings_df = ratings.count()

check write went ok

print("Dataframe count: {}".format(num_ratings_df))
print("ES index count: {}".format(num_ratings_es))
plz help me

[hung96ad]

I also got the same error

[pxike]

the student despair is real, i also had this issue, luckily for me i solved it on my own , the cause of the issue is that spark 3 is compiled in scala 2.12 , and the connectors in elasticsearch are compiled in 2.11, since elasticsearch website doesn't have that in their site (for some reason like wtf ?), u can search that in maven and install it manually,
mvn dependency:get -DgroupId=org.elasticsearch -DartifactId=elasticsearch-spark-30_2.12 -Dversion=8.4.3 -Dtransitive=false
then check the local maven repo
/home/YourUserName/.m2/repository

either that or get a spark that is compiled in scala 2.11, which is spark 2.x versions but then the python version won't work cuz u will need python 3.4 or something which no longer supported , it's such a mean problem