Make FallbackStorage cleanup multithreaded #173

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft

EnricoMi wants to merge 1 commit into fallback-storage-cleanup-wait-on-shutdown from fallback-storage-cleanup-threads

Draft

Make FallbackStorage cleanup multithreaded #173

Make FallbackStorage cleanup multithreaded

GitHub Actions / Report test results failed Feb 12, 2026 in 0s

64784 tests run, 1130 skipped, 125 failed.

Annotations

Check failure on line 206 in org/apache/spark/sql/SQLQueryTestSuite

github-actions / Report test results

SQLQueryTestSuite.udf/postgreSQL/udf-case.sql - Scalar Pandas UDF

org.scalatest.exceptions.TestFailedException: udf/postgreSQL/udf-case.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<Two:string,i:int,f:double,i:int,j:int>"), but got Some("struct<>") Schema did not match for query #30
SELECT '' AS `Two`, *
  FROM CASE_TBL a, CASE2_TBL b
  WHERE udf(COALESCE(f,b.i) = 2): -- !query
SELECT '' AS `Two`, *
  FROM CASE_TBL a, CASE2_TBL b
  WHERE udf(COALESCE(f,b.i) = 2)
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkRuntimeException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"BOOLEAN\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 62,
    "stopIndex" : 85,
    "fragment" : "udf(COALESCE(f,b.i) = 2)"
  } ]
}

Raw output


            sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: udf/postgreSQL/udf-case.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<Two:string,i:int,f:double,i:int,j:int>"), but got Some("struct<>") Schema did not match for query #30
SELECT '' AS `Two`, *
  FROM CASE_TBL a, CASE2_TBL b
  WHERE udf(COALESCE(f,b.i) = 2): -- !query
SELECT '' AS `Two`, *
  FROM CASE_TBL a, CASE2_TBL b
  WHERE udf(COALESCE(f,b.i) = 2)
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkRuntimeException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"BOOLEAN\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 62,
    "stopIndex" : 85,
    "fragment" : "udf(COALESCE(f,b.i) = 2)"
  } ]
}
	at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
	at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
	at org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1564)
	at org.scalatest.Assertions.assertResult(Assertions.scala:847)
	at org.scalatest.Assertions.assertResult$(Assertions.scala:842)
	at org.scalatest.funsuite.AnyFunSuite.assertResult(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$readGoldenFileAndCompareResults$3(SQLQueryTestSuite.scala:681)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:630)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:628)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:936)
	at org.apache.spark.sql.SQLQueryTestSuite.readGoldenFileAndCompareResults(SQLQueryTestSuite.scala:676)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueries$13(SQLQueryTestSuite.scala:418)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.Assertions.withClue(Assertions.scala:1065)
	at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
	at org.scalatest.funsuite.AnyFunSuite.withClue(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueries(SQLQueryTestSuite.scala:414)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1(SQLQueryTestSuite.scala:219)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1$adapted(SQLQueryTestSuite.scala:217)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueriesWithSparkConfigDimensions(SQLQueryTestSuite.scala:217)
	at org.apache.spark.sql.SQLQueryTestSuite.runSqlTestCase(SQLQueryTestSuite.scala:243)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$createScalaTestCase$6(SQLQueryTestSuite.scala:206)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 206 in org/apache/spark/sql/SQLQueryTestSuite

github-actions / Report test results

SQLQueryTestSuite.udf/postgreSQL/udf-join.sql - Scalar Pandas UDF

org.scalatest.exceptions.TestFailedException: udf/postgreSQL/udf-join.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<xxx:string,udf(i):int,udf(j):int,udf(t):string>"), but got Some("struct<>") Schema did not match for query #28
SELECT udf('') AS `xxx`, udf(i), udf(j), udf(t)
  FROM J1_TBL AS tx: -- !query
SELECT udf('') AS `xxx`, udf(i), udf(j), udf(t)
  FROM J1_TBL AS tx
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkNumberFormatException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"INT\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 34,
    "stopIndex" : 39,
    "fragment" : "udf(j)"
  } ]
}

Raw output


            sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: udf/postgreSQL/udf-join.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<xxx:string,udf(i):int,udf(j):int,udf(t):string>"), but got Some("struct<>") Schema did not match for query #28
SELECT udf('') AS `xxx`, udf(i), udf(j), udf(t)
  FROM J1_TBL AS tx: -- !query
SELECT udf('') AS `xxx`, udf(i), udf(j), udf(t)
  FROM J1_TBL AS tx
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkNumberFormatException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"INT\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 34,
    "stopIndex" : 39,
    "fragment" : "udf(j)"
  } ]
}
	at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
	at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
	at org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1564)
	at org.scalatest.Assertions.assertResult(Assertions.scala:847)
	at org.scalatest.Assertions.assertResult$(Assertions.scala:842)
	at org.scalatest.funsuite.AnyFunSuite.assertResult(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$readGoldenFileAndCompareResults$3(SQLQueryTestSuite.scala:681)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:630)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:628)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:936)
	at org.apache.spark.sql.SQLQueryTestSuite.readGoldenFileAndCompareResults(SQLQueryTestSuite.scala:676)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueries$13(SQLQueryTestSuite.scala:418)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.Assertions.withClue(Assertions.scala:1065)
	at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
	at org.scalatest.funsuite.AnyFunSuite.withClue(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueries(SQLQueryTestSuite.scala:414)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1(SQLQueryTestSuite.scala:219)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1$adapted(SQLQueryTestSuite.scala:217)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueriesWithSparkConfigDimensions(SQLQueryTestSuite.scala:217)
	at org.apache.spark.sql.SQLQueryTestSuite.runSqlTestCase(SQLQueryTestSuite.scala:243)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$createScalaTestCase$6(SQLQueryTestSuite.scala:206)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 206 in org/apache/spark/sql/SQLQueryTestSuite

github-actions / Report test results

SQLQueryTestSuite.udf/udf-except-all.sql - Scalar Pandas UDF

org.scalatest.exceptions.TestFailedException: udf/udf-except-all.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<udf(c1):int>"), but got Some("struct<>") Schema did not match for query #4
SELECT udf(c1) FROM tab1
EXCEPT ALL
SELECT udf(c1) FROM tab2: -- !query
SELECT udf(c1) FROM tab1
EXCEPT ALL
SELECT udf(c1) FROM tab2
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkNumberFormatException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"INT\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 8,
    "stopIndex" : 14,
    "fragment" : "udf(c1)"
  } ]
}

Raw output


            sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: udf/udf-except-all.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<udf(c1):int>"), but got Some("struct<>") Schema did not match for query #4
SELECT udf(c1) FROM tab1
EXCEPT ALL
SELECT udf(c1) FROM tab2: -- !query
SELECT udf(c1) FROM tab1
EXCEPT ALL
SELECT udf(c1) FROM tab2
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkNumberFormatException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"INT\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 8,
    "stopIndex" : 14,
    "fragment" : "udf(c1)"
  } ]
}
	at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
	at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
	at org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1564)
	at org.scalatest.Assertions.assertResult(Assertions.scala:847)
	at org.scalatest.Assertions.assertResult$(Assertions.scala:842)
	at org.scalatest.funsuite.AnyFunSuite.assertResult(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$readGoldenFileAndCompareResults$3(SQLQueryTestSuite.scala:681)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:630)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:628)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:936)
	at org.apache.spark.sql.SQLQueryTestSuite.readGoldenFileAndCompareResults(SQLQueryTestSuite.scala:676)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueries$13(SQLQueryTestSuite.scala:418)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.Assertions.withClue(Assertions.scala:1065)
	at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
	at org.scalatest.funsuite.AnyFunSuite.withClue(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueries(SQLQueryTestSuite.scala:414)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1(SQLQueryTestSuite.scala:219)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1$adapted(SQLQueryTestSuite.scala:217)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueriesWithSparkConfigDimensions(SQLQueryTestSuite.scala:217)
	at org.apache.spark.sql.SQLQueryTestSuite.runSqlTestCase(SQLQueryTestSuite.scala:243)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$createScalaTestCase$6(SQLQueryTestSuite.scala:206)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 206 in org/apache/spark/sql/SQLQueryTestSuite

github-actions / Report test results

SQLQueryTestSuite.udf/udf-except.sql - Scalar Pandas UDF

org.scalatest.exceptions.TestFailedException: udf/udf-except.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<udf(k):string,udf(v):int>"), but got Some("struct<>") Schema did not match for query #2
SELECT udf(k), udf(v) FROM t1 EXCEPT SELECT udf(k), udf(v) FROM t2: -- !query
SELECT udf(k), udf(v) FROM t1 EXCEPT SELECT udf(k), udf(v) FROM t2
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkNumberFormatException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"INT\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 53,
    "stopIndex" : 58,
    "fragment" : "udf(v)"
  } ]
}

Raw output


            sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: udf/udf-except.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<udf(k):string,udf(v):int>"), but got Some("struct<>") Schema did not match for query #2
SELECT udf(k), udf(v) FROM t1 EXCEPT SELECT udf(k), udf(v) FROM t2: -- !query
SELECT udf(k), udf(v) FROM t1 EXCEPT SELECT udf(k), udf(v) FROM t2
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkNumberFormatException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"INT\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 53,
    "stopIndex" : 58,
    "fragment" : "udf(v)"
  } ]
}
	at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
	at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
	at org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1564)
	at org.scalatest.Assertions.assertResult(Assertions.scala:847)
	at org.scalatest.Assertions.assertResult$(Assertions.scala:842)
	at org.scalatest.funsuite.AnyFunSuite.assertResult(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$readGoldenFileAndCompareResults$3(SQLQueryTestSuite.scala:681)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:630)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:628)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:936)
	at org.apache.spark.sql.SQLQueryTestSuite.readGoldenFileAndCompareResults(SQLQueryTestSuite.scala:676)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueries$13(SQLQueryTestSuite.scala:418)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.Assertions.withClue(Assertions.scala:1065)
	at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
	at org.scalatest.funsuite.AnyFunSuite.withClue(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueries(SQLQueryTestSuite.scala:414)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1(SQLQueryTestSuite.scala:219)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1$adapted(SQLQueryTestSuite.scala:217)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueriesWithSparkConfigDimensions(SQLQueryTestSuite.scala:217)
	at org.apache.spark.sql.SQLQueryTestSuite.runSqlTestCase(SQLQueryTestSuite.scala:243)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$createScalaTestCase$6(SQLQueryTestSuite.scala:206)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 206 in org/apache/spark/sql/SQLQueryTestSuite

github-actions / Report test results

SQLQueryTestSuite.udf/udf-group-analytics.sql - Scalar Pandas UDF

org.scalatest.exceptions.TestFailedException: udf/udf-group-analytics.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<udf(a):int,udf(b):int,sum(b):bigint>"), but got Some("struct<>") Schema did not match for query #2
SELECT udf(a), udf(b), SUM(b) FROM testData GROUP BY udf(a), b WITH CUBE: -- !query
SELECT udf(a), udf(b), SUM(b) FROM testData GROUP BY udf(a), b WITH CUBE
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkNumberFormatException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"INT\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 16,
    "stopIndex" : 21,
    "fragment" : "udf(b)"
  } ]
}

Raw output


            sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: udf/udf-group-analytics.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<udf(a):int,udf(b):int,sum(b):bigint>"), but got Some("struct<>") Schema did not match for query #2
SELECT udf(a), udf(b), SUM(b) FROM testData GROUP BY udf(a), b WITH CUBE: -- !query
SELECT udf(a), udf(b), SUM(b) FROM testData GROUP BY udf(a), b WITH CUBE
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkNumberFormatException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"INT\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 16,
    "stopIndex" : 21,
    "fragment" : "udf(b)"
  } ]
}
	at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
	at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
	at org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1564)
	at org.scalatest.Assertions.assertResult(Assertions.scala:847)
	at org.scalatest.Assertions.assertResult$(Assertions.scala:842)
	at org.scalatest.funsuite.AnyFunSuite.assertResult(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$readGoldenFileAndCompareResults$3(SQLQueryTestSuite.scala:681)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:630)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:628)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:936)
	at org.apache.spark.sql.SQLQueryTestSuite.readGoldenFileAndCompareResults(SQLQueryTestSuite.scala:676)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueries$13(SQLQueryTestSuite.scala:418)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.Assertions.withClue(Assertions.scala:1065)
	at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
	at org.scalatest.funsuite.AnyFunSuite.withClue(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueries(SQLQueryTestSuite.scala:414)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1(SQLQueryTestSuite.scala:219)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1$adapted(SQLQueryTestSuite.scala:217)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueriesWithSparkConfigDimensions(SQLQueryTestSuite.scala:217)
	at org.apache.spark.sql.SQLQueryTestSuite.runSqlTestCase(SQLQueryTestSuite.scala:243)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$createScalaTestCase$6(SQLQueryTestSuite.scala:206)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 206 in org/apache/spark/sql/SQLQueryTestSuite

github-actions / Report test results

SQLQueryTestSuite.udf/udf-group-by.sql - Scalar Pandas UDF

org.scalatest.exceptions.TestFailedException: udf/udf-group-by.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<count(udf(a)):bigint,udf(count(b)):bigint>"), but got Some("struct<>") Schema did not match for query #2
SELECT COUNT(udf(a)), udf(COUNT(b)) FROM testData: -- !query
SELECT COUNT(udf(a)), udf(COUNT(b)) FROM testData
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkNumberFormatException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"INT\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 14,
    "stopIndex" : 19,
    "fragment" : "udf(a)"
  } ]
}

Raw output


            sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: udf/udf-group-by.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<count(udf(a)):bigint,udf(count(b)):bigint>"), but got Some("struct<>") Schema did not match for query #2
SELECT COUNT(udf(a)), udf(COUNT(b)) FROM testData: -- !query
SELECT COUNT(udf(a)), udf(COUNT(b)) FROM testData
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkNumberFormatException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"INT\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 14,
    "stopIndex" : 19,
    "fragment" : "udf(a)"
  } ]
}
	at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
	at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
	at org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1564)
	at org.scalatest.Assertions.assertResult(Assertions.scala:847)
	at org.scalatest.Assertions.assertResult$(Assertions.scala:842)
	at org.scalatest.funsuite.AnyFunSuite.assertResult(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$readGoldenFileAndCompareResults$3(SQLQueryTestSuite.scala:681)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:630)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:628)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:936)
	at org.apache.spark.sql.SQLQueryTestSuite.readGoldenFileAndCompareResults(SQLQueryTestSuite.scala:676)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueries$13(SQLQueryTestSuite.scala:418)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.Assertions.withClue(Assertions.scala:1065)
	at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
	at org.scalatest.funsuite.AnyFunSuite.withClue(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueries(SQLQueryTestSuite.scala:414)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1(SQLQueryTestSuite.scala:219)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1$adapted(SQLQueryTestSuite.scala:217)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueriesWithSparkConfigDimensions(SQLQueryTestSuite.scala:217)
	at org.apache.spark.sql.SQLQueryTestSuite.runSqlTestCase(SQLQueryTestSuite.scala:243)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$createScalaTestCase$6(SQLQueryTestSuite.scala:206)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 206 in org/apache/spark/sql/SQLQueryTestSuite

github-actions / Report test results

SQLQueryTestSuite.udf/udf-intersect-all.sql - Scalar Pandas UDF

org.scalatest.exceptions.TestFailedException: udf/udf-intersect-all.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<udf(k):int,v:int>"), but got Some("struct<>") Schema did not match for query #2
SELECT udf(k), v FROM tab1
INTERSECT ALL
SELECT k, udf(v) FROM tab2: -- !query
SELECT udf(k), v FROM tab1
INTERSECT ALL
SELECT k, udf(v) FROM tab2
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkNumberFormatException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"INT\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 8,
    "stopIndex" : 13,
    "fragment" : "udf(k)"
  } ]
}

Raw output


            sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: udf/udf-intersect-all.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<udf(k):int,v:int>"), but got Some("struct<>") Schema did not match for query #2
SELECT udf(k), v FROM tab1
INTERSECT ALL
SELECT k, udf(v) FROM tab2: -- !query
SELECT udf(k), v FROM tab1
INTERSECT ALL
SELECT k, udf(v) FROM tab2
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkNumberFormatException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"INT\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 8,
    "stopIndex" : 13,
    "fragment" : "udf(k)"
  } ]
}
	at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
	at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
	at org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1564)
	at org.scalatest.Assertions.assertResult(Assertions.scala:847)
	at org.scalatest.Assertions.assertResult$(Assertions.scala:842)
	at org.scalatest.funsuite.AnyFunSuite.assertResult(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$readGoldenFileAndCompareResults$3(SQLQueryTestSuite.scala:681)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:630)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:628)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:936)
	at org.apache.spark.sql.SQLQueryTestSuite.readGoldenFileAndCompareResults(SQLQueryTestSuite.scala:676)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueries$13(SQLQueryTestSuite.scala:418)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.Assertions.withClue(Assertions.scala:1065)
	at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
	at org.scalatest.funsuite.AnyFunSuite.withClue(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueries(SQLQueryTestSuite.scala:414)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1(SQLQueryTestSuite.scala:219)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1$adapted(SQLQueryTestSuite.scala:217)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueriesWithSparkConfigDimensions(SQLQueryTestSuite.scala:217)
	at org.apache.spark.sql.SQLQueryTestSuite.runSqlTestCase(SQLQueryTestSuite.scala:243)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$createScalaTestCase$6(SQLQueryTestSuite.scala:206)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 206 in org/apache/spark/sql/SQLQueryTestSuite

github-actions / Report test results

SQLQueryTestSuite.udf/udf-special-values.sql - Scalar Pandas UDF

org.scalatest.exceptions.TestFailedException: udf/udf-special-values.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<udf(x):int>"), but got Some("struct<>") Schema did not match for query #0
SELECT udf(x) FROM (VALUES (1), (2), (NULL)) v(x): -- !query
SELECT udf(x) FROM (VALUES (1), (2), (NULL)) v(x)
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkNumberFormatException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"INT\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 8,
    "stopIndex" : 13,
    "fragment" : "udf(x)"
  } ]
}

Raw output


            sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: udf/udf-special-values.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<udf(x):int>"), but got Some("struct<>") Schema did not match for query #0
SELECT udf(x) FROM (VALUES (1), (2), (NULL)) v(x): -- !query
SELECT udf(x) FROM (VALUES (1), (2), (NULL)) v(x)
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkNumberFormatException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"INT\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 8,
    "stopIndex" : 13,
    "fragment" : "udf(x)"
  } ]
}
	at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
	at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
	at org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1564)
	at org.scalatest.Assertions.assertResult(Assertions.scala:847)
	at org.scalatest.Assertions.assertResult$(Assertions.scala:842)
	at org.scalatest.funsuite.AnyFunSuite.assertResult(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$readGoldenFileAndCompareResults$3(SQLQueryTestSuite.scala:681)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:630)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:628)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:936)
	at org.apache.spark.sql.SQLQueryTestSuite.readGoldenFileAndCompareResults(SQLQueryTestSuite.scala:676)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueries$13(SQLQueryTestSuite.scala:418)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.Assertions.withClue(Assertions.scala:1065)
	at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
	at org.scalatest.funsuite.AnyFunSuite.withClue(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueries(SQLQueryTestSuite.scala:414)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1(SQLQueryTestSuite.scala:219)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1$adapted(SQLQueryTestSuite.scala:217)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueriesWithSparkConfigDimensions(SQLQueryTestSuite.scala:217)
	at org.apache.spark.sql.SQLQueryTestSuite.runSqlTestCase(SQLQueryTestSuite.scala:243)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$createScalaTestCase$6(SQLQueryTestSuite.scala:206)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 206 in org/apache/spark/sql/SQLQueryTestSuite

github-actions / Report test results

SQLQueryTestSuite.udf/udf-window.sql - Scalar Pandas UDF

org.scalatest.exceptions.TestFailedException: udf/udf-window.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<udf(val):int,cate:string,count(val) OVER (PARTITION BY cate ORDER BY udf(val) ASC NULLS FIRST ROWS BETWEEN CURRENT ROW AND CURRENT ROW):bigint>"), but got Some("struct<>") Schema did not match for query #1
SELECT udf(val), cate, count(val) OVER(PARTITION BY cate ORDER BY udf(val) ROWS CURRENT ROW) FROM testData
ORDER BY cate, udf(val): -- !query
SELECT udf(val), cate, count(val) OVER(PARTITION BY cate ORDER BY udf(val) ROWS CURRENT ROW) FROM testData
ORDER BY cate, udf(val)
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkNumberFormatException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"INT\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 8,
    "stopIndex" : 15,
    "fragment" : "udf(val)"
  } ]
}

Raw output


            sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: udf/udf-window.sql - Scalar Pandas UDF
Python: 3.11 Pandas: 3.0.0 PyArrow: 23.0.0
Expected Some("struct<udf(val):int,cate:string,count(val) OVER (PARTITION BY cate ORDER BY udf(val) ASC NULLS FIRST ROWS BETWEEN CURRENT ROW AND CURRENT ROW):bigint>"), but got Some("struct<>") Schema did not match for query #1
SELECT udf(val), cate, count(val) OVER(PARTITION BY cate ORDER BY udf(val) ROWS CURRENT ROW) FROM testData
ORDER BY cate, udf(val): -- !query
SELECT udf(val), cate, count(val) OVER(PARTITION BY cate ORDER BY udf(val) ROWS CURRENT ROW) FROM testData
ORDER BY cate, udf(val)
-- !query schema
struct<>
-- !query output
org.apache.spark.SparkNumberFormatException
{
  "errorClass" : "CAST_INVALID_INPUT",
  "sqlState" : "22018",
  "messageParameters" : {
    "ansiConfig" : "\"spark.sql.ansi.enabled\"",
    "expression" : "'nan'",
    "sourceType" : "\"STRING\"",
    "targetType" : "\"INT\""
  },
  "queryContext" : [ {
    "objectType" : "",
    "objectName" : "",
    "startIndex" : 8,
    "stopIndex" : 15,
    "fragment" : "udf(val)"
  } ]
}
	at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
	at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
	at org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1564)
	at org.scalatest.Assertions.assertResult(Assertions.scala:847)
	at org.scalatest.Assertions.assertResult$(Assertions.scala:842)
	at org.scalatest.funsuite.AnyFunSuite.assertResult(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$readGoldenFileAndCompareResults$3(SQLQueryTestSuite.scala:681)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:630)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:628)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:936)
	at org.apache.spark.sql.SQLQueryTestSuite.readGoldenFileAndCompareResults(SQLQueryTestSuite.scala:676)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueries$13(SQLQueryTestSuite.scala:418)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.Assertions.withClue(Assertions.scala:1065)
	at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
	at org.scalatest.funsuite.AnyFunSuite.withClue(AnyFunSuite.scala:1564)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueries(SQLQueryTestSuite.scala:414)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1(SQLQueryTestSuite.scala:219)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$runQueriesWithSparkConfigDimensions$1$adapted(SQLQueryTestSuite.scala:217)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.apache.spark.sql.SQLQueryTestSuite.runQueriesWithSparkConfigDimensions(SQLQueryTestSuite.scala:217)
	at org.apache.spark.sql.SQLQueryTestSuite.runSqlTestCase(SQLQueryTestSuite.scala:243)
	at org.apache.spark.sql.SQLQueryTestSuite.$anonfun$createScalaTestCase$6(SQLQueryTestSuite.scala:206)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 45 in org/apache/spark/sql/connect/pipelines/EndToEndAPISuite

github-actions / Report test results

EndToEndAPISuite.SQL Pipeline with mv, st, and flows

java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Raw output


            sbt.ForkMain$ForkError: java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:126)
	at org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$1(APITest.scala:115)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
	at org.apache.spark.SparkTestSuite.withTempDir(SparkTestSuite.scala:193)
	at org.apache.spark.SparkTestSuite.withTempDir$(SparkTestSuite.scala:191)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir$(SQLTestUtils.scala:79)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$1(EndToEndAPISuite.scala:45)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.runWithInstrumentation(PipelineTest.scala:149)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.$anonfun$test$1(PipelineTest.scala:138)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 45 in org/apache/spark/sql/connect/pipelines/EndToEndAPISuite

github-actions / Report test results

EndToEndAPISuite.SQL Pipeline with CTE

java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Raw output


            sbt.ForkMain$ForkError: java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:126)
	at org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$2(APITest.scala:140)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
	at org.apache.spark.SparkTestSuite.withTempDir(SparkTestSuite.scala:193)
	at org.apache.spark.SparkTestSuite.withTempDir$(SparkTestSuite.scala:191)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir$(SQLTestUtils.scala:79)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$1(EndToEndAPISuite.scala:45)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.runWithInstrumentation(PipelineTest.scala:149)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.$anonfun$test$1(PipelineTest.scala:138)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 45 in org/apache/spark/sql/connect/pipelines/EndToEndAPISuite

github-actions / Report test results

EndToEndAPISuite.SQL Pipeline with subquery

java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Raw output


            sbt.ForkMain$ForkError: java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:126)
	at org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$3(APITest.scala:159)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
	at org.apache.spark.SparkTestSuite.withTempDir(SparkTestSuite.scala:193)
	at org.apache.spark.SparkTestSuite.withTempDir$(SparkTestSuite.scala:191)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir$(SQLTestUtils.scala:79)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$1(EndToEndAPISuite.scala:45)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.runWithInstrumentation(PipelineTest.scala:149)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.$anonfun$test$1(PipelineTest.scala:138)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 45 in org/apache/spark/sql/connect/pipelines/EndToEndAPISuite

github-actions / Report test results

EndToEndAPISuite.SQL Pipeline with join

java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Raw output


            sbt.ForkMain$ForkError: java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:126)
	at org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$4(APITest.scala:178)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
	at org.apache.spark.SparkTestSuite.withTempDir(SparkTestSuite.scala:193)
	at org.apache.spark.SparkTestSuite.withTempDir$(SparkTestSuite.scala:191)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir$(SQLTestUtils.scala:79)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$1(EndToEndAPISuite.scala:45)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.runWithInstrumentation(PipelineTest.scala:149)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.$anonfun$test$1(PipelineTest.scala:138)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 45 in org/apache/spark/sql/connect/pipelines/EndToEndAPISuite

github-actions / Report test results

EndToEndAPISuite.SQL Pipeline with aggregation

java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Raw output


            sbt.ForkMain$ForkError: java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:126)
	at org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$5(APITest.scala:196)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
	at org.apache.spark.SparkTestSuite.withTempDir(SparkTestSuite.scala:193)
	at org.apache.spark.SparkTestSuite.withTempDir$(SparkTestSuite.scala:191)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir$(SQLTestUtils.scala:79)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$1(EndToEndAPISuite.scala:45)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.runWithInstrumentation(PipelineTest.scala:149)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.$anonfun$test$1(PipelineTest.scala:138)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 45 in org/apache/spark/sql/connect/pipelines/EndToEndAPISuite

github-actions / Report test results

EndToEndAPISuite.SQL Pipeline with table properties

java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Raw output


            sbt.ForkMain$ForkError: java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:126)
	at org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$6(APITest.scala:215)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
	at org.apache.spark.SparkTestSuite.withTempDir(SparkTestSuite.scala:193)
	at org.apache.spark.SparkTestSuite.withTempDir$(SparkTestSuite.scala:191)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir$(SQLTestUtils.scala:79)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$1(EndToEndAPISuite.scala:45)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.runWithInstrumentation(PipelineTest.scala:149)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.$anonfun$test$1(PipelineTest.scala:138)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 45 in org/apache/spark/sql/connect/pipelines/EndToEndAPISuite

github-actions / Report test results

EndToEndAPISuite.SQL Pipeline with schema

java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Raw output


            sbt.ForkMain$ForkError: java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:126)
	at org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$7(APITest.scala:241)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
	at org.apache.spark.SparkTestSuite.withTempDir(SparkTestSuite.scala:193)
	at org.apache.spark.SparkTestSuite.withTempDir$(SparkTestSuite.scala:191)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir$(SQLTestUtils.scala:79)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$1(EndToEndAPISuite.scala:45)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.runWithInstrumentation(PipelineTest.scala:149)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.$anonfun$test$1(PipelineTest.scala:138)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 45 in org/apache/spark/sql/connect/pipelines/EndToEndAPISuite

github-actions / Report test results

EndToEndAPISuite.Pipeline with Python and SQL

java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Raw output


            sbt.ForkMain$ForkError: java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:126)
	at org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$8(APITest.scala:284)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
	at org.apache.spark.SparkTestSuite.withTempDir(SparkTestSuite.scala:193)
	at org.apache.spark.SparkTestSuite.withTempDir$(SparkTestSuite.scala:191)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir$(SQLTestUtils.scala:79)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$1(EndToEndAPISuite.scala:45)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.runWithInstrumentation(PipelineTest.scala:149)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.$anonfun$test$1(PipelineTest.scala:138)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 45 in org/apache/spark/sql/connect/pipelines/EndToEndAPISuite

github-actions / Report test results

EndToEndAPISuite.Pipeline referencing internal datasets

java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Raw output


            sbt.ForkMain$ForkError: java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:126)
	at org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$9(APITest.scala:330)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
	at org.apache.spark.SparkTestSuite.withTempDir(SparkTestSuite.scala:193)
	at org.apache.spark.SparkTestSuite.withTempDir$(SparkTestSuite.scala:191)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir$(SQLTestUtils.scala:79)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$1(EndToEndAPISuite.scala:45)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.runWithInstrumentation(PipelineTest.scala:149)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.$anonfun$test$1(PipelineTest.scala:138)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 45 in org/apache/spark/sql/connect/pipelines/EndToEndAPISuite

github-actions / Report test results

EndToEndAPISuite.Pipeline referencing external datasets

java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Raw output


            sbt.ForkMain$ForkError: java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:126)
	at org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$10(APITest.scala:369)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
	at org.apache.spark.SparkTestSuite.withTempDir(SparkTestSuite.scala:193)
	at org.apache.spark.SparkTestSuite.withTempDir$(SparkTestSuite.scala:191)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir$(SQLTestUtils.scala:79)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$1(EndToEndAPISuite.scala:45)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.runWithInstrumentation(PipelineTest.scala:149)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.$anonfun$test$1(PipelineTest.scala:138)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 45 in org/apache/spark/sql/connect/pipelines/EndToEndAPISuite

github-actions / Report test results

EndToEndAPISuite.Python Pipeline with materialized_view, create_streaming_table, and append_flow

java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Raw output


            sbt.ForkMain$ForkError: java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:126)
	at org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$11(APITest.scala:417)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
	at org.apache.spark.SparkTestSuite.withTempDir(SparkTestSuite.scala:193)
	at org.apache.spark.SparkTestSuite.withTempDir$(SparkTestSuite.scala:191)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir$(SQLTestUtils.scala:79)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$1(EndToEndAPISuite.scala:45)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.runWithInstrumentation(PipelineTest.scala:149)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.$anonfun$test$1(PipelineTest.scala:138)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 45 in org/apache/spark/sql/connect/pipelines/EndToEndAPISuite

github-actions / Report test results

EndToEndAPISuite.Python Pipeline with temporary_view

java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Raw output


            sbt.ForkMain$ForkError: java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:126)
	at org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$12(APITest.scala:452)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
	at org.apache.spark.SparkTestSuite.withTempDir(SparkTestSuite.scala:193)
	at org.apache.spark.SparkTestSuite.withTempDir$(SparkTestSuite.scala:191)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir$(SQLTestUtils.scala:79)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$1(EndToEndAPISuite.scala:45)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.runWithInstrumentation(PipelineTest.scala:149)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.$anonfun$test$1(PipelineTest.scala:138)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 45 in org/apache/spark/sql/connect/pipelines/EndToEndAPISuite

github-actions / Report test results

EndToEndAPISuite.Python Pipeline with parquet sink

java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Raw output


            sbt.ForkMain$ForkError: java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:126)
	at org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$14(APITest.scala:498)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
	at org.apache.spark.SparkTestSuite.withTempDir(SparkTestSuite.scala:193)
	at org.apache.spark.SparkTestSuite.withTempDir$(SparkTestSuite.scala:191)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir$(SQLTestUtils.scala:79)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$1(EndToEndAPISuite.scala:45)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.runWithInstrumentation(PipelineTest.scala:149)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.$anonfun$test$1(PipelineTest.scala:138)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 45 in org/apache/spark/sql/connect/pipelines/EndToEndAPISuite

github-actions / Report test results

EndToEndAPISuite.Python Pipeline with json sink

java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Raw output


            sbt.ForkMain$ForkError: java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:126)
	at org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$14(APITest.scala:498)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
	at org.apache.spark.SparkTestSuite.withTempDir(SparkTestSuite.scala:193)
	at org.apache.spark.SparkTestSuite.withTempDir$(SparkTestSuite.scala:191)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir$(SQLTestUtils.scala:79)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$1(EndToEndAPISuite.scala:45)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.runWithInstrumentation(PipelineTest.scala:149)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.$anonfun$test$1(PipelineTest.scala:138)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 45 in org/apache/spark/sql/connect/pipelines/EndToEndAPISuite

github-actions / Report test results

EndToEndAPISuite.Python Pipeline with partition columns

java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Raw output


            sbt.ForkMain$ForkError: java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:126)
	at org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$15(APITest.scala:535)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
	at org.apache.spark.SparkTestSuite.withTempDir(SparkTestSuite.scala:193)
	at org.apache.spark.SparkTestSuite.withTempDir$(SparkTestSuite.scala:191)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir$(SQLTestUtils.scala:79)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$1(EndToEndAPISuite.scala:45)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.runWithInstrumentation(PipelineTest.scala:149)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.$anonfun$test$1(PipelineTest.scala:138)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Check failure on line 45 in org/apache/spark/sql/connect/pipelines/EndToEndAPISuite

github-actions / Report test results

EndToEndAPISuite.Python Pipeline with cluster columns

java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Raw output


            sbt.ForkMain$ForkError: java.lang.RuntimeException: Pipeline update process failed with exit code 1.
Output: 
Error: Traceback (most recent call last):
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/cli.py", line 36, in <module>
    from pyspark.pipelines.block_session_mutations import block_session_mutations
  File "/home/runner/work/spark/spark/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
    from pyspark.sql.connect.catalog import Catalog
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/catalog.py", line 29, in <module>
    from pyspark.sql.connect.dataframe import DataFrame
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/dataframe.py", line 75, in <module>
    import pyspark.sql.connect.plan as plan
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/plan.py", line 51, in <module>
    import pyspark.sql.connect.proto as proto
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 21, in <module>
    from pyspark.sql.connect.proto import base_pb2 as spark_dot_connect_dot_base__pb2
  File "/home/runner/work/spark/spark/python/pyspark/sql/connect/proto/base_pb2.py", line 29, in <module>
    _runtime_version.ValidateProtobufRuntimeVersion(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 100, in ValidateProtobufRuntimeVersion
    _ReportVersionError(
  File "/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
    raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading spark/connect/base.proto: gencode 6.33.5 runtime 6.33.0. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.awaitPipelineTermination(EndToEndAPISuite.scala:126)
	at org.apache.spark.sql.pipelines.utils.APITest.$anonfun$$init$$17(APITest.scala:571)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2(EndToEndAPISuite.scala:47)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$2$adapted(EndToEndAPISuite.scala:45)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1(SQLTestUtils.scala:81)
	at org.apache.spark.sql.test.SQLTestUtils.$anonfun$withTempDir$1$adapted(SQLTestUtils.scala:80)
	at org.apache.spark.SparkTestSuite.withTempDir(SparkTestSuite.scala:193)
	at org.apache.spark.SparkTestSuite.withTempDir$(SparkTestSuite.scala:191)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.org$apache$spark$sql$test$SQLTestUtils$$super$withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir(SQLTestUtils.scala:80)
	at org.apache.spark.sql.test.SQLTestUtils.withTempDir$(SQLTestUtils.scala:79)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.withTempDir(PipelineTest.scala:39)
	at org.apache.spark.sql.connect.pipelines.EndToEndAPISuite.$anonfun$test$1(EndToEndAPISuite.scala:45)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.runWithInstrumentation(PipelineTest.scala:149)
	at org.apache.spark.sql.pipelines.utils.PipelineTest.$anonfun$test$1(PipelineTest.scala:138)
	at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
	at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
	at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
	at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
	at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)
	at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)
	at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

View more details on GitHub Actions

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make FallbackStorage cleanup multithreaded #173

Uh oh!

Uh oh!

Make FallbackStorage cleanup multithreaded #173

Uh oh!

64784 tests run, 1130 skipped, 125 failed.

Annotations

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

github-actions / Report test results

Re-running jobs...

Make FallbackStorage cleanup multithreaded #173

Are you sure you want to change the base?

Uh oh!