scalatest depedency

[rosireddyr]

It looks fast-tests project is dependent on scalatest package. Where can I find this jar package file?
I am trying to write test cases in azure synapse notebook, so even basic tests are also failing

[MrPowers]

@rosireddyr - scalatest is a test dependency, see here.

If you want to use scalatest and spark-fast-tests in an Azure Synapse notebook, you'll have to install both libraries.

[rosireddyr]

Thanks @MrPowers .
Not sure there is a way to install packages in synapse. But I am downloading the jar packages, uploading into synapse workspace packages and configuring app pools with these packages. Just now found scalatest package at this location, Hope this is the right url.
https://oss.sonatype.org/content/groups/public/org/scalatest/scalatest_2.13/3.2.15/

Still I get this error
:38: error: object scalatest is not a member of package org
import org.scalatest.funsuite.AnyFunSuite

[MrPowers]

@rosireddyr scalatest made some breaking changes. I think you're using syntax that's from a legacy version of scalatest. You probably need to change your import statement. I actually wrote about that particular breaking change in this blog post.

[rosireddyr]

Thanks @MrPowers .

Previous issue is resolved. Now I am got into fast-tests related issue. Below is the code and the issue.

import org.apache.spark.sql.types._
import org.apache.spark.sql.Row
import org.apache.spark.sql.functions._
import com.github.mrpowers.spark.fast.tests
import com.github.mrpowers.spark.fast.tests.DatasetComparer
import org.scalatest.FunSpec

import org.apache.spark.sql.SparkSession

trait SparkSessionTestWrapper {

lazy val spark: SparkSession = {
SparkSession
.builder()
.master("local")
.appName("spark session")
.config("spark.sql.shuffle.partitions", "1")
.getOrCreate()
}

}

class DatasetSpec extends FunSpec with SparkSessionTestWrapper with com.github.mrpowers.spark.fast.tests.DatasetComparer {

import spark.implicits._

it("aliases a DataFrame") {

val sourceDF = Seq(
  ("jose"),
  ("li"),
  ("luisa")
).toDF("name")

val actualDF = sourceDF.select(col("name").alias("student"))

val expectedDF = Seq(
  ("jose"),
  ("li"),
  ("luisa")
).toDF("student")

com.github.mrpowers.spark.fast.tests.assertSmallDatasetEquality(actualDF, expectedDF)

}
}

:18: error: class DatasetComparer needs to be a trait to be mixed in
class DatasetSpec extends FunSpec with SparkSessionTestWrapper with com.github.mrpowers.spark.fast.tests.DatasetComparer {

                                                                                                            ^

:38: error: object assertSmallDatasetEquality is not a member of package com.github.mrpowers.spark.fast.tests

       com.github.mrpowers.spark.fast.tests.assertSmallDatasetEquality(actualDF, expectedDF)

[MrPowers]

yea, you extend the DatasetComparer trait, so it should be assertSmallDatasetEquality, not com.github.mrpowers.spark.fast.tests.assertSmallDatasetEquality. Can you try to just use the example that's in the README?

[rosireddyr]

yes, I am using readme example and its not working as expected, so trying with minor changes in the same code.

Now I see only below error.

:19: error: class DatasetComparer needs to be a trait to be mixed in
class DatasetSpec extends FunSpec with SparkSessionTestWrapper with DatasetComparer {

SampleNotebook.txt