Scala module 2.10.0 requires Jackson Databind version >= 2.10.0 and < 2.11.0 - scala

Question

I have a sbt projects and I want to make a test with scala test and shared spark session. Several weeks ago my project started to make an error.
java.lang.ExceptionInInitializerError
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)
.....
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.10.0 requires Jackson Databind version >= 2.10.0 and < 2.11.0
at com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:61)
at com.fasterxml.jackson.module.scala.JacksonModule.setupModule$(JacksonModule.scala:46)
There is a very simple test
import org.apache.spark.sql.QueryTest.checkAnswer
import org.apache.spark.sql.Row
import org.apache.spark.sql.test.SharedSparkSession
class SparkTestSpec extends SharedSparkSession {
import testImplicits._
test("join - join using") {
val df = Seq(1, 2, 3).toDF("int")
checkAnswer(df, Row(1) :: Row(2) :: Row(3) :: Nil)
}
}
And sbt config
ThisBuild / scalaVersion := "2.12.10"
val sparkVersion = "3.1.0"
val scalaTestVersion = "3.2.1"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-sql" % sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion % Test,
"org.apache.spark" %% "spark-sql" % sparkVersion % Test classifier "tests",
"org.apache.spark" %% "spark-catalyst" % sparkVersion % Test,
"org.apache.spark" %% "spark-catalyst" % sparkVersion % Test classifier "tests",
"org.apache.spark" %% "spark-hive" % sparkVersion % Test,
"org.apache.spark" %% "spark-hive" % sparkVersion % Test classifier "tests",
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-core" % sparkVersion % Test classifier "tests",
"log4j" % "log4j" % "1.2.17",
"org.slf4j" % "slf4j-log4j12" % "1.7.30",
"org.scalatest" %% "scalatest" % scalaTestVersion % Test,
"org.scalatestplus" %% "scalacheck-1-14" % "3.2.2.0",
)

Answer 1

This is a very classic issue with Jackson. The error tells you that you need to have a single version of Jackson across all your dependencies but it's not the case.
Usually you have both Spark and another library pulling transitively Jackson in different versions.
What you need to do is:
run sbt dependencyTree to identify which library is pulling Jackson and which version
define a dependencyOverrides to force the same Jackson version for all Jackson libraries (which version is up to you depending on compatibility with the other libraries needing it)