In my build.sbt files I'm stating that I want to use version 18.9 from a library:
val finagleVersion = "18.9.0"
<zip>
lazy val commonDependencies = Seq(
<zip>,
"com.twitter" %% "finagle-core" % finagleVersion,
but this seems to be ignored when I run sbt with
scalacOptions ++= (compilerOptions :+ "-Ylog-classpath"),
which outputs all the jars used at compile time. And there I see that for every finagle dependency including core the 19.3 version is used:
C:\Users\<me>\.coursier\cache\v1\https\<me>%40<company repo>\artifactory\Central-cache\com\twitter\finagle-core_2.12\19.3.0\finagle-core_2.12-19.3.0.jar
Where is this "preference" for the latest versions coming from?
After using evicted and seeing which library overrides the version you want, you can opt to use dependencyOverrides. For example:
dependencyOverrides += "com.twitter" %% "finagle-core" % "18.9.0"
You do have to be careful though as the library that depends on Finagle too may require the newer version and break if you use the older version. That is why you should really check first which library is evicting the old version, and validate if it's ok to do so.
Also important, this is a livy-only feature so the override won't be present in the published pom.xml!
Related
Does anyone have a bare-bones zio-grpc server, with codegen in the project also, working with Scala 3?
I started with the HelloWorld project from their repo and attempted to get it to build with scalaVersion := "3.1.0"
Here is the relevant section in plugins.sbt:
libraryDependencies ++= Seq(
"com.thesamet.scalapb.zio-grpc" % "zio-grpc-codegen_2.13" % zioGrpcVersion,
"com.thesamet.scalapb" % "compilerplugin_2.13" % "0.11.1"
)
excludeDependencies ++= Seq(
ExclusionRule("org.scala-lang.modules", "scala-collection-compat_2.12"),
ExclusionRule("com.thesamet.scalapb", "protoc-bridge_2.12")
)
and in build.sbt:
libraryDependencies ++= Seq(
"io.grpc" % "grpc-netty" % grpcVersion,
"com.thesamet.scalapb" % "scalapb-runtime-grpc_2.13" % scalapb.compiler.Version.scalapbVersion
)
excludeDependencies ++= Seq(
ExclusionRule("org.scala-lang.modules", "scala-collection-compat_2.12"),
ExclusionRule("com.thesamet.scalapb", "protoc-bridge_2.12")
)
Since Scala 3 can use 2.13 libraries, that's what I'm doing. (Of the three zio-grpc-related libs, one, zio-grpc-codegen, does not have a Scala 3 version, so 2.13 must be used at least for that one.)
I get this error from sbt with the versions as above:
java.lang.NoSuchMethodError: scala.package$.Seq()Lscala/collection/immutable/Seq$;
at protocbridge.gens$.java(gens.scala:17)
at protocbridge.gens$.(gens.scala:11)
If I remove either of the scala-collection-compat exclusions, we get
[error] Modules were resolved with conflicting cross-version suffixes in ProjectRef(uri("file:/Users/xxx/dev/zio-grpc/examples/helloworld/project/"), "helloworld-build"):
[error] com.thesamet.scalapb:protoc-bridge _2.12, _2.13
[error] com.thesamet.scalapb:compilerplugin _3, _2.13
In short, I cannot find any permutation of Scala 2.13/3 versions of zio-grpc-codegen, compilerplugin_3, and scalapb-runtime-grpc that will not give some sbt conflicting cross-version suffix error.
TL;DR: this is not possible yet as some of the code you are using rely on macros and is not yet published for Scala3.
SBT plugins runs with Scala 2.12 no matter which Scala version is used in your project, thus you don't have to try to use plugins with _2.13 or _3 suffix, just use the regular syntax that will actually pick _2.12 artifacts.
That is, in plugins.sbt:
libraryDependencies ++= Seq(
"com.thesamet.scalapb.zio-grpc" %% "zio-grpc-codegen" % zioGrpcVersion,
"com.thesamet.scalapb" %% "compilerplugin" % "0.11.8"
)
(No need for any exclusion).
You can confirm this by looking at sbt logs and you should see that it downloads plugins for version 2.12 of Scala:
...
https://somerepository.com/com/thesamet/scalapb/zio-grpc/zio-grpc-codegen_2.12/0.5.1/zio-grpc-codegen_2.12-0.5.1.pom
https://somerepository.com/com/thesamet/scalapb/compilerplugin_2.12/0.11.8/compilerplugin_2.12-0.11.8.pom
...
Once you do that, you'll get an error as following dependencies do not exist:
com.thesamet.scalapb:scalapb-runtime_3:0.11.1
com.thesamet.scalapb:scalapb-runtime-grpc_3:0.11.1
com.thesamet.scalapb.zio-grpc:zio-grpc-core_3:0.5.0
For the 1st and 2nd, you just have to update the compilerplugin version to 0.11.8 as I did already above (the compilerplugin version is used for the main dependency scalapb-runtime-grpc).
For the 3rd, unfortunately it's not published yet for Scala 3. One attempt is to force the _2.13 version for this one with something like that in the build.sbt:
libraryDependencies += ("com.thesamet.scalapb.zio-grpc" %% "zio-grpc-core" % "0.5.1") cross CrossVersion.for3Use2_13
excludeDependencies += "com.thesamet.scalapb.zio-grpc" % "zio-grpc-core_3"
But this doesn't compile with some errors related to macros, and that is something that is not compatible between Scala 2.13 and 3. You cannot workaround that.
Remember you can check available versions of a lib for a Scala version on Maven central:
https://mvnrepository.com/artifact/com.thesamet.scalapb/compilerplugin
https://mvnrepository.com/artifact/com.thesamet.scalapb.zio-grpc/zio-grpc-core
I haven't released zio-grpc for Scala 3 as some tests related to Has were failing and were tricky to fix. At the same ZIO 2 is coming out and deprecated Has. There's a version of zio-grpc with ZIO 2 and Scala 3 support coming soon.
I am having a hard time understanding the multiple version numbers going into the build.sbt file for spark programs.
1. version
2. scalaVersion
3. spark version?
4. revision number.
There are multiple compatibility between these versions as well.
Can you please explain how to decide these versions for my project.
I hope the following SBT lines and their comments will be sufficient to explain your question.
// The version of your project itself.
// You can change this value whenever you want,
// e.g. everytime you make a production release.
version := "0.1.0"
// The Scala version your project uses for compile.
// If you use spark, you can only use a 2.11.x version.
// Also, because Spark includes its own Scala in runtime
// I recommend you use the same one;
//you can check which one your Spark instance uses in the spark-shell.
scalaVersion := "2.11.12"
// The spark version the project uses for compile.
// Because you wont generate an uber jar with Spark included,
// but deploy your jar to an spark cluster instance.
// This version must match with the remote one, unless you want weird bugs...
val SparkVersion = "2.3.1"
// Note, I use a val with the Spark version
// to make it easier to include several Spark modules in my project,
// this way, if I want/have to change the Spark version,
// I only have to modify one line,
// and avoid strange erros because I changed some versions, but not others.
// Also note the 'Provided' modifier at the end,
// it indicates SBT that it shouldn't include the Spark bits in the generated jar
// neither in package nor assembly tasks.
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % SparkVersion % Provided,
"org.apache.spark" %% "spark-sql" % SparkVersion % Provided,
)
// Exclude Scala from the assembly jar, because spark already includes it.
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
You should also take care of the SBT version, that is the version of the SBT used in your project. You set it in the "project/build.properties" file.
sbt.version=1.2.3
Note:
I use the sbt-assembly plugin, to generate a jar with all dependencies included except Spark and Scala. This is usefull if you use other libraries like the MongoSparkConnector for example.
I am building my first spark application.
http://spark.apache.org/downloads.html tells me that Spark 2.x is built against Scala 2.11.
On the Scala site https://www.scala-lang.org/download/all.html I am seeing the versions from 2.11.0 - 2.11.11
So here is my question: what exactly does the 2.11 on the Spark site mean. Is it any Scala version in the 2.11.0 - 2.11.11 range?
Another question: Can I build my Spark apps using the latest Scala 2.12.2? I assume that Scala is backward compatible, so Spark libraries built with Scala say 2.11.x can be used/called in Scala 2.12.1 applications. Am I correct?
Scala is not backwards compatible, as you assume. You must use scala 2.11 with spark unless you rebuild spark under scala 2.12 (which is an option if you want to use the latest Scala version, but requires more work to get everything working).
When considering compatibility, you need to consider both source compatibility and binary compatibility. Scala does tend to be source backwards compatible, so you can rebuild your jar under a newer version, but it is not binary backward compatible, so you can't use a jar built with an old version with code from a new version.
This is just major versions, so scala 2.10, 2.11, 2.12 etc. are all major versions and are not binary compatible (even if they are source compatible). Within a major version though compatibility is maintained, so Scala 2.11 is compatible with all versions 2.11.0 - 2.11.11 (plus any future 2.11 revisions will also be compatible)
It is for this reason that you will see most Scala libraries have separate releases for each major Scala version. You have to make sure that any library you use provides a jar for the version you are using, and that you use that jar and not one for a different version. If you use SBT %% will handle selecting the correct version for you but with maven you need to make sure to use the correct artifact name. The versions are typically prepended with _2.10, _2.11, and _2.12 referring to the scala version the jar is built for.
For anyone who wants to get jump started, this is the versioning pair I've used.
scalaVersion := "2.11.12"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.3.2",
"org.apache.spark" %% "spark-sql" % "2.3.2"
)
I used these versions of Scala and Spark and it worked OK for my need:
scalaVersion := "2.12.8"
libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.4.0"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.0"
Some libraries need 2.11 version of Scala, and in this case one should use the versions mentioned by #the775.
NOTE : This is an old answer, it is no longer available now, as newer versions of Scala and Spark exist.
I want to add a Java library (e.g. Apache PDFBox) to an sbt project.
This is the Ivy dependency:
dependency org="org.apache.pdfbox" name="pdfbox" rev="1.8.2"
I first tried to do the following:
resolvers += "Sonatype releases" at "http://oss.sonatype.org/content/repositories/releases/"
libraryDependencies += "org.apache.pdfbox" %% "pdfbox" % "1.8.2"
But it gives me errors of the type
[warn] ==== public: tried [warn]
http://repo1.maven.org/maven2/org/apache/pdfbox/pdfbox_2.10/1.8.2/pdfbox_2.10-1.8.2.pom
So I understand that with this syntax I can just manage Scala dependencies. I am sure that there is a way to manage Java dependencies, but how?
I tried to search in Google for "sbt add java dependencies" but did not find (recognize) a relevant result.
You should replace the %% (double percent) with single one.
libraryDependencies += "org.apache.pdfbox" % "pdfbox" % "1.8.2"
The double-percent is a convenience operator, and causes adding the _+scalaVersion postfix inside the path, which is _2.10 in your case. Single percent should fix the problem.
Short answer:
Use
libraryDependencies += "org.apache.pdfbox" % "pdfbox" % "1.8.2"
For java libraries, and
libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.8"
For Scala libraries, where the difference is the double % for the scala library.
Long answer:
Scala is not backward compatible across major version, so a library compiled for scala 2.12.x cannot be used by a project written in scala 2.13.x.
So when writing a scala library, you will need to compile and publish it one time per scala major version you would like to support. When using a library in a project, you would then have to pick the version compiled for the same Scala major version as your are using. Doing this manually would be cumbersome, so SBT has built in support for it.
When publishing a library, you can add the crossScalaVersions key to SBT like
crossScalaVersions := Seq( "2.10.6", "2.11.11", "2.12.3" )
And then publish with sbt +publish. This will make SBT build and publish a version of the library for both scala 2.10.6, 2.11.11 and 2.12.3. Note that the minor number is in-relevant, when it comes to compatibility for libraries. The published libraries, will have the name suffixed with _2.10, _2.11 and _2.12 to show what scala version it is for. An alternative to using the SBT build in support for this, is to use the experimental plugin sbt-projectmatrix as this gives a lot more options, and often faster builds.
When using a library sbt can also help your use the one compiled for the correct scala version, and thats where %% comes into play. When specifying a library without the _ suffix in the name, but instead use %%, then sbt will fill in suffix matching the Scala major version your use, and thereby fetch the correct version of the library.
I have started working with Actors, and was following a simple example as mention in the Getting Started Guide.
Specs:
Scala Version: 2.9.2
Akka Version: 2.0
I ran the example and it ran well. Then I changed by sbt build script to:
name := "PracAkka"
scalaVersion := "2.9.2"
resolvers += "Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/"
libraryDependencies += "com.typesafe.akka" % "akka-actor_2.10" % "2.1.2"
i.e. I started using Akka 2.1.2. There were small changes and as per the migration guide, I made the respective changes. But still I am getting the below error:
class file needed by Props is missing. reference type ClassTag of package reflect refers to nonexisting symbol.
What do I need to change?
The documentation is quite clear I'd say: http://doc.akka.io/docs/akka/2.1.2/project/migration-guide-2.0.x-2.1.x.html
(I.e. Akka 2.1.x is for Scala 2.10)
From the sbt documentation
If you use groupID %% artifactID % revision rather than groupID %
artifactID % revision (the difference is the double %% after the
groupID), sbt will add your project's Scala version to the artifact
name. This is just a shortcut. You could write this without the %%:
libraryDependencies += "org.scala-tools" % "scala-stm_2.9.1" % "0.3"
Assuming the scalaVersion for your build is 2.9.1, the following is
identical:
libraryDependencies += "org.scala-tools" %% "scala-stm" % "0.3"
The idea is that many dependencies are compiled for multiple Scala
versions, and you'd like to get the one that matches your project.
Your script seems to be getting Akka for scala 2.10 in "akka-actor_2.10" try renaming to your scala version.