I have a simple demo Scala application that reads from a file and outputs to the screen. I am trying to build this application using sbt and submit it to DataStax Spark. The SBT instructions in the DataStax documentation seems to be incomplete. https://docs.datastax.com/en/dse/6.0/dse-dev/datastax_enterprise/spark/sparkJavaApi.html
Using this as-is did not work because of the lacking link to datastax repo.
After searching around for a bit, found a sample build.sbt file from https://github.com/datastax/SparkBuildExamples/blob/master/scala/sbt/dse/build.sbt which went the furthest.
This one is failing here:
[error] unresolved dependency: org.apache.directory.api#api-ldap-codec-standalone;1.0.0.2.dse: not found
[error] unresolved dependency: org.apache.directory.api#api-ldap-extras-codec;1.0.0.2.dse: not found
[error] unresolved dependency: org.apache.directory.api#api-ldap-net-mina;1.0.0.2.dse: not found
[error] unresolved dependency: org.apache.directory.api#api-ldap-codec-core;1.0.0.2.dse: not found
[error] unresolved dependency: org.apache.directory.api#api-ldap-extras-aci;1.0.0.2.dse: not found
[error] unresolved dependency: org.apache.directory.api#api-ldap-extras-codec-api;1.0.0.2.dse: not found
[error] unresolved dependency: org.apache.directory.api#api-ldap-model;1.0.0.2.dse: not found
[error] unresolved dependency: org.apache.directory.api#api-asn1-ber;1.0.0.2.dse: not found
[error] unresolved dependency: org.apache.directory.api#api-util;1.0.0.2.dse: not found
[error] unresolved dependency: org.apache.directory.api#api-asn1-api;1.0.0.2.dse: not found
[error] unresolved dependency: org.apache.directory.api#api-i18n;1.0.0.2.dse: not found
The key sections of build.sbt is:
scalaVersion := "2.11.8"
resolvers += Resolver.mavenLocal // for testing
resolvers += "DataStax Repo" at "https://repo.datastax.com/public-repos/"
val dseVersion = "6.0.0"
libraryDependencies += "com.datastax.dse" % "dse-spark-dependencies" % dseVersion % "provided" exclude(
"org.slf4j", "log4j-over-slf4j", "org.apache.directory.api")
libraryDependencies ++= Seq(
"junit" % "junit" % "4.12" % "test"
).map(_.excludeAll(
ExclusionRule("org.slf4j","log4j-over-slf4j"),
ExclusionRule("org.slf4j","slf4j-log4j12"))
) // Excluded to allow for Cassandra to run embedded
Seems to be a broken dependency. Can you please advise.
Please try with following dependency
scalaVersion := "2.11.8"
libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "2.0.9"
Related
Recently, started learning and using Spark and Scala. Ran into the following dependency conflict issue
Scala version : 2.13.8
SBT version : 1.6.2
Spark version required : 3.2.1
[error] (update) found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
[error]
[error] * org.scala-lang.modules:scala-parser-combinators_2.13:2.1.1 (early-semver) is selected over 1.1.2
[error] +- ch.epfl.scala:hello-world_2.13:1.0 (depends on 2.1.1)
[error] +- org.apache.spark:spark-catalyst_2.13:3.2.1 (depends on 1.1.2)
[error]
libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.3.0"
... other libraries ...
dependencyOverrides += "org.scala-lang.modules" %% "scala-parser-combinators" % "2.1.1"
Add this dependencyOverrides in your build.sbt file
This is my build.sbt file
lazy val myproject = (project in file(".")).settings(
name := "Slick3Test2",
organization := "com.foo",
version := "1.2",
scalaVersion := "2.11.8",
libraryDependencies ++= Seq(
"com.typesafe.slick" %% "slick" % "3.2.0"
),
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("slick.**" -> "shaded.#1").inAll
),
artifact in (Compile, assembly) := {
val art = (artifact in (Compile, assembly)).value
art.copy(`classifier` = Some("assembly"))
},
addArtifact(artifact in (Compile, assembly), assembly)
)
I published this locally using sbt publish-local
Now I referenced this is another project build.sbt
name := "Slick2Test"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"mysql" % "mysql-connector-java" % "5.1.36",
"com.typesafe.slick" % "slick_2.11" % "2.1.0",
"com.foo" %% "Slick3Test2" % "1.2"
)
But I get an error
[info] Resolving jline#jline;2.12.1 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.foo#Slick3Test2_2.11;1.2: java.text.ParseException: inconsistent module descriptor file found in '/Users/foo/.ivy2/local/com.foo/Slick3Test2_2.11/1.2/ivys/ivy.xml': bad module name: expected='Slick3Test2_2.11' found='slick3test2_2.11';
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] com.abhi:Slick3Test2_2.11:1.2 (/Users/foo/IdeaProjects/Slick2Test/build.sbt#L7-12)
[warn] +- default:slick2test_2.11:1.0
[trace] Stack trace suppressed: run 'last *:ssExtractProject' for the full output.
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[error] (*:ssExtractProject) sbt.ResolveException: unresolved dependency: com.foo#Slick3Test2_2.11;1.2: java.text.ParseException: inconsistent module descriptor file found in '/Users/foo/.ivy2/local/com.foo/Slick3Test2_2.11/1.2/ivys/ivy.xml': bad module name: expected='Slick3Test2_2.11' found='slick3test2_2.11';
[error] (*:update) sbt.ResolveException: unresolved dependency: com.foo#Slick3Test2_2.11;1.2: java.text.ParseException: inconsistent module descripto
Your error has nothing to do with publishing FAT jar to your local repository!
I believe the error is pretty descriptive itself:
com.foo#Slick3Test2_2.11;1.2: java.text.ParseException: inconsistent module descriptor file found in '/Users/foo/.ivy2/local/com.foo/Slick3Test2_2.11/1.2/ivys/ivy.xml': bad module name: expected='Slick3Test2_2.11' found='slick3test2_2.11';
Especially: expected='Slick3Test2_2.11' found='slick3test2_2.11';
That being said, your dependency should probably look like:
libraryDependencies ++= Seq(
...
"com.foo" %% "slick3test2" % "1.2"
)
If you are not certain what is the artifactID of your library, you can always check it on your own, at:
${user.home}/.ivy2/local
Also keep in mind that local repository has precedence before cache and what is more, if some library version is found at you local cache, it will not be fetched from external servers.
Newbie in scala here, going through a scala tutorial.
I am trying to use scalatest as mentioned in the tutorial but getting this error:
[error] (*:update) sbt.ResolveException: unresolved dependency: org.scalatest#scalatest;3.0.1: not found
> scalaVersion
[info] 2.10.6
My build.sbt file looks like this:
name := "tasky"
version := "0.1.0"
libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.1"
libraryDependencies += "org.scalatest" % "scalatest" % "3.0.1" % "test"
Also as mentioned here, I have created the global.sbt and plugins.sbt file at the respective locations with the required details. But, I am still getting the error:
[info] Updating {file:/home/tushar/lsbt/tasky/}tasky...
[info] Resolving org.scalatest#scalatest;3.0.1 ...
[warn] module not found: org.scalatest#scalatest;3.0.1
[warn] ==== local: tried
[warn] /home/tushar/.ivy2/local/org.scalatest/scalatest/3.0.1/ivys/ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/org/scalatest/scalatest/3.0.1/scalatest-3.0.1.pom
[warn] ==== local-preloaded-ivy: tried
[warn] /home/tushar/.sbt/preloaded/org.scalatest/scalatest/3.0.1/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:////home/tushar/.sbt/preloaded/org/scalatest/scalatest/3.0.1/scalatest-3.0.1.pom
[warn] ==== Artima Maven Repository: tried
[warn] http://repo.artima.com/releases/org/scalatest/scalatest/3.0.1/scalatest-3.0.1.pom
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.scalatest#scalatest;3.0.1: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] org.scalatest:scalatest:3.0.1 (/home/tushar/lsbt/tasky/built.sbt#L4-5)
[warn] +- default:tasky_2.10:0.1.0
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: org.scalatest#scalatest;3.0.1: not found
Any pointers to resolve this issue?
you should use %% instead of a single % as shown below. the %% changes the artifact from scalatest to scalatest_2.10 since the version of scala used in the project is 2.10.
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.1" % "test"
You can either use %% or specify the artifact_id explicitly as below
libraryDependencies += "org.scalatest" % "scalatest_2.10" % "3.0.1" % "test"
the former approach is recommended over the later.
Explanation
Scala's minor version's are not backward compatible. so a library compiled with 2.10 version cannot be used in a project using scala 2.11. So a library has to be published by compiling it for each versions separately (2.10, 2.11 and 2.12 for eg).
thus all Scala artifacts are published with scala version suffixed at the end. Thus scalatest has a artifact scalatest_2.10,scalatest_2.11 and scalatest_2.12.
In order to make it easier to select the right version of the artifact in your SBT build depending on your scala version , SBT provides a convient symbol %% that will change the artifact id by appending the scala version for you. Thus if "org.scalatest" %% "scalatest" % "3.0.1" would translate to org.scalatest#scalatest_2.10;3.0.1 on your build dynamically.
If you are using maven repository then it should be
libraryDependencies += "org.scalatest" % "scalatest_2.11" % "3.0.1"
And your scalactic should be
libraryDependencies += "org.scalactic" % "scalactic_2.11" % "3.0.1"
Unable to resolve dependencies while trying to use latest Apache Spark 2.0.2 and Scala 2.11.7 together
I need to use the latest version of spark since it has support for SparkSession. Below configuration working fine for spark 1.6.2. But I want for spark 2.0.*
This is my dependency configuration
scalaVersion := "2.11.7"
val sparkVersion = "2.0.2"
libraryDependencies ++= Seq(
ws ,
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion,
"org.apache.spark" %% "spark-streaming" % sparkVersion,
"org.mongodb.spark" % "mongo-spark-connector_2.11" % "2.0.0",
specs2 % Test
)
And this is error message:
Error:Error while importing SBT project:
[info] Resolving org.ow2.asm#asm-tree;4.1 ...
[info] Resolving org.ow2.asm#asm-analysis;4.1 ...
[info] Resolving org.ow2.asm#asm-util;4.1 ...
[info] Resolving jline#jline;2.12.1 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.glassfish.hk2#hk2-utils;2.22.2: not found
[warn] :: org.glassfish.hk2#hk2-locator;2.22.2: not found
[warn] :: javax.validation#validation-api;${javax.validation.version}: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: org.glassfish.hk2#hk2-utils;2.22.2: not found
[error] unresolved dependency: org.glassfish.hk2#hk2-locator;2.22.2: not found
[error] unresolved dependency: javax.validation#validation-api;${javax.validation.version}: not found
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: org.glassfish.hk2#hk2-utils;2.22.2: not found
[error] unresolved dependency: org.glassfish.hk2#hk2-locator;2.22.2: not found
[error] unresolved dependency: javax.validation#validation-api;${javax.validation.version}: not found
[error] Total time: 81 s, completed 3 Dec, 2016 8:51:18 PM
Correct me if anything wrong in applying dependencies. Thank you
There's probably an error in the dependency publication chain. One option would be to exclude the broken dependencies - add exclude("org.glassfish.hk2", "hk2-utils") exclude("org.glassfish.hk2", "hk2-locator") exclude("javax.validation", "validation-api") to whatever Spark dependency that's bringing these in - and explicitly add the working versions of those three dependencies.
Also, I don't see version 2.22.2 for org.glassfish.hk2#hk2-utils or org.glassfish.hk2#hk2-locator.
Excluding at the org level (as suggested in How to exclude commons logging dependency of spring with ivy? ) helped me.
I had to use something like this :
<dependency org="org.glassfish.jersey.core" name="jersey-client" rev="2.3" conf="binaries->default">
<exclude org="org.glassfish.hk2"/>
</dependency>
<dependency org="org.glassfish.hk2" name="hk2-utils" rev="2.2.0-b14" conf="binaries->default"/>
<dependency org="org.glassfish.hk2" name="hk2-locator" rev="2.2.0-b14" conf="binaries->default"/>
<dependency org="org.glassfish.hk2" name="hk2-api" rev="2.2.0-b14" conf="binaries->default"/>
In PROJECT/plugins.sbt:
addSbtPlugin("com.typesafe.sbteclipse" % "sbtsclipse-plugin" % "4.0.0")
In PROJECT/build.sbt:
name := "FileSearcher"
version := "0.1"
scalaVersion := "2.11.8"
sbt version : 0.13.1
Below is an error I am getting:
[error] (*:update) sbt.ResolveException: unresolved dependency:
com.typesafe.sbteclipse#sbtsclipse-plugin;4.0.0: not found
[error] Could not create Eclipse project files:
[error] Error evaluating task 'scalacOptions': error
[error] Error evaluating task 'externalDependencyClasspath': error
The error was due to a typo:
addSbtPlugin("com.typesafe.**sbteclipse**" % "**sbtsclipse-plugin**" % "4.0.0")