I need to use symbols from the kafka project in spark (to use the DefaultDecoder rather than the StringDecoder). Since these symbols are in kafka I need to link both kafka and spark in my sbt project. Here is reduced sbt file that isolates my exact problem:
name := """spark-kafka"""
version := "1.0"
scalaVersion := "2.10.4"
lazy val root = (project in file("."))
libraryDependencies ++= Seq(
"org.apache.kafka" % "kafka_2.10" % "0.8.2.0",
"com.typesafe.scala-logging" %% "scala-logging-slf4j" % "2.1.2",
"org.apache.spark" %% "spark-core" % "1.2.1" % "provided"
)
If I try and build this with sbt compile, I get this error:
> compile
[info] Updating {file:/home/rick/go/src/defend7/sparksprint/tools/consumer/}root...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[error] impossible to get artifacts when data has not been loaded. IvyNode = org.slf4j#slf4j-api;1.6.1
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) java.lang.IllegalStateException: impossible to get artifacts when data has not been loaded. IvyNode = org.slf4j#slf4j-api;1.6.1
[error] Total time: 8 s, completed Feb 21, 2015 1:37:05 PM
This is the same error I get in my less-isolated larger sbt project so that is why I think this smaller sbt file has isolated just the problem I am facing.
I have tried to understand what 'data' sbt is talking about in 'impossible to get artifacts when data has not been loaded' and have also tried some of the common remedies (such as explicitly including slf4j-api 1.6.1 in my libraryDependencies) but this has gotten me nowhere so far.
I'm really stuck and would be very grateful for any help. Thanks!
It looks like a conflict resolution problem somewhere deep in Ivy. It might be fixed by manually excluding slf4j dependency from Kafka and explicitly adding dependency on latest version:
libraryDependencies ++= Seq(
"org.apache.kafka" % "kafka_2.10" % "0.8.2.0" excludeAll(
ExclusionRule(organization = "org.slf4j")
),
"com.typesafe.scala-logging" %% "scala-logging-slf4j" % "2.1.2",
"org.slf4j" % "slf4j-api" % "1.7.10",
"org.apache.spark" %% "spark-core" % "1.2.1" % "provided"
)
Related
I have installed Flink, Scala and sbt
Flink Version: 1.9.1
Scala Version: 2.10.6
Sbt Version: 1.3.7
I made relevant changes in build.sbt.
Compile command is failing
Here is the relevant information.
Any information is greatly appreciated
**Versions Information
[osboxes#osboxes local]$ scala -version
Scala code runner version 2.10.6 -- Copyright 2002-2013, LAMP/EPFL
[osboxes#osboxes local]$ flink --version
Version: 1.9.1, Commit ID: 4d56de8
[osboxes#osboxes readcsvfile]$ sbt -version
sbt version in this project: 1.3.7
sbt script version: 1.3.7
** build.sbt changes
val flinkVersion = "1.9.1"
val flinkDependencies = Seq(
"org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided")
** Compile Errors
sbt:readCsvfile> compile
[info] Updating
[info] Resolved dependencies
[warn]
[warn] Note: Unresolved dependencies path:
[error] stack trace is suppressed; run last update for the full output
[error] (update) sbt.librarymanagement.ResolveException: Error downloading org.apache.flink:flink-streaming-scala_2.13:1.9.1
[error] Not found
[error] Not found
[error] not found: /home/osboxes/.ivy2/local/org.apache.flink/flink-streaming-scala_2.13/1.9.1/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/flink/flink-streaming-scala_2.13/1.9.1/flink-streaming-scala_2.13-1.9.1.pom
[error] Error downloading org.apache.flink:flink-scala_2.13:1.9.1
[error] Not found
[error] Not found
[error] not found: /home/osboxes/.ivy2/local/org.apache.flink/flink-scala_2.13/1.9.1/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/flink/flink-scala_2.13/1.9.1/flink-scala_2.13-1.9.1.pom
[error] Total time: 4 s, completed Jan 30, 2020 3:59:12 PM
sbt:readCsvfile>
Few points I want to mention here regarding the SBT dependencies issues are:
Please add scalaVersion := "2.12.11" in build.sbt file like this, which includes the Scala version in your SBT dependencies automatically due to this%%.
name := "flink-streaming-demo"
scalaVersion := "2.12.11"
val flinkVersion = "1.10.0"
libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
If you want Scala version specific SBT dependencies then use % like this:
libraryDependencies += "org.apache.flink" % "flink-scala_2.12" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" % "flink-streaming-scala_2.12" % flinkVersion % "provided"
In worst case if all these does not work then simply delete or rename these existing .sbt and .ivy2 hidden folder in your system home directory, where your all dependecies and plugins get sotred after downloading from maven central and then refresh/build the SBT project.
SBT dependency format
libraryDependencies += groupID % artifactID % revision % configuration
Meaning of % and %%
%: A method used to construct an Ivy Module ID from the strings you supply.
%%: When used after the groupID, it automatically adds your project’s Scala version (such as _2.12) to the end of the artifact name.
NOTE: To get more details click here.
summing up the comments since perhaps it is a bit hard to know what you should do
In general, if you get an "Unresolved dependencies" error, look at mvnrepository.com, search for your artifact:
https://mvnrepository.com/artifact/org.apache.flink/flink-scala
This tells you (second column) which Scala versions are supported by it. In this case, the library is available for 2.11.x and 2.12.x.
Thus, you have to use a Scala version compatible with that in your build, in build.sbt:
ThisBuild / scalaVersion := "2.12.10"
I'm new to Scala and working with IntelliJ Community Edition.
I open a new project and edit the build.sbt file to contain the following:
name := "test"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.locationtech.geotrellis" %% "geotrellis-raster" % "1.0.0" % "1.1.0" ,
"org.locationtech.geotrellis" % "geotrellis-shapefile_2.11" % "1.1.0"
).map(_
.exclude("com.azavea.geotrellis", "geotrellis-proj4_2.11")
.exclude("com.azavea.geotrellis", "geotrellis-raster_2.11")
)
After that, I refresh the project, and I get the following errors in the error file:
[error] (*:ssExtractDependencies) java.lang.IllegalArgumentException: Cannot add dependency 'org.locationtech.geotrellis#geotrellis-raster_2.11;1.0.0' to configuration '1.1.0' of module default#test_2.11;1.0 because this configuration doesn't exist!
[error] (*:update) java.lang.IllegalArgumentException: Cannot add dependency 'org.locationtech.geotrellis#geotrellis-raster_2.11;1.0.0' to configuration '1.1.0' of module default#test_2.11;1.0 because this configuration doesn't exist!
What am I doing wrong here?
Main issue seems to be two different version numbers on one of your dependencies.
"org.locationtech.geotrellis" %% "geotrellis-raster" % "1.0.0" % "1.1.0"
Should be
"org.locationtech.geotrellis" %% "geotrellis-raster" % "1.0.0"
or
"org.locationtech.geotrellis" %% "geotrellis-raster" % "1.1.0"
if you want the 1.1.0 version
I have a problem, I want to build my scala project with sbt.
I have scala 2.11.8, sbt about print
[info] Loading project definition from /home/xxx
[info] Set current project to base (in build file:/home/xxx)
[info] This is sbt 0.13.8
[info] The current project is {file:/home/xxx}
[info] The current project is built against Scala 2.11.8
[info]
[info] sbt, sbt plugins, and build definitions are using Scala 2.10.4
but still, when I run sbt compile I have exception with :
[error] during phase: typer
[error] library version: version 2.10.4
[error] compiler version: version 2.10.4
can someone help me?
Thanks
EDIT Forgot build.sbt file
build.sbt
name := "myproject"
version := "0.1"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"io.jvm.uuid" %% "scala-uuid" % "0.2.1",
"org.scala-lang.modules" %% "scala-parser-combinators" % "1.0.4",
"org.scalatest" % "scalatest_2.11" % "2.2.6" % "test",
"com.github.nscala-time" %% "nscala-time" % "2.12.0",
"ch.qos.logback" % "logback-classic" % "1.1.7",
"com.typesafe.scala-logging" %% "scala-logging" % "3.4.0",
"org.scala-lang.modules" %% "scala-xml" % "1.0.4",
"org.scalacheck" %% "scalacheck" % "1.13.0" % "test"
)
I don't have any build.scala for this project
For Fedora 23. I initially installed sbt with the bintray repository. I had to remove it and re install sbt without the bintray repo.
I was not working so I re-re installed sbt avec bintray repository and it works. Some configuration somewhere was messed up I think...
I am a new user to Scala, following the way to create a scala sbt project.
https://www.youtube.com/watch?v=Ok7gYD1VbNw
After adding
libraryDependencies += "org.scalatest" % "scalatest_2.11" % "2.2.4" % "test"
to build.sbt, and refreshed the project, I got this msg.
[warn] Multiple dependencies with the same organization/name but different versions. To avoid conflict, pick one version:
[warn] * org.scala-lang:scala-reflect:(2.11.2, 2.11.7)
[warn] * org.scala-lang.modules:scala-xml_2.11:(1.0.2, 1.0.4)
And in build.sbt, thw word 'scalatest' is red that means it's an unsolved dependencies.
Should I change something in Project Setting, like scala sdk?
Best Regard!
You could regard adding those dependencies:
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % "2.11.7",
"org.scala-lang.modules" % "scala-xml_2.11" % "1.0.4"
)
It forces compiler to choose concrete version of libraries. It solves problem for me.
I was able to resolve this by excluding these from the scalatest dependency.
libraryDependencies ++= Seq(
"org.scalatest" % "scalatest_2.11" % "2.2.4" % "test"
exclude("org.scala-lang", "scala-reflect")
exclude("org.scala-lang.modules", "scala-xml_2.11")
)
I use Cassandra Spark connector in Scala.
Here is my sample code that is working to save data to Cassandra.
val data = rdd.map
(f => new CassandraRow(IndexedSeq("pk", "count"), IndexedSeq(f._1, f._2.toString())))
data.saveToCassandra("keyspace", "table")
Then, I try to use TTL. Here is my sample code for adding TTL.
adding import
import com.datastax.spark.connector.writer.{TTLOption, WriteConf}
and adding TTL to saveToCassandra
data.saveToCassandra
("keyspace", "table", writeConf = WriteConf(ttl = TTLOption.constant(604800))
However, when I compiled it, it gets Errors.
Error Messages
[error] bad symbolic reference. A signature in TTLOption.class refers to term streaming
[error] in package org.apache.spark which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling TTLOption.class.
[error] one error found
[error] (compile:compile) Compilation failed
[error] Total time: 4 s, completed Mar 31, 2015 11:55:14 AM
build.sbt
name := "Cassandra"
version := "1.0"
scalaVersion := "2.10.4"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.2.1"
libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "1.2.1"
libraryDependencies += "com.datastax.spark" % "spark-cassandra-connector_2.10" % "1.2.0-rc2"
Is there problem with my code??
This is a known bug in the current connector relating to a dependency on the
streaming spark libraries.
https://datastax-oss.atlassian.net/browse/SPARKC-113
If this is blocking and you need a workaround you can just include the spark-streaming library as a dependency in the build file.
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.2.1"