Here is my build.sbt
resolvers ++= Seq("Apache Development Snapshot Repository" at "https://repository.apache.org/content/repositories/snapshots/",
Resolver.mavenLocal)
name := "flink-wiki-edits"
version := "1.0"
scalaVersion := "2.11.7"
val flinkVersion = "1.1.3"
val flinkDependencies = Seq(
"org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-connector-wikiedits"% flinkVersion )
lazy val root = (project in file(".")).
settings(
libraryDependencies ++= flinkDependencies
)
mainClass in assembly := Some("com.ta.WikipediaAnalytics")
// make run command include the provided dependencies
run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run))
// exclude Scala library from assembly
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
and the console output snippet of $ sbt assembly
vagrant#vagrant-ubuntu-trusty-64:/workspace/flink-wiki-edits$ sbt assembly
[info] Loading project definition from /workspace/flink-wiki-edits/project
[info] Set current project to flink-wiki-edits (in build file:/workspace/flink-wiki-edits/)
[info] Including: flink-connector-wikiedits_2.11-1.1.3.jar
[info] Including: flink-streaming-java_2.11-1.1.3.jar
[info] Including: flink-core-1.1.3.jar
[info] Including: flink-annotations-1.1.3.jar
[info] Including: jsr305-1.3.9.jar
[info] Including: commons-lang3-3.3.2.jar
[info] Including: slf4j-api-1.7.7.jar
[info] Including: slf4j-log4j12-1.7.7.jar
[info] Including: log4j-1.2.17.jar
[info] Including: force-shading-1.1.3.jar
[info] Including: flink-metrics-core-1.1.3.jar
[info] Including: kryo-2.24.0.jar
[info] Including: minlog-1.2.jar
where I see lot of libraries that are part of the
"org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
lineage, I assumed that provided will exclude the libraries while assembling the fat-jar, let me know if I have overlooked something.
This is caused by flink-connector-wikiedits library has dependency flink-streaming-java_2.11, see: flink-streaming-java_2.11 pom:
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.11</artifactId>
<version>1.1.3</version>
<scope>compile</scope>
</dependency>
and also flink-streaming-java_2.11 has flink-core ... dependency, see:
http://search.maven.org/#artifactdetails%7Corg.apache.flink%7Cflink-streaming-java_2.11%7C1.1.3%7Cjar
Related
I'm new in scala world (coming from android world), I have created scala project with play-framework, everything works fine now I need to add database and for that I decided to chose slick, but when I'm trying to add dependency like that
libraryDependencies ++= Seq(
"com.typesafe.slick" %% "slick" % "3.1.0",
"org.slf4j" % "slf4j-nop" % "1.6.4"
)
I'm getting this error log
Error:Error while importing SBT project:<br/>...<br/><pre>[info] Resolving com.typesafe#jse_2.10;1.2.3 ...
[info] Resolving org.scala-sbt#run;0.13.15 ...
[info] Resolving org.scala-sbt.ivy#ivy;2.3.0-sbt-48dd0744422128446aee9ac31aa356ee203cc9f4 ...
[info] Resolving com.typesafe.play#play-exceptions;2.6.5 ...
[info] Resolving org.scala-sbt#test-interface;1.0 ...
[info] Resolving com.jcraft#jsch;0.1.50 ...
[info] Resolving org.scala-lang#scala-compiler;2.10.6 ...
[info] Resolving jline#jline;2.14.3 ...
[info] Resolving org.scala-sbt#compiler-ivy-integration;0.13.15 ...
[info] Resolving org.scala-sbt#incremental-compiler;0.13.15 ...
[info] Resolving org.scala-sbt#logic;0.13.15 ...
[info] Resolving com.typesafe#config;1.3.1 ...
[info] Resolving org.scala-sbt#main-settings;0.13.15 ...
[info] Resolving com.lightbend.play#play-file-watch_2.10;1.0.0 ...
[info] Resolving com.typesafe.play#sbt-plugin;2.6.5 ...
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output.
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: com.typesafe.slick#slick_2.12;3.1.0: not found
[error] (*:update) sbt.ResolveException: unresolved dependency: com.typesafe.slick#slick_2.12;3.1.0: not found
[error] Total time: 4 s, completed Sep 28, 2017 12:21:57 PM</pre><br/>See complete log in file:/home/dev-00/.IntelliJIdea2017.1/system/log/sbt.last.log
here is my hole sbt build script
name := """play-scala-starter-example"""
version := "1.0-SNAPSHOT"
lazy val root = (project in file(".")).enablePlugins(PlayScala)
resolvers += Resolver.sonatypeRepo("snapshots")
scalaVersion := "2.12.2"
libraryDependencies ++= Seq(
"com.typesafe.slick" %% "slick" % "3.1.0",
"org.slf4j" % "slf4j-nop" % "1.6.4"
)
libraryDependencies ++= Seq(
jdbc,
"com.typesafe.play" %% "anorm" % "2.5.3"
)
libraryDependencies += guice
libraryDependencies += "org.scalatestplus.play" %% "scalatestplus-play" % "3.0.0" % Test
libraryDependencies += "com.h2database" % "h2" % "1.4.194"
libraryDependencies += jdbc
libraryDependencies += "mysql" % "mysql-connector-java" % "5.1.41"
From what I find on maven, it seems only few versions of slick are compiled against scala 2.12:
3.2.1
3.2.0
2.1.0
see: http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22com.typesafe.slick%22%20AND%20a%3A%22slick_2.12%22
Either use a more recent version of slick or another scala version (2.11.x).
Add the following line to your build.sbt
resolvers += "typesafe" at "http://repo.typesafe.com/typesafe/releases/"
I am starting to learn spark sql I am using the following dependencies in sbt. I am getting errors
name := "sparkLearning"
version := "1.0"
scalaVersion := "2.11.8"
val sparkVersion = "1.6.1"
val sqlVersion = "1.3.1"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" % "spark-sql" % sqlVersion
)
I am getting an error.
Error:Error while importing SBT project:<br/>...<br/><pre>[info] Resolving com.thoughtworks.paranamer#paranamer;2.6 ...
[info] Resolving org.scala-sbt#completion;0.13.15 ...
[info] Resolving org.scala-sbt#control;0.13.15 ...
[info] Resolving org.scala-sbt#sbt;0.13.15 ...
[info] Resolving org.scala-sbt#run;0.13.15 ...
[info] Resolving org.scala-sbt.ivy#ivy;2.3.0-sbt-48dd0744422128446aee9ac31aa356ee203cc9f4 ...
[info] Resolving org.scala-sbt#test-interface;1.0 ...
[info] Resolving com.jcraft#jsch;0.1.50 ...
[info] Resolving org.scala-lang#scala-compiler;2.10.6 ...
[info] Resolving jline#jline;2.14.3 ...
[info] Resolving org.scala-sbt#compiler-ivy-integration;0.13.15 ...
[info] Resolving org.scala-sbt#incremental-compiler;0.13.15 ...
[info] Resolving org.scala-sbt#logic;0.13.15 ...
[info] Resolving org.scala-sbt#main-settings;0.13.15 ...
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-sql;1.3.1: not found
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-sql;1.3.1: not found
[error] Total time: 15 s, completed 27-Jul-2017 15:29:52
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384M; support was removed in 8.0
Please let me know how to resolve this.
correct form for you sbt file is
name := "sparkLearning"
version := "1.0"
scalaVersion := "2.11.8"
val sparkVersion = "1.6.1"
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.10" % sparkVersion,
"org.apache.spark" % "spark-sql_2.10" % sparkVersion
)
I would suggest you to use latest spark versions which should be compatible with scala 2.11.8
name := "sparkLearning"
version := "1.0"
scalaVersion := "2.11.8"
val sparkVersion = "2.2.0"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion
)
After adding
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.0" % "test"
to build.sbt, and refreshed the project, I got this msg.
SBT project import
[warn] Multiple dependencies with the same organization/name but
different versions. To avoid conflict, pick one version:
[warn] * org.scala-lang.modules:scala-xml_2.11:(1.0.5, 1.0.4)
Changing the above to
libraryDependencies ++= Seq(
"org.scalatest" %% "scalatest" % "3.0.0" % "test"
exclude("org.scala-lang", "scala-reflect")
exclude("org.scala-lang.modules", "scala-xml_2.11")
)
solves the issue. However, instead of excluding scala-xml_2.11 version 1.0.5 from scalatest, I would like to force the scala compiler to use scala-xml_2.11 version 1.0.5 instead of version 1.0.4. (I researched the versions at https://mvnrepository.com.) Thus I tried substituting scalaVersion := "2.11.8" for
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % "2.11.8",
"org.scala-lang.modules" % "scala-xml_2.11" % "1.0.5"
)
This however results in
SBT project import
[warn] Binary version (2.11) for dependency
org.scala-lang#scala-reflect;2.11.8
[warn] in default#myproject$sources_javadoc_2.10;1.0 differs
from Scala binary version in project (2.10).
[warn] Binary version (2.11) for dependency
org.scala-lang#scala-library;2.11.8
[warn] in default#myproject$sources_javadoc_2.10;1.0 differs
from Scala binary version in project (2.10).
[warn] Multiple dependencies with the same organization/name but
different versions. To avoid conflict, pick one version:
[warn] * org.scala-lang:scala-library:(2.11.8, 2.10.4)
[warn] * org.scala-lang:scala-reflect:(2.11.8, 2.10.4)
[warn] [FAILED ]
com.artima.supersafe#supersafe_2.10.4;1.1.0!supersafe_2.10.4.jar(src):
(0ms)
[warn] ==== local: tried
[warn]
/home/user/.ivy2/local/com.artima.supersafe/supersafe_2.10.4/1.1.0/srcs/supersafe_2.10.4-sources.jar
[warn] ==== activator-local: tried [warn] /Development/Activator/activator-dist-1.3.10/repository/com.artima.supersafe/supersafe_2.1...
(show balloon)
What am I supposed to do?
Edit: What else I tried and did not work:
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
"org.scala-lang.modules" %% "scala-xml" % "1.0.5"
)
// ScalaTest
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.0" % "test"
[warn] Multiple dependencies with the same organization/name but
different versions. To avoid conflict, pick one version: [warn] *
org.scala-lang.modules:scala-xml_2.11:(1.0.5, 1.0.4)
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
"org.scala-lang.modules" %% "scala-xml_2.11" % "1.0.5"
)
// ScalaTest
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.0" % "test"
Error:Error while importing SBT project:...[info]
Resolving org.scala-sbt#run;0.13.8 ... [info] Resolving
org.scala-sbt#task-system;0.13.8 ... [info] Resolving
org.scala-sbt#tasks;0.13.8 ... [info] Resolving
org.scala-sbt#tracking;0.13.8 ... [info] Resolving
org.scala-sbt#cache;0.13.8 ... [info] Resolving
org.scala-sbt#testing;0.13.8 ... [info] Resolving
org.scala-sbt#test-agent;0.13.8 ... [info] Resolving
org.scala-sbt#test-interface;1.0 ... [info] Resolving
org.scala-sbt#main-settings;0.13.8 ... [info] Resolving
org.scala-sbt#apply-macro;0.13.8 ... [info] Resolving
org.scala-sbt#command;0.13.8 ... [info] Resolving
org.scala-sbt#logic;0.13.8 ... [info] Resolving
org.scala-sbt#precompiled-2_8_2;0.13.8 ... [info] Resolving
org.scala-sbt#precompiled-2_9_2;0.13.8 ... [info] Resolving
org.scala-sbt#precompiled-2_9_3;0.13.8 ... [trace] Stack trace
suppressed: run 'last *:update' for the full output. [trace] Stack
trace suppressed: run 'last :ssExtractDependencies' for the full
output. [error] (:update) sbt.ResolveException: unresolved
dependency: org.scala-lang.modules#scala-xml_2.11_2.11;1.0.5: not
found [error] (*:ssExtractDependencies) sbt.ResolveException:
unresolved dependency:
org.scala-lang.modules#scala-xml_2.11_2.11;1.0.5: not found [error]
Total time: 4 s, completed 01.10.2016 17:46:55
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
"org.scala-lang.modules" %% "scala-xml" % "1.0.5"
)
// ScalaTest
libraryDependencies ++= Seq(
"org.scalatest" %% "scalatest" % "3.0.0" % "test"
exclude("org.scala-lang", "scala-reflect")
exclude("org.scala-lang.modules", "scala-xml")
)
[warn] Multiple dependencies with the same organization/name but
different versions. To avoid conflict, pick one version: [warn] *
org.scala-lang.modules:scala-xml_2.11:(1.0.5, 1.0.4)
My build.sbt
name := "MyProject"
version := "0.1.0"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
"org.scala-lang.modules" %% "scala-xml" % "1.0.5"
)
// ScalaTest
//libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.0"
//libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.0" % "test"
libraryDependencies ++= Seq(
"org.scalatest" %% "scalatest" % "3.0.0" % "test"
exclude("org.scala-lang", "scala-reflect")
exclude("org.scala-lang.modules", "scala-xml_2.11")
)
The way to exclude specific transitive dependency is this:
Run sbt evicted to figure out which of the project dependencies is pulling in outdated library, let's assume the problematic library is: com.typesafe.slick.
Add the following exclude (the parentheses are important):
("com.typesafe" %% "slick" % "3.1.1").exclude("org.scala-lang.modules", "scala-xml_2.11")
Add this normally as you would be listing dependencies.
This will prevent sbt from including any version of scala-xml that was coming in as a transitive dependency of Slick.
Don't substitute; you need both scalaVersion and libraryDependencies.
Though use
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
"org.scala-lang.modules" %% "scala-xml" % "1.0.5"
)
to avoid bugs when you eventually change scalaVersion.
By removing scalaVersion you get default scalaVersion := "2.10.4" (with your version/settings of SBT, at least) but your libraryDependencies still require 2.11.
Following is the core of the project/build.sbt for a scalatra/spark project :
val ScalaVersion = "2.11.6"
val ScalatraVersion = "2.4.0-RC2-2"
// ivyScala := ivyScala.value map { _.copy(overrideScalaVersion = true)}
lazy val project = Project (
"keywordsservlet",
file("."),
settings = ScalatraPlugin.scalatraSettings ++ scalateSettings ++ Seq(
organization := Organization,
name := Name,
version := Version,
scalaVersion := ScalaVersion,
resolvers += Classpaths.typesafeReleases,
resolvers += "Scalaz Bintray Repo" at "http://dl.bintray.com/scalaz/releases",
libraryDependencies ++= Seq(
// "org.scala-lang" % "scala-reflect" % ScalaVersion,
"org.apache.spark" % "spark-core_2.11" % "1.4.1",
"org.scalatra" %% "scalatra" % ScalatraVersion,
"org.scalatra" %% "scalatra-scalate" % ScalatraVersion,
"org.scalatra" %% "scalatra-specs2" % ScalatraVersion % "test",
"ch.qos.logback" % "logback-classic" % "1.1.2" % "runtime",
"org.eclipse.jetty" % "jetty-webapp" % "9.2.10.v20150310" % "container",
"javax.servlet" % "javax.servlet-api" % "3.1.0" % "provided"
),
Here is the sbt output: notice it is loading a 2.10 target !
$ sbt
[info] Loading project definition from /shared/keywords/project
[info] Updating {file:/shared/keywords/project/}keywords-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 1 Scala source to /shared/keywords/project/target/scala-2.10/sbt-0.13/classes...
[info] Set current project to KeywordsServlet (in build file:/shared/keywords/)
So what is happening here?
There is a difference between the version of Scala that you are using for your project and the version of Scala that sbt itself uses.
sbt 0.13 can compile 2.9, 2.10 and 2.11 (and 2.12). However, when it compiles your build.sbt or Build.scala files, sbt 0.13 uses Scala 2.10.
Similarly, all of the plugins that sbt uses are compiled with 2.10.
On the other hand, sbt 0.12 used Scala 2.9.
I know there is scalacOptions in sbt. But options add via addCompilerPlugin does not exists in scalacOptions.
I type show scalacOptions in sbt console and there is nothing like -Xplugin.
So how can I get real scalac options when I am writing an sbt plugin
My sbt info
> libraryDependencies
[info] List(org.scala-lang:scala-library:2.10.4, org.scala-lang:scala-reflect:2.10.4, io.netty:netty:3.9.0.Final, net.sandrogrzicic:scalabuff-runtime:1.3.7, org.scalamacros:quasiquotes:2.0.0, org.specs2:specs2:2.3.11:test, org.scalamacros:paradise:2.0.0:plugin->default(compile))
> show scalacOptions
[info] List(-feature, -deprecation, -language:implicitConversions, -language:dynamics)
[success] Total time: 0 s, completed Jul 3, 2014 4:04:09 PM
> version
[info] 0.0.1-SNAPSHOT
> sbt
sbtBinaryVersion sbtClearOnFailure sbtDependency sbtPlugin sbtPopOnFailure sbtResolver sbtStashOnFailure
> sbtVersion
[info] 0.13.5
>
my build.sbt
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % "2.10.4",
"io.netty" % "netty" % "3.9.0.Final",
"net.sandrogrzicic" %% "scalabuff-runtime" % "1.3.7",
"org.scalamacros" %% "quasiquotes" % "2.0.0",
"org.specs2" %% "specs2" % "2.3.11" % "test")
scalacOptions ++= Seq(
"-feature",
"-deprecation",
"-language:implicitConversions",
"-language:dynamics"
)
parallelExecution in Test := false
addCompilerPlugin("org.scalamacros" % "paradise" % "2.0.0" cross CrossVersion.full)
Finally I found compile:scalacOptions actually contains the plugin settings
Are you sure that you have actually managed to add the compiler plugin?
With sbt 0.13.5 and an empty project with this in build.sbt:
addCompilerPlugin("org.scala-lang.plugins" % "continuations" % "2.10.1")
I get the plugin in both libraryDependencies and scalacOptions:
> libraryDependencies
[info] List(org.scala-lang:scala-library:2.10.4, org.scala-lang.plugins:continuations:2.10.1:plugin->default(compile))
> show scalacOptions
[info] List(-Xplugin:/Users/johan/.ivy2/cache/org.scala-lang.plugins/continuations/jars/continuations-2.10.1.jar)