When I try to run ( through SBT) my scala program I run into a bunch of errors.
Here's an excerpt:
[error] missing or invalid dependency detected while loading class file 'IterableUtils.class'.
[error] Could not access type ScalaObject in package scala,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'IterableUtils.class' was compiled against an incompatible version of scala.
[error] missing or invalid dependency detected while loading class file 'AsBooleanTrait.class'.
[error] Could not access type ScalaObject in package scala,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
.....
I did add the scala-library.jar to the classpath but to no avail. Does anyone know what might be missing?
Ps. used -> new SBT project (Intellij) on osx
edit: here's the build.sbt
name := "test"
version := "1.0"
scalaVersion := "2.11.8"
resolvers += "Scales XML" at "https://mvnrepository.com/artifact/org.scalesxml/scales-xml_2.9.1"
libraryDependencies += "org.scalesxml" % "scales-xml_2.9.1" % "0.3-RC7"
SBT is version 0.13.8
Edit 2:
Figured it out. I was trying to run a class (with a main method) without creating an instance... After changing it to an Object things work a lot better :)
Edit 3:
Spoke too soon. I turns out it has to do with setting the scalaVersion in build.sbt. When I leave that entire line out, it no longer complains about the missing dependencies. When I put it back in I get the errors mentioned above back. I tried setting it to 2.11.7 as well ( after installing that with brew install scala) but to no avail.
scalaVersion := "2.11.8"
libraryDependencies += "org.scalesxml" % "scales-xml_2.9.1" % "0.3-RC7"
You can't use a library compiled for Scala 2.9.1 with Scala 2.11.*. Write "org.scalesxml" %% "scales-xml" % some-version instead, which will look for scales-xml_2.11. See http://www.scala-sbt.org/0.13/docs/Cross-Build.html.
Related
i was compiling my scalafx code in intellij in a sbt file and they kept showing me this error, please help i tried everything but its still not working
[info] Updating
[info] Resolved dependencies
[warn]
[warn] Note: Unresolved dependencies path:
[error] stack trace is suppressed; run last update for the full output
[error] (update) sbt.librarymanagement.ResolveException: Error downloading org.scalafx:scalafx_2.13:8.0.192-R14
[error] Not found
[error] Not found
[error] not found: /Users/xxxxx/.ivy2/local/org.scalafx/scalafx_2.13/8.0.192-R14/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/scalafx/scalafx_2.13/8.0.192-R14/scalafx_2.13-8.0.192-R14.pom
this is my built.sbt
name := "practical4b"
version := "0.1"
scalaVersion := "2.13.1"
// https://mvnrepository.com/artifact/org.scalafx/scalafx
libraryDependencies += "org.scalafx"%%"scalafx"%"8.0.192-R14"
The problem here is that there is no release of ScalaFX 8/R14 that works with Scala 2.13.
Each Scala release, with the exception of bug-fix releases, has different binary compatibility requirements. So if you're using a Scala-built library with, say, Scala 2.13, you need the build of that library created for the same Scala version. Libraries created from Java code typically have binary compatibility with any version of Scala.
To determine which Scala versions are supported, they are listed in the column under Scala on Maven Central (the site linked to in your code above the libraryDependencies property):
You can resolve this problem either by switching to Scala 2.12, by changing the Scala version as follows:
scalaVersion := "2.12.12"
Or, you can use a more recent version of ScalaFX, that will work with Scala 2.13, by changing the library dependency as follows:
libraryDependencies += "org.scalafx" %% "scalafx" % "12.0.2-R18"
to use R18, or:
libraryDependencies += "org.scalafx" %% "scalafx" % "14-R19"
for R19.
A lot will depend upon which version of Java you are using. If you need to use Scala 2.13, with one of the more recent ScalaFX releases, I would recommend using Java 11 LTS. If you must use Java 8, then stay with the ScalaFX 8/R14 release and use Scala 2.12. Let me know if you have any questions.
I just started a clean project in Scala using SBT and have run into some issues once attempting to add dependencies. Specifically I tried adding cats-core to my dependencies in my build.sbt file:
scalaVersion := "2.13.1"
name := "hello-world"
libraryDependencies += "org.typelevel" %% "cats-core" % "2.1.0"
My sbt version is set to 1.3.7 in build.properties. However once I run compile, it shoots a bunch of angry errors saying it can't resolve the dependency, and rightfully so since the paths it is using don't exist:
[error] sbt.librarymanagement.ResolveException: Error downloading org.typelevel:cats-core:2.1.0
[error] Not found
[error] Not found
[error] not found: /home/lambda/.ivy2/local/org.typelevel/cats-core/2.1.0/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/typelevel/cats-core/2.1.0/cats-core-2.1.0.pom
I'm a bit confused as I don't seem to be running into this issue on other machines on the same network, however this persists on my local machine despite deleting ~/.ivy2 and ~/.sbt. I've also tried completely reinstalling sbt, and even building it through a clean Docker image, but the same error persists. It looks to me to be an issue with ivy trying to look for an ivy.xml although I'm fairly certain cats uses a POM.xml, but I'm unsure how to resolve this. Looking for any help resolving this!
I'm new for spark, following tutorial to learn. I have installed
openjdk version "1.8.0_121"(web-binary)
Hadoop 2.8.0 (web-binary)
scala version 2.11.8 (apt)
and spark version 2.1.1 (web-binary-pre-build-with-hadoop 2.6.0 or later).
I runned SparkPi example and successed. But, some error appears, when I try to package my first spark app with sbt 0.13.15(apt), which was installed by the way org said.
I know must be a mistake about settings somewhere, but fail to find out in this link. Could anyone help me? Thanks :)
My project is like :
---SparkApp
|---simple.sbt
|---src
|---main
|---scala
|--- SimpleApp.scala
The dot sbt file in my project is :
name := "Simple Project"
version := "0.13.15"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.1"
Error Log is like this :
hadoop#master:~/Mycode/SparkApp$ sbt package
[warn] Executing in batch mode.
[warn] For better performance, hit [ENTER] to switch to interactive mode, or
[warn] consider launching sbt without any commands, or explicitly passing 'shell'
[info] Loading project definition from /home/hadoop/Mycode/SparkApp/project
[info] Set current project to Simple Project (in build file:/home/hadoop/Mycode/SparkApp/)
[info] Compiling 1 Scala source to /home/hadoop/Mycode/SparkApp/target/scala-2.11/classes...
[error] missing or invalid dependency detected while loading class file 'SparkContext.class'.
[error] Could not access term akka in package <root>,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'SparkContext.class' was compiled against an incompatible version of <root>.
[error] one error found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 2 s, completed May 16, 2017 1:08:53 PM
Some hints might be the problem is :
When I type spark-shell, I got this Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131) , which is different with when I type java -version openjdk version "1.8.0_121". Would this be the problem?
I didn't do anything after install sbt. Should I do something for setting? like let sbt know where my scala and spark is located. How?
I didn't have maven, should I?
------------------------ Second edit -------------------
After add -Ylog-classpath in dot sbt file, like this link said. I got a very long classpath print out which is too long to show here. Problem unsolved yet.
As noted, I provide the SimpleApp.scala :
/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
object SimpleApp {
def main(args: Array[String]) {
val logFile = "file:///usr/local/spark/README.md" // Should be some file on your system
val conf = new SparkConf().setAppName("Simple Application")
val sc = new SparkContext(conf)
val logData = sc.textFile(logFile, 2).cache()
val numAs = logData.filter(line => line.contains("a")).count()
val numBs = logData.filter(line => line.contains("b")).count()
println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
}
}
tl;dr If you want to develop Spark applications you don't have to install Spark.
Having Spark installed locally does help a lot in your early days as a Spark developer (with tools like spark-shell and spark-submit), but is not required yet highly recommended.
In other words, what you've installed as a Spark package has nothing to do with what you can and want to use while developing a Spark application.
In sbt-managed Scala projects, you define what you want to use as a dependency, including Spark dependency, in libraryDependencies setting as follows:
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.1"
And to my great surprise, you did that.
It appears that you use two different project directories to explain what you're doing ~/Mycode/SparkApp (in which you execute sbt package) and ---Pro (of which you show build.sbt).
Assuming your simple.sbt looks as follows:
name := "Simple Project"
version := "0.13.15"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.1"
I could find one issue only which is version setting that I believe is 0.13.15 to reflect the version of sbt.
Please note that they are not in any way related and version is the version of your application while the version of sbt to use in the project is defined in project/build.properties that (given the latest version of sbt 0.13.15) should be as follows:
sbt.version = 0.13.15
The issue you face while executing sbt package (in /home/hadoop/Mycode/SparkApp) is that your application defines dependency on Akka as you can see in the error message:
[info] Set current project to Simple Project (in build file:/home/hadoop/Mycode/SparkApp/)
[info] Compiling 1 Scala source to /home/hadoop/Mycode/SparkApp/target/scala-2.11/classes...
[error] missing or invalid dependency detected while loading class file 'SparkContext.class'.
[error] Could not access term akka in package <root>,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'SparkContext.class' was compiled against an incompatible version of <root>.
[error] one error found
[error] (compile:compileIncremental) Compilation failed
As of Spark 1.6 or so, Akka is no longer in use by Spark so I guess the project somehow reference Akka libraries that it should not if they're for Spark.
Lots of guesswork which I hope we'll sort out soon.
Every now and then I start things from scratch to make sure I know what all the intricate details of setting up a project are. I have a simple app like the following but I get some dependency issues that don't seem to point me anywhere. I'm using scala verions 2.11.
My SBT:
name := "Helios"
version := "1.0"
scalaVersion := "2.11.8"
resolvers += "Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/"
libraryDependencies ++= Seq(
"com.typesafe.akka" % "akka-actor" % "2.0.2",
"com.typesafe.akka" % "akka-slf4j" % "2.0.5")
My Sample Class
import com.echostar.ese.helios.core.Asset
import akka.actor._
class NSPSG extends Actor {
def receive = {
case a: Asset => {
println(s"NSPSG Received asset: ${a}")
}
case _ => println("Unexpected message received")
}
}
(Asset class is just a case class with id and title in it.)
Error Message:
C:\PROJECTS\active\Helios>sbt compile
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
[info] Loading global plugins from C:\Users\dana.murad\.sbt\0.13\plugins
[info] Loading project definition from C:\PROJECTS\active\Helios\project
[info] Set current project to Helios (in build file:/C:/PROJECTS/active/Helios/)
[info] Updating {file:/C:/PROJECTS/active/Helios/}helios...
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.
[info] Compiling 3 Scala sources to C:\PROJECTS\active\Helios\target\scala-2.11\classes...
[error] missing or invalid dependency detected while loading class file 'package.class'.
[error] Could not access type ScalaObject in package scala,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'package.class' was compiled against an incompatible version of scala.
[error] missing or invalid dependency detected while loading class file 'Actor.class'.
[error] Could not access type ScalaObject in package scala,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'Actor.class' was compiled against an incompatible version of scala.
[error] C:\PROJECTS\active\Helios\src\main\scala\com\echostar\ese\helios\workers\NSPSG.scala:9: illegal inheritance;
[error] self-type com.echostar.ese.helios.workers.NSPSG does not conform to akka.actor.Actor's selftype akka.actor.Actor
[error] class NSPSG extends Actor {
[error] ^
[error] three errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 3 s, completed Apr 4, 2016 9:19:45 AM
My main App is just a println for now. It's not even calling this actor.
Am I using the wrong version of akka with scala 2.11? -Ylog-classpath didn't help
Don't know what fixed it but here's a list of things I did and the project compiles now.
Changed akka dependency line to this (added double percent and changed version back to 2.4)
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % "2.4.0"
)
Removed my Run configuration and added it back again. ( I think the path to main showed as red (invalid) so redoing it helped resolve it. I changed my package name and I don't think intelliJ picked up the rename well in build-conf)
Started another test project from IntelliJ menu (Akka Main in Scala) and that took a bit to download all dependencies. So maybe my project needed those and wasn't download them?
Removed commented out lines (got rid of a semi-colon). I don't think this did anything but full disclosure, technically I did touch the code even though my actor definition is exactly the same.
Now I somehow messed up my global sbt plugins (~/.sbt/plugins/build.sbt). They were always fine retrieved against Scala 2.9.1 which seems to be the version that sbt 0.11.3 wants, and all the plugins (sbt-gpg-plugin, sbt-idea-plugin) are published against 2.9.1.
Now whatever I do, it persistently tries to find them built against 2.9.2:
[warn] Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes.
[warn] com.github.mpeltonen:sbt-idea:1.0.0 (sbtVersion=0.11.3, scalaVersion=2.9.2)
[warn] com.jsuereth:xsbt-gpg-plugin:0.6 (sbtVersion=0.11.3, scalaVersion=2.9.2)
...
[error] {file:...}default-50be6e/*:update: sbt.ResolveException: unresolved dependency: com.github.mpeltonen#sbt-idea;1.0.0: not found
How can I fix this, so sbt retrieves the plugins for Scala 2.9.1 as before?
For the sake of completeness, this is how my files look after the suggestions:
// project-home/build.sbt
scalaVersion := "2.9.2"
...
// project-home/project/plugins.sbt
resolvers += "less is" at "http://repo.lessis.me"
addSbtPlugin( "me.lessis" % "ls-sbt" % "0.1.1" )
scalaVersion := "2.9.1" // "just in case it helps"
// ~/.sbt/plugins/build.sbt
scalaVersion := "2.9.1" // "just in case it helps"
resolvers += "sbt-idea-repo" at "http://mpeltonen.github.com/maven/"
resolvers += Resolver.url( "sbt-plugin-releases", url( "http://scalasbt.artifactoryonline.com/scalasbt/sbt-plugin-releases" ))( Resolver.ivyStylePatterns )
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.0.0")
addSbtPlugin( "com.jsuereth" % "xsbt-gpg-plugin" % "0.6" )
What is even worse, the problem persists, even after I removed ~/.sbt/plugins/build.sbt. So there are no more references (at least visible to me) to either sbt-idea or xsbt-gpg-plugin. Still I cannot compile any project any more, because sbt still tries to find those two plugins. Epic fail.
You could provide the Scala version of plugin. I didn't use ~/.sbt/, but I think it will works too.
The following is my project configuration using Scala 2.9.2 as my project compiler, and using some pluign that are compiled from Scala 2.9.1. Since Scala 2.9.1 and Scala 2.9.2 is binary compatible, I don't encounter any problem yet.
// MyProject/build.sbt
name := "MyProject"
version := "0.1"
scalaVersion := "2.9.2"
The following is plugin configuration:
// File: MyProject/project/plugins.sbt
import sbt._
import Defaults._
resolvers += Resolver.url("sbt-plugin-releases",
new URL("http://scalasbt.artifactoryonline.com/scalasbt/sbt-plugin-releases/"))(
Resolver.ivyStylePatterns)
// Resolved to:
//
// http://..../com.untyped/sbt-less/scala_2.9.1/sbt_0.11.3/0.4/jars/sbt-less.jar
//
libraryDependencies += sbtPluginExtra(
m = "com.untyped" % "sbt-less" % "0.4", // Plugin module name and version
sbtV = "0.11.3", // SBT version
scalaV = "2.9.1" // Scala version compiled the plugin
)
Some references:
SBT plugins are versioned to the scala version they were built with.
You're not the only one... Again, no answers.
And another case study, this may have answers.
A possibly relevant quote? " Remember to also remove the project/plugins -directory, because if the directory exists, project/plugins.sbt is ignored."
You can also specify sbtVersion and scalaVersion. There is an overloaded addSbtPlugin -
addSbtPlugin(dependency : sbt.ModuleID, sbtVersion : scala.Predef.String, scalaVersion : scala.Predef.String)
Ok, I went back to all the events that could have led to this. And because I found that other projects were still building, I started to understand that the problem with this particular project B was that it appeared that the two plugins were regular dependencies of project A which it depended on.
When sbt refused to acknowledge the presence of the GPG plugin which I uncommented in ~/.sbt/plugins/build.sbt, I renamed that file to ~/.sbt/plugins.sbt, and back again. That intermediate position somehow meant that the plugins were not any more plugins (despite being added as addSbtPlugin), but regular dependencies for the project A when I publish that.
I tried to find that messed up A's ivy.xml across all known Ivy2 cache directories. It was not possible to find the one that sbt was obviously seeing.
Long story, short answer: I had to bump the version of A artificially to get rid of the wrong ivy.xml cached somewhere in a hidden place. That forced sbt to re-find A with the correct ivy.xml (not depending any more on the plugins).
Now I'm just worried what happens if I put my global plugin settings into place :-#
Future advise: Never ever ever put anything into ~/.sbt/plugins.sbt. If you need sbt to understand that you changed ~/.sbt/plugins/build.sbt: Reboot your computer. Don't touch any file.