ScalaPB and shading with SBT isn't working - scala

I'm trying to get https://scalapb.github.io/">ScalaPB to work with Spark. In order to do this I need to shade my protocol buffers within SBT using code like below
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.protobuf.**" -> "shadeproto.#1").inAll
)
This doesn't work within spark-shell. In fact, it doesn't even work within the Scala REPL. Reproduced below are my errors within the REPL.
Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_102).
Type in expressions for evaluation. Or try :help.
scala> val x = zefr.commons.proto.Video.VideoProto.descriptor
error: missing or invalid dependency detected while loading class file 'VideoProto.class'.
Could not access term google in package com,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'VideoProto.class' was compiled against an incompatible version of com.
error: missing or invalid dependency detected while loading class file 'VideoProto.class'.
Could not access term protobuf in value com.google,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'VideoProto.class' was compiled against an incompatible version of com.google.
error: missing or invalid dependency detected while loading class file 'GeneratedMessageCompanion.class'.
Could not access term google in package com,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'GeneratedMessageCompanion.class' was compiled against an incompatible version of com.
error: missing or invalid dependency detected while loading class file 'GeneratedMessageCompanion.class'.
Could not access term protobuf in value com.google,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'GeneratedMessageCompanion.class' was compiled against an incompatible version of com.google.
error: missing or invalid dependency detected while loading class file 'VideoProto.class'.
Could not access term Descriptors in value com.protobuf,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'VideoProto.class' was compiled against an incompatible version of com.protobuf.
error: missing or invalid dependency detected while loading class file 'VideoProto.class'.
Could not access term protobuf in value com.google,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'VideoProto.class' was compiled against an incompatible version of com.google.
error: missing or invalid dependency detected while loading class file 'VideoProto.class'.
Could not access term google in package com,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'VideoProto.class' was compiled against an incompatible version of com.
error: missing or invalid dependency detected while loading class file 'VideoProto.class'.
Could not access type Descriptor in value com.Descriptors,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'VideoProto.class' was compiled against an incompatible version of com.Descriptors.
Here is a fully copy of my build.sbt file
scalaVersion := "2.11.8"
organization := "zefr.commons"
name := "commons-proto"
val nexus = "http://nexus.zefr.com/repository/maven"
resolvers ++= Seq(
Resolver.sonatypeRepo("releases"),
Resolver.sonatypeRepo("snapshots"),
"Neuxs Snapshots" at s"$nexus-snapshots",
"Neuxs Releases" at s"$nexus-releases"
)
conflictManager := ConflictManager.strict
PB.targets in Compile := Seq(
scalapb.gen() -> (sourceManaged in Compile).value
)
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.protobuf.**" -> "shadeproto.#1").inAll
)
And project/protos.sbt
addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.3")
libraryDependencies += "com.trueaccord.scalapb" %% "compilerplugin" % "0.5.45-p3"
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")
Can anyone please help me understand what I'm missing here?
Thanks
EDIT
In order to see if this was a problem that I was introducing I downloaded this project and built it as is, only changing the Scala version to 2.11.8 and I get the same error.
scala> val x = com.example.protos.demo.Person.descriptor
error: missing or invalid dependency detected while loading class file 'Person.class'.
Could not access term google in package com,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Person.class' was compiled against an incompatible version of com.
error: missing or invalid dependency detected while loading class file 'Person.class'.
Could not access term protobuf in value com.google,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Person.class' was compiled against an incompatible version of com.google.
error: missing or invalid dependency detected while loading class file 'GeneratedMessageCompanion.class'.
Could not access term google in package com,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'GeneratedMessageCompanion.class' was compiled against an incompatible version of com.
error: missing or invalid dependency detected while loading class file 'GeneratedMessageCompanion.class'.
Could not access term protobuf in value com.google,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'GeneratedMessageCompanion.class' was compiled against an incompatible version of com.google.
error: missing or invalid dependency detected while loading class file 'Person.class'.
Could not access term Descriptors in value com.protobuf,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Person.class' was compiled against an incompatible version of com.protobuf.
error: missing or invalid dependency detected while loading class file 'Person.class'.
Could not access term protobuf in value com.google,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Person.class' was compiled against an incompatible version of com.google.
error: missing or invalid dependency detected while loading class file 'Person.class'.
Could not access term google in package com,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Person.class' was compiled against an incompatible version of com.
error: missing or invalid dependency detected while loading class file 'Person.class'.
Could not access type Descriptor in value com.Descriptors,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Person.class' was compiled against an incompatible version of com.Descriptors.

Related

Can't compile scala/SBT project

When I try to run ( through SBT) my scala program I run into a bunch of errors.
Here's an excerpt:
[error] missing or invalid dependency detected while loading class file 'IterableUtils.class'.
[error] Could not access type ScalaObject in package scala,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'IterableUtils.class' was compiled against an incompatible version of scala.
[error] missing or invalid dependency detected while loading class file 'AsBooleanTrait.class'.
[error] Could not access type ScalaObject in package scala,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
.....
I did add the scala-library.jar to the classpath but to no avail. Does anyone know what might be missing?
Ps. used -> new SBT project (Intellij) on osx
edit: here's the build.sbt
name := "test"
version := "1.0"
scalaVersion := "2.11.8"
resolvers += "Scales XML" at "https://mvnrepository.com/artifact/org.scalesxml/scales-xml_2.9.1"
libraryDependencies += "org.scalesxml" % "scales-xml_2.9.1" % "0.3-RC7"
SBT is version 0.13.8
Edit 2:
Figured it out. I was trying to run a class (with a main method) without creating an instance... After changing it to an Object things work a lot better :)
Edit 3:
Spoke too soon. I turns out it has to do with setting the scalaVersion in build.sbt. When I leave that entire line out, it no longer complains about the missing dependencies. When I put it back in I get the errors mentioned above back. I tried setting it to 2.11.7 as well ( after installing that with brew install scala) but to no avail.
scalaVersion := "2.11.8"
libraryDependencies += "org.scalesxml" % "scales-xml_2.9.1" % "0.3-RC7"
You can't use a library compiled for Scala 2.9.1 with Scala 2.11.*. Write "org.scalesxml" %% "scales-xml" % some-version instead, which will look for scales-xml_2.11. See http://www.scala-sbt.org/0.13/docs/Cross-Build.html.

Trying to compile a project with Scalatra and failing

I'm starting a new project using Scalatra and when trying to build the process the compiler says:
[info] Compiling 13 Scala sources to /home/mauricio/projects/scala/id2-search-proxy/target/scala-2.11/classes...
[error] missing or invalid dependency detected while loading class file 'ScalatraServlet.class'.
[error] Could not access term servlet in package javax,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'ScalatraServlet.class' was compiled against an incompatible version of javax.
[error] missing or invalid dependency detected while loading class file 'ScalatraServlet.class'.
[error] Could not access type HttpServlet in value javax.http,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'ScalatraServlet.class' was compiled against an incompatible version of javax.http.
[error] missing or invalid dependency detected while loading class file 'DynamicScope.class'.
[error] Could not access term servlet in package javax,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'DynamicScope.class' was compiled against an incompatible version of javax.
[error] missing or invalid dependency detected while loading class file 'RequestResponseScope.class'.
[error] Could not access term servlet in package javax,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'RequestResponseScope.class' was compiled against an incompatible version of javax.
[error] missing or invalid dependency detected while loading class file 'ScalatraContext.class'.
[error] Could not access term servlet in package javax,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'ScalatraContext.class' was compiled against an incompatible version of javax.
[error] missing or invalid dependency detected while loading class file 'DynamicScope.class'.
[error] Could not access type HttpServletResponse in value javax.http,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'DynamicScope.class' was compiled against an incompatible version of javax.http.
[error] missing or invalid dependency detected while loading class file 'DynamicScope.class'.
[error] Could not access type HttpServletRequest in value javax.http,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'DynamicScope.class' was compiled against an incompatible version of javax.http.
[error] missing or invalid dependency detected while loading class file 'SessionSupport.class'.
[error] Could not access term servlet in package javax,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'SessionSupport.class' was compiled against an incompatible version of javax.
[error] missing or invalid dependency detected while loading class file 'SessionSupport.class'.
[error] Could not access type HttpSession in value javax.http,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'SessionSupport.class' was compiled against an incompatible version of javax.http.
[error] missing or invalid dependency detected while loading class file 'ServletApiImplicits.class'.
[error] Could not access term servlet in package javax,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'ServletApiImplicits.class' was compiled against an incompatible version of javax.
[error] missing or invalid dependency detected while loading class file 'Route.class'.
[error] Could not access term servlet in package javax,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'Route.class' was compiled against an incompatible version of javax.
[error] missing or invalid dependency detected while loading class file 'ScalatraServlet.class'.
[error] Could not access type ServletConfig in value javax.servlet,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'ScalatraServlet.class' was compiled against an incompatible version of javax.servlet.
[error] missing or invalid dependency detected while loading class file 'ServletApiImplicits.class'.
[error] Could not access type ServletContext in value javax.servlet,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'ServletApiImplicits.class' was compiled against an incompatible version of javax.servlet.
[error] missing or invalid dependency detected while loading class file 'ServletApiImplicits.class'.
[error] Could not access type HttpSession in value javax.http,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'ServletApiImplicits.class' was compiled against an incompatible version of javax.http.
[error] missing or invalid dependency detected while loading class file 'ServletApiImplicits.class'.
[error] Could not access type HttpServletResponse in value javax.http,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'ServletApiImplicits.class' was compiled against an incompatible version of javax.http.
[error] missing or invalid dependency detected while loading class file 'ServletApiImplicits.class'.
[error] Could not access type HttpServletRequest in value javax.http,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'ServletApiImplicits.class' was compiled against an incompatible version of javax.http.
[error] 16 errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 6 s, completed 23/09/2015 13:03:32
I can't find these broken references anywhere at the scalatra source code or at the files I have here, all imports look right. Any idea what's going on here?
My SBT file is like this:
name := "id2-search-proxy"
version := "1.0"
scalaVersion := "2.11.7"
lazy val versions = new {
val scalatra = "2.4.0-RC2-2"
val specs2 = "3.6.4-20150916230958-8339115"
}
resolvers ++= Seq(
Resolver.sonatypeRepo("releases"),
"Twitter Maven" at "https://maven.twttr.com"
)
libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.1.3"
libraryDependencies += "com.ning" % "async-http-client" % "1.9.31"
libraryDependencies += "commons-io" % "commons-io" % "2.4"
libraryDependencies += "io.argonaut" %% "argonaut" % "6.1-M4"
libraryDependencies += "oauth.signpost" % "signpost-core" % "1.2.1.2"
libraryDependencies += "oauth.signpost" % "signpost-commonshttp4" % "1.2.1.2"
libraryDependencies += "org.mockito" % "mockito-core" % "1.10.19" % Test
libraryDependencies += "org.specs2" %% "specs2-core" % versions.specs2 % Test
libraryDependencies += "org.specs2" %% "specs2-mock" % versions.specs2 % Test
libraryDependencies += "org.scalatra" %% "scalatra" % versions.scalatra
libraryDependencies += "org.scalatra" %% "scalatra-specs2" % versions.scalatra % Test
resolvers += "twitter-repo" at "http://maven.twttr.com/"
resolvers += "scalaz-bintray" at "http://dl.bintray.com/scalaz/releases"
scalacOptions in Test ++= Seq("-Yrangepos", "-Ylog-classpath")
Since servlet-api was set to provided it would be loaded on Idea but wouldn't be available during compilation, directly depending on it fixed the issue:
libraryDependencies += "javax.servlet" % "javax.servlet-api" % "3.1.0"

Play 2.3 got incompatible errors for DefaultReads

I am trying to update play to 2.3 from 2.2, but I got several errors complaining about DefaultReads incompatible.
[error] Could not access term time in package java,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'DefaultReads.class' was compiled against an incompatible version of java.
[error] (jsObj \ "items").asOpt[JsArray] match {
The problem is play-cache dependency. I shouldn't use the latest one. After I use the one matched with Play version, it works.
"com.typesafe.play" % "play-cache_2.11" % "2.3.8"
As suggested by #richard-close, the issue was caused in my case by having a jdk7 used by sbt.
You can check that with
sbt
> eval System.getProperty("java.version")
[info] ans: String = 1.7.0_80
> eval System.getProperty("java.home")
[info] ans: String = /usr/lib/jvm/java-7-oracle/jre

Run sbt with -Ylog-classpath

I got an error like this when I run 'sbt compile':
missing or invalid dependency detected while loading class file 'DefaultReads.class'.
[error] Could not access term time in package java,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
How can I pass -Ylog-classpath to sbt?
-Y is just a scalac parameter. There are two types as per the doc -Y being private and -X being advanced. The difference is that private ones can be removed without prior notice and you should be aware of that fact.
To add scalac options add to your build.sbt file this line:
scalacOptions += "-Ylog-classpath"
Similarly you can add javac options with javacOptions.

ScalaFX: HelloWorld compilation error

I'm new to Scala and SBT so I might be missing something obvious.
I was trying to compile the HelloWorld example on http://www.scalafx.org/docs/quickstart/
I created a file build.sbt containing:
scalaVersion := "2.11.5"
libraryDependencies += "org.scalafx" %% "scalafx" % "8.0.0-R4"
and a file src/main/scala/ScalaFXHelloWorld.scala containing the code from linked page.
However, when running sbt run I get:
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
[info] Set current project to scalafx (in build file:/home/kvbx/Projects/ScalaFX/)
[info] Compiling 1 Scala source to /home/kvbx/Projects/ScalaFX/target/scala-2.11/classes...
[error] missing or invalid dependency detected while loading class file 'Color.class'.
[error] Could not access term javafx in package <root>,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'Color.class' was compiled against an incompatible version of <root>.
[error] missing or invalid dependency detected while loading class file 'Color.class'.
[error] Could not access term scene in value javafx,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'Color.class' was compiled against an incompatible version of javafx.
[error] missing or invalid dependency detected while loading class file 'Stage.class'.
[error] Could not access term javafx in package <root>,
...
...
I'm running sbt 0.13.7 and scala 2.11.5 on openjdk 1.8.0_31 on Archlinux
JavaFX isn't part of OpenJDK 8. I installed openjfx. Works. (Thanks Jasper)