How do I set predictionIO engine template's Scala version for engine build? (PredictionIO-0.11.0-incubating scala.version=2.11.6 spark.version=2.1.0) - scala

When I try to build one of the predictionIO engines (https://github.com/peoplehum/BagOfWords_SentimentAnalysis_Template), I get error [INFO] [Engine$] Looking for an engine...
[ERROR] [Engine$] No engine found. Your build might have failed. Aborting.
I see that the build process created target/scala-2.10/santimentanalysis_bagofwords_model_2.10-1.0.jar
and target/scala-2.10/SantimentAnalysis_BagOfWords_Model-assembly-1.0-deps.jar but should have created them in target/scala-2.11.
Why is the engine building for scala-2.10 instead of scala-2.11?
I built PredictionIO with the following to handle my environment:
./make-distribution.sh -Dscala.version=2.11.6 -Dspark.version=2.1.0 -Delasticsearch.version=2.3.3
ubuntu1604vm:~/mnt/PredictionIO-0.11.0-incubating/vendors/BagOfWords_SentimentAnalysis_Template$ pio build --verbose
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/b/mnt/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/b/mnt/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[INFO] [Engine$] Using command '/home/b/mnt/PredictionIO-0.11.0-incubating/sbt/sbt' at /home/b/mnt/PredictionIO-0.11.0-incubating/vendors/BagOfWords_SentimentAnalysis_Template to build.
[INFO] [Engine$] If the path above is incorrect, this process will fail.
[INFO] [Engine$] Uber JAR disabled. Making sure lib/pio-assembly-0.11.0-incubating.jar is absent.
[INFO] [Engine$] Going to run: /home/b/mnt/PredictionIO-0.11.0-incubating/sbt/sbt package assemblyPackageDependency in /home/b/mnt/PredictionIO-0.11.0-incubating/vendors/BagOfWords_SentimentAnalysis_Template
[INFO] [Engine$] [info] Loading project definition from /home/b/mnt/PredictionIO-0.11.0-incubating/vendors/BagOfWords_SentimentAnalysis_Template/project
[INFO] [Engine$] [info] Set current project to SantimentAnalysis_BagOfWords_Model (in build file:/home/b/mnt/PredictionIO-0.11.0-incubating/vendors/BagOfWords_SentimentAnalysis_Template/)
[INFO] [Engine$] [success] Total time: 0 s, completed Apr 30, 2017 7:53:12 PM
[INFO] [Engine$] [info] Including from cache: scala-library-2.10.5.jar
[INFO] [Engine$] [info] Checking every *.class/*.jar file's SHA-1.
[INFO] [Engine$] [info] Merging files...
[INFO] [Engine$] [warn] Merging 'META-INF/MANIFEST.MF' with strategy 'discard'
[INFO] [Engine$] [warn] Strategy 'discard' was applied to a file
[INFO] [Engine$] [info] Assembly up to date: /home/b/mnt/PredictionIO-0.11.0-incubating/vendors/BagOfWords_SentimentAnalysis_Template/target/scala-2.10/SantimentAnalysis_BagOfWords_Model-assembly-1.0-deps.jar
[INFO] [Engine$] [success] Total time: 1 s, completed Apr 30, 2017 7:53:13 PM
[INFO] [Engine$] Compilation finished successfully.
[INFO] [Engine$] Looking for an engine...
[ERROR] [Engine$] No engine found. Your build might have failed. Aborting.
results of pio status
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/mnt/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar!/org/slf4j/i mpl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/mnt/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar!/org/slf4j/impl/StaticLogger Binder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[INFO] [Management$] Inspecting PredictionIO...
[INFO] [Management$] PredictionIO 0.11.0-incubating is installed at /home/mnt/PredictionIO-0.11.0-incubating
[INFO] [Management$] Inspecting Apache Spark...
[INFO] [Management$] Apache Spark is installed at /home/mnt/PredictionIO-0.11.0-incubating/vendors/spark-2.1.0-bin-hadoop2.7
[INFO] [Management$] Apache Spark 2.1.0 detected (meets minimum requirement of 1.3.0)
[INFO] [Management$] Inspecting storage backend connections...
[INFO] [Storage$] Verifying Meta Data Backend (Source: PGSQL)...
[INFO] [Storage$] Verifying Model Data Backend (Source: PGSQL)...
[INFO] [Storage$] Verifying Event Data Backend (Source: PGSQL)...
[INFO] [Storage$] Test writing to Event Store (App Id 0)...
[INFO] [Management$] Your system is all ready to go.
env
PIO_SCALA_VERSION=2.11.6
PIO_FS_BASEDIR=/home/tmp/.pio_store
PIO_ELASTICSEARCH_VERSION=2.3.3
PIO_HADOOP_VERSION=2.7
PIO_SPARK_VERSION=2.1.0
PIO_HOME=/home/mnt/PredictionIO-0.11.0-incubating

To set a predictionIO engine template's Scala version for engine pio build, edit the engine's build.sbt
For the engine I was working with, I added
scalaVersion := "2.11.6" to build.sbt
to match my configuration.. PredictionIO-0.11.0-incubating scala.version=2.11.6 spark.version=2.1.0)
PredictionIO-0.11.0-incubating/vendors/BagOfWords_SentimentAnalysis_Template/build.sbt:
assemblySettings
name := "SantimentAnalysis_BagOfWords_Model"
version := "1.0"
scalaVersion := "2.11.6"
libraryDependencies ++= Seq(
"org.apache.predictionio" %% "apache-predictionio-core" % "0.10.0-incubating" % "provided",
"org.apache.spark" %% "spark-core" % "1.5.1" % "provided",
"org.apache.spark" %% "spark-mllib" % "1.5.1" % "provided"
)

Related

PredictionIO Not able to build pio

I am trying to use UR with predictionio-0.11.0-incubating, with following configs:
elasticsearch.version=1.7.6
scala.version=2.11.8
It is giving an error message [ERROR] [Engine$] No engine found. Your build might have failed. Aborting.
It is not listing any detailed exception. Please find below the logs:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/neha/incubator-predictionio-0.11.0-incubating/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/neha/incubator-predictionio-0.11.0-incubating/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[INFO] [Engine$] Using command '/home/neha/incubator-predictionio-0.11.0-incubating/PredictionIO-0.11.0-incubating/sbt/sbt' at /home/neha/projects/universal-recommender to build.
[INFO] [Engine$] If the path above is incorrect, this process will fail.
[INFO] [Engine$] Uber JAR disabled. Making sure lib/pio-assembly-0.11.0-incubating.jar is absent.
[INFO] [Engine$] Going to run: /home/neha/incubator-predictionio-0.11.0-incubating/PredictionIO-0.11.0-incubating/sbt/sbt package assemblyPackageDependency in /home/neha/projects/universal-recommender
[INFO] [Engine$] Compilation finished successfully.
[INFO] [Engine$] Looking for an engine...
[ERROR] [Engine$] No engine found. Your build might have failed. Aborting.
Initial question posted at https://github.com/PredictionIO/template-scala-parallel-universal-recommendation/issues/63.
You must make sure the below mentioned points :
You are running pio build command in the same directory where your engine.json is placed
Mention scala version scalaVersion := "2.11.8" in your build.sbt

PredictionIO Universal Recommender Engine

I have pulled "zephrax/docker-prediction.io-universal-recommender" docker image and follow the instruction in https://github.com/PredictionIO/template-scala-parallel-universal-recommendation, when I do pio build it stops in this line and not going on :
root#0016c0672bc4:/universal-recommender-0.6.0# pio build --verbose
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/PredictionIO-0.11.0-
incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-
incubating.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/PredictionIO-0.11.0-
incubating/lib/pio-assembly-0.11.0-
incubating.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[INFO] [Engine$] Using command '/PredictionIO-0.11.0-
incubating/sbt/sbt' at /universal-recommender-0.6.0 to build.
[INFO] [Engine$] If the path above is incorrect, this process will
fail.
[INFO] [Engine$] Uber JAR disabled, but current working directory does
not look like an engine project directory. Please delete lib/pio-
assembly-0.11.0-incubating.jar manually.
[INFO] [Engine$] Going to run: /PredictionIO-0.11.0-incubating/sbt/sbt
package assemblyPackageDependency in /universal-recommender-0.6.0
[INFO] [Engine$] [info] Loading project definition from /universal-
recommender-0.6.0/project
[INFO] [Engine$] [info] Updating {file:/universal-recommender-
0.6.0/project/}universal-recommender-0-6-0-build...
[INFO] [Engine$] [info] Resolving org.scalariform#sbt-
scalariform;1.6.0 ...
[info] Resolving org.scalariform#sbt-scalariform;1.6.0
...
what is the problem ? how can I find the log in this docker image?
thank you

How to deal with library eviction when writing sbt plugin?

I try to write a simple sbt plugin which is using jsch. Due to the security reasons (ecdsa) my requirement is to use jsch in 0.1.53. I have declared the dependency in my build.sbt file:
libraryDependencies += "com.jcraft" % "jsch" % "0.1.53"
Unfortunately org.scala-sbt:ivy 2.3.0 introduces dependency on 0.1.46. SBT says that there was an eviction of the older version though in the runtime when using my plugin I can see that it's still running against 0.1.46 jsch.
[info] com.jcraft:jsch
[info] - 0.1.46
[info] status: release
[info] publicationDate: Thu Feb 02 13:39:27 CET 2012
[info] resolver: sbt-chain
[info] artifactResolver: sbt-chain
[info] evicted: true
[info] evictedData: latest-revision
[info] homepage: http://www.jcraft.com/jsch/
[info] isDefault: false
[info] configurations: default, compile, runtime, default(compile), master
[info] licenses: (Revised BSD,Some(http://www.jcraft.com/jsch/LICENSE.txt))
[info] callers: org.scala-sbt:ivy:0.13.11
[info] - 0.1.53
[info] status: release
[info] publicationDate: Fri Jun 05 17:32:00 CEST 2015
[info] resolver: sbt-chain
[info] artifactResolver: sbt-chain
[info] evicted: false
[info] homepage: http://www.jcraft.com/jsch/
[info] isDefault: false
[info] configurations: default, compile, runtime, default(compile), master
[info] licenses: (Revised BSD,Some(http://www.jcraft.com/jsch/LICENSE.txt))
[info] callers: com.evojam:sbt-ssh:0.1.1-SNAPSHOT (scalaVersion=2.10, sbtVersion=0.13), org.scala-sbt:ivy:0.13.11
I use sbt in version 0.13.11.
I have added my plugin to the test project. When running my sbt task I get clear log message:
[info] Local version string: SSH-2.0-JSCH-0.1.46
and an error about unsupported algorithm which leads me to the conclusion that my version of jsch has been overridden somehow.
The question is how to deal with library eviction when writing sbt plugin?
May be you could try with:
dependencyOverrides += "com.jcraft" % "jsch" % "0.1.53"
But I suspect the sbt version of jsch is loaded in the jvm classpath before your plugin.
There is a sbt issue to update to ivy 2.4; that would also update the used jsch version: https://github.com/sbt/sbt/issues/1920

Why are transitive dependencies not detected?

I have Scala library based on SBT, which I publish to Maven repository:
organization := "com.mycompany"
name := "mylib"
version := "0.0.1"
scalaVersion := "2.10.6"
crossScalaVersions := Seq("2.10.6", "2.11.7")
scalacOptions ++= Seq("-feature", "-unchecked", "-deprecation")
libraryDependencies ++= Seq(
"org.scalaj" %% "scalaj-http" % "2.2.0",
"org.json4s" %% "json4s-jackson" % "3.3.0",
"org.slf4j" % "slf4j-api" % "1.7.13",
"org.slf4j" % "slf4j-simple" % "1.7.13"
)
isSnapshot := true
publishMavenStyle := true
publishTo := {
Some(s3resolver.value("My Repo", s3("mybucket")).withMavenPatterns)
}
I include this library into another project:
libraryDependencies ++= Seq(
"com.mycompany" %% "mylib" % "0.0.1"
)
Running sbt sbt-dependency dependencyTree only shows:
[info] Done updating.
[info] default:another-project_2.10:1.2 [S]
[info] +-com.mycompany:mylib_2.10:0.0.1
[info]
I'm not able to see all the 3rd party dependencies: org.scalaj, org.json4s, etc.
EDIT: Moreover, when building an assembly, these dependencies are missing from the uberjar as well.
The .pom file deployed to the Maven repository do contain all the mentioned dependencies, while ~/.ivy2/cache/com.mycompany/mylib_2.10/ivy-0.0.1.xml does not.
Running sbt about in mylib/ shows:
[info] Loading global plugins from /home/michael/.sbt/0.13/plugins
[info] Loading project definition from /home/michael/Dev/projects/mylib/project
[info] Set current project to mylib (in build file:/home/michael/Dev/projects/mylib/)
[info] This is sbt 0.13.9
[info] The current project is {file:/home/michael/Dev/projects/mylib/}mylib 0.0.1
[info] The current project is built against Scala 2.10.6
[info] Available Plugins: sbt.plugins.IvyPlugin, sbt.plugins.JvmPlugin, sbt.plugins.CorePlugin, sbt.plugins.JUnitXmlReportPlugin, com.typesafe.sbteclipse.plugin.EclipsePlugin, ohnosequences.sbt.SbtS3Resolver
[info] sbt, sbt plugins, and build definitions are using Scala 2.10.5
Running sbt about in otherproject/ shows:
[info] Loading global plugins from /home/michael/.sbt/0.13/plugins
[info] Loading project definition from /home/michael/Dev/projects/otherproject/project
[info] Set current project to otherproject (in build file:/home/michael/Dev/projects/otherproject/)
[info] Updating {file:/home/michael/Dev/projects/otherproject/}otherproject...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] This is sbt 0.13.9
[info] The current project is {file:/home/michael/Dev/projects/otherproject/}otherproject 1.2
[info] The current project is built against Scala 2.10.6
[info] Available Plugins: sbt.plugins.IvyPlugin, sbt.plugins.JvmPlugin, sbt.plugins.CorePlugin, sbt.plugins.JUnitXmlReportPlugin, com.typesafe.sbteclipse.plugin.EclipsePlugin, ohnosequences.sbt.SbtS3Resolver, net.virtualvoid.sbt.graph.DependencyGraphPlugin, sbtassembly.AssemblyPlugin
[info] sbt, sbt plugins, and build definitions are using Scala 2.10.5
Here's the output from publishing to a local directory:
[info] Loading global plugins from /home/michael/.sbt/0.13/plugins
[info] Loading project definition from /home/michael/Dev/projects/mylib/project
[info] Set current project to mylib (in build file:/home/michael/Dev/projects/mylib/)
[info] Updating {file:/home/michael/Dev/projects/mylib/}mylib...
[info] Packaging /home/michael/Dev/projects/mylib/target/scala-2.10/mylib_2.10-0.0.1-sources.jar ...
[info] Done packaging.
[info] Wrote /home/michael/Dev/projects/mylib/target/scala-2.10/mylib_2.10-0.0.1.pom
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] :: delivering :: com.mycompany#mylib_2.10;0.0.1 :: 0.0.1 :: integration :: Sat Jan 09 16:10:20 IST 2016
[info] delivering ivy file to /home/michael/Dev/projects/mylib/target/scala-2.10/ivy-0.0.1.xml
[info] Main Scala API documentation to /home/michael/Dev/projects/mylib/target/scala-2.10/api...
[info] Packaging /home/michael/Dev/projects/mylib/target/scala-2.10/mylib_2.10-0.0.1.jar ...
[info] Done packaging.
model contains 9 documentable templates
[info] Main Scala API documentation successful.
[info] Packaging /home/michael/Dev/projects/mylib/target/scala-2.10/mylib_2.10-0.0.1-javadoc.jar ...
[info] Done packaging.
[info] published mylib_2.10 to /home/michael/Dev/projects/mylib/releases/com/mycompany/mylib_2.10/0.0.1/mylib_2.10-0.0.1.pom
[info] published mylib_2.10 to /home/michael/Dev/projects/mylib/releases/com/mycompany/mylib_2.10/0.0.1/mylib_2.10-0.0.1.jar
[info] published mylib_2.10 to /home/michael/Dev/projects/mylib/releases/com/mycompany/mylib_2.10/0.0.1/mylib_2.10-0.0.1-sources.jar
[info] published mylib_2.10 to /home/michael/Dev/projects/mylib/releases/com/mycompany/mylib_2.10/0.0.1/mylib_2.10-0.0.1-javadoc.jar
[success] Total time: 4 s, completed Jan 9, 2016 4:10:23 PM
What am I doing wrong?
Your build.sbt file contains the line
publishMavenStyle := true
According to the documentation
a POM is generated by the makePom action and published to the
repository instead of an Ivy file
I can suppose that ivy.xml is just not generated and the one that you see is a residue from a previous run, when publishMavenStyle was not (yet) set.
Since your artifact is published into the Maven repository, have you tried to remove the one from ~/.ivy2/cache/com.mycompany/mylib_2.10 and check the result of dependency tree listing?

Why does Play 2.3.4 and jacoco4sbt > 2.1.4 fail with NoSuchMethodError?

I'm trying to use jacoco4sbt in my Play 2.3.4 (scala 2.10.4) app but only 2.1.4 seems to work. When I use .5 or .6, I get the following error:
[error] (main/jacoco:fullClasspath) java.lang.NoSuchMethodError:
org.objectweb.asm.MethodVisitor.visitMethodInsn(ILjava/lang/String;Ljava/lang/String;Ljava/lang/String;Z)V
I tried to add asm-all (versions 4.1 and 5.0.3) to the dependencies but this does not help at all.
So what could be the problem? Do I need to add any extra dependency?
They have indeed changed the ASM version to 5.0.1, and even with this I couldn't reproduce your issue.
project/build.properties
sbt.version=0.13.7-M3
project/plugins.sbt
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.3.4")
plugins/jacoco4sbt.sbt
addSbtPlugin("de.johoop" % "jacoco4sbt" % "2.1.6")
build.sbt
name := """play-jacoco"""
version := "1.0-SNAPSHOT"
enablePlugins(PlayScala)
scalaVersion := "2.11.2"
libraryDependencies ++= Seq(
jdbc,
anorm,
cache,
ws
)
pipelineStages := Seq(rjs, digest, gzip)
jacoco.settings
When I ran jacoco:cover it worked fine.
[play-uglify] $ jacoco:cover
[info] Compiling 2 Scala sources to /Users/jacek/sandbox/play-uglify/target/scala-2.11/test-classes...
SLF4J: The following set of substitute loggers may have been accessed
SLF4J: during the initialization phase. Logging calls during this
SLF4J: phase were not honored. However, subsequent logging calls to these
SLF4J: loggers will work as normally expected.
SLF4J: See also http://www.slf4j.org/codes.html#substituteLogger
SLF4J: org.apache.http.impl.client.DefaultHttpClient
SLF4J: com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl
SLF4J: com.gargoylesoftware.htmlunit.javascript.StrictErrorReporter
SLF4J: com.gargoylesoftware.htmlunit.DefaultCssErrorHandler
SLF4J: com.gargoylesoftware.htmlunit.javascript.configuration.JavaScriptConfiguration
SLF4J: com.gargoylesoftware.htmlunit.WebClient
SLF4J: com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine
SLF4J: com.gargoylesoftware.htmlunit.WebResponse
[info] IntegrationSpec
[info]
[info] Application should
[info] + work from within a browser
[info]
[info] Total for specification IntegrationSpec
[info] Finished in 29 ms
[info] 1 example, 0 failure, 0 error
[info] ApplicationSpec
[info]
[info] Application should
[info] + send 404 on a bad request
[info] + render the index page
[info]
[info] Total for specification ApplicationSpec
[info] Finished in 29 ms
[info] 2 examples, 0 failure, 0 error
[info] Passed: Total 3, Failed 0, Errors 0, Passed 3
[info]
[info] ------- Jacoco Coverage Report --------
[info]
[info] Lines: 57.5% (>= required 0.0%) covered, 34 of 80 missed, OK
[info] Instructions: 72.12% (>= required 0.0%) covered, 522 of 1872 missed, OK
[info] Branches: 27.78% (>= required 0.0%) covered, 26 of 36 missed, OK
[info] Methods: 81.94% (>= required 0.0%) covered, 41 of 227 missed, OK
[info] Complexity: 76.73% (>= required 0.0%) covered, 57 of 245 missed, OK
[info] Class: 57.14% (>= required 0.0%) covered, 12 of 28 missed, OK
[info] Check /Users/jacek/sandbox/play-uglify/target/scala-2.11/jacoco for detail report
[info]
[success] Total time: 6 s, completed Oct 9, 2014 7:42:31 AM
Where do we differ?
After looking at Jacek answer and started removing things from my project, I found that this was cause by some custom task to run PMD that I was not aware of, which probably depends on a different asm version.
To fix this I just changed the dependency in project/plugins.sbt to exclude asm dependency:
libraryDependencies ++= Seq(
// (...)
"net.sourceforge.pmd" % "pmd" % "5.1.3" exclude("org.ow2.asm", "asm")
)
and both Jacoco and PMD started working again.