Spark Job Server Keep getting Invalid Jar - spark-jobserver

I did the build using sbt job-server-api/packages and sbt job-server-tests/packages.
I am able to extend/unzip the jar using jar -cxf command
curl --data-binary /opt/hadoop/spark-jobserver/job-server-api/target/scala-2.10/job-server-api_2.10-0.6.1-SNAPSHOT.jar localhost:8090/jars/test
{
"status": "ERROR",
"result": "Jar is not of the right format"
}
curl --data-binary /opt/hadoop/spark-jobserver/job-server-tests/target/scala-2.10/job-server-tests_2.10-0.6.1-SNAPSHOT.jar localhost:8090/jars/test
{
"status": "ERROR",
"result": "Jar is not of the right format"
}
sbt version
[info] Loading project definition from /opt/hadoop/spark-jobserver/project
Missing bintray credentials /root/.bintray/.credentials. Some bintray features depend on this.
Missing bintray credentials /root/.bintray/.credentials. Some bintray features depend on this.
Missing bintray credentials /root/.bintray/.credentials. Some bintray features depend on this.
Missing bintray credentials /root/.bintray/.credentials. Some bintray features depend on this.
[info] Set current project to root (in build file:/opt/hadoop/spark-jobserver/)
[info] job-server-tests/*:version
[info] 0.6.1-SNAPSHOT
[info] job-server-extras/*:version
[info] 0.6.1-SNAPSHOT
[info] job-server-api/*:version
[info] 0.6.1-SNAPSHOT
[info] akka-app/*:version
[info] 0.6.1-SNAPSHOT
[info] job-server/*:version
[info] 0.6.1-SNAPSHOT
[info] root/*:version
[info] 0.6.1-SNAPSHOT
I am using SPARK Version 1.5.
I am able to view Manifest.
cat MANIFEST.MF
Manifest-Version: 1.0
Implementation-Vendor: spark.jobserver
Implementation-Title: job-server-tests
Implementation-Version: 0.6.1-SNAPSHOT
Implementation-Vendor-Id: spark.jobserver
Specification-Vendor: spark.jobserver
Specification-Title: job-server-tests
Implementation-URL: https://github.com/spark-jobserver/spark-jobserver
Specification-Version: 0.6.1-SNAPSHOT
Can you help me here to resolve the jar format issue?

Put an # before the jar file path as below:
curl --data-binary #/opt/hadoop/spark-jobserver/job-server-api/target/scala-2.10/job-server-api_2.10-0.6.1-SNAPSHOT.jar localhost:8090/jars/test

Related

How to change sbt version for project

I have build.properties file in the project with the following line:
sbt.version=1.2.8
build.properties file is in the same folder as build.sbt file.
But when I enter the command "sbt about", it reports 1.0.3. Following is the actual output of the command:
/Users/kale> sbt about
[info] Loading settings from idea.sbt ...
[info] Loading global plugins from /Users/kale/.sbt/1.0/plugins
[info] Loading settings from plugins.sbt ...
[info] Loading project definition from
/Users/kale/workspace/gr8Project/myapp/project
[info] Loading settings from build.sbt ...
[info] Set current project to myapp (in build
file:/Users/kale/workspace/gr8Project/myapp/)
[info] This is sbt 1.0.3
[info] The current project is
{file:/Users/kale/workspace/gr8Project/myapp/}myapp 0.1-SNAPSHOT
[info] The current project is built against Scala 2.12.8
[info] Available Plugins: sbt.plugins.IvyPlugin,
sbt.plugins.JvmPlugin, sbt.plugins.CorePlugin,
sbt.plugins.JUnitXmlReportPlugin,
sbt.plugins.Giter8TemplatePlugin, sbtassembly.AssemblyPlugin,
com.lucidchart.sbt.scalafmt.ScalafmtCorePlugin,
com.lucidchart.sbt.scalafmt.ScalafmtPlugin,
com.lucidchart.sbt.scalafmt.ScalafmtSbtPlugin,
scoverage.ScoverageSbtPlugin,
net.virtualvoid.sbt.graph.DependencyGraphPlugin,
sbtdocker.DockerPlugin, Build, Docker
[info] sbt, sbt plugins, and build definitions are using Scala
2.12.4
How can I change sbt version to 1.2.8?
your project structure should be <root>/project/build.properties, and then you should change the sbt version property (see https://github.com/pedrorijo91/play-slick3-steps/blob/master/project/build.properties as an example)
Use sbt about to know the version in use.

Why does sbt 1.0.2 hang after compilation?

I have installed sbt 1.0.2 and tried building the "hello project", but it hangs after done compilation.
I waited for 2 hours and it does not complete. what am I missing here?
HOSTPC:IdeaProjects XYZ2017$ sbt new sbt/scala-seed.g8
[info] Loading settings from plugins.sbt ...
[info] Loading global plugins from /Users/XYZ2017/.sbt/1.0/plugins
[info] Set current project to ideaprojects (in build file:/Users/XYZ2017/IdeaProjects/)
A minimal Scala project.
name [Scala Seed Project]: hello
Template applied in ./hello
HOSTPC:IdeaProjects XYZ2017$ ls
hello
HOSTPC:IdeaProjects XYZ2017$ cd hello/
HOSTPC:hello XYZ2017$ sbt
[info] Loading settings from plugins.sbt ...
[info] Loading global plugins from /Users/XYZ2017/.sbt/1.0/plugins
[info] Loading project definition from /Users/XYZ2017/IdeaProjects/hello/project
[info] Updating {file:/Users/XYZ2017/IdeaProjects/hello/project/}hello-build...
[info] Done updating.
[info] Compiling 1 Scala source to /Users/XYZ2017/IdeaProjects/hello/project/target/scala-2.12/sbt-1.0/classes ...
[info] Done compiling.

sbt doesn't find class of managed dependency

I try to include the build of jep (https://mvnrepository.com/artifact/jep/jep/2.24) into this scala project using sbt: https://github.com/shadaj/scalapy
So, instead of building jep manually via an unmanaged dependency, I want to include it as a managed dependency. Therefore I just included:
resolvers += "jep" at "https://mvnrepository.com/artifact/"
libraryDependencies += "jep" % "jep" % "2.24"
in the build.sbt
The .jar is getting downloaded and included in my .ivy2 folder but when compiling, sbt seems not to find it:
> compile
[info] Updating {file:/C:/scalapy_indp/scalapy/}scalapy...
[info] Resolving org.sonatype.oss#oss-parent;9 ...
[info] downloading https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.12.1/scala-library-2.12.1.jar ...
[info] [SUCCESSFUL ] org.scala-lang#scala-library;2.12.1!scala-library.jar (1406ms)
[info] downloading https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.12.1/scala-reflect-2.12.1.jar ...
[info] [SUCCESSFUL ] org.scala-lang#scala-reflect;2.12.1!scala-reflect.jar (1156ms)
[info] downloading https://repo1.maven.org/maven2/jep/jep/2.24/jep-2.24.jar ...
[info] [SUCCESSFUL ] jep#jep;2.24!jep.jar (188ms)
[info] downloading https://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.12.1/scala-compiler-2.12.1.jar ...
[info] [SUCCESSFUL ] org.scala-lang#scala-compiler;2.12.1!scala-compiler.jar (2469ms)
[info] downloading https://repo1.maven.org/maven2/org/scala-lang/modules/scala-xml_2.12/1.0.6/scala-xml_2.12-1.0.6.jar ...
[info] [SUCCESSFUL ] org.scala-lang.modules#scala-xml_2.12;1.0.6!scala-xml_2.12.jar(bundle) (360ms)
[info] downloading https://repo1.maven.org/maven2/jline/jline/2.14.1/jline-2.14.1.jar ...
[info] [SUCCESSFUL ] jline#jline;2.14.1!jline.jar (250ms)
[info] Done updating.
[info] Compiling 16 Scala sources to C:\scalapy_indp\scalapy\target\scala-2.12\classes...
[info] 'compiler-interface' not yet compiled for Scala 2.12.1. Compiling...
[info] Compilation completed in 15.285 s
[error] C:\scalapy_indp\scalapy\src\gen\scala\me\shadaj\scalapy\py\ObjectTupleReaders.scala:2: not found: object jep
[error] import jep.Jep
[error] ^
The errors of course keep coming for every Jep appearance in the code.
I'm absolutely not familiar with scala or sbt, I'm just trying to get this running without having to use unmanaged dependencies i.e. having it platform independent. Building jep manually seems to be platform dependent regarding the files which are being create:
If the build succeeds it will create a directory jep/build which will
contain a jep.jar and the compiled C library of Jep, typically named
jep.so or jep.dll depending on your platform.
see https://github.com/mrj0/jep/wiki/Getting-Started
The dependency is correctly setup, but the import is wrong.
Try
import org.nfunk.jep.JEP
instead.
--
Actually, I just realized those may be two different projects. The dependencies you included is this one: http://sens.cse.msu.edu/Software/jep-2.23/doc/website/doc/doc_usage.htm

maven dependency not found in repository

I'm trying to run the project, which uses gwt-dev.
The instructions are at: http://neiliscoding.blogspot.ru/2012/05/how-to-setup-examples-for-use-in-gxt-3.html?showComment=1362999279386
when I create dependency in terminal like: mvn install:install-file -DgroupId=com.google.gwt -DartifactId=gwt-dev -Dversion=2.5.0 -Dpackaging=jar -Dfile='/home/mikhail/Загрузки/libraries/gwt-2.5.0/gwt-dev.jar'
and add dependency to the pom.xml:
<dependency>
<groupId>com.google.gwt</groupId>
<artifactId>gwt-dev</artifactId>
<version>2.5.0</version>
</dependency>
maven says:
[WARNING] You should not declare gwt-dev as a project dependency. This may introduce complex dependency conflicts
Downloading: http://repo1.maven.org/maven2/com/google/gwt/gwt-dev/2.5.0/gwt-dev-2.5.0-linux-libs.zip
[INFO] Unable to find resource 'com.google.gwt:gwt-dev:zip:linux-libs:2.5.0' in repository central (http://repo1.maven.org/maven2)
else if I remove it maven says:
[INFO] using GWT jars from project dependencies : 2.5.0
Downloading: http://repo1.maven.org/maven2/com/google/gwt/gwt-dev/2.5.0/gwt-dev-2.5.0-linux-libs.zip
[INFO] Unable to find resource 'com.google.gwt:gwt-dev:zip:linux-libs:2.5.0' in repository central (http://repo1.maven.org/maven2)
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
So I have Build error in both cases. How to deal with such situation? help me please.
I didn't managed with this problem yet but I've just found that I used 1.2 version of maven plugin. I'm not shure it is the main problem, but I'd like to deal with it anyway. so, sinse I've added this plugin according to your first link, maven drops build error:
`
[INFO] Scanning for projects...
[INFO] artifact org.apache.maven.plugins:maven-archetype-plugin: checking for updates from central
[INFO] artifact org.apache.maven.plugins:maven-archetype-plugin: checking for updates from central
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Default Project
[INFO] task-segment: [org.apache.maven.plugins:maven-archetype-plugin:RELEASE:generate] (aggregator-style)
[INFO] ------------------------------------------------------------------------
[INFO] Preparing archetype:generate
[INFO] No goals needed for project - skipping
[INFO] [archetype:generate {execution: default-cli}]
[INFO] Generating project in Batch mode
[INFO] Archetype repository missing. Using the one from [org.codehaus.mojo:gwt-maven-plugin:2.5.1-rc1] found in catalog remote
[WARNING] Property module is missing. Add -Dmodule=someValue
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Archetype org.codehaus.mojo:gwt-maven-plugin:2.5.1-rc1 is not configured
Property module is missing.
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4 seconds
[INFO] Finished at: Tue Mar 12 15:18:50 MSK 2013
[INFO] Final Memory: 27M/428M
`
and by the way, I use intellij IDEA
GWT stopped maintaining separate platform dependent libs for windows,linux and mac long time ago. GWT 1.7 was the last of such platform dependent libs. So "linux-libs" does not make sense in your error. I am guessing you are using a really old gwt-maven-plugin .
Move to latest gwt-maven-plugin.
Use <scope>provided</scope> for gwt-dev dependency to avoid it being pushed into WEB-INF/lib ( harmless but redundant )
Use global/central repository instead of installing it user repo locally.
You can also start off using GWT samples for maven projects here.
Note - your query might be a duplicate/similart to NetBeans + GWT 2.2.0 + Maven compile problem
Edit for new GWT release
GWT 2.5.1 has been released and also make note gwt-maven-plugin is of 2.5.0 ( 1 step behind GWT :)

Squeryl - "CustomTypesMode is not a member of package org.squeryl.customtypes"?

I've built Squeryl framework with SBT and am trying to use it with Scala 2.8 and NetBeans 6.9. I've defined an entity model as it is desctibed in Squeryl intro, but the IDE and compiler complain that "CustomTypesMode is not a member of package org.squeryl.customtypes". What might the reason be?
You don't have to build Squeryl, you can download the jar from http://github.com/max-l/Squeryl/downloads.
If you use sbt, you can drop the jar in the lib directory (I just tried and was able to compile the example using import org.squeryl.customtypes.CustomTypesMode._.)
Alternately you should be able to this line to your sbt project definition:
val squeryl = "org.squeryl" % "squeryl_2.8.0" % "0.9.4beta8"
Then run update:
$ sbt
[info] Recompiling project definition...
[info] Source analysis: 1 new/modified, 0 indirectly invalidated, 0 removed.
[info] Building project test 1.0 against Scala 2.8.0
[info] using MainProject with sbt 0.7.4 and Scala 2.7.7
> update
[info]
[info] == update ==
[info] downloading http://scala-tools.org/repo-releases/org/squeryl/squeryl_2.8.0/0.9.4beta8/squeryl_2.8.0-0.9.4beta8.jar ...
[info] [SUCCESSFUL ] org.squeryl#squeryl_2.8.0;0.9.4beta8!squeryl_2.8.0.jar (1425ms)
[info] downloading http://repo1.maven.org/maven2/cglib/cglib-nodep/2.2/cglib-nodep-2.2.jar ...
[info] [SUCCESSFUL ] cglib#cglib-nodep;2.2!cglib-nodep.jar (909ms)
[info] :: retrieving :: test#test_2.8.0 [sync]
[info] confs: [compile, runtime, test, provided, system, optional, sources, javadoc]
[info] 2 artifacts copied, 0 already retrieved (1268kB/60ms)
[info] == update ==
[success] Successful.