I am trying to inject a custom handler to jetty.
I have written the handler in my application code which is packaged as a war.
package com.foo.bar
import javax.servlet.http.{HttpServletRequest, HttpServletResponse}
import org.eclipse.jetty.server.Request
import org.eclipse.jetty.server.handler.AbstractHandler
// scalastyle:off println
class CustomJettyHandler extends AbstractHandler {
override def handle(target: String, baseRequest: Request,
request: HttpServletRequest, response: HttpServletResponse): Unit = {
println("This is a custom jetty handler")
}
}
// scalastyle:on println
I have then injected this handler in the jetty.xml file:
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_3.dtd">
<Configure id="Server" class="org.eclipse.jetty.server.Server">
........
........
<Call name="insertHandler">
<Arg>
<New id="CustomJettyHandler" class="com.foo.bar.CustomJettyHandler"/>
</Arg>
</Call>
........
I am now running jetty in standalone mode. Note that I am passing the location where CustomJettyHandler.class resides to jetty-start.jar.
java -server -Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.StdErrLog -jar lib/jetty-start.jar OPTIONS=All --lib=lib/* --lib=webapps/root/WEB-INF/classes/com/foo/bar/* etc/jetty.xml etc/jetty-jmx.xml --debug
In my application logs, I can see jetty loading my custom handler to its classpath but then eventually failing because of ClassNotFoundException. Can someone point out where could this be going wrong?
.......
rawlibref = webapps/root/WEB-INF/classes/com/foo/bar/*
expanded = webapps/root/WEB-INF/classes/com/foo/bar/*
getPaths('webapps/root/WEB-INF/classes/com/foo/bar/*')
Using relative path pattern: glob:**/webapps/root/WEB-INF/classes/com/foo/bar/*
Found [webapps/root/WEB-INF/classes/com/foo/bar/CustomJettyHandler.class] /Users/...path_to_application.../webapps/root/WEB-INF/classes/com/foo/bar/CustomJettyHandler.class
Adding classpath component: /Users/...path_to_application.../webapps/root/WEB-INF/classes/com/foo/bar/CustomJettyHandler.class
.......
.......
.......
URLClassLoader.url[33] = file:/Users/...path_to_application.../webapps/root/WEB-INF/classes/com/foo/bar/CustomJettyHandler.class
Loaded 34 URLs into URLClassLoader
class org.eclipse.jetty.xml.XmlConfiguration - 9.4.24.v20191120
Command Line Args: /var/folders/z5/dmt38gq54kxcrrzpgvbl5m_c0000gp/T/start_6046998329479549547.properties /Users/...path_to_application.../etc/jetty.xml /Users/...path_to_application.../etc/jetty-jmx.xml
2020-05-27 14:46:13.676:INFO::main: Logging initialized #477ms to org.eclipse.jetty.util.log.StdErrLog
2020-05-27 14:46:13.934:INFO:oeju.TypeUtil:main: JVM Runtime does not support Modules
2020-05-27 14:46:14.007:WARN:oejx.XmlConfiguration:main: Config error at <Call name="insertHandler"><Arg>| <New id="CustomJettyHandler" class="com.foo.bar.CustomJettyHandler"/>| </Arg></Call> java.lang.ClassNotFoundException: com.foo.bar.CustomJettyHandler in file:///Users/...path_to_application.../etc/jetty.xml
2020-05-27 14:46:14.007:WARN:oejx.XmlConfiguration:main:
java.security.PrivilegedActionException: java.lang.ClassNotFoundException: com.foo.bar.CustomJettyHandler
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1837)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:218)
at org.eclipse.jetty.start.Main.start(Main.java:491)
at org.eclipse.jetty.start.Main.main(Main.java:77)
Caused by:
java.lang.ClassNotFoundException: com.foo.bar.CustomJettyHandler
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.eclipse.jetty.util.Loader.loadClass(Loader.java:64)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.newObj(XmlConfiguration.java:1028)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.itemValue(XmlConfiguration.java:1638)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.value(XmlConfiguration.java:1539)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.access$500(XmlConfiguration.java:369)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration$AttrOrElementNode.getList(XmlConfiguration.java:1768)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration$AttrOrElementNode.getList(XmlConfiguration.java:1744)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.call(XmlConfiguration.java:919)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:512)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:454)
at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:354)
at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$0(XmlConfiguration.java:1874)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1837)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:218)
at org.eclipse.jetty.start.Main.start(Main.java:491)
at org.eclipse.jetty.start.Main.main(Main.java:77)
Where did you get this command line from?
java -server \
-Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.StdErrLog\
-jar lib/jetty-start.jar\
OPTIONS=All \
--lib=lib/* \
--lib=webapps/root/WEB-INF/classes/com/foo/bar/* \
etc/jetty.xml \
etc/jetty-jmx.xml \
--debug
That's not valid for use with Jetty 9.x's start.jar
Some advice
don't use XML directly on the java command line, that's the responsibility of the start.jar and the jetty-home module system (order is SUPER IMPORTANT).
Your choices of etc/jetty.xml and etc/jetty-jmx.xml is an incomplete list of xmls. (you are missing all dependent XML files)
don't edit the standard Jetty XML files, leave them be, otherwise you'll complicate upgrades later. Use XML to inject your behavior instead (see below for example)
OPTIONS is not supported by Jetty 9.x (that's old school Codehaus / Jetty 6 behavior)
Your usage of --lib= is discouraged, it only supports fully qualified paths to jars or directories with exploded class trees (not relative paths, no globs supported).
-Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.StdErrLog is a harsh way of setting up logging. Create a jetty-logging.properties files and make sure it's present on the classpath.
Example contents of jetty-logging.properties
org.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.StdErrLog
org.eclipse.jetty.LEVEL=INFO
#org.eclipse.jetty.deploy.LEVEL=DEBUG
Do this instead.
Create an injection based XML for your new handler.
my-handler.xml
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_3.dtd">
<Configure id="Server" class="org.eclipse.jetty.server.Server">
<Call name="insertHandler">
<Arg>
<New id="CustomJettyHandler" class="com.foo.bar.CustomJettyHandler" />
</Arg>
</Call>
</Configure>
Next create your jetty-base directory properly (for jetty's start.jar)
# Create your "jetty-base" directory
$ mkdir /path/to/myjettybase
$ cd /path/to/myjettybase
# Establish the basic files / directories / modules that you want to use
# You can find the configuration in start.ini or start.d/*.ini
$ java -jar /path/to/jetty-home/start.jar --add-to-start=http,jmx,deploy,ext,resources
# Copy your custom handler JAR into place
$ cp /path/to/my-handlers.jar /path/to/myjettybase/lib/ext/
# Copy your custom handler XML into place
$ cp /path/to/my-handler.xml /path/to/myjettybase/etc/
# Ensure that the custom handler XML is loaded into the jetty instance at the right point in the XML load order by declaring it to be used in a custom INI
$ mkdir start.d
$ echo "etc/my-handler.xml" >> start.d/my-handlers.ini
# Copy your jetty-logging.properties into place
$ cp /path/to/my-jetty-logging.properties /path/to/myjettybase/resources/jetty-logging.properties
# verify that your configuration looks sane (including the server classpath)
$ cd /path/to/myjettybase
$ java -jar /path/to/jetty-home/start.jar --list-config
# run your instance
$ cd /path/to/myjettybase
$ java -jar /path/to/jetty-home/start.jar
But that's not all, seeing as you seem to want to use jetty-home from a maven-style project (or project layout), you can do that too!
An example project showing this can be found at ...
https://github.com/jetty-project/servlet-error-page-handling
That maven project is also a valid jetty-base directory suitable for execution by a jetty-home archive somewhere else on your machine.
Related
I have a job definition in a java configuration file. When I try and run the job from gradle task, I get IOException parsing XML Document from class path. How do I run a job using CommandLineJobRunner without XML configuration?
Gradle
task executeJob(type: JavaExec) {
main = 'org.springframework.batch.core.launch.support.CommandLineJobRunner'
classpath = sourceSets.test.runtimeClasspath
args = ["--job_path", "C:\\dev\\git\\ncf-bulk-order\\src\\main\\java\\com\\shelter\\NCFBulkOrder\\Jobs\\NCFBulkOrderConfig.java",
"--job_id", "Response",
"--next",
"--customParam", "value"]
}
StackTrace
org.springframework.beans.factory.BeanDefinitionStoreException: IOException parsing XML document from class path resource [--job_path]; nested exception is java.io.FileNotFoundException: class path resource [--job_path] cannot be opened because it does not exist
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:344)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:252)
at org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:127)
at org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:93)
at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:613)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:514)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
at org.springframework.batch.core.launch.support.CommandLineJobRunner.start(CommandLineJobRunner.java:290)
at org.springframework.batch.core.launch.support.CommandLineJobRunner.main(CommandLineJobRunner.java:565)
Caused by: java.io.FileNotFoundException: class path resource [--job_path] cannot be opened because it does not exist
at org.springframework.core.io.ClassPathResource.getInputStream(ClassPathResource.java:172)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:330)
... 14 common frames omitted
The CommandLineJobRunner expects either the XML file or the Java configuration class containing the Spring application context where your job is defined. Here is an example with a Java config class:
java -cp your/class/path \
org.springframework.batch.core.launch.support.CommandLineJobRunner \
com.example.MyJobConfiguration \
myJob
So in your command, you don't need to pass the --job_path flag.
I am connecting to Linux server using SFTP protocol using citrus FTP component. the following command that was provide in the documentation was executing fine.
<send endpoint="ftpClient">
<message>
<payload>
<ftp:command>
<ftp:signal>MKD</ftp:signal>
<ftp:arguments>test</ftp:arguments>
</ftp:command>
</payload>
</message>
</send>
but when I provide another command like PWD as shown below
<send endpoint="ftpClient">
<message>
<payload>
<ftp:command>
<ftp:signal>PWD</ftp:signal>
<ftp:arguments></ftp:arguments>
</ftp:command>
</payload>
</message>
</send>
I am getting error. but when i execute the above command using SSH protocol the command was executing fine
<send endpoint="sftpClient">
<message>
<payload>
<ftp:command>
<ftp:signal>pwd</ftp:signal>
<ftp:arguments></ftp:arguments>
</ftp:command>
</payload>
</message>
</send>
I am unable to figure out the problem and another thing is how execute two commands in a sequence using citrus ftp component for example ex: I want to move to a particular path and have see the size of files in that path for this we have to use "CD" command first and next we have "ls -l" how can I achive this using citrus.
Here is the stack trace of the error
TEST FAILED ssh_connection_Test <com.consol.citrus.samples.todolist> Nested exception is:
com.consol.citrus.exceptions.CitrusRuntimeException: Failed to read ftp XML object from source
at com.consol.citrus.ftp.message.FtpMarshaller.unmarshal(FtpMarshaller.java:120)
at com.consol.citrus.ftp.message.FtpMessage.getCommand(FtpMessage.java:429)
at com.consol.citrus.ftp.message.FtpMessage.getPayload(FtpMessage.java:384)
at com.consol.citrus.ftp.client.FtpClient.send(FtpClient.java:108)
at com.consol.citrus.actions.SendMessageAction.doExecute(SendMessageAction.java:125)
at com.consol.citrus.actions.AbstractTestAction.execute(AbstractTestAction.java:42)
at com.consol.citrus.TestCase.executeAction(TestCase.java:234)
at com.consol.citrus.TestCase.doExecute(TestCase.java:153)
at com.consol.citrus.actions.AbstractTestAction.execute(AbstractTestAction.java:42)
at com.consol.citrus.Citrus.run(Citrus.java:403)
at com.consol.citrus.testng.AbstractTestNGCitrusTest.invokeTestMethod(AbstractTestNGCitrusTest.java:124)
at com.consol.citrus.testng.AbstractTestNGCitrusTest.run(AbstractTestNGCitrusTest.java:108)
at com.consol.citrus.testng.AbstractTestNGCitrusTest.run(AbstractTestNGCitrusTest.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:212)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:707)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
at org.testng.TestRunner.privateRun(TestRunner.java:767)
at org.testng.TestRunner.run(TestRunner.java:617)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
at org.testng.SuiteRunner.run(SuiteRunner.java:240)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
at org.testng.TestNG.run(TestNG.java:1057)
at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:204)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:175)
Caused by: org.springframework.oxm.UnmarshallingFailureException: JAXB unmarshalling exception; nested exception is javax.xml.bind.UnmarshalException
- with linked exception:
[org.xml.sax.SAXParseException; lineNumber: 2; columnNumber: 37; cvc-enumeration-valid: Value 'PWD' is not facet-valid with respect to enumeration '[OPEN, ABOR, ACCT, ALLO, APPE, CDUP, CWD, DELE, EPRT, EPSV, FEAT, HELP, LIST, MDTM, MFMT, MKD, MLSD, MLST, MODE, NLST, NOOP, PASS, PASV, PORT, PWD, QUIT, REIN, REST, RETR, RMD, RNFR, RNTO, SITE, SMNT, STAT, STOR, STOU, STRU, SYST, TYPE, USER]'. It must be a value from the enumeration.]
at org.springframework.oxm.jaxb.Jaxb2Marshaller.convertJaxbException(Jaxb2Marshaller.java:909)
at org.springframework.oxm.jaxb.Jaxb2Marshaller.unmarshal(Jaxb2Marshaller.java:782)
at org.springframework.oxm.jaxb.Jaxb2Marshaller.unmarshal(Jaxb2Marshaller.java:751)
at com.consol.citrus.ftp.message.FtpMarshaller.unmarshal(FtpMarshaller.java:95)
... 36 more
The error information you are giving is not related to the PWD signal. It fails because you send the signal ls -ltr which is not a supported signal when using the FTP client.
In case you want to list the files with FTP on the server you need to use the LIST signal.
Please do not mix up SSH and FTP commands and signals. Of course you can also open a SSH session and fire up the list files command with ls. But then you need to use the Citrus SSH client.
I am trying to use Apache Zeppelin (0.7.2, net install running locally on a Mac) to explore data loaded from an s3 bucket. The data seems to load just fine, as the command:
val p = spark.read.textFile("s3a://sparkcookbook/person")
gives the result:
p: org.apache.spark.sql.Dataset[String] = [value: string]
However, when I try to call methods on the object p, I get an error. For example:
p.take(1)
results in:
java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.rdd.RDDOperationScope$
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
at org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:225)
at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:308)
at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)
at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2371)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)
at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2765)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$execute$1(Dataset.scala:2370)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collect(Dataset.scala:2377)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2113)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2112)
at org.apache.spark.sql.Dataset.withTypedCallback(Dataset.scala:2795)
at org.apache.spark.sql.Dataset.head(Dataset.scala:2112)
at org.apache.spark.sql.Dataset.take(Dataset.scala:2327)
My conf/zeppelin-env.sh is the same as the default, except that I have amazon access key and secret key environment variables defined there. In the Spark interpreter in the Zeppelin notebook, I have added the following artifacts:
org.apache.hadoop:hadoop-aws:2.7.3
com.amazonaws:aws-java-sdk:1.7.9
com.fasterxml.jackson.core:jackson-core:2.9.0
com.fasterxml.jackson.core:jackson-databind:2.9.0
com.fasterxml.jackson.core:jackson-annotations:2.9.0
(I think only the first two are necessary). The two commands above work fine in the Spark shell, just not in the Zeppelin notebook (see How to use s3 with Apache spark 2.2 in the Spark shell for how that was set up).
So it seems that there is a problem with one of the Jackson libraries. Maybe I'm using the wrong artifacts above for the Zeppelin interpreter?
UPDATE: Following the advice in the proposed answer below, I removed the jackson jars that came with Zeppelin, and replaced them with the following:
jackson-annotations-2.6.0.jar
jackson-core-2.6.7.jar
jackson-databind-2.6.7.jar
And replaced the artifacts with these, so my artifacts are now:
org.apache.hadoop:hadoop-aws:2.7.3
com.amazonaws:aws-java-sdk:1.7.9
com.fasterxml.jackson.core:jackson-core:2.6.7
com.fasterxml.jackson.core:jackson-databind:2.6.7
com.fasterxml.jackson.core:jackson-annotations:2.6.0
The error I get, however, from running the above commands is the same.
UDPATE2: As per I removed the jackson libraries from the list of artifacts, since they are already now in the jars/ folder - the only added artifacts are now the aws artifacts above. I then cleaned the classpath by entering the following in the notebook (as per the instructions):
%spark.dep
z.reset()
I get a different error now:
val p = spark.read.textFile("s3a://sparkcookbook/person")
p.take(1)
p: org.apache.spark.sql.Dataset[String] = [value: string]
java.lang.NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer$.handledType()Ljava/lang/Class;
at com.fasterxml.jackson.module.scala.deser.NumberDeserializers$.<init>(ScalaNumberDeserializersModule.scala:49)
at com.fasterxml.jackson.module.scala.deser.NumberDeserializers$.<clinit>(ScalaNumberDeserializersModule.scala)
at com.fasterxml.jackson.module.scala.deser.ScalaNumberDeserializersModule$class.$init$(ScalaNumberDeserializersModule.scala:61)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.<init>(DefaultScalaModule.scala:20)
at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<init>(DefaultScalaModule.scala:37)
at com.fasterxml.jackson.module.scala.DefaultScalaModule$.<clinit>(DefaultScalaModule.scala)
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
at org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:225)
at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:308)
at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)
at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2371)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)
at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2765)
UPDATE3: As per the suggestion in a comment to the proposed answer below, I cleaned the class path by deleting all the files in the local repo:
rm -rf local-repo/*
I then restarted the Zeppelin server. To check the class path, I executed the following in the notebook:
val cl = ClassLoader.getSystemClassLoader
cl.asInstanceOf[java.net.URLClassLoader].getURLs.foreach(println)
This gave the following output (I include only the jackson libraries from the output here, otherwise the output is too long to paste):
...
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/local-repo/2CT9CPAA9/jackson-annotations-2.1.1.jar
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/local-repo/2CT9CPAA9/jackson-annotations-2.2.3.jar
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/local-repo/2CT9CPAA9/jackson-core-2.1.1.jar
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/local-repo/2CT9CPAA9/jackson-core-2.2.3.jar
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/local-repo/2CT9CPAA9/jackson-core-asl-1.9.13.jar
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/local-repo/2CT9CPAA9/jackson-databind-2.1.1.jar
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/local-repo/2CT9CPAA9/jackson-databind-2.2.3.jar
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/local-repo/2CT9CPAA9/jackson-jaxrs-1.9.13.jar
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/local-repo/2CT9CPAA9/jackson-mapper-asl-1.9.13.jar
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/local-repo/2CT9CPAA9/jackson-xc-1.9.13.jar
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/lib/jackson-annotations-2.6.0.jar
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/lib/jackson-core-2.6.7.jar
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/lib/jackson-databind-2.6.7.jar
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/jackson-annotations-2.6.5.jar
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/jackson-core-2.6.5.jar
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/jackson-core-asl-1.9.13.jar
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/jackson-databind-2.6.5.jar
file:/Users/shafiquejamal/allfiles/scala/spark/zeppelin-0.7.2-bin-netinst/jackson-mapper-asl-1.9.13.jar
file:/Users/shafiquejamal/allfiles/scala/spark/spark-2.1.0-bin-hadoop2.7/jars/jackson-annotations-2.6.5.jar
file:/Users/shafiquejamal/allfiles/scala/spark/spark-2.1.0-bin-hadoop2.7/jars/jackson-core-2.6.5.jar
file:/Users/shafiquejamal/allfiles/scala/spark/spark-2.1.0-bin-hadoop2.7/jars/jackson-core-asl-1.9.13.jar
file:/Users/shafiquejamal/allfiles/scala/spark/spark-2.1.0-bin-hadoop2.7/jars/jackson-databind-2.6.5.jar
file:/Users/shafiquejamal/allfiles/scala/spark/spark-2.1.0-bin-hadoop2.7/jars/jackson-jaxrs-1.9.13.jar
file:/Users/shafiquejamal/allfiles/scala/spark/spark-2.1.0-bin-hadoop2.7/jars/jackson-mapper-asl-1.9.13.jar
file:/Users/shafiquejamal/allfiles/scala/spark/spark-2.1.0-bin-hadoop2.7/jars/jackson-module-paranamer-2.6.5.jar
file:/Users/shafiquejamal/allfiles/scala/spark/spark-2.1.0-bin-hadoop2.7/jars/jackson-module-scala_2.11-2.6.5.jar
file:/Users/shafiquejamal/allfiles/scala/spark/spark-2.1.0-bin-hadoop2.7/jars/jackson-xc-1.9.13.jar
file:/Users/shafiquejamal/allfiles/scala/spark/spark-2.1.0-bin-hadoop2.7/jars/json4s-jackson_2.11-3.2.11.jar
file:/Users/shafiquejamal/allfiles/scala/spark/spark-2.1.0-bin-hadoop2.7/jars/parquet-jackson-1.8.1.jar
...
It seems that multiple versions are fetched from the repo. Should I exclude the older versions? If so, how do I do that?
Use this jar versions;
aws-java-sdk-1.7.4.jar
hadoop-aws-2.6.0.jar
like in this script : https://github.com/2dmitrypavlov/sparkDocker/blob/master/zeppelin.sh
do not use package but download the jars and put them in a path, let's say in "/root/jars/" then edit your zeppelin-env.sh;
then run this command from zeppelin/conf dir;
echo 'export SPARK_SUBMIT_OPTIONS="--jars /root/jars/mysql-connector-java-5.1.39.jar,/root/jars/aws-java-sdk-1.7.4.jar,/root/jars/hadoop-aws-2.6.0.jar"'>>zeppelin-env.sh
after that restart zeppelin.
The code at the link above is pasted below (just in case the link becomes stale):
#!/bin/bash
# Download jars
cd /root/jars
wget http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.39/mysql-connector-java-5.1.39.jar
cd /usr/share/
wget http://archive.apache.org/dist/zeppelin/zeppelin-0.7.1/zeppelin-0.7.1-bin-all.tgz
tar -zxvf zeppelin-0.7.1-bin-all.tgz
cd zeppelin-0.7.1-bin-all/conf
cp zeppelin-env.sh.template zeppelin-env.sh
echo 'export MASTER=spark://'$MASTERZ':7077'>>zeppelin-env.sh
echo 'export SPARK_SUBMIT_OPTIONS="--jars /root/jars/mysql-connector-java-5.1.39.jar,/root/jars/aws-java-sdk-1.7.4.jar,/root/jars/hadoop-aws-2.6.0.jar"'>>zeppelin-env.sh
echo 'export ZEPPELIN_NOTEBOOK_STORAGE="org.apache.zeppelin.notebook.repo.VFSNotebookRepo, org.apache.zeppelin.notebook.repo.zeppelinhub.ZeppelinHubRepo"'>>zeppelin-env.sh
echo 'export ZEPPELINHUB_API_ADDRESS="https://www.zeppelinhub.com"'>>zeppelin-env.sh
echo 'export ZEPPELIN_PORT=9999'>>zeppelin-env.sh
echo 'export SPARK_HOME=/usr/share/spark'>>zeppelin-env.sh
cd ../bin/
./zeppelin.sh
You are probably using a too recent Jackson version. Even spark 2.3 is still on `2.6.7. Downgrade, and make sure that all your jackson JARs are consistent.
I have a problem when running JUnit Plug-in tests in Eclipse 4.3 and 4.2 and possible other versions. This worked in version 3.6 so I do not know what has changed.
The test are written using JUnit 4 and using parameterized tests (though I have tested with normal tests and the problem is the same). The test cases are written in YAML format and inflated at runtime using SnakeYAML.
When running the test as a normal JUnit test everything works fine, but when running the test as a JUnit Plug-in tests the test fails since it can no longer inflate objects from Yaml using SnakeYAML.
This is the code that reads test cases from YAML
public static ArrayList<StartsWithCommentTestCase> readTests(String filename) {
// set up how Java types match to the YAML file
Constructor testCaseConstructor = new Constructor(StartsWithCommentTestCase.class);
TypeDescription testCaseDesc = new TypeDescription(StartsWithCommentTestCase.class);
testCaseConstructor.addTypeDescription(testCaseDesc);
Yaml yamlParser = new Yaml(testCaseConstructor);
// read all the documents in the test file
String tests = TestUtils.readFile(filename);
ArrayList<StartsWithCommentTestCase> testCases = new ArrayList<StartsWithCommentTestCase>();
for (Object testCase : yamlParser.loadAll(new StringReader(tests))) {
testCases.add((StartsWithCommentTestCase) testCase);
}
return testCases;
}
#Test
public void bla(){
List<StartsWithCommentTestCase> tests = readTests(TEST_FILES_DIR + "starts-with-comment.yaml");
for(StartsWithCommentTestCase test : tests){
boolean actual = ToggleCommentHandler.startsWithComment( test.getLine() );
assertEquals( test.getName(), test.getExpected(), actual );
}
}
And this is the stacktrace that is received when running as a JUnit Plug-in test
Can't construct a java object for tag:yaml.org,2002:pde.test.tests.StartsWithCommentTestCase;exception=Class not found: pde.test.tests.StartsWithCommentTestCase
in "<reader>", line 4, column 1:
name: Empty line
^
at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:325)
at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:181)
at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:140)
at org.yaml.snakeyaml.constructor.BaseConstructor.getData(BaseConstructor.java:109)
at org.yaml.snakeyaml.Yaml$1.next(Yaml.java:317)
at pde.test.tests.StartsWithCommentTest.readTests(StartsWithCommentTest.java:86)
at pde.test.tests.StartsWithCommentTest.bla(StartsWithCommentTest.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunner.main(RemotePluginTestRunner.java:62)
at org.eclipse.pde.internal.junit.runtime.CoreTestApplication.run(CoreTestApplication.java:23)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.eclipse.equinox.internal.app.EclipseAppContainer.callMethodWithException(EclipseAppContainer.java:587)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:198)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:354)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:181)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:636)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:591)
at org.eclipse.equinox.launcher.Main.run(Main.java:1450)
at org.eclipse.equinox.launcher.Main.main(Main.java:1426)
Caused by: org.yaml.snakeyaml.error.YAMLException: Class not found: pde.test.tests.StartsWithCommentTestCase
at org.yaml.snakeyaml.constructor.Constructor.getClassForNode(Constructor.java:625)
at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.getConstructor(Constructor.java:313)
at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:323)
... 49 more
The strange thing (for me at least) is that I can make a StartsWithCommentTestCase object with
new StartsWithCommentTestCase()
without any problem also when running as a plugin test. In otherwords the class is avaiable at runtime, but for some reason not available to SnakeYAML.
Any pointers on this would be very helpful :)
Edit 1
The first lines of the YAML file:
#all the lines must have newlines in them since newlines are returned
#by Document.get()
---
name: Empty line
line:
expected: false
---
name: "Line where first character is #"
line: "#
"
expected: true
Edit 2
Added more code to the sample.
It looks like the Junit plugin don't know how to read the SnakeYAML code. Are you sure the class-loader of SnakeYAML is correctly initialized when you run the tests through the plug in?
And how does the mentioned line 4 look in your YAML code? Is it empty?
Maybe the junit plugin uses another jar dependency than your ordinary setup. Which maven binary/path does the junit plugin use? Is this the same as your ordinary one? If you have different m2-configurations or even different repos there will be strange issues.
I found out that this is because I had moved the SnakeYAML dependency to another plugin. SnakeYAML was then part of another plugin with a different class loader than the test plugin.
The solution was to add the following to the META-INF/MANIFEST.MF file in the dependeny plugin
Eclipse-BuddyPolicy: registered
And the following to the META-INF/MANIFEST.MF file in the test plugin
Eclipse-RegisterBuddy: <symbolic name of the dependency plugin>
Now on to see if I can move the SnakeYAML dependency back into a lib/ folder or similar.
I've used the java -classpath wildcard expansion feature previously and successfully. I'm currently experiencing a strange problem with it.
The wildcard is supposed to expand to every jar in the named folder. Here's a quote from Oracle:
Class path entries can contain the basename wildcard character *,
which is considered equivalent to specifying a list of all the files
in the directory with the extension .jar or .JAR. For example, the
class path entry foo/* specifies all JAR files in the directory
named foo. A classpath entry consisting simply of * expands to a
list of all the jar files in the current directory.
This is a link to Oracle doc on Java 6 on the subject of classpath.
The behaviour I am seeing contradicts this. Here are 3 runs. The first explicitly names the jar and so it works. The others use a wildcard and fail. Why? This matters to me because I rely on wildcards (elsewhere) and so an understanding of this unexpected behaviour is important to me.
#!/bin/bash
printf "The EV is...\n"
echo $CLASSPATH
printf "The working directory is...\n"
pwd
printf "Directory listing...\n"
ls
printf "END of directory listing.\n"
printf "Test with named jar.\n"
java -javaagent:../sizeof/sizeof.jar -classpath ./testsizeof.jar info.zqxj.test.Tester
printf "Test with star.\n"
java -javaagent:../sizeof/sizeof.jar -classpath * info.zqxj.test.Tester
printf "Test with dot slash star.\n"
java -javaagent:../sizeof/sizeof.jar -classpath ./* info.zqxj.test.Tester
The output:
The EV is...
The working directory is...
/home/b/Documents/workspace/testsizeof
Directory listing...
bin run.sh src testsizeof.jar
END of directory listing.
Test with named jar.
40
Test with star.
Exception in thread "main" java.lang.NoClassDefFoundError: run/sh
Caused by: java.lang.ClassNotFoundException: run.sh
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: run.sh. Program will exit.
Test with dot slash star.
Exception in thread "main" java.lang.NoClassDefFoundError: //run/sh
Caused by: java.lang.ClassNotFoundException: ..run.sh
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: ./run.sh. Program will exit.
Solution, double quote the classpath argument. Example: -classpath "*" This is necessary on the command line as well as inside a bash script.
A subsequent addendum:
Furthermore, note that -classpath "~/folder/*" fails but -classpath ~/folder/"*" is good. Quote the wildcard but do not quote the ~. It seems that you need the operating system to interpret ~ but you need to quote the * wildcard because you want to pass it to java for expansion in Java-fashion.
Note also that you should not ask java to expand *.jar because that will have an unintended result. The Java spec says that the correct wildcard is just the * alone.