I know this question might have already been answered but i havn't found a proper answer.
I am using hadoop mapreduce on eclipse and i want to create an executable jar to put it on a linux server where there is my hdfs cluster. The problem when i run the jar on my hdfs cluster and eclipse, there is exception appearing caused by java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory. So I added an external jar common-logging but when i run it again and another NoClassDefFoundError appeared but this time with log4j-level and so on. I wondered how many external jar i had to add but i don't think that it's the best solution. Does anyone have the same problem and how did you solve it ?
Thank you very much.
Usually, this error occurs when a previous exception occurred during logging initialisation.
Hadoop is quite fickle about dependencies, you should either make sure that your third party libs are part of your hadoop classpath or create a shaded jar with all dependencies in your job jar. You should use maven or assembly to create that jar.
Related
I'm building a job in Scala to run on a Flink Cluster, that will store data in AWS S3, and I've some problems related to dependencies.
I've checked most of the question previously asked here, and to fix this I needed to add flink-s3-fs-hadoop-1.9.1.jar jar files to the $FLINK_HOME/plugins in order to run my job successfully:
My question is, should this be detected as been inside the fatjar generate by sbt assembly ? The files are inside the jar, but for some reason the Flink Cluster can't see them.
I know that in the documentation says that flink-s3-fs-hadoop-1.9.1.jar should download to $FLINK_HOME/plugins folder.
Filesystems cannot be bundled in the user-jar, they must be present either in /lib or /plugins.
The components that use filesystems aren't necessarily aware of the user-jar.
I am trying to run my job using SpringBatch-CommandLineJobRunner on Linux, I have the below jars in classpath.
Can anyone please take a look and see If I am missing any jars and tell me the missing jars. In our project they dont want to use any tools like Maven or STS, we are just using eclipse, our jobs run well from eclipse. On Linux we get the below errors.. we are using below command
java -cp org.springframework.batch.core.launch.support.CommandLineJobRunner springbatch/config/applicationBatchContextPlan.xml crossRefAllPlansJob
Exception in thread "main" java.lang.NoClassDefFoundError: org/springframework/batch/core/launch/support/CommandLineJobRunner
Jars in Classpath on linux :
springbatch/lib/aopalliance-1.0.jar
springbatch/lib/commons-collections-3.2.1.jar
springbatch/lib/commons-lang3-3.3.2.jar
springbatch/lib/commons-lang3-3.3.2-javadoc.jar
springbatch/lib/commons-logging-1.2.jar
springbatch/lib/javax.batch-api-1.0.jar
springbatch/lib/log4j-1.2.17.jar
springbatch/lib/ojdbc6.jar
springbatch/lib/spring-aop-4.1.7.RELEASE.jar
springbatch/lib/spring-aspects-4.1.7.RELEASE.jar
springbatch/lib/spring-batch-core-3.0.4.RELEASE.jar
springbatch/lib/spring-batch-infrastructure-3.0.4.RELEASE.jar
springbatch/lib/spring-beans-4.1.7.RELEASE.jar
springbatch/lib/spring-context-4.1.7.RELEASE.jar
springbatch/lib/spring-core-4.1.7.RELEASE.jar
springbatch/lib/spring-expression-4.1.7.RELEASE.jar
springbatch/lib/spring-jdbc-4.1.7.RELEASE.jar
springbatch/lib/spring-retry-1.1.2.RELEASE.jar
springbatch/lib/spring-tx-4.1.7.RELEASE.jar
Also, I want to know if there is any other way we can run spring batch jobs on linux without using the CommandLineJobRunner.
Based looking at this post and this post, you need to manually include all the jars on your classpath.
So basically:
java -cp ... org.springframework.batch.core.launch.support.CommandLineJobRunner springbatch/config/applicationBatchContextPlan.xml crossRefAllPlansJob
The first post has a handy shell script to help with this.
In terms of other deployment options, you can deploy a spring batch application anywhere you can deploy spring (e.g. servlet container). Generally though, if it is the principal thing going on in the app, a plain old jar is probably the simplest option.
The problem is resolved since a couple of days. The jars in the classpath were not getting picked up properly.After a couple of retries, it did and its working as expected now
Thanks to everyone for helping.
Thanks
Lakshmi
Hi I'm building a program using optaplanner, and everything seems to be imported and working fine, but when I run and debug the code, I get a ClassNotFoundException "org.optaplanner.core.impl.solver.DefaultSolver". I believe I've imported all the jar files correctly, but just to be sure, what jar file is the DefaultSolver class in?
As a follow up, I keep getting prompted to edit the source lookup path. This happens after running SolverConfig.buildSolver() line 156: "DefaultSolver solver = new DefaultSolver();" I'm running OptaPlanner 6.1.0.Final.
Found the problem, I realized that I had both imported the examples jar file into my buildpath and at one point imported a class from the examples jar file when I was messing around with the examples. Rereading the manual I saw it explicitly said not to import the examples jar file, so I unimported that and found the culprit pretty fast. Thanks for your help.
Both the class SolverConfig and the class DefaultSolver are in the jar optaplanner-core-*.jar.
So this doesn't make sense: if the jar is in the classpath, it should find DefaultSolver. If the jar is not in the classpath, it shouldn't even find SolverConfig. What you're seeing can never happen in a normal classpath.
Are you using anything like OSGi, JBoss modules, jigsaw, Karaff or other classloading magic?
I am developing a web app (on tomcat 7) that needs to create a periodic task and so I chose Quartz Scheduler.But every time I start server it throws
java.lang.NoClassDefFoundError: javax/transaction/UserTransaction and
java.lang.ClassNotFoundException: javax.transaction.UserTransaction.
I rechecked many times in classpath and it is surely has this UserTransaction class in javaee.jar by this line of code System.out.println( System.getProperty( "java.class.path" ) );
So could anyone please help me point it out ? . Maybe I am missing something.
Probably (you did not state which javaee.jar you are using) you have the version from Oracle that contains only the API. This jar is suitable for compiling, not for running your application.
Check if you have javaee-api jar in your classpath. Also, you need to check if you have multiple jars carrying the same class. You can see here what all jars have javax/transaction/UserTransaction. For any futher debugging you can follow this
You may solve it just by adding jta.jar
I'm using Eclipse 3.4 with WTP 3.0.2 and running a fairly large Dynamic Web Project. I've set up the project so that I can access it at http://127.0.0.1:8080/share/ but whenever I do, I get the following error:
java.lang.NoSuchMethodError: javax.servlet.jsp.tagext.TagAttributeInfo.(Ljava/lang/String;ZLjava/lang/String;ZZ)V
at org.apache.jasper.compiler.TagLibraryInfoImpl.createAttribute(TagLibraryInfoImpl.java:572)
at org.apache.jasper.compiler.TagLibraryInfoImpl.createTagInfo(TagLibraryInfoImpl.java:401)
at org.apache.jasper.compiler.TagLibraryInfoImpl.parseTLD(TagLibraryInfoImpl.java:248)
at org.apache.jasper.compiler.TagLibraryInfoImpl.(TagLibraryInfoImpl.java:162)
at org.apache.jasper.compiler.Parser.parseTaglibDirective(Parser.java:423)
at org.apache.jasper.compiler.Parser.parseDirective(Parser.java:492)
at org.apache.jasper.compiler.Parser.parseElements(Parser.java:1552)
at org.apache.jasper.compiler.Parser.parse(Parser.java:126)
at org.apache.jasper.compiler.ParserController.doParse(ParserController.java:211)
at org.apache.jasper.compiler.ParserController.parse(ParserController.java:100)
at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:155)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:295)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:276)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:264)
at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:563)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:303)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Unknown Source)
As none of the above files is my own, pointing out the cause of the problem is quite hard. Any ideas where to start looking?
I ended up answering my own question: the problem was that among the necessary JARs that I had added to Tomcat was a conflicting servlet.jar. When I removed this, the error disappeared.
Did you set a Tomcat path in "Preferences->Tomcat->Advanced->Tomcat base" ?
Try to clean that path (getting back to default configuration), and check if that does solve the problem.
most probably the tomcat plugin in the eclipse cause the conflict problem, i manage to solve the problem by getting the same version jar file and override it in the tomcat plugin in the eclipse.
I had similar problem and I had fixed this problem by Ensuring that i have correct version of servlet.jar in the classpath that is being taken by my application and also i had kept old J2EE.jar file in the classpath and that was causing the main problem. Hence I removed it from the classpath to ensure that it uses by default files.
I had the same problem running Eclipse Helios, with Maven handling dependencies and using Jetty as webserver. After updating to Spring 3.1 I suddenly got this problem, but only on my local development machine.
I first deleted the spring and jetty folders in my local maven repository and updated the dependencies, but that didn't improve the situation.
Then I just deleted the servlet-api and servlet-api-2.5 folders that comes with jetty (but leaving everything else), I got it to work.
All hail the magic of the classpath.