The instructions for running kafka from the quick start page are not working for me.
http://kafka.apache.org/07/quickstart.html
Kafka builds fine
05:55:01/kafka-0.8.1-src:58 $sbt package
[info] Set current project to kafka-0-8-1-src (in build file:/shared/kafka-0.8.1-src/)
[info] Packaging /shared/kafka-0.8.1-src/target/scala-2.10/kafka-0-8-1-src_2.10-0.1-SNAPSHOT.jar ...
[info] Done packaging.
[success] Total time: 0 s, completed Apr 17, 2014 5:55:07 AM
But does not run fine..
05:55:07/kafka-0.8.1-src:59 $bin/zookeeper-server-start.sh config/zookeeper.properties
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/zookeeper/server/quorum/QuorumPeerMain
Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.server.quorum.QuorumPeerMain
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Similar errors occur for kafka-server-start.sh and all the other scripts inside bin
You downloaded kafka-0.8.1-src.tgz from download page. The instructions on quickstart link are meant for Binary download . Download one from the Binary downloads section of http://kafka.apache.org/downloads.html page. Now try .It should work. Or if you want to build from the src.tgz package you downloaded, then run ./gradlew jar. It will download all the required dependencies.
I was facing same problem On Windows 10, What i have done is :
Don't download/install Zookeeper separately, download only kafka_2.12-1.1.0 (or higher version)
Create temp folder , ( like this E:\DevApplications\kafka\temp)
Open zookeeper.properties ( i have it # E:\DevApplications\kafka\kafka_2.12-1.1.0\config)
Update dataDir (for me : dataDir=E:/DevApplications/kafka/temp) Note the forward slash
Open CMD and run zookeeper-server-start.bat with zookeeper.properties as second parameter like
.\zookeeper-server-start.bat ..\..\config\zookeeper.properties
Once zookeeper is started start kafka server by typing
.\kafka-server-start.bat ..\..\config\server.properties
Hope this helps.
To add to the Chandra Kant solutions, if you have proxy connection in your network then please use the below command
./gradlew -Dhttp.proxyHost=<PROXY-HOST> -Dhttp.proxyPort=<PROXY-PORT> jar
Thanks #Chandra kant it helped me a lot
You can also reach this exception if you try starting Kafka 0.9.0.0 running a java version less than java 1.7. Set your $JAVA_HOME to 1.7 or above and make sure JAVA_HOME/bin is on your classpath.
Related
I'm using Tomcat v7.0 and with Eclipse IDE for a Webapp. When trying to start the server , I get the below ClassNotFoundException
java.lang.NoClassDefFoundError: org/apache/tomcat/util/digester/Rule
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2585)
at java.lang.Class.getConstructor0(Class.java:2885)
at java.lang.Class.newInstance(Class.java:350)
at org.apache.catalina.startup.Bootstrap.init(Bootstrap.java:239)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:426)
Caused by: java.lang.ClassNotFoundException: org.apache.tomcat.util.digester.Rule
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 6 more
On searching, I found the apache-tomcat-util 5.23 jar containing the above class but after adding the jar to the launch configuration of Tomcat I get a series of other errors. The same server installation works good in a different work space with a different webapp. What is the issue here?
You have broken your installation, probably before you found apache-tomcat-util-5.23.jar and most certainly after adding that JAR file to your installation. That JAR you added was from a Tomcat version that is many years old and incompatible with your newer version.
Remove your existing Tomcat installation (but save the webapps/ directory if you put anything in there you want to keep) and reinstall.
If you are still getting that error, check to make sure that your IDE is configured correctly: Tomcat comes with every JAR it needs to launch, so if it won't start, it's because of a misconfiguration with the IDE's integration.
In my case the only problem was version of Apache Tomcat which I used is Apache 7.0 and when I change to Apache 8.0 It will run perfectly fine.
the exception "java.lang.NoClassDefFoundError: org/apache/tomcat/util/digester/Rule" may occur because the server installation was stopped before it was finished.
Make sure of the % of installation when you install the server since nothing appears when we click install and it seems immediate, but in reality it takes a few seconds.
No problem :-)
All I did was download wildfly-8.1.0.CR2 and extract it. standalone.bat and add-user.bat work but jboss-cli.bat does not.
F:\wildfly-8.1.0.CR2\bin>jboss-cli
java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no jansi64-1.9 in
java.library.path, no jansi-1.9 in java.library.path, no jansi in java.library.path,
D:\pgarner\AppData\Local\Temp\jansi-64-1.9.dll: The application has failed to start
because its side-by-side configuration is incorrect. Please see the application event
log or use the command-line sxstrace.exe tool for more detail]
at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:184)
at org.fusesource.hawtjni.runtime.Library.load(Library.java:142)
at org.fusesource.jansi.internal.Kernel32.<clinit>(Kernel32.java:37)
at org.fusesource.jansi.WindowsAnsiOutputStream.<clinit>(WindowsAnsiOutputStream.java:52)
at org.jboss.aesh.terminal.WindowsTerminal.init(WindowsTerminal.java:53)
at org.jboss.aesh.console.Console.setTerminal(Console.java:193)
at org.jboss.aesh.console.Console.reset(Console.java:154)
at org.jboss.aesh.console.Console.<init>(Console.java:105)
at org.jboss.aesh.console.Console.<init>(Console.java:101)
at org.jboss.as.cli.impl.Console$Factory.getConsole(Console.java:85)
at org.jboss.as.cli.impl.Console$Factory.getConsole(Console.java:78)
at org.jboss.as.cli.impl.CommandContextImpl.initBasicConsole(CommandContextImpl.java:349)
at org.jboss.as.cli.impl.CommandContextImpl.<init>(CommandContextImpl.java:296)
at org.jboss.as.cli.impl.CommandContextFactoryImpl.newCommandContext(CommandContextFactoryImpl.java:76)
at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:273)
at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:253)
at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.jboss.modules.Module.run(Module.java:312)
at org.jboss.modules.Main.main(Main.java:460)
Press any key to continue . . .
When I start up Wildfly using standalone.bat I see the following entry for java.library.path in server.log:
java.library.path = F:\Java\jdk1.7.0_45\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;F:\WANdisco\uberSVN\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;F:\GnuPG\pub;F:\7-Zip;"E:\WebTest\build\bin";F:\WANdisco\uberSVN\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;F:\GnuPG\pub;F:\7-Zip;.
The following file does indeed appear on my file system when I attempt to run jboss-cli:
D:\pgarner\AppData\Local\Temp\jansi-64-1.9.dll
I also tried using wildfly-8.0.0.Final instead of wildfly-8.1.0.CR2 and the same exact problem happened.
How to resolve this problem? I assumed the CLI should just work right out of the box after extracting all the files out of the zip file.
We're facing the same error and the problem is due to the jansi dll dependencies. In fact you need to install the Microsoft Visual C++ 2008 Redistributable Package corresponding to your platform. For x64, you can follow this link :
http://www.microsoft.com/en-US/download/details.aspx?id=2092
Have a similar problem. I can start Wildfly and deploy my application without errors, but everytime I redeploy my application, I get the following error:
Caused by:
java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no
jansi64-1.9 in java.library.path, no jansi-1.9 in java.library.path,
no jansi in java.library.path, Native Library
C:\Users\zb\AppData\Local\Temp\jansi-64-1.9.dll already loaded in
another classloader]
It seems this jansi library is stuck after deploy.
Restarting the server helps temporarily.
I am trying to run Scalding sample word count example. I have followed this github link for steps:-
https://github.com/twitter/scalding/wiki/Getting-Started
But I am getting ClassNotFoundException. Below is my StackTrace:-
[cloudera#localhost scalding-develop]$ **sudo scripts/scald.rb --local WordCount --input input.txt --output ./someOutputFile.tsv**
can not find /root/.sbt/boot/scala-2.9.3/lib/scala-library.jar appending SBT_VERSION [0.12.0] to SBT_HOME
scripts/scald.rb:139: warning: already initialized constant SBT_HOME
scripts/scald.rb:140: warning: already initialized constant SCALA_LIB_DIR
Exception in thread "main" java.lang.Throwable: If you know what exactly caused this error, please consider contributing to GitHub via following link.
https://github.com/twitter/scalding/wiki/Common-Exceptions-and-possible-reasons#javalangclassnotfoundexception
at com.twitter.scalding.Tool$.main(Tool.scala:146)
at com.twitter.scalding.Tool.main(Tool.scala)
Caused by: java.lang.ClassNotFoundException: WordCount
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:188)
at com.twitter.scalding.Job$.apply(Job.scala:39)
at com.twitter.scalding.Tool.getJob(Tool.scala:49)
at com.twitter.scalding.Tool.run(Tool.scala:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at com.twitter.scalding.Tool$.main(Tool.scala:132)
... 1 more
Please let me know where exactly is the issue?
Thanks.
Check if your JDK's bin directory is there in your PATH. I had a similar issue and I had use update-alternatives to install java but had not included /usr/lib/jvm/jdk-1.7.0/bin in my PATH.
Check - Running your Scalding jobs in Eclipse article at
http://hokiesuns.blogspot.co.uk/2012/07/running-your-scalding-jobs-in-eclipse.html
Using Maven for Scalding should be your first step into this new brave world.
After a few months give sbt a try and see if you like it
Don't go directly to sbt as its usage is not that trivial
I suggest not using scald.rb.
Instead use sbt, check this github repo, https://github.com/deanwampler/activator-scalding. It includes all the necessary basic setup.
Regarding Apache-Kafka messaging queue.
I have downloaded Apache Kafka from the Kafka download page. I've extracted it to /opt/apache/installed/kafka-0.7.0-incubating-src.
The quickstart page says you need to start zookeeper and then start Kafka by running: >bin/kafka-server-start.sh config/server.properties
I'm using a separate Zookeeper server, so i edited config/server.properties to point to that Zookeeper instance.
When i run Kafka, as instructed in the quickstart page, I get the following error:
Exception in thread "main" java.lang.NoClassDefFoundError: kafka/Kafka
Caused by: java.lang.ClassNotFoundException: kafka.Kafka
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Could not find the main class: kafka.Kafka. Program will exit.
I used telnet to make sure the Zookeeper instance is accessible from the machine that Kafka runs on. Everything is OK.
Why am i getting this error?
You must first build Kafka by running the following commands:
> ./sbt update
> ./sbt package
Only then will Kafka be ready for use.
You should know that
./sbt update
./sbt package
will produce Kafka binaries for Scala 2.8.0 by default. If you need it for a different version, you need to do
./sbt "++2.9.2 update"
./sbt "++2.9.2 package"
replacing 2.9.2 with the desired version number. This will make the appropriate binaries. In general, when you switch versions, you should run
./sbt clean
to clean up the binaries from previous versions.
Actually, in addition, you might also need to perform this command
./sbt "++2.9.2 assembly-package-dependency"
This command resolves all the dependencies for running Kafka, and creates a jar that contains just these. Then the start scripts would add this to the class path and you should have all your desired classes.
It seems that without the SCALA_VERSION environment variable, the executable doesn't know how to load the libraries necessary. Try the following from the Kafka installation directory:
SCALA_VERSION=2.9.3 bin/kafka-server-start.sh config/server.properties
See http://kafka.apache.org/documentation.html#quickstart.
Just to add to the previous answer, if you're running IntelliJ, and want to run Kafka inside IntelliJ and/or step through it, make sure to run
> ./sbt idea
I spent easily half a day trying to create the IntelliJ project from scratch, and it turns out that single command was all I needed to get it working. Also, make sure you have the Scala plugin for IntelliJ installed.
You can also use the binary downloads provided by Apache.
For example download kafka version - 0.9.0.1 from this link.
For other version download from link2 and download the binary versions instead. These are already built version. Not need to build again using Scala.
Use are using the source download instead.
You've downloaded the source version. Download the binary package of Kafka and proceed with your testing.
You can find following two options on Kafka downloads page
https://kafka.apache.org/downloads.html
Source download:
Binary downloads
You have downloaded "kafka-0.7.0-incubating-src" it' source code
Download the binary package of Kafka
Scala 2.10 - kafka_2.10-0.10.1.1.tgz (asc, md5)
I am trying to build my project using Eclipse on Windows and execute on a Linux cluster. The project depends on some external jars, which I enclosed using eclipse's "Export->Runnable JAR -> Package required library into jar" build option. I checked the jar contains the classes within a folder structure, and the external jars are in the root folder.
On Hadoop standalone, Cygwin and Linux, this works fine but on an actual Hadoop Linux cluster it fails, when it tries to access a class from the first external jar, throwing up a ClassNotFoundException.
Is there a way to force Hadoop to search the jar, I thought this would work.
10/07/16 11:44:59 INFO mapred.JobClient: Task Id : attempt_201007161003_0005_m_000001_0, Status : FAILED
Error: java.lang.ClassNotFoundException: org.jfree.data.xy.XYDataset
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at org.akintayo.analysis.ecg.preprocess.ReadPlotECG.plotECG(ReadPlotECG.java:27)
at org.akintayo.analysis.ecg.preprocess.BuildECGImages.writeECGImages(BuildECGImages.java:216)
at org.akintayo.analysis.ecg.preprocess.BuildECGImages.converSingleECGToImage(BuildECGImages.java:305)
at org.akintayo.analysis.ecg.preprocess.BuildECGImages.main(BuildECGImages.java:457)
at org.akintayo.hadoop.HadoopECGPreprocessByFile$MapTest.map(HadoopECGPreprocessByFile.java:208)
at org.akintayo.hadoop.HadoopECGPreprocessByFile$MapTest.map(HadoopECGPreprocessByFile.java:1)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Java can not use jars that are in other jar:/ (classloaders can't handle this)
So what you have to do is to install those packages separately on each machine in cluster, or if not possible add jars on the run, to do this you have to add option -libjars mylib.jar when running hadoop jar myjar.jar -libjars mylib.jar and this should work.
Wojtek's answer is correct. Using -libjars will put your external jars in the distributed cache and make them available to all of your Hadoop nodes.
However, if your external jars are not changing frequently, you may find it more convenient to copy the jar files to the node's hadoop/lib manually. Once you restart Hadoop your external jar will be added to the classpath of your jobs.