In trying to learn about Apache Storm, I have been following Unboxing Apache Storm series (https://www.youtube.com/watch?v=QoEyXKIKZKY&list=PLeUBsMTwZBi3rPzowug5-PdGA_cmD9QbE&index=1), with the exception that I am working under Windows rather than an Linux system. For the windows specific installation, I followed https://www.techgeeknext.com/apache/install-apache-storm#google_vignette. I am using Eclipse to set up the main and other classes that contain the toy example in the series (video 7/Creating First Java Project). However, once I try to run the example, I get the error:
Unable to canonicalize address localhost/:2000 because it's not resolvable.
I thought the error was because I had the clientPort in zookeeper.properties and storm.zookeeper.port in storm.yaml set to the default of 2181, so I changed it to 2000, but I still get the same error. I am using JDK 1.8.333, apache storm 1.2.3, eclipse 2022-03 (4.23.0), python 3.9.7, and zookeeper 3.4.14. If anyone has any suggestions, that would be great.
Related
I am running Pycharm on my MacBook.
Client settings:
Python Interpreter -> Python 3.7 (dtabricks-connect-6.4)
Cluster settings:
Databricks Runtime Version -> 6.4 (includes Apache Spark 2.4.5, Scala 2.11)
It worked well for months but suddenly, without any updates made, I cant run my python script from Pycharm against databricks cluster anymore.
The Error is ...
Caused by: `java.lang.IllegalArgumentException: The cluster is running server version `dbr-6.4` but this client only supports Set(dbr-5.5)....`
I restarted Pycharm, I witched back and for the interpreter, I restarted the cluster and I even restarted my MacBook but it didnt helped. The Error message is simply false, because both, cluster and client are using the SAME version. I can see that when I execute my python script that the cluster is being started but fails at the end.
pyenv activate databricks-connect-6-4
pip freeze
Cython==0.29.21
databricks-connect==6.4.0
numpy==1.19.2
pandas==1.0.1
py4j==0.10.7
pyarrow==0.13.0
pycountry==20.7.3
python-dateutil==2.8.1
pytz==2020.1
six==1.15.0
It looks like that it was caused by some internal changes on the server side, and this prevents databricks-connect from working. You can always disable this check by setting environment variable DEBUG_IGNORE_VERSION_MISMATCH to value 1 (export DEBUG_IGNORE_VERSION_MISMATCH=1 in the console before executing databricks-connect test) - you can also set this environment variable in the PyCharm.
Update: it should be fixed by Databricks Connect 6.4.2 that was just released.
Thanks for raising this. The Databricks-Connect team has acknowledged this issue and we are working on a patch to address this issue. Will keep you posted. In the meantime you can use the DEBUG_IGNORE_VERSION_MISMATCH as Alex pointed out.
Update: A compatible db-connect client has been released to fix this problem: version 6.4.2 (https://pypi.org/project/databricks-connect/6.4.2/, install with: pip install databricks-connect==6.4.2).
I am using cygwin on window.
and Running follows command on cygwin for starting ksql server.
bash ksql-server-start .properties
Error: Could not find or load main class Folder.confluent-5.5.1-2.12.tar.confluent-5.5.1-2.12.confluent-5.5.1.logs
Thanks.
No Windows compatible scripts are provided with ksqlDB. You may be able to 'fix' the scripts that do come with it to work on windows. Alternatively, you can just run ksqlDB and the CLI in docker, which is the supported routed for Microsoft windows. See the ksqlDB microsite for instructions on how to get something up and running in docker, or the Kafka examples microsite.
I searched this forum already, no working answer is found for my case:
I installed java 1.8
I downloaded the binary version of zookeeper-3.5.5 from https://www.apache.org/dist/zookeeper/zookeeper-3.5.5/apache-zookeeper-3.5.5-bin.tar.gz
I changed the zoo_sample.cfg to zoo.cfg, no changes made to the cfg
I tried to start zookeeper as root:
[root#pocdnr1n1 apache-zookeeper-3.5.5-bin]# bin/zkServer.sh start conf/zoo.cfg
I received error in log:
Could not find the main class: org.apache.zookeeper.server.quorum.QuorumPeerMain. Program will exit.
Thanks.
I think I have found the root cause, posting it here for future readers:
The culprit is Java.
I had an old java version on the node, this can be verified by running java -version
In my case the java was 1.6, what I did is to reset the environment to add the new java 1.8 to it:
# export JAVA_HOME=/opt/jdk1.8.0_151
# export JRE_HOME=/opt/jdk1.8.0_151/jre/
# export PATH=$PATH:/opt/jdk1.8.0_151/bin:/opt/jdk1.8.0_151/jre/bin
You should add the PATH to the .bash_profile so that it becomes permanent.
Now after that you run source .bash_profile, now you have set java 1.8 as the default java. again, you can confirm that by running java -version
Run your zookeeper again and it will be started as expected.
The common missings here are:
downloaded the non-binary zookeeper
java is too low (1.6 doesn't work, 1.8 is recommended)
environment should be set to ensure java 1.8 is picked up as default version
I hope this helps.
If you look at ZooKeeper Administrator's Guide - Required Software for 3.5.5 it says:
ZooKeeper runs in Java, release 1.7 or greater (JDK 7 or greater, FreeBSD support requires openjdk7).
which affirms what you found out. Your Java version was too low.
What worked for me was rebuilding with ./gradlew jar -PscalaVersion-2.13.10
I was using kafka straight from source from github.com/apache/kafka and that step was given to me when I cloned a fresh copy of the repo to start from scratch.
I had done a git pull on my old version, so it broke the dependencies, which I forgot I must have at one point installed.
How to run Apache Storm in Single Node on Windows OS? Can anyone provide a link for that?
Install Java
Download and install a JDK (Storm works with both Oracle and OpenJDK 6/7). For this setup I used JDK 7 from Oracle.
I installed Java in:
C:\Java\jdk1.7.0_45\
Install Python
To test the installation, we’ll be deploying the “word count” sample from the storm-starter project which uses a multi-lang bolt written in python. I used python 2.7.6 which can be downloaded here.
I installed python in:
C:\Python27\
Install and Run Zookeeper
Download Apache Zookeeper 3.3.6 and extract it. Configure and run Zookeeper with the following commands:
> cd zookeeper-3.3.6
> copy conf\zoo_sample.cfg conf\zoo.cfg
> .\bin\zkServer.cmd
Install Storm
The changes that allow Storm to run seamlessly on Windows have not been officially released yet, but you can download a build with those changes incorporated here.
(Source branch for that build can be found here).
Extract that file to the location of your choice. I chose C:.
Configure Environment Variables
On Windows Storm requires the STORM_HOME and JAVA_HOME environment variables to be set, as well as some additions to the PATH variable:
JAVA_HOME:
C:\Java\jdk1.7.0_45\
STORM_HOME:
C:\storm-0.9.1-incubating-SNAPSHOT-12182013\
PATH: (add)
%STORM_HOME%\bin;%JAVA_HOME%\bin;C:\Python27;C:\Python27\Lib\site-packages\;C:\Python27\Scripts\;
PATHEXT: (add)
.PY
Start Nimbus, Supervisor, and Storm UI Daemons
For each deamon open a separate command prompt.
Nimbus
cd %STORM_HOME%
storm nimbus
Supervisor
cd %STORM_HOME%
storm supervisor
Storm UI
cd %STORM_HOME%
storm ui
Verify that Storm is running by opening http://localhost:8080/ in a browser.
Deploy the “Word Count” Topology
Either build the storm-starter project from source, or download a pre-built jar
Deploy the Word Count topology to your local cluster with the storm jar command:
storm jar storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.WordCountTopology WordCount -c nimbus.host=localhost
I have been developing a Java program in Netbeans/Ubuntu 12.04 using OpenJdk. I have got this program to run successfully using the .jar in Linux, but when I copy the .jar to my Windows 7 virtual machine I get the following error;
"Could not find the main class: testprogram.Main. Program will exit."
Also this program uses no external libraries.
How did you start your program? Did you use java -jar yourjarfile.jar?
Maybe your Manifest file does not defined applycation entry point. You can study this topic here.
My fix to this problem was to use JDK 6 as opposed to JDK 7, still unsure as to why this error was produced, but this change has fixed the problem.