I have created a jar containing class files used for SASL callbacks and is present in my application directory.
Is there a way to link this jar with Kafka start up (using kafka-server-start.sh script) without having to explicitly copy this jar under Kafka's libs directory?
Environment: RHEL
You can export the CLASSPATH variable prior to running kafka-server-start, or you can simply symlink the file
Related
I have installed Kafka in Windows.
Also unpacked connector files jar.
Where to specify path to these files and how to configure the connector and launch it?
In connect properties file, there is plugin.path, which should be set to the absolute path of the parent directory of all connectors folders.
https://docs.confluent.io/current/connect/managing/community.html
I created a shadow jar which included Kafka-client library. But this jar fails to work when placed in $cassaandra_home/conf/trigger directory. Is there a way to add the external jar of Kafka separately and then link it to the main jar?
It might have to be added to Cassandra's CLASSPATH environment variable. You can do this by adding a line in the cassandra-env.sh file, referencing its location:
CLASSPATH="$CLASSPATH:$CASSANDRA_HOME/lib/cassandra-ldap-3.11.4.jar"
The above line allows the use of Instaclustr's Cassandra LDAP Authenticator, referencing it from within Cassandra's lib/ dir (on each node). Give that a try.
I am using Kafka connect to create a MQTT Kafka connection.I put all the kafka MQTT connector specific jar downloaded from confluent site to "/data" folder. And accordingly update the "connect-standalone.properties" file to reflect the plugin path i.e
plugin.path=/opt/kafka_2.11-2.1.1/libs,/data
When I run the Kafka Connect
./connect-standalone.sh ../config/connect-standalone.properties ../config/connect-mqtt-source.properties
I get following error :
[2019-07-18 10:26:05,823] INFO Loading plugin from:
/data/kafka-connect-mqtt-1.2.1.jar
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:220)
[2019-07-18 10:26:05,829] ERROR Stopping due to error
(org.apache.kafka.connect.cli.ConnectStandalone:128)
java.lang.NoClassDefFoundError:
com/github/jcustenborder/kafka/connect/utils/VersionUtil
at io.confluent.connect.mqtt.MqttSourceConnector.version(MqttSourceConnector.java:29)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor(DelegatingClassLoader.java:344)
screenshot as below :
Please note that "connect-utils-0.3.140.jar" is present in "/data" folder as highlighted by red underlines.
Now If I make a soft link screenshot below or copy all the jars from "/data" folder and update the plugin path to :
plugin.path=/opt/kafka_2.11-2.1.1/libs
Kafka connect works perfectly fine.
Any help why it does not work in the first scenario i.e kafka connector specific jars in different folders
From Kafka Connect user guide on Confluent page:
...
Kafka Connect isolates each plugin from one another so that libraries in one plugin are not affected by the libraries in any other plugins. This is very important when mixing and matching connectors from multiple providers.
A Kafka Connect plugin is:
an uber JAR containing all of the classfiles for the plugin and its third-party dependencies in a single JAR file; or
a directory on the file system that contains the JAR files for the plugin and its third-party dependencies.
In your case you have to put plugin jars in one folder, ex /data/pluginName not directly in /data/
More details can be found here: Installing Plugins
I am running WordCount example in eclipse luna 3.8. My job is running fine on localjobrunner but I want it to run on yarn cluster because want to access hadoop logs. Somewhere I read that if job is running on local then it do not create logs until it submit to the resource manager. Submitting job to resource manager is possible only when job is running on yarn.
My working environment:
hadoop-2.6.0 running as pseudo distribute mode.
eclipse luna 3.8.
Any help will be appreciated.
Initialize the Job with YARN specific configurations. Add these configurations in the driver,
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://localhost:8020");
conf.set("mapreduce.framework.name", "yarn");
conf.set("yarn.resourcemanager.address", "localhost:8032");
You need yarn-site.xml and core-site.xml correctly on your classpath as well as all yarn and mapreduce jars(dependencies). Now you may have those jars from maven or something but you most likely missing those config files. You can set these on classpath from "Run as configuration" in eclipse. I assume you have local hadoop installation with these configuration files and you can run hadoop commands. In that case you can point your classpath to that installation's conf dir and lib dirs. It may be tedious but start with just pointing to conf dir (which contains core-site and yarn-site) first and see if that works. If not then also exclude your eclipses local dependencies (maven or similar) of yarn and mapreduce and explicitly set them from your installation dir. check this article for setting classpath for hadoop1:
https://letsdobigdata.wordpress.com/2013/12/07/running-hadoop-mapreduce-application-from-eclipse-kepler/
Here's another article from MapR (ignore mapr client related setup)
https://mapr.com/blog/basic-notes-on-configuring-eclipse-as-a-hadoop-development-environment-for-mapr/
You can do similar steps for hadoop2(yarn) but basic idea is your application runtime has to pickup correct jars and config files on classpath to be able to successfully deploy it on cluster.
i am wondering what is use of jbossall-client.jar ?
Up to JBoss version 4.2.3 these file contains client code for application.
But since JBoss 5.0 these file contains only dependences against other libraries from server client directory. If you want use it you must put also other jar file on in the same directory as jbossall-client.jar. These is excerpt from readme.txt file from jbossall file:
This jar file contains a classpath reference to various client jar files used by jboss client applications.
Each of the jar files in the following list must available in the same directory as the jbossall-client.jar, Otherwise they will not be found by the classloader.
In readme.txt you can also find the list jar files against which jbossall-client has dependencies.
It's a bundling of all JBoss client code into a single JAR, for those who don't want to bother with selecting the individual smaller JARs.