Want to upgrade the zookeeper from 3.4.14 to recent/3.5.6. I have followed the link for upgrade and downloaded the zookeeper jar.
but still on restarting server, it is failing in loading the data.
Tried with snapshot.trust.empty=true flag in configuration but in this case it is not able to load the previous data.
Worked by adding snapshot.0 file in version directory in zookeeper data directory.
Related
I have recently updated the logging jar of our application from log4j-1.2.17.jar to the latest log4j-1.2-api-2.18.0.jar.After configuring the latest .my kafka server and zookeeper server unable to start
log4j-1.2-api-2.16.0.jar
log4j-api-2.16.0.jar
log4j-core-2.16.0.jar
log4j-slf4j-impl-2.16.0.jar
slf4j-api-1.7.30.jar
how to resolve this issue after update the log4j
You cannot just upgrade JARs and hope things will work. Instead, upgrade all of Kafka and Zookeeper, as I believe they both use reload4j now.
https://issues.apache.org/jira/browse/KAFKA-13660
https://issues.apache.org/jira/browse/ZOOKEEPER-4626
I am trying to use kafka connect in a docker container with a custom connector (PROGRESS_DATADIRECT_JDBC_OE_ALL.jar) to connect to an openedge database.
I have put the JAR file in the plugin path (usr/share/java) but it won't load as a connector.
COPY Openedge/PROGRESS_DATADIRECT_JDBC_OE_ALL.jar /usr/share/java/progress
I can load another (standard) connector by putting it in the plugin path. This works
COPY confluentinc-kafka-connect-jdbc-10.3.2 /usr/share/java/confluentinc-kafka-connect-jdbc-10.3.2
A little lost on how to move forward and I'm very new to kafka. My main sources of information are
openedge to kafka streaming and How to use Kafka connect
#OneCricketeer had the solution. As a retro for me and hopefully helpful to someone else, here are my steps to make this work.
Copy the JDBC Connector to CONNECT_PLUGIN_PATH and install with confluent hub install:
COPY confluentinc-kafka-connect-jdbc-10.3.2.zip /usr/share/java
RUN confluent-hub install --no-prompt /usr/share/java/confluentinc-kafka-connect-jdbc-10.3.2.zip
Copy the driver (I ended up using openedge.jar) to the path where other jars are located (like sqllite) according to #OneCricketeer suggestion.
COPY Openedge/openedge.jar /usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/lib
Verify with this by enabling DEBUG as suggested by this page
Finally add a .properties file to create the connector. In my case based on the one in “openedge to kafka streaming” link above
JDBC Drivers are not Connect plugins, nor are they connectors themselves.
You'd need to set the JVM CLASSPATH environment variable for detecting JDBC Drivers, as with any Java process.
The instructions on the linked site suggest you should copy the JDBC Drivers into the directory for the existing Confluent JDBC connector. While you could use a Docker COPY command, the better way would be to use confluent-hub install
Trying upgrade from Kafka 0.8.x to Kafka 0.10.x
I did everything what the Kafka documents tell
(http://kafka.apache.org/documentation.html#upgrade_10)
But I still don't see the Kafka being upgraded. I did:
Insert "inter.broker.protocol.version" and "log.message.format.version" into server.properties
Stop broker -> update "inter.broker.protocol.version" to have current Kafka version -> start broker. Repeated this on all the brokers in the cluster.
Update protocol version to be "0.10.0.0" and did a restart on all the brokers.
I dont think Kafka upgrade ever happened.
I still find the old jars and I dont find the new bash scripts that are to be available when you upgrade to 0.10.0.0
Not sure if I did it right.
Any help would be appreciated.
It looks like you skipped Step 2:
Upgrade the brokers. This can be done a broker at a time by simply bringing it down, updating the code, and restarting it.
After adding inter.broker.protocol.version and log.message.format.version to the config, for each broker, you need to stop it, put the new JARs (for 1.0) and restart it.
You can get the latest 1.0 JAR from http://kafka.apache.org/downloads#1.0.2
I was trying to install and configure kafka manager in my kafka cluster but facing issue while building kafka manager binary as below.
./sbt clean dist.
Server is not connected with internet so not able to download required binary and handing with error:
getting scala version x.x.x
Kindly help to install and configure kafka manager offline.
Thanks
You can run sbt in offline mode by setting below parameter:
$ sbt "set offline := true" run
And make sure you have all the required dependencies and components in the local .ivy cache .ivy2/cache in order to build the project offline.
I have an exisiting Kafka Cluster. I want to install the Kafka REST Proxy:
https://github.com/confluentinc/kafka-rest
If I install confluent does that come with Kafka? I am afraid if I still it on my master Kafka node confluent will override all my settings and mess up my Kafka cluster.
How do you install Kafka REST when you have an existing Kafka cluster?
This is not made clear on their website. I have CentOS and was going to try:
sudo yum install confluent-platform-oss-2.11
Any help would be great....
Download the Confluent Platform tarball, extract it, (or preferrably use APT/YUM) then only configure and run the REST proxy via kafka-rest-start
I wouldn't recommend using APT/YUM to install the entire confluent platform if you already have an existing Kafka. You might be able to only install kafka-rest using it, though.
Alternatively, backup your existing Kafka and Zookeeper property files, then place the Confluent Platform on top of the existing files, keeping the original files. If your Kafka is an old release, take this as a good opportunity to schedule an upgrade. Downloading Confluent isn't going to overwrite anything for the upstream Apache projects version for the corresponding release. If anything, it's an extension