I have the following configuration:
Spring-integration-kafka 1.3.1.RELEASE
I have a custom kafka-sink and a custom kafka-source
The configuration I want to have:
I'd like to still using Spring-integration-kafka 1.3.1.RELEASE with my custom kafka-sink.
I'm changing my kafka-source logic to use Spring-integration-kafka-2.1.0.RELEASE. I noticed the way to implement a consumer/producer is way different to prior versions of Spring-integration-kafka.
My question is: could I face some compatibily issues?
I'm using Rabbit.
You should be ok then; it would probably work with the newer kafka jars in the source's /lib directory since each module is loaded in its own classloader so there should be no clashes with the xd/lib jars.
However, you might have to remove the old kafka jars from the xd/lib directory (which is why I asked about the message bus).
Related
The problem is simple: i want to print all topics from apache kafka after installing kafka module on karaf. I need to get properties from cfg file which is located in jbossfuse/etc and create a KafkaConsumer object. I want to implement BundleActivator to be able to start method in the moment of installation module.
The question is: how can i get properties from the config file?
I found some solution here: some solution, they said " you can use ConfigAdimn service from OSGi spec. ". How can i use it? All examples with the code are welcome
Karaf uses Felix-FileInstall to read config files: http://felix.apache.org/documentation/subprojects/apache-felix-file-install.html
So if there is a file named kafka.cfg, it will pick it up and register a config with the ConfigAdmin-Service under the pid 'kafka'.
You can fetch the ConfigAdmin-Service and fetch the config using an Activator and read that config from there, but I strongly recommend to use DeclarativeServices or Blueprint instead to interact with the OSGi-Framework, both support injection of configuration if it is available.
Because otherwise you have to deal yourself with the following topics:
there is no ConfigAdmin (yet), maybe because your bundle starts earlier)
the ConfigAdmin changes (for example due to a package refresh or update)
the configuration is not yet registered (because felix has not read it yet)
the configuration gets updated (for example somone changes the file)
In my project camus is used for long time and it is never get updated.
The camus project uses kafka version 0.8.2.2. I want to find a workaround to use kafka 1.0.0.
So I cloned the directory and updated the dependency. When I do that the Message here requires additional parameters here.
As given in the github link above, the code compiles but the messages are not read from the kafka due to the condition here.
Is it possible to update the kafka dependency along with appropriate data constructors of kafka.message.Message and make it work.
My Presto plugin has 2 components: some UDFs (for basic MD5 / SHA1 hashing) and an EventListener (for logging queries using FluentD logger)
During development (single-node Presto cluster), I added them under a single Plugin class, bundled a single JAR and faced no problem
During deployment I found a pitfall: the UDFs must be registered with all nodes whereas (my particular) EventListener must be registered only with master node
Now I have two options
1. Bundle them together in single JAR
We can control registration of UDFs / EventListeners via external config file (different configs for master & slave nodes). As more UDFs, EventListeners and other SPIs are added, a single JAR paired with tweaked config file with achieve the desired result.
2. Bundle them as separate JARs
We can create different Plugin classes for UDFs / EventListener and provide corresponding classpaths in META-INF.services/com.facebook.spi.Plugin file through Jenkins. We'll then have different JARs for different components: one JAR for all UDFs, one JAR for all EventListeners etc. However as more functionalities are added in future, we might end up having lots of different JARs.
My questions are
What are the pros and cons of both techniques?
Is there an alternate approach?
I'm currently on Presto 0.194 but will soon be upgrading to Presto 0.206
Either way works. You can do whichever is easiest for you. There's actually a third option in the middle, which is to have multiple Plugin implementations in a single JAR (you would list all implementations in the META-INF/services file).
EventListener is actually used on both the coordinator and workers. Query events happen on the coordinator and split events happen on the workers. However, if you only care about query events, you only need it on the coordinator.
You can deploy the event plugin on both coordinator and workers but only configure it on the coordinator. The code will only be used if you configure it by adding an event-listener.properties file with a event-listener.name property that matches the name you return in your EventListenerFactory.getName() method.
I am trying to execute a sample producer consumer application using Apache Kafka. I downloaded it from https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.0.0/kafka-0.10.0.0-src.tgz . Then I started following the steps given in http://www.javaworld.com/article/3060078/big-data/big-data-messaging-with-kafka-part-1.html.
When I tried to run bin/zookeeper-server-start.sh config/zookeeper.properties, I am getting Error: Could not find or load main class config.zookeeper.properties I googled about the issue but didn't get any useful information on this. Can anyone help me to continue?
You've downloaded the source package. Download the binary package of Kafka and do testing.
You have to download the binary version from the official Kafka web site.
Assuming you have the correct binary version check to see that you do not already have CLASSPATH defined in your environment. If you do and the defined CLASSPATH has a space in it (e.g.C:\Program Files\<>) then neither zookeeper or kafka will start.
To solve this either delete your existing CLASSPATH or modify the startup script that builds the zookeeper and kafka CLASSPATH values, putting your CLASSPATH entry in double quotes before the path is built
I have created a set of applications in akka in the multi-jvm. Following all the conventions on the docs http://doc.akka.io/docs/akka/snapshot/dev/multi-jvm-testing.html I can run them using multi-jvm:run {application name}.
This behaves perfectly but the applications now require remote akka features. To do this I need to change settings in application.conf as mentioned here: http://doc.akka.io/docs/akka/snapshot/scala/remoting.html
My problem is that I do not now how to give each of these multi-jvm test applications an application.conf file of their own. I'm not sure if their is a file-system based convention, or it has to be done in code. Either solution would solve the problem in theory.
I suggest that you define the configuration in code in the test using ConfigFactory.parseString and use that in MultiNodeConfig.commonConfig.
Note that you can use define specific config per node with MultiNodeConfig.nodeConfig.
The application.conf's are loaded from the class path, but you can override them with system properties as described here: https://github.com/typesafehub/config#standard-behavior
So for each JVM, you can specify the necessary System properties(which is kind of ugly), or you can drop an application.conf in the classpath of each JVM that overrides the reference.conf's in the libraries.