I want to write tests for spring kafka producer and consumer. I have tried multiple ways:
EmbeddedKafka annotation
EmbeddedKafkaRule
EmbeddedKafkaBroker
etc...
Every time I get one or the other error and all the examples posted on GitHub don't seem to run at all. I checked the spring kafka versions for compatibility as well.
Can someone share an example code base that was recently written and has seen it run successfully?
There are hundreds of tests in the framework itself.
This is probably the most extensive one...
https://github.com/spring-projects/spring-kafka/blob/1b9a9451feea7cca16903f1c990c74c6be9b8ffb/spring-kafka/src/test/java/org/springframework/kafka/annotation/EnableKafkaIntegrationTests.java#L164-L176
Related
I need to write integration tests and mock reactive Kafka consumer. I see there're ways to do it with blocking Kafka, like using #EmbeddedKafka but was not able to find information about reactive
As linked in the comments, reactor-kafka project themselves use TestContainers for integration tests. If you don't run tests in an environment with Docker, then spring-kafka-test's EmbeddedKafka, or junit-kafka project should still work with Reactive-based Kafka clients since you really only need bootstrap.servers property to point at any valid broker server.
Regarding mocks, you don't need an actual broker; that's the point of mocking. Their source code does include mock classes, too.
I have a Spring cloud stream application which I need to make an integration test for (to be specific using cucumber). The application communicate with other services using Kafka message broker. From what I know I could make this work using either a kafka testcontainers or using spring provided embedded kafka. But what I don't know is which one would be the best solution so are there anything that the testcontainer could do but embedded can't or the other way around? (use cases or example would be appreciate!)
p.s This integration should be able to run on ci/cd pipeline.
It is called embedded for a reason. It really can be only accessed from the process which spawned it. With Testcontainers you really can reuse existing container and have access to it from the other process. But that's probably to exotic.
I guess with properly configured Testcontainers you can reach as much as possible similarities with the prod you'd deploy your solution. The embedded Kafka might be limited in some areas, e.g. SSL configuration or so.
My project has a lot of Spring Batch Jobs.
I have requirement to create externalized configuration for message brokers (example Kafka, rabbitMQ etc.).
I want to use spring cloud stream since it has various binders to solve this problem.
Hence i wanted to understand if there is an integration between the two frameworks, please explain.
Currently, at my company we are migrating from Kafka 0.8 to 0.11, brokers migration steps and clearly stated in kafka documentation here
What I am stuck in is, upgrading the kafka clients (producers, consumers, spark-streaming), I don't find any documentation/ articles listing out clearly what are the required changes or steps to follow to upgarde the client, all what I found is the java doc Producer Client
What I did so far is to change the kafka client version in my gradle to kafka-clients-0.11.0.0, and everything from the compilation point of view went fine with no code changes at all.
What I seek help with is, is there any expected problems I should take care of, any pointers for client changes other than the kafka-client version?
I went through lots of experiments to get this done.
For the consumers and producers, I just used the kafka consumers and producers 0.11.0.
The trick part was replacing spark-streaming, spark-streaming latest version only support upto kafka 0.10.X, which doesn't contains any updates related to the new broker.
What I recommend here, if you are about to write an application from scratch and your main goal is realtime streaming go for kafka-streaming API, it is just AWESOME!, if you already have spark streaming app (which was my case), you should either judge which is more important than the other wether to get stuck with the kafka-broker version 10.X and spark-streaming which was [experimental][1] btw.
The benefits of having the streaming inside kafka not spark the following:
Kafka streaming is a normal jar that can be injected in any java application, so you don't care that much about deployment, and environment
Auto-scaling is so easy when using kafka-streaming using any scaleset provided by any cloud service provider, unlike scaling a HDP cluster.
Monitoring using something like prometheus would be much easier.
I am new to kafka.
I have downloaded kafka 2.9.2-0.8.1.1. I have started zookeeper, broker, producer and consumer from command prompt. I successfully created a topic and sent a message.
Now I want to run the producer from eclipse. But I dont know how to to do that. I found some links like http://vulab.com/blog/?p=611 to do this but I am still not able to run it. Is this the correct process mentioned in the link? Do i really need to create maven project in eclipse? Or is there any other way to do that?
You need to use the Kafka Java API for creating kafka producer (if you intent to use JAVA). Using maven is highly recomended as it will help managing all the dependencies for you. But you are free to bypass maven if you are ready to manage all the required JARs by yourself.
kafka wiki is another good place to look at.