Any pointers for creating Kafka endpoint support? - apache-kafka

I have a fairly immediate need to support Kafka integration testing using the Citrus Framework. I was thinking of taking the existing jms module as an example/framework and using Spring Kafka. Any pointers or gotchas that I should be aware of? I am willing, assuming I'm successful, to donate the module back to the project.

Here is an example of how you can use a Kafka Camel component with Citrus:
#Bean
public CamelContext camelKafkaAdapterContext() throws Exception {
SpringCamelContext context = new SpringCamelContext();
context.addRouteDefinition(new RouteDefinition()
.from("kafka:localhost:9092?topic=test&zookeeperHost=localhost&zookeeperPort=2181&serializerClass=kafka.serializer.StringEncoder")
.to("log:com.consol.citrus.camel?level=DEBUG")
.to("seda:kafka-buffer"));
return context;
}
#Bean
public CamelEndpoint kafkaEndpoint(CamelContext camelContext) {
CamelEndpoint endpoint = new CamelEndpoint();
endpoint.getEndpointConfiguration().setCamelContext(camelContext);
endpoint.getEndpointConfiguration().setEndpointUri("seda:kafka-buffer");
return endpoint;
}
You first define a Camel Context which will be startet when you run any test with Citrus. After it is instantiated, this Camel component will read from the configured topic and send all messages into a buffer seda:kafka-buffer (seda is used only as an example). After which you can use a Citrus CamelEndpoint to read messages from that buffer inside any test.
receive(action -> action.endpoint(kafkaEndpoint)
.messageType(MessageType.JSON)
.payload(...);
Note, this is just an example I have assembled. I haven't tested this exact setup, but it will work once you configure the Camel Context correctly.

Related

Cannot get custom store connected to a Transformer with Spring Cloud Stream Binder Kafka 3.x

Cannot get custom store connected to my Transformer in Spring Cloud Stream Binder Kafka 3.x (functional style) following examples from here.
I am defining a KeyValueStore as a #Bean with type StoreBuilder<KeyValueStore<String,Long>>:
#Bean
public StoreBuilder<KeyValueStore<String,Long>> myStore() {
return Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore("my-store"), Serdes.String(),
Serdes.Long());
}
#Bean
#DependsOn({"myStore"})
public MyTransformer myTransformer() {
return new MyTransformer("my-store");
}
In debugger I can see that the beans get initialised.
In my stream processor function then:
return myStream -> {
return myStream
.peek(..)
.transform(() -> myTransformer())
...
MyTransformer is declared as
public class MyTransformer implements Transformer<String, MyEvent, KeyValue<KeyValue<String,Long>, MyEvent>> {
...
#Override
public void init(final ProcessorContext context) {
this.context = context;
this.myStore = context.getStateStore(storeName);
}
Getting the following error when application context starts up from my unit test:
Caused by: org.apache.kafka.streams.errors.StreamsException: Processor KSTREAM-TRANSFORM-0000000002 has no access to StateStore my-store as the store is not connected to the processor. If you add stores manually via '.addStateStore()' make sure to connect the added store to the processor by providing the processor name to '.addStateStore()' or connect them via '.connectProcessorAndStateStores()'. DSL users need to provide the store name to '.process()', '.transform()', or '.transformValues()' to connect the store to the corresponding operator, or they can provide a StoreBuilder by implementing the stores() method on the Supplier itself. If you do not add stores manually, please file a bug report at https://issues.apache.org/jira/projects/KAFKA.
In the application startup logs when running my unit test, I can see that the store seems to get created:
2021-04-06 00:44:43.806 INFO [ main] .k.s.AbstractKafkaStreamsBinderProcessor : state store my-store added to topology
I'm already using pretty much every feature of the Spring Cloud Stream Binder Kafka in my app and from my unit test, everything works very well. Unexpectedly, I got stuck at adding the custom KeyValueStore to my Transformer. It would be great, if you could spot an error in my setup.
The versions I'm using right now:
org.springframework.boot:spring-boot:jar:2.4.4
org.springframework.kafka:spring-kafka:jar:2.6.7
org.springframework.kafka:spring-kafka-test:jar:2.6.7
org.springframework.cloud:spring-cloud-stream-binder-kafka-streams:jar:3.0.4.RELEASE
org.apache.kafka:kafka-streams:jar:2.7.0
I've just tried with
org.springframework.cloud:spring-cloud-stream-binder-kafka-streams:jar:3.1.3-SNAPSHOT
and the issue seems to persist.
In your processor function, when you call .transform(() -> myTransformer()), you also need to provide the state store names in order for this to be connected to that transformer. There are some overloaded transform methods in the KStream API that takes state store names as a vararg. I wonder if this is the issue that you are running into. You may want to change that call to .transform(() -> myTransformer(), "myStore").

Spring boot Kafka request-reply scenario

I am implementing POC of request/reply scenario in order to move event-based microservice stack with using Kafka.
There is 2 options in spring.
I wonder which one is better to use. ReplyingKafkaTemplate or cloud-stream
First is ReplyingKafkaTemplate which can be easily configured to have dedicated channel to reply topics for each instance.
record.headers().add(new RecordHeader(KafkaHeaders.REPLY_TOPIC, provider.getReplyChannelName().getBytes()));
Consumer should not need to know replying topic name, just listens a topic and returns with given reply topic.
#KafkaListener(topics = "${kafka.topic.concat-request}")
#SendTo
public ConcatReply listen(ConcatModel request) {
.....
}
Second option is using combination of StreamListener, spring-integration and IntegrationFlows. Gateway should be configured and reply topics should be filtered.
#MessagingGateway
public interface StreamGateway {
#Gateway(requestChannel = START, replyChannel = FILTER, replyTimeout = 5000, requestTimeout = 2000)
String process(String payload);
}
#Bean
public IntegrationFlow headerEnricherFlow() {
return IntegrationFlows.from(START)
.enrichHeaders(HeaderEnricherSpec::headerChannelsToString)
.enrichHeaders(headerEnricherSpec -> headerEnricherSpec.header(Channels.INSTANCE_ID ,instanceUUID))
.channel(Channels.REQUEST)
.get();
}
#Bean
public IntegrationFlow replyFiltererFlow() {
return IntegrationFlows.from(GatewayChannels.REPLY)
.filter(Message.class, message -> Channels.INSTANCE_ID.equals(message.getHeaders().get("instanceId")) )
.channel(FILTER)
.get();
}
Building reply
#StreamListener(Channels.REQUEST)
#SendTo(Channels.REPLY)
public Message<?> process(Message<String> request) {
Specifying reply channel is mandatory. So receieved reply topics are filtered according to instanceID which a kind of workaround (might bloat up the network). On the other hand, DLQ scenario is enabled with adding
consumer:
enableDlq: true
Using spring cloud streams looks promising in terms of interoperability with RabbitMQ and other features, but not officially supports request reply scenario right away. Issue is still open, not rejected also. (https://github.com/spring-cloud/spring-cloud-stream/issues/1800)
Any suggestions are welcomed.
Spring Cloud Stream is not designed for request/reply; it can be done, it is not straightforward and you have to write code.
With #KafkaListener the framework takes care of everything for you.
If you want it to work with RabbitMQ too, you can annotate it with #RabbitListener as well.

Camel keep sending messages to queue via JMS after 1 minute

I am currently learning Camel and i am not sure if we can send messages to a activemq queue/topic from camel at fixed interval.
Currently i have created code in Scala which looks up the database and create a message and sends it to queue after every minute can we do this in camel.
We have a timer component in camel but it does not produce the message. I was thinking something like this.
from("timer://foo?fixedRate=true&period=60000")
.to("customLogic")
.to("jms:myqueue")
Timer will kick after a minute.
Custom logic will do database look up and create a message
Finally send to jms queue
I am very new to Camel so some code will be really helpful thanks
Can you please point me to how can i create this customeLogic method that can create a message and pass it to next ".to("jms:myqueue")". Is there some class that in need to inherit/implement which will pass the the message etc.
I guess your question is about how to hook custom java logic into your camel route to prepare the JMS message payload.
The JMS component will use the exchange body as the JMS message payload, so you need to set the body in your custom logic. There are several ways to do this.
You can create a custom processor by implementing the org.apache.camel.Processor interface and explicitly setting the new body on the exchange:
Processor customLogicProcessor = new Processor() {
#Override
public void process(Exchange exchange) {
// do your db lookup, etc.
String myMessage = ...
exchange.getIn().setBody(myMessage);
}
};
from("timer://foo?fixedRate=true&period=60000")
.process(customLogicProcessor)
.to("jms:myqueue");
A more elegant option is to make use of Camel's bean binding:
public class CustomLogic {
#Handler
public String doStuff() {
// do your db lookup, etc.
String myMessage = ...
return myMessage;
}
}
[...]
CustomLogic customLogicBean = new CustomLogic();
from("timer://foo?fixedRate=true&period=60000")
.bean(customLogicBean)
.to("jms:myqueue");
The #Handler annotation tells Camel which method it should call. If there's only one qualifying method you don't need that annotation.
Camel makes the result of the method call the new body on the exchange that will be passed to the JMS component.

Putting output from a Spring Batch item writer onto a Spring Integration message channel

What is the best way to take output from a Spring Batch item writer - obviously run as part of a job - and to place it onto a Spring Integration message channel? I understand that one option is to code a Spring Batch listener (extending ItemWriteListener), which would be injected with a message channel, and then to attach the listener to the batch job. Is there another option?
One option to inject to the ItemWriter a Spring Integration <gateway>:
<int:gateway id="itemWriterGateway" service-interface="com.my.proj.MyGateway"/>
class MyItemWriter implements ItemWriter<Foo> {
#Autowired
private MyGateway gateway;
public void write(List<Foo> items) {
for(Foo foo : items) {
this.gateway.send(foo);
}
}
}
Another option to use ItemWriterAdapter:
<bean id="myItemWriter" class="org.springframework.batch.item.adapter.ItemWriterAdapter">
<property name="targetObject" ref="itemWriterGateway"/>
<property name="targetMethod" value="send"/>
</bean>
I won't argue that StepListener may have value for you too. But here there is need to understand what you want to achieve using Spring Integration from Batch Job...
UPDATE
Actually, Listener won't help you, because the ItemWriter doesn't return anything and it encapsulates its logic around underlying resource.
As you say that you are using StaxEventItemWriter and its Resource is some output file, maybe there is a reason to decouple your logic and read that result File using <int-file:inbound-channel-adapter>...
UPDATE 2
) why won't the listener help
Because you need a result of ItemReader, otherwise you have to build your XML one more time in the SI flow from items (ItemWriteListener#afterWrite).
How could the the StaxEventItemWriter output, and the inbound adapter connect to, a resource which is held in memory?
How does your resource is in memory, if StaxEventItemWriter requires that it should exists?
From other side you can share that resource with ItemWriter and some SI endpoint. And right: you can do it from ChunkListener#afterChunk and get your resource from, e.g. jobParameters and send it to the SI Gateway.

JAX-WS SoapHandler with large messages: OutOfMemoryError

Using JAX-WS 2, I see an issue that others have spoken about as well. The issue is that if a SOAP message is received inside a handler, and that SOAP message is large - whether due to inline SOAP body elements that happen to have lots of content, or due to MTOM attachments - then it is dangerously easy to get an OutOfMemoryError.
The reason is that the call to getMessage() seems to set off a chain of events that involve reading the entire SOAP message on the wire, and creating an object (or objects) representing what was on the wire.
For example:
...
public boolean handleMessage(SOAPMessageContext context)
{
// for a large message, this will cause an OutOfMemoryError
System.out.println( context.getMessage().countAttachments() );
...
My question is: is there a known mechanism/workaround for dealing with this? Specifically, it would be nice to access the SOAP part in a SOAP message without forcing the attachments (if MTOM for example) to also be vacuumed up.
For those who run their app on JBoss 6 & 7 (with Apache CXF)... I was able to troubleshoot the problem by implementing my handler from the LogicalHandler interface instead of the SOAPHandler.
In this case your handleMessage() method would get the LogicalMessageContext context (instead of SOAPMessageContext) in the arguments that has no issues with the context.getMessage() call
There's actually a JAX-WS RI (aka Metro) specific solution for this which is very effective.
See https://javaee.github.io/metro/doc/user-guide/ch02.html#efficient-handlers-in-jax-ws-ri. Unfortunately that link is now broken but you can find it on WayBack Machine. I'll give the highlights below:
The Metro folks back in 2007 introduced an additional handler type, MessageHandler<MessageHandlerContext>, which is proprietary to Metro. It is far more efficient than SOAPHandler<SOAPMessageContext> as it doesn't try to do in-memory DOM representation.
Here's the crucial text from the original blog article:
MessageHandler:
Utilizing the extensible Handler framework provided by JAX-WS
Specification and the better Message abstraction in RI, we introduced
a new handler called MessageHandler to extend your Web Service
applications. MessageHandler is similar to SOAPHandler, except that
implementations of it gets access to MessageHandlerContext (an
extension of MessageContext). Through MessageHandlerContext one can
access the Message and process it using the Message API. As I put in
the title of the blog, this handler lets you work on Message, which
provides efficient ways to access/process the message not just a DOM
based message. The programming model of the handlers is same and the
Message handlers can be mixed with standard Logical and SOAP handlers.
I have added a sample in JAX-WS RI 2.1.3 showing the use of
MessageHandler to log messages and here is a snippet from the sample:
public class LoggingHandler implements MessageHandler<MessageHandlerContext> {
public boolean handleMessage(MessageHandlerContext mhc) {
Message m = mhc.getMessage().copy();
XMLStreamWriter writer = XMLStreamWriterFactory.create(System.out);
try {
m.writeTo(writer);
} catch (XMLStreamException e) {
e.printStackTrace();
return false;
}
return true;
}
public boolean handleFault(MessageHandlerContext mhc) {
.....
return true;
}
public void close(MessageContext messageContext) { }
public Set getHeaders() {
return null;
}
}
(end quote from 2007 blog post)
You can find a full example in the Metro GitHub repo.
What JAX-WS implementation runtime are you using? If there's a way to do this using the runtime built into WebSphere I'm certain there's a way to do this cleanly in other runtimes like Axis2 (proper), Apache CXF, and Metro/RI.
I am using the other way to reduce the memory costing, which is Message Accessor.
Instead of using context.getMessage(), I changed it to this way:
Object accessor = context.get("jaxws.message.accessor");
if (accessor != null) {
baosInString = accessor.toString();
}
Base on advice from IBM website. http://www-01.ibm.com/support/docview.wss?uid=swg1PM21151