could you please advise , how can I stop sending to my 3rd kafka topic, when the control reaches the catch block, currently the message is sent to both error topic as well as the topic to which it should send in case of normal processing. A snippet of code is as below:
#Component
public class Abc {
private final StreamBridge streamBridge;
public Abc (StreamBridge streamBridge)
this.streamBridge = streamBridge;
#Bean
public Function<KStream<String, KafkaClass>, KStream<String,KafkaClass>> hiProcess() {
return input -> input.map((key,value) -> {
try{
KafkaClass stream = processFunction();
}
catch(Exception e) {
Message<KakfaClass> mess = MessageBuilder.withPayload(value).build();
streamBridge.send("errProcess-out-0". mess);
}
return new KeyValue<>(key, stream);
})
}
}
This can be implemented using the following pattern:
KafkaClass stream;
return input -> input
.branch((k, v) -> {
try {
stream = processFunction();
return true;
}
catch (Exception e) {
Message<KakfaClass> mess = MessageBuilder.withPayload(value).build();
streamBridge.send("errProcess-out-0". mess);
return false;
}
},
(k, v) -> true)[0]
.map((k, v) -> new KeyValue<>(k, stream));
Here, we are using the branching feature (API) of KStream to split your input into two paths - normal flow and the one causing the errors. This is accomplished by providing two filters to the branch method call. The first filter is the normal flow in which you call the processFunction method and get a response back. If we don't get an exception, the filter returns true, and the result of the branch operation is captured in the first element of the output array [0] which is processed downstream in the map operation in which it sends the final result to the outbound topic.
On the other hand, if it throws an exception, it sends whatever is necessary to the error topic using StreamBridge and the filter returns false. Since the downstream map operation is only performed on the first element of the array from branching [0], nothing will be sent outbound. When the first filter returns false, it goes to the second filter which always returns true. This is a no-op filter where the results are completely ignored.
One downside of this particular implementation is that you need to store the response from processFunction in an instance field and then mutate on each incoming KStream record so that you can access its value in the final map method where you send the output. However, for this particular use case, this may not be an issue.
Related
Especially if the signal processing needs to invoke an/some activities, how can I achieve that?
I tried to return data or exception but it doesn't work.
Data cannot be returned from signal method. Throwing exception will block workflow execution.
Common mistakes
It's wrong to return data in a signal method, or throw an exception -- because signal method is meant to be Asynchronous. The processing must be like Kafka processing messages and you can't return the result via the method returning.
So below code will NOT work:
public class SampleWorkflow{
public Result mySignalMethod(SignalRequest req){
Result result = activityStub.execute(req)
if(...){
throw new RuntimeException(...)
}
return result
}
}
What should you do
What you must do:
Make sure signal don't return anything
Use a query method to return the results
In signal method processing, store the results into workflow state so that query can return the states
A bonus if you also use the design pattern to store signal request into a queue, and let workflow method to process the signal. This will give you some benefits
Guarantee FIFO ordering of signal processing
Make sure reset workflow won't run into issues -- after reset, signals will be preserved and moved to earlier position of the workflow history. Sometimes workflow are not initialized to replay the signals.
Also make exception handling easier
See this design pattern in sample code: Cadence Java sample/Temporal java sample
If we applied all above, the sample code should be like below :
public class SampleWorkflow{
private Queue<SignalRequest> queue = new Queue<>();
private Response<Result> lastSignalResponse;
public void myWorkflowMethod(){
Async.procedure(
() -> {
while (true) {
Workflow.await(() -> !queue.isEmpty());
final SignalRequest req =
queue.poll();
// alternatively, you can use async to start an activity:
try{
Result result = activityStub.execute(req);
}catch (ActivityException e){
lastSignalResponse = new Response( e );
}
if(...){
lastSignalResponse = new Response( new RuntimeException(...) );
}else{
lastSignalResponse = new Response( result);
}
}
});
...
}
public Response myQueryMethod(){
return lastSignalResponse;
}
public Result mySignalMethod(SignalRequest req){
queue.add(req)
}
}
And in the application code, you should signal and then query the workflow to get the result:
workflowStub.mySignalMethod(req)
Response response = workflowStub.myQueryMethod()
Follow this sample-Cadence / sample-Temporal if you want to use aysnc activity
Why
Signal is executed via Workflow decision task(Workflow task in Temporal). A decision task cannot return result. In current design, there is no mechanism to let a decision task return result to application code.
Throw exception in workflow code will either block the decision task or fail the workflow).
Query method is designed to return result. -- However, query cannot schedule activity or modify workflow states.
It's a missing part to let app code to make a synchronous API call to update and return data. It needs a complicated design: https://github.com/temporalio/proposals/pull/53
We have a use-case, where we have to read some messages into the KStreams, then transform the message and produce it to another topic conditionally.
In our use-case, for transformation of object, we make an downstream API call. If the API call is succesfull, then produce to newTopic1, else produce to newTopic2. How can the same be achieved ??
As of now, we are using the below style of producing the enriched (i.e. transformed objects) to the new Kafka topic using to method provided by Streams API.
KStream<String, Customer> transformedStream = sourceKafkaStream
.mapValues(cust -> {
try {
logger.info("Hitting to the downstream to fetch additional information, will take few seconds.");
Thread.sleep(7000);
return RecordEnrichmentAndTransformationBuilder.enrichTheCustomer(cust);
} catch (InterruptedException e) {
e.printStackTrace();
}
return cust;
});
.to('newTopic1', Produced.with(AppSerdes.String(), AppSerdes.serdeForEnrichedCustomer()));
Appreciate the response on this.
Using DSL api you use KStream::filter or KStream:to(TopicNameExtractor<K, V> topicExtractor, Produced<K, V> produced).
Sample code will looks like that if both format are the same:
KStream<String, Customer> transformedStream = sourceKafkaStream
.mapValues(cust -> {
try {
logger.info("Hitting to the downstream to fetch additional information, will take few seconds.");
Thread.sleep(7000);
return RecordEnrichmentAndTransformationBuilder.enrichTheCustomer(cust);
} catch (InterruptedException e) {
e.printStackTrace();
}
return cust;
});
.to((key, value, recordContext) -> topicNameCalculation(key, value), Produced.with(AppSerdes.String(), AppSerdes.serdeForEnrichedCustomer()));
topicNameCalculation(...) will based on the key and the value chose the proper topic.
One Notice
Generally it is not good approach to make external call in Kafka Streams.
I would like to merge two Single<MyData> such that if one of them fails but the other one succeeds then the error of the one that failed and the emission from the other one are reported, and then the resulting Single<MyData> (or Observable<MyData>) completes.
If both Single<MyData> fail then the result should also fail and also be marked as failed.
What I would like to have at the end is:
If both succeed then the emitted values and a producer marked as completed.
If one succeeds and the other fails, the emitted value, the thrown error and the producer marked as complete.
If all fail, the errors and the producer marked as failed.
It's like an 'OR' operation
This is not possible. There is only a single terminal event allowed. The contract for Single is success|error. If you need to receive a next event as well, you should consider to use Observable instead. The contract for Observable is next* complete|error, but you'll still not get a complete.
Observable.mergeDelayError(single1.toObservable(), single2.toObservable())
This can be accomplished with Single.create(SingleOnSubscribe). If your return type is Single<MyData> only one of the responses can be returned, but you could also modify this to instead return a Single<List<MyData>> or some other RxJava structure like Flowable<MyData> that supports multiple returns. In this example, the Single<MyData> returns whichever call returns last because that was the simplest to implement.
public Single<MyData> getCombinedSingle(List<Single<MyData>> singles) {
return Single.create(new SingleOnSubscribe<MyData> {
private boolean encounteredError = false;
private MyData myData;
#Override
public void subscribe(#NonNull Emitter<MyData> emitter) {
List<Disposable> disposables = new ArrayList<>();
Consumer<MyData> myDataConsumer = myData -> {
this.MyData = myData;
checkForFinish(emitter, disposables);
}
Consumer<Throwable> throwableConsumer = throwable -> {
throwable.printStackTrace();
encounteredError = true;
checkForFinish(emitter, disposables);
}
for (Single single: singles) {
disposables.put(single.subscribe(myDataConsumer, throwableConsumer);
}
}
private void checkForFinish(SingleEmitter<MyData> emitter, List<Disposable> disposables) {
if (disposables1.stream().allMatch(Disposable::isDisposed)) {
if (encounteredError) {
emitter.onError(new Throwable());
} else {
emitter.onSuccess(myData);
}
}
}
}
}
This could be modified to return a Throwable from the original Singles if needed.
I have create one observable
Observable<Map<Integer, String>> observable = Observable.create(s -> {
try {
System.out.println("getMultipleCitiesName ==="+Thread.currentThread().getName());
List<String> cityIdsString = new ArrayList<>();
for (Integer cityId : cityIds) {
cityIdsString.add(cityId.toString());
}
MultiValueMap<String, String> formParams = new LinkedMultiValueMap<>();
formParams.put("cityIds[]", cityIdsString);
// Call the Location Client to call the API
Response<Map<Integer, String>> response = locationClient.getMultipleCitiesName(formParams);
s.onNext(response.getData());
} catch (Exception e) {
System.out.println("Inside Exception CITY NEW");
s.onError(e);
}
s.onComplete();
});
Now I want to add onErrorReturnItem() and retry() both.
So I tried it two ways
a) observable = observable.onErrorReturnItem(new HashMap<>()).retry(3);
b) observable = observable.retry(3).onErrorReturnItem(new HashMap<>());
b) is working (meaning retry and onErrorReturnItem) both are working
while in a) retry is not working?
Why is that
Order of operators matters, this is how you have to interpret the chains:
(a) you're saying something like Let take whatever comes from observable and if some error happens upstream then let have an observable that emit an empty HashMap(i.e calling onErrorReturnItem) and then retry on this observable(if some error is emitted). but no error is emitted by the observable onErrorReturnItem it just emit the empty HashMap followed by an onComplete signal. i.e whatever the error happens upstream is hidden by onErrorReturnItem and retry would never be signaled an onError to start retrying.
(b) you've saying completely the opposite, Let take whatever comes from observable and if some error happens upstream let retry, if retry can't get any valid item after 3 attempts, then signal onError downstream and onErrorReturnItem will return an empty HashMap.
Now it should be clear why (b) tend be what you're expecting to happens.
I am using rabbitmq and I want to make sure that if I have a connection problem in the client, the messages that I posted won't be lost. I simulate it with eclipse: I do system.exit the program of fetching after 100 messages. I posted 1000 messages. The second run I don't limit the number of messages and it returns me 840 messages with 3 times. Can you help me?
the code of the producer is:
public void run() {
String json =SimpleQueueServiceSample.getFromList();
while (!(json.equals(""))){
json =SimpleQueueServiceSample.getFromList();
try {
c.basicPublish("", "test",
MessageProperties.PERSISTENT_TEXT_PLAIN, json.getBytes());
} catch (IOException e) {
e.printStackTrace();
}
}
try {
c.waitForConfirmsOrDie();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
the code of the consumber is:
QueueingConsumer consumer = new QueueingConsumer(channel);
channel.basicConsume(QUEUE_NAME, true, consumer);
while (true) {
System.out.println(count++);
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String message = new String(delivery.getBody());
System.out.println(" [x] Received '" + message + "'");
}
So the challenge for your scenario is how you're handling the acknowledgements.
channel.basicConsume(QUEUE_NAME, true, consumer);
Is the problem. The second parameter of true is the auto-acknowledge field.
To fix that, use:
channel.basicConsume(QUEUE_NAME, false, consumer);
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
//...
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
}
It looks like you're using RabbitMQ's tutorials, and your code snippet is from part one. If you look at part two, they start talking about acknowledgements and setting up quality of service to provide round-robin dispatch.
It's worth pointing out that the basicConsume() and nextDelivery() combination rely upon a hidden queue that lives within the consumer. So when you call basicConsume() several messages are pulled down to the client to local storage.
The benefit at that approach is that it avoids additional network overhead from calling for each individual message. The problem is that it can put more messages within your local consumer than you wish and you may lose messages if the consumer drops before processing all of the messages in the local hidden queue.
If you truly want your consumers only working on one message a time so that nothing is lost, you probably want to look at the basicGet() method instead of the basicConsume().