Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 1 year ago.
Improve this question
I need help for an archetecture issue.
I develope a cep system based on kafka technology with java.
CEP should have followed characteristics:
distributed (cluster)
scalable
fault-tolerance
CEP should make followed actions:
create events from different sources, which is actually are multi-partitioned kafka-topics (ETL-part)
analyze sequences of that events and if they are matched for a special patterns (scenario) put reaction-record to some store (analyze-part)
every X period query this store to do some communication with a client if it's the time (schedule-part)
During X period if a cancel-event appears, so I remove a reaction record from store.
I created that system using KafkaStreams library. But the archetecture as a result is not so good.
KafkaStreams use RocksDB in backend to store states. There are many problems with managing stores in cluster mode and having a consistent data. Also I cant to make sql-queries to them to resque from iterating every record in store to check if the time for reaction is heppen.
I'm not an architect and I only one who is busy in this task. I was adviced to look at KafkaStreams and Flink for create cep programm. But in fact are these technologies really fit?
There are no question for an ETL-part.
But how can I built an analyze-part and (it's more interesting) query-part? What tools can I use?
I'm grateful for any help and advices
[UPD]
About queries and stores:
We need to check if the time to send a communication is heppen. If it's true so communicate with a person: push-message, email or any other chanel.
select
...where event_time + wait_time < now
After that we need update that record in store to next message of this scenario (and make this algorithm until the person go to last message of scenario or does the cancel action)
Sequence of scenario A:
ev A -> ev B -> ev C -> ev D -----> start scenario -----> ev E or msg c was sent -----> cancel scenario
Messages for scenario A:
msg a (send after wait_time: 10 minutes)
msg b (send after wait_time: 1 day)
msg c (send after wait_time: 7 days) - last
update
... where user_id = xxx and scenario_id = A
If action was made in 2nd point, so we also need to update userStore (there are some information about users, including special counters; they are help to not spam the client and no sending messagies to him at night)
update
... where user_id = xxx
I wrote an engine for CEP with some rules, which I save in special store - scenarioStore.
Thus, there are a several stores:
initialStore (keep last event in scenario sequencies with message parameters, waiting the time to be sent) - ev D
scenarioStore (sequences of events by scenarios) - CEP rules
messageStore (texts and other properties of messages) - msg rules
userStore (information about users)
You can definitely do complex event processing CEP with Kafka Streams. There are even open-source libraries for that kafkastreams-cep.
Kafka Streams framework supports interactive-queries where you can query your state stores to retrieve required data. You can add a REST layer to make it queryable from REST API. Please, see a code example WordCountInteractiveQueriesExample.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 2 years ago.
Improve this question
I'm working on STM32F103RB Nucleo Board. I want to know how CAN messages are segregated in FIFO upon reception of data?. And what happens after FIFO is full(more than 3 messages)?.
When you configure a filter bank, you also specify the receive mailbox (you have 2 of them). Messages which are accepted by one filter bank goes into the associated mailbox.
FIFO (mailbox) overrun can trigger an interrupt if enabled. The behavior of the FIFO and the fate of the incoming messages are determined by the RFLM bit of the CAN->MCR register.
RFLM = 0 -> The last (3rd) message is overwritten (destroyed) by the new arriving messages. The first (oldest) 2 messages are preserved until you read them.
RFLM = 1 -> FIFO is locked. New arriving messages are discarded. The oldest 3 messages are preserved.
And what happens after FIFO is full(more than 3 messages)?
Then you are basically done for - you'll lose data upon Rx FIFO overflow, which is often unacceptable in CAN real-time systems. So in case your MCU is too busy to always meet the 3 message deadline, you would have to implement some ugly system with interrupts + ring buffers.
This is one reason why CAN controllers from somewhere around the late 90s/early 2000s started to use some 5 to 8 message rx buffers. BxCAN is apparently ancient, since it is worse than those 20+ years old controllers.
Hopefully you can DMA the messages, which is much prettier than the mentioned interrupt/ring buffer complexity. If that's not an option, then you should perhaps go for a modern CAN controller instead. Bascially any other CAN controller on the market has a larger rx FIFO than this one.
I'm trying to understand zookeeper's internal.
Suppose a 3-servers zookeeper cluster, the leader server send a proposal(say setdata: foo=1) to two followers and then crashed, but at least one follower record this proposal to its transaction log file. According "Zab paper" says, the other two server can still form a valid quorum and elect a new leader. And the new leader can still propose and commit this proposal(setdata: foo=1).
My question is in this situation, the client think this request is not completed(because of the leader crash and not respond or the client timeout), but in fact it is still success in the zookeeper cluster. Is this an inconsistent?
In fact this is an inconsistent, but it's not a problem.
In zookeeper programmer guide,there is a line:
If a client gets a successful return code, the update will have been applied. On some failures (communication errors, timeouts, etc) the client will not know if the update has applied or not. We take steps to minimize the failures, but the only guarantee is only present with successful return codes. (This is called the monotonicity condition in Paxos.)
This means you know your update succeed when you gets a successful return code, but when you can't know whether it succeed or failed when you don't receive the return code.
But this is not a problem, when your update failed because of leader crash, you can just retry the update operation. This time your update will failed because the the version you specified is behind the actual version number and you will be notified. Then you can call get method to retrive the data to see whether the data equals you specified value.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
We are working on web application, that has a feature to generate metrics based on how user is using the app. We are exploring to use Storm to process the user events and generate metrics.
The high level approach we are planning :
On client side (Browser), a Java script component to capture user events and post the event to server, and event message will be posted to RabbitMQ.
Storm spout consumes message from RabbitMQ.
Storm bolt process the message and computes metrics.
Finally metrics get saved to MongoDB
Question :
Bolt has to accumulate event's metrics before saving to MongoDB because of two reasons, need to avoid IO load on MongoDB and metrics logic has dependency on multiple events. So we need to have intermittent persistence for Bolt, and not impacting performance.
How can we add temporary persistence within storm topology while we calculate statistics on the data pulled from RabbitMQ, and then save metrics to permanent persistence MongoDB, only on some interval or some other logical trigger.
Please clarify if I don't fully answer your question but the general gist of your query seems to echo the theme: how can we persist within our storm topology while we calculate statistics on the data pulled from RabbitMQ?
Luckily for you, Storm has already considered this question and developed Storm-Trident, which performs real time aggregation on incoming tuples and allows the topology to persist the aggregated state for DRPC queries and for situations requiring high availability and persistence.
For example, in your particular scenario, you would have this kind of TridentTopology:
TridentTopology topology = new TridentTopology();
TridentState metricsState = topology.newSpout(new RabbitMQConsumer())
.each(new Fields("rawData"), new ComputeMetricsFunction(), new Fields("output"))
.groupBy(new Fields("output"))
.persistentAggregate(new MemoryMapState.Factory(), new AggregatorOfYourChoice(), new Fields("aggregationResult"))
Note: the code isn't 100% accurate but should be considered more as pseudo-code. See Nathan's word count example for code specific implementation (https://github.com/nathanmarz/storm/wiki/Trident-tutorial).
I need to choose a new Queue broker for my new project.
This time I need a scalable queue that supports pub/sub, and keeping message ordering is a must.
I read Alexis comment: He writes:
"Indeed, we think RabbitMQ provides stronger ordering than Kafka"
I read the message ordering section in rabbitmq docs:
"Messages can be returned to the queue using AMQP methods that feature
a requeue
parameter (basic.recover, basic.reject and basic.nack), or due to a channel
closing while holding unacknowledged messages...With release 2.7.0 and later
it is still possible for individual consumers to observe messages out of
order if the queue has multiple subscribers. This is due to the actions of
other subscribers who may requeue messages. From the perspective of the queue
the messages are always held in the publication order."
If I need to handle messages by their order, I can only use rabbitMQ with an exclusive queue to each consumer?
Is RabbitMQ still considered a good solution for ordered message queuing?
Well, let's take a closer look at the scenario you are describing above. I think it's important to paste the documentation immediately prior to the snippet in your question to provide context:
Section 4.7 of the AMQP 0-9-1 core specification explains the
conditions under which ordering is guaranteed: messages published in
one channel, passing through one exchange and one queue and one
outgoing channel will be received in the same order that they were
sent. RabbitMQ offers stronger guarantees since release 2.7.0.
Messages can be returned to the queue using AMQP methods that feature
a requeue parameter (basic.recover, basic.reject and basic.nack), or
due to a channel closing while holding unacknowledged messages. Any of
these scenarios caused messages to be requeued at the back of the
queue for RabbitMQ releases earlier than 2.7.0. From RabbitMQ release
2.7.0, messages are always held in the queue in publication order, even in the presence of requeueing or channel closure. (emphasis added)
So, it is clear that RabbitMQ, from 2.7.0 onward, is making a rather drastic improvement over the original AMQP specification with regard to message ordering.
With multiple (parallel) consumers, order of processing cannot be guaranteed.
The third paragraph (pasted in the question) goes on to give a disclaimer, which I will paraphrase: "if you have multiple processors in the queue, there is no longer a guarantee that messages will be processed in order." All they are saying here is that RabbitMQ cannot defy the laws of mathematics.
Consider a line of customers at a bank. This particular bank prides itself on helping customers in the order they came into the bank. Customers line up in a queue, and are served by the next of 3 available tellers.
This morning, it so happened that all three tellers became available at the same time, and the next 3 customers approached. Suddenly, the first of the three tellers became violently ill, and could not finish serving the first customer in the line. By the time this happened, teller 2 had finished with customer 2 and teller 3 had already begun to serve customer 3.
Now, one of two things can happen. (1) The first customer in line can go back to the head of the line or (2) the first customer can pre-empt the third customer, causing that teller to stop working on the third customer and start working on the first. This type of pre-emption logic is not supported by RabbitMQ, nor any other message broker that I'm aware of. In either case, the first customer actually does not end up getting helped first - the second customer does, being lucky enough to get a good, fast teller off the bat. The only way to guarantee customers are helped in order is to have one teller helping customers one at a time, which will cause major customer service issues for the bank.
It is not possible to ensure that messages get handled in order in every possible case, given that you have multiple consumers. It doesn't matter if you have multiple queues, multiple exclusive consumers, different brokers, etc. - there is no way to guarantee a priori that messages are answered in order with multiple consumers. But RabbitMQ will make a best-effort.
Message ordering is preserved in Kafka, but only within partitions rather than globally. If your data need both global ordering and partitions, this does make things difficult. However, if you just need to make sure that all of the same events for the same user, etc... end up in the same partition so that they are properly ordered, you may do so. The producer is in charge of the partition that they write to, so if you are able to logically partition your data this may be preferable.
I think there are two things in this question which are not similar, consumption order and processing order.
Message Queues can -to a degree- give you a guarantee that messages will get consumed in order, they can't, however, give you any guarantees on the order of their processing.
The main difference here is that there are some aspects of message processing which cannot be determined at consumption time, for example:
As mentioned a consumer can fail while processing, here the message's consumption order was correct, however, the consumer failed to process it correctly, which will make it go back to the queue. At this point the consumption order is intact, but the processing order is not.
If by "processing" we mean that the message is now discarded and finished processing completely, then consider the case when your processing time is not linear, in other words processing one message takes longer than the other. For example, if message 3 takes longer to process than usual, then messages 4 and 5 might get consumed and finish processing before message 3 does.
So even if you managed to get the message back to the front of the queue (which by the way violates the consumption order) you still cannot guarantee they will also be processed in order.
If you want to process the messages in order:
Have only 1 consumer instance at all times, or a main consumer and several stand-by consumers.
Or don't use a messaging queue and do the processing in a synchronous blocking method, which might sound bad but in many cases and business requirements it is completely valid and sometimes even mission critical.
There are proper ways to guarantuee the order of messages within RabbitMQ subscriptions.
If you use multiple consumers, they will process the message using a shared ExecutorService. See also ConnectionFactory.setSharedExecutor(...). You could set a Executors.newSingleThreadExecutor().
If you use one Consumer with a single queue, you can bind this queue using multiple bindingKeys (they may have wildcards). The messages will be placed into the queue in the same order that they were received by the message broker.
For example you have a single publisher that publishes messages where the order is important:
try (Connection connection2 = factory.newConnection();
Channel channel2 = connection.createChannel()) {
// publish messages alternating to two different topics
for (int i = 0; i < messageCount; i++) {
final String routingKey = i % 2 == 0 ? routingEven : routingOdd;
channel2.basicPublish(exchange, routingKey, null, ("Hello" + i).getBytes(UTF_8));
}
}
You now might want to receive messages from both topics in a queue in the same order that they were published:
// declare a queue for the consumer
final String queueName = channel.queueDeclare().getQueue();
// we bind to queue with the two different routingKeys
final String routingEven = "even";
final String routingOdd = "odd";
channel.queueBind(queueName, exchange, routingEven);
channel.queueBind(queueName, exchange, routingOdd);
channel.basicConsume(queueName, true, new DefaultConsumer(channel) { ... });
The Consumer will now receive the messages in the order that they were published, regardless of the fact that you used different topics.
There are some good 5-Minute Tutorials in the RabbitMQ documentation that might be helpful:
https://www.rabbitmq.com/tutorials/tutorial-five-java.html