Suppose you have two agent types:
Agent Type 1 with a population of 10
Agent Type 2 with a population of 1
Suppose Type 2 has a statechart with two states as follows:
Agent Type 2 statechart
If all 10 agents of Type 1 send the same message simultaneously or at least with intervals smaller than the timeout transition shown in the image, what happens to the messages received while the the agent of Type 2 is in the state "evaluateLenderDecision"? Are the messages discarded or queued until the state "waitingForLender" is reached again?
First I suggest you watch this youtube video I made that explains how messages are sent. https://www.youtube.com/watch?v=Fe2U8IAhlHM
The messages using send or deliver are received in the connections object where the message is redirected to the different state charts that you define there.
In your case, you should maybe generate a queue yourself with all the messages that have been received (using a collection maybe)
If your messages are sent at the same time, 9 of your 10 agents will have their message discarded from your statechart point of view since there will be no statechart waiting for a message after the first one is received, but not from your connections point of view... All messages are received effectively.
Related
I have a set of requirements as below:
Message 'T' arrives, must wait for 5 seconds for corresponding message in 'A' to arrive (with same key). If it comes within 5 seconds, then send joined values and send downstream. If it does not come within 5 seconds, send only 'T' message downstream.
Message 'A' arrives, must wait for 5 seconds for corresponding message in 'T' to arrive (with same key). If it comes within 5 seconds, then send joined values and send downstream. If it does not come within 5 seconds, send only 'A' message downstream.
My current thinking was to do a KStream-KStream Sliding Window OUTER join. However, that does not wait for 5 seconds before sending the (T, null) or (null, T) message downstream (that is done instantly).
I need to wait for a timeout to happen, and if a join did not occur, then send the unjoined message through.
I've attached a diagram to help make sense of the cases. I am trying to use DSL as much as possible.
Any help appreciated.
Okay I found a fairly hacky solution that i'm still evaluating, but will work for this scenario.
I can simply groupByKey at the end and then suppress until window expires, with an unbounded buffer.
I am working on a project wherein I have multiple push sockets( created dynamically ) connected to a single pull socket. Here, each push socket sends a unique kind of data to the pull socket and the data size for each messages is less than 1000Bytes and each push socket sends upto 20,000 to 30,0000 such messages. During the peak load the number of push sockets can scale upto 100. So here, there are 100 push sockets connecting to one PULL receiver and each push socket sending 20-30K of messages with size almost 1000Bytes.
Now, On the PULL side receiver I process these each unique packets and put it into the respective queue, For Ex: If I have 50 PUSH sockets sending in data than on my receiver side there will be 50 queues processing the data. This is required as each queue holds unique set of data which needs to be processed uniquely. Here, I cannot have multiple PULL receivers as the data would then be routed in a round-robin fashion and one set of unique data might go into another one which I don't need.
Here, the problem I am facing is sometimes I see I don't receive all the data sent by the PUSH sockets. I believe there is some data loss that I happening. In order to control this I have setup HWM on both the sides. I tried the below combinations:
1) SNDHWM set to 100 and RCVHWM set to 100
2) SNDHWM set to 10 and RCVHWM set to 10
3) SNDHWM set to 10 and RCVHWM set to 100
Apart from the above configs I have also set the context threads on sender side to be 3 and on the receiver side to be 1. Here, on the sender side there is only one context created from which multiple push sockets are created dynamically as in when the data receives.
But, in all these combinations I find still some packets are missing.
Can someone guide me as in which parameters I need to configure in ZMQ so that I don't end up in packet loss and all the packets I receive on my receiver side. Also, I somehow believe my receiver side processing might be slow due to which this could have happened.
Please pen your thoughts here and guide me if possible.
In my project, I am using a statechart inside a population of agents named 'vehicle'. There are 5 vehicles. All vehicles have same speed. There is one more agent named 'sender'. Now what happens in my project is, that sender sends different messages to different vehicles. i.e. messages are m1,m2,m3,m4,m5 which are sent to vehicles having id v1,v2,v3,v4,v5. Messages are sent from sender, each time when a cyclic event triggers in the sender.
Lets take an example, when the event triggers in sender, it sends a message m1 to v1. The statechart receives the message m1 and uses it till the finishpoint of statechart. Similarly after a certain interval message m2 is sent to v2 and so on. Each message received is assigned to a variable named 'messageVal' in vehicle agent.
Click here to view statechart.
Currently it is working fine.
Here is the problem that occurs when I assign different speeds to vehicles.
The statechart for vehicle v1 when is still processing in the middle, while the event in the sender triggers again and sends another message m2 to vehicle v2. In this way, the variable value of 'messageVal' variable for vehicle v1, which should have been m1, becomes m2 now. and so on. Hence, the Results i receive for each vehicle, at the end, are incorrect.
Here is a sample Result:
vehicle=v1, message=m2
vehicle=v2, message=m2
I don't have any problem with the logic behind the event. All I need to know is, that does anyone know any method to allow multiple vehicles that have different speeds, to use statechart simultaneously without the values of ‘messageVal’ variable being altered till statechart's finishpoint?
If there is any confusion in understanding my question, I can further elaborate it. Thankyou.
Any idea how to make kafka-to-kafka mirroring but with a sampling (for example only 10% of the messages)?
You could use MirrorMakerMessageHandler (which is configured by message.handler parameter):
https://github.com/apache/kafka/blob/1.0/core/src/main/scala/kafka/tools/MirrorMaker.scala#L430
The handler itself would need to make a decision whether to forward a message. A simple implementation would be just a counter of messages received, and forwarding if 0 == counter % 10.
However this handler is invoked for every message received, so it means that you'd be receiving all of messages & throwing away 90% of them.
The alternative is to modify main loop, where the mirror maker consumer receives the message, and forwards it to producers (that send the message to mirror cluster) is here
https://github.com/apache/kafka/blob/1.0/core/src/main/scala/kafka/tools/MirrorMaker.scala#L428
You would need to modify the consumer part to either-or:
forward only N-th (10th) message/offset
seek to only N-th message in log
I prefer the former idea, as in case of multiple MM instances in the same consumer group, you would still get reasonable behaviour. Second choice would demand more work from you to handle reassignments.
Also, telling which message is from 10% is non-trivial, I just assumed that it's every 10th message received.
[org.jgroups.protocols.pbcast.NAKACK] (requester=, local_addr=) message ::port not found in retransmission table of :port:
(size=xxxx, missing=x, highest stability=xxxxx)]
NAKACK (or its newer cousin, NAKACK2) provide reliable transmission of messages to the cluster. To do this, every messages gets a sequence number (seqno) and receivers deliver the message to the application in seqno order.
Every cluster member has a table of all other members and their messages (conceptually a list). When member P sends messages P21, P22 and P23, a receiver R first looks up the message list for R, then adds P21-P23 to the list.
However, in your case, the list for R was not found. This means that R was not a cluster member (anymore).
For example, if we have cluster {P,Q,R,T}, and member R leaves or is excluded because it was suspected (e.g. we didn't receive a heartbeat for a period of time), then messages P21-23 will be dropped by any receiver.
This is because JGroups only allows cluster members to send and receive messages.
How can a member get excluded?
This is likely done by on of the failure detection protocols (e.g. FD_ALL or FD).
Another possibility is that your thread pools were clogged and failure detection heartbeat messages were dropped, leading to false suspicions.
Also, long GC pauses can cause this.
Fixes:
Increase the timeouts in FD_ALL or FD. The timeout should be longer than the longest GC cycle. Note that it will now take longer to detect hung members.
Size your thread pools, e.g. make sure that the max number of threads are big and the queue is disabled.
Note that false suspicions can happen, but MERGE3 should rememdy a split cluster later on.