I have been using Spring XD for a while for continuous ingestion of sensor data and it works perfectly.
The new requirement that I have is the ability to "replay" portions of that data. In my particular case it would be reading from MongoDB (with a certain query), generate a UDP packet with a certain filed of the entry and send it to a SocketAddress in a fixed interval of time.
The first attempt that I am implementing is through spring-batch job. The reader is simple since it is just querying MongoDB for the data, but I am concern about the UDP portion. It does not feel natural to use spring-batch for sending UDP packets, so I would like to know if anybody can suggest me an idea for implementing this.
Thanks
You could use a custom XD source with a MongoDB Inbound Channel Adapter piped to a custom sink using a UDP Outbound Channel Adapter.
Related
I am new in using omnet+ + , I am trying use INET framework to simulate queueing model M/M/1 on server-side based on FIFO discipline
Firstly, I found INET support external output queue such as FIFO a DopeTailqueue, I try to edit it and to make it work on input data not output, but I am not sure if I am in the right way.
My question, how can I simulate the M/M/1 queue in which layer and how can I calculate the arrival rate and processing rate?
As your task has nothing to do with actual protocols (just a generic queuing question), you would be better off with the queuing tutorial in the base OMNeT++ installation (omnetpp/samples/queuenet). You can quite easily assemble various queueing models.
I need to insert large bulk data to elasticsearch regulary via scala code. When googling, I found to use logstash for large insertion rate but logstash doesn't have any java libraries or Api to call so I tried to connect to it via http client. I don't know it is a good approach to send large data with http protocol or better to use other approaches for example using broker, queues, redis, etc.
I know last versions of logstash(6.X,7.x) enables uses of persistent queue so it can be another solution to use logstash queue but again through http or tcp protocol.
Also note that reliability is the first priority for me since data must not be lost and there should be a mechanism to return response in code so to handle success or failure.
I would appreciate any ideas.
Update
It seems using http is robust and has acknowledgement mechanism based on here but if taking this approach, what http client libs in scala is more appropriate as I need to send bulk data in sequence of key value format and handle response in none-blocking way?
It may sound overkill but introducing a buffering layer between scala code and logstash may prove helpful as you can get rid of heavy HTTP calls and rely on lightweight protocol transport.
Consider adding Kafka between your scala code and logstash for queuing of messages. Logstash can reliably process the messages from Kafka using TCP transport and bulk insert into ElasticSearch. On the other hand, you can put messages into Kafka from your scala code in build (batches) to make the whole pipeline work efficiently.
With that being said, if you don't have a volume in let say 10,000 msgs/sec then you can also consider fiddling around with logstash HTTP input plugin by tweaking threads and using multiple logstash processes. This is to reduce the complexity of adding another moving piece (Kafka) into your architecture.
We use Spring integration for TCP socket communication with the hardware.
The client would be sending a sequence number to uniquely identify a message.
My requirement is to store these sequence numbers part of the socket message and validate them for non repetitive sequence numbers.
I went thru IdempotentReceiver, sounds like what i wanted.
But I need a durable and faster mechanism to store it, before unexpected shutdown of service and use the in memory cache for retrieving the latest sequence number.
Thank you in advance.!
You can use PropertiesPersistingMetadataStore for idempotent receiver:
The PropertiesPersistingMetadataStore is backed by a properties file and a PropertiesPersister.
By default, it only persists the state when the application context is closed normally. It implements Flushable so you can persist the state at will, be invoking flush().
See more in its JavaDocs.
I have captured SNMP traps/informs from my network (a mix of V1 and V2c) and I wish to write a camel pipeline to replay them in order to test my trap processing engine. In order to do this, I must resend the traps with the source IP of the original sender (since that is part of the criteria for identification and correct processing of a trap).
I thought I would send the resulting UDP datagrams using Netty (although I'm open to writing a stand-alone component or using Mina or any other approach, but the use of the Camel SNMP component does not immediately seem appropriate). I have implemented similar functionality in Python, and I needed to write to a raw socket. I have looked through the netty component source in camel and was not able to see how I might use it unmodified to use a raw socket.
Does anyone out there have an example of using netty with raw sockets that they could share, bonus points if it includes some reference to camel or better still if it calls out a way I can do this with something more high-level than a raw socket (ie a regular UDP datagram with some kind of modifier to set the source IP).
Many thanks
I found Slick 3.0 introduced a new feature called streaming
http://slick.typesafe.com/doc/3.0.0-RC1/database.html#streaming
I'm not familiar with Akka. streaming seems a lazy or async value, but it is not very clear for me to understand why it is useful, and when will it be useful..
Does anyone have ideas about this?
So lets imagine the following use case:
A "slow" client wants to get a large dataset from the server. The client sends a request to the server which loads all the data from the database, stores it in memory and then passes it down to the client.
And here we're faced with problems: The client handles the data not so fast as we wanted => we can't release the memory => this may result in an out of memory error.
Reactive streams solve this problem by using backpressure. We can wrap Slick's publisher around the Akka source and then "feed" it to the client via Akka HTTP.
The thing is that this backpressure is propagated through TCP via Akka HTTP down to the publisher that represents the database query.
That means that we only read from the database as fast as the client can consume the data.
P.S This just a little aspect where reactive streams can be applied.
You can find more information here:
http://www.reactive-streams.org/
https://youtu.be/yyz7Keg1w9E
https://youtu.be/9S-4jMM1gqE