Since this feature is relatively new (mongo 3.6) I found very few java examples.
My questions:
1. What is the best practices for watching change streams?
2. Does it have to be a blocking call to watch the stream? (This means a thread per collection which is less desired)
This is the example I encountered:
http://mongodb.github.io/mongo-java-driver/3.6/driver/tutorials/change-streams/
The blocking call is:
collection.watch().forEach(printBlock);
Thanks,
Rotem.
Change streams make a lot more sense when you look at them in the context of reactive streams. It took me a while to realize this concept has a much broader existence than just the MongoDB driver.
I recommend reviewing the article above and then looking at the example provided here. The two links helped clear things up, and provided insight on how to write code leveraging the reactive streams Mongo driver, which is non-blocking.
Use mongo reactive driver so that it will be non-blocking. And we used this approach and running in production for last one month, no issue.
Related
I have started exploring Quarkus with MongoDB client. I came across a lot of boiler plate code in the documentation. (esp. BSON)
refer: https://quarkus.io/guides/mongo-guide
Can't we make use of the annotations to generate the code.
Is this intentional to avoid the Reflection overhead.
We have a lot of good things coming up for the MongoDB support in our next release (0.24.0 planned for Wednesday): it's called MongoDB with Panache.
You can see what it does in the documentation of the master branch here: https://github.com/quarkusio/quarkus/blob/master/docs/src/main/asciidoc/mongodb-panache-guide.adoc .
So I would say, wait for 0.24.0 and give a try to this new feature (or you can build master and give it a try already), feedback is very welcome.
If you have implemented publish-subscribe pattern using Curator/Zookeeper could you please share your experience? We're currently doing proof-of-concept and your feedback would help big time!
Found this Pub-Sub Example link which shows an example using Curator. The note in that example says "it is not meant for production". Are they saying it's not a good pattern to use with Curator and they're only using it as an example to show Curator features? Or are they saying the pattern works well with Curator but the example is not something you would implement in Production?
Would also help to know pros and cons besides the well-known 1MB limitation on a znode size.
Your help will be greatly appreciated!
Came across this Curator Tech Note link which talks about using Zookeeper as a Queue source. Looks like most (if not all) bullets on that page would apply to using ZooKeeper as a publish-subscribe source as well.
So the verdict is ZooKeeper would make a bad publish-subscribe model.
I'm in the process of developing my next app, and I'm really interested in using polyglot persistence. I like the idea of being able to query different data structures for different services. I'm essentially wanting to sync MongoDB, Neo4j/Titan, SQL, and maybe Cassandra/Hbase.
Currently, I'm wrapping everything in a try/catch block and rolling them all back if one fails. However, this is taxing my write times. I've also looked into AMQP systems like Kafka or ZeroMQ, but these seem more big data centric, whereas my app is still small and I want to keep it efficient.
Has anyone had experience with this? Is a MQ a good idea for a small app or am I prematurely optimizing?
Thanks
I know quite a bit about ZeroMQ, but not a lot about the database servers you mention.
First, you're a bit confused about ZeroMQ. Although it is derived from experience with AMQP, it uses the ZMTP wire protocol. That was custom designed during ZeroMQ development [but other applications do now use it].
ZeroMQ is a small and very fast MQ library that is symmetrical for all nodes; it is very good for small apps. The problem here is that you need something on the other systems that talks ZMTP, whether it's ZeroMQ or a bridge. If you intend making plugins or the like for the other systems then fine.
I presume though, that you are using JMS to talk to the other systems without intending to develop add-ons for them. In which case you're probably stuck with JMS. Kafka is a new one that I haven't caught up with, but RabbitMQ is a good, fast, and small, broker. FWIW. There are a great many broker comparisons out there for you to find. Many are dodgy in the sense that one small tweak of a setting can affect the performance greatly and are not necessarily comparing apples with apples. If want to compare broker performance in your environment, there isn't much of a shortcut to doing it yourself.
One thing that is confusing me is how you expect a broker to help your rollback performance. You'll still need to do the rollback in essentially the same manner, albeit asyncronously via the broker.
I work at CloudBoost.io (https://www.cloudboost.io) and we build a layer that sits on top of databases and give you the power of Polyglot Persistence persistence. We integrate MongoDB, ElasticSearch, Redis, Cassandra, and Neo4j and give you the one single APIwhere you can query / store your data. We automatically shard your data into various databases based on your query / storage request patterns.
Let me know if this helps. :)
For syncing data from MongoDB to Neo4j there is now the Neo4j Doc Manager project. It works by monitoring MongoDB for operations and converts the document operation into a property graph model and immediately writes to Neo4j.
I'm using plain old Java 1.6, and am interested in both these libraries.
Having read the documentation, I'm not sure of the differences, if any. Can anyone explain it, or point me at some relevant info? Thanks in advance!
RxJava does a lot more than the ListenableFutures. I'm not familiar with ListenableFutures, but from the docs it seems that it is simply Futures with callbacks and a few simple methods to compose them. On the other hand, RxJava (or the original Reactive Extensions for .NET, which are a huge inspiration to RxJava) models also sequences of values over time - data in motion, basically anything from a stream of mouse moves to a stream of network packets or database results. It also provides various scheduling strategies and many combinators to compose the streams. A good introduction to RxJava and even comparison to the futures is at the wiki page of the project. You can also have a look at Intro to Rx for an introduction to the general concept.
At a couple of places there is state that akka is somehow "real-time". E.g.:
http://doc.akka.io/docs/akka/2.0/intro/what-is-akka.html
Unfortunately I was not able to find a deeper explanation in which way akka is "real-time". So this is the question:
In which way is akka real-time?
I assume akka is not really a real-time computing system in the sense of the following definition, isn't it?: https://en.wikipedia.org/wiki/Real-time_computing
No language built on the JVM can be real-time in the sense that it's guaranteed to react within a certain amount of time unless it is using a JVM that supports real-time extensions (and takes advantage of them). It just isn't technically possible--and Akka is no exception.
However, Akka does provide support for running things quickly and with pretty good timing compared to what is possible. And in the docs, the other definitions of real-time (meaning online, while-running, with-good-average-latency, fast-enough-for-you-not-to-notice-the-delay, etc.) may be used on occasion.
Since akka is a message driven system, the use of real-time relates to one of the definition of the wikipedia article you mention in the domain of data transfer, media processing and enterprise systems, the term is used to mean 'without perceivable delay'.
"real time" here equates to "going with the flow": events/messages are efficiently processed/consumed as they are produced (in opposition to "batch processing").
Akka can be a foundation for a soft real-time system, but not for a hard one, because of the limitations of the JVM. If you scroll a bit down in the Wikipedia article, you will find the section "Criteria for real-time computing", and there is a nice explanation about the different "real-timeness" criteria.
systems that are subject to a "real-time constraint"— e.g. operational
deadlines from event to system response.
en.wikipedia.org/wiki/Real-time_computing
The akka guys might be reffering to features like futures that allow you to add a time constraint on expectations from a computation.
Also the clustering model of akka may be used to mean an online system which is real-time(Abstracted so as to look like its running locally).
My take is that the Akka platform can support a form of real-time constraint by delivering responsive applications through the use of (I'm quoting here):
Asynchronous, non-blocking and highly performant event-driven programming model
Fault tolerance through supervisor hierarchies with “let-it-crash” semantics
Definition of time-out policies in the response delivery
As already said, all these features combined provides a platform with a form of response time guarantee, especially compared to mainstream applications and tools available nowadays on the JVM.
It's still arguable to claim that Akka could be strictly defined as a real-time computing system, as per wikipedia's definition.
For such claims to be proven, you would better refer to the Akka team itself.