I want to synchronize data among a set of REST API servers(Spring Boot based API cluster) periodically. Any instance in the cluster should be able to broadcast new information to all others.
I don't want to use a DB here. I am trying to find a lightweight library that can be used inside the API for this purpose. Is it possible to use Atomoix/Hazelcast/ZooKeeper for this purpose? If so, it will be really helpful if someone can post a sample code - if possible.
My thanks in advance.
In Hazelcast you can do it through WAN replication.
It is an enterprise feature you have to buy a license.
Hazelcast can be used for this use-case. Each of the REST instances will create an embedded Hazelcast member within its JVM. Hazelcast members then discover each other and form the cluster. Your REST apps will use the IMap or ReplicatedMap service - a distributed key-value store (IMap can store more data, ReplicatedMap is faster). Once you write a data to the IMap all other instances see it right away.
See the code sample here: https://docs.hazelcast.com/hazelcast/latest/getting-started/get-started-java.html#complete-code-samples
This feature and the Spring integration are open-source.
Related
I need a solution preferably something inbuilt (rather than creating my own application) which would help management search through multiple/all topics in Kafka. We are using Confluent Platform. Basically user should be able to search a keyword in a UI and it should search current log of multiple/all Kafka topics and return the data. All the topics in our environment use json to communicate.
So this search would enable us to track flow for example, multiple microservices send data from one system to another system and this flow can be tracked via a correlation id which is present in all the jsons. So if someone searchers this correlation id he should be able to see the messages involved in the flow. This search would have more use cases later on.
We need a solution which would have minimal coding involved. We would prefer to use a UI like Kibana.
On basic reading I suspect below solutions but not really sure as I am new to Confluent (used open-source Apache Kafka earlier):
Sol 1: use ksqldb. (need more help on how to use it)
Sol 2: Stream all topics data using Kafka Connect to Elastic Search by using inbuilt plugin and use Kibana on top of Elastic.
Kindly help to find the best case alternative.
You could use Elastic, sure.
You could also use Splunk, though.
There is also the pdk tool offered by Pilosa that creates a distributed index over Kafka events. (no affiliation)
Another option would be distributed tracing using interceptors between clients, not "on all topics", which sounds like what you actually need
From the MongoLab's documentation, they recommend:
MongoLab databases can be accessed by your application code in two ways.
The first method - the one we strongly recommend - is to connect using one of the MongoDB drivers (as described above). You do not need
to use our API if you use the driver.
The second method, which you should use only if you cannot connect via one of the MongoDB drivers, is via MongoLab’s RESTful data API.
Why do they recommend using the driver rather than their REST API? One reason I can think of is portability across different MongoDB providers. Are there any other reasons? Wouldn't it be more beneficial for MongoLab to "vendor lock-in" customers with their API?
The points that #WiredPrairie and #Stennie brought up around security are correct. *When you use our REST API, you expose your API key to the client. Currently, anyone with the API key can modify your database. As a result, we only recommend using the REST API with public data, e.g. all the locations for taco trucks in the country.
By writing your own app tier, you can keep credentials to your database from being exposed to the client.
If you have any more questions, email us at support#mongolab.com. Happy to help!
-Chris#MongoLab
p.s. thanks #WiredPrairie and #Stennie
I'm pretty new with an Ember so for the start I have a noob question - is it possible to use Local Storage and REST adapter at the same time?
For example, if I want to do a login via API, if login is success the server will return an API key which is used for later communication with a service. Is it possible to store that information locally on the client and to retrieve it when necessary but also, for other models, to use REST adapter?
If this is not a good way to handle such case, which one would you propose and is there any kind of example which would me lead me in the right direction?
Thanks to the people from #emberjs, I found out that there is a wonderful ember-auth authentication framework for the Ember.js which does what I need.
We would like to use OrientDB Graph in an Azure environment. Does anybody has experience using it? We also would like to know if high availability from OrientDB is required under Azure cloud? Azure already offers high availability for Azure storage, Azure Drive and SQL. I understand that they have replications and load balancing built in.
This is super important because we prefer not to get into the business of replications and infrastructure management.
Thanks
So you can spin up 2 or more machines and install OrientDB on them, then configure them together as a distributed cluster. However I haven't been able to find any way that is simpler, easier to do. I am interested in this topic too.
Azure does have features such as geo-replication, which is protects your data against a major data-center incident but doesn't provide any performance benefit and will not make it highly available.
Although pretty reliable, occasionally Microsoft will reboot servers for updates, so to protect against downtime you can use affinity groups so that, of your 2 or more servers, one will always be online. This however does need to be used in conjunction with database replication and ideally load balancing.
It's also worth noting that OrientDB recommends clusters have an odd number of servers as this can prevent conflicts when synchronising data after a communication issue between the servers.
I am using it in amazon and I had to create a java project to monitor http requests inserts and queries. The queries are very fast but takes longer inserting data .
I recommend this type of graph database mode to decrease the time of the queries. Also if you have empty fields OrientDB manages very well compared to other databases .
If you need help with the java project can response to this post and I´ll help u.
I hope it helps. Good luck.
I'd like to be able to have an admin app and a client app for my project. Ideally, I'd like to be able to have a shared MongoDB collection. How would I be able to accomplish this?
I tried creating collections with the same name in two different apps, but found that Meteor will keep the data separate. Any idea what I can do? Thanks.
export MONGO_URL=mongodb://localhost:3002/meteor
Then run meteor app, it will change the default database meteor uses. So share databases or collections won't be a problem!
For administrative reason, I would use a individual MongoDB server managed by myself other than using meteor's internal MongoDB.
A reasonable question and probably worth a discussion in excess of this answer:
The MongoDB connection is handled by the Meteor application process itself and this is - as far as I read and understood - part of Meteors philosophy targeting an approach that might be described like: One data source serves one application belonging to it but many clients subscribing to it.
This in mind, combining "admin" and "client" clients in one application (i.e. your Meteor app) is probably the preferred way.
From a server administrative view, however, connections are handled by Meteor in that way that there is always the default local data source which resides in your project directory (.meteor/local/db, try meteor mongo --url to obtain the mongo connection string while the meteor application process is running). But nevertheless one may specify an optional data source string for deployment purposes like described in these deployment instructions.
So you would need to choose a somewhat creepy way of "local development deployment" for your intended setup to get working. Or you go and hack the sources and... no, forget it. You probably want your application and clients to take advantage of e.g. realtime UI updates (publish) and that is why the Meteor application is tied to an "application data source" and vice-versa by now. When connecting from another app, events that trigger changes in the model would not be transported across those applications. The mongoDB instance itself of course isn't aware of that.
I'm sure the core team won't expose the data source connection to a configuration section for considered reasons unless they extend their architecture with some kind of module concept which provides a common service layer of core Model/Collections abstraction across Meteor instances - at least supporting awareness of publish/subscribe events.
Try this DDP test I hacked together for a way to bridge two apps (server A and B).
Both servers can manipulate data, but data is only stored in one collection on server A.
See this link as well