I have created a instance of mongodb on AWS EC2 Instance.
I have 2 Spring boot/microservices connected to this database. One is just inserting the data and other is fetching data. In my microservices, I dont have any delete operation and no such code is provided atany time so by mistake data getting deleted scenario will not occure.
**But some how mongodb database is getting reset/cleared. All database is getting deleted.**
I have checked the mongodb configuration and I haven't explicitly changed anything.
Mongodb is community edition.
On spring boot microservices, I am directly using the Mongorepository
JPA to insert or fetch the data.
I dont have any constraints in POJOs.
Can someone point out what might be the issue? Does mongodb has any
default setting for resetting the database?
Related
I have an environment on AWS with a RDS Postgresql9.6 and a Spring Boot v1.2.7RELEASE application running on a EC2 Instance. Now I want to fetch about 10.000 entries from a table of the Postgresql DB, which takes about 1 minute. If I do this locally It takes about a second to fetch the entities.
I would expect that the fetching would just take some more time than locally like 2 or 3 seconds.
Actually the request takes 1 minute.
To determine if the problem maybe is caused by a bad query I did
explain analyze SELECT * FROM view_name where uuid ='4e663553-4271-4d7d-8de9-d7b746787cc6' which tells me that the execution of the query itself just takes 300ms.
Therefore I thought the performance Issue comes from transmitting the data from the DB to the application. But I don't know how to evaluate this or even how to improve this.
To reproduce this I guess you need a AWS environment with a RDS and an application which just uses Hibernate to fetch from the RDS a table with approximately 10.000 entries.
EDIT 1
Persistence and DataSource Configuration.
We are using hibernate and have the the following configuration:
hibernate.default_batch_fetch_size=8
hibernate.jdbc.fetch_size=10
hibernate.jdbc.batch_size=8
hibernate.cache.use_query_cache=true
hibernate.cache.use_second_level_cache=true
hibernate.cache.region.factory_class=org.hibernate.cache.redis.SingletonRedisRegionFactory
hibernate.cache.use_structured_entries=true
hibernate.max_fetch_depth=10
hibernate.transaction.factory_class=org.hibernate.engine.transaction.internal.jdbc.JdbcTransactionFactory
javax.persistence.sharedCache.mode=ENABLE_SELECTIVE
I should also note that we use ElastiCache Redis with version 2.8.24.
I am trying to make a PoC for MongoDB to DocDB migration with DMS.
I've set up a MongoDB instance with some dummy data and an empty DocDB. Source and Target endpoints are also set in DMS and both of them are connecting successfully to my databases.
When I create a migration task in DMS everything seems to be working fine. All existing data is successfully replicated from the MongoDB instance to DocDB and the migration task state is at "Load complete, replication ongoing".
At this point I tried creating new entries in existing collections as well as creating new empty collections in MongoDB but nothing happens in DocDB. If I understand correctly the replication should be real time and anything I create should be replicated instantly?
Also there are no indication of errors or warnings whatsoever... I don't suppose its a connectivity issue to the databases since the initial data is being replicated.
Also the users I am using for the migration have admin privileges in both databases.
Does anyone have any suggestions?
#PPetkov - could you check the following?
1. Check if right privileges were assigned to the user in the MongoDB endpoint according to https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.MongoDB.html.
2. Check if replicate set to capture changes was appropriately configured in the MongoDB instance.
3. Once done, try to search for "]E:" or "]W:" in the CloudWatch logs to understand if there are any noticeable failures/warnings.
I tried to have spring batch meta data tables in Mongo database but its not working correctly. I referred and used below mentioned github project to configure JobRepository to store job data in Mongodb. This GitHub project is updated last 3 years ago and looks discontinued.
https://github.com/vfouzdar/springbatch-mongoDao
https://jbaruch.wordpress.com/2010/04/27/integrating-mongodb-with-spring-batch/
Currently my application uses in-memory tables for spring batch and functional part is done. But I want job data to be stored in Mongodb.
I have already used Mysql for spring batch job data but in current application don't want mysql.
If anybody has any other solution/link which can help me, please share.
One of our clients have a server running a MongoDB instance and we have to build an analytical application using the data stored in their MongoDB database which changes frequently.
Clients requirements are:
That we do not connect to their MongoDB instance directly or run another instance of MongoDB on their server but just somehow run our own MongoDB instance on our machine in our office using their MongoDB database directory with read only access remotely.
We've suggested deploying a REST application, getting a copy of their database dump but they did not want that. They just want us to run our own MongoDB intance which is hooked up with the MongoDB instance directory. Is this even possible ?
I've been searching for a solution for the past two days and we have to submit a solution by Monday. I really need some help.
I think this is normal request because analytical queries could cause too much load on the production server. It is pretty normal to separate production and analytical databases.
The easiest option is to use MongoDB replication. Set up MongoDB replica set with production database instance as primary and analytical database instance as secondary, also configure the analytical instance to never become primary.
If it is not possible to use replication - for example client doesn't want this, the servers could not connect directly to each other... - there is another option. You can read oplog from remote database and apply operations to your database instance. This is exactly the low level mechanism how replica set works, but you can do it manually too. For example MMS (Mongo Monitoring Sevice) Backup uses reading oplog for online backups of MongoDB.
Update: mongooplog could be the right tool for real-time application of replication oplog pulled from remote server on local server.
I don't think that running two databases that points to the same database files is possible or even recommended.
You could use mongorestore to restore from their data files directly, but this will only work if their mongod instance is not running (because mongorestore will need to lock the directory).
Another solution will be to do file system snapshots and then restore to your local database.
The downside to this backup/restore solutions is that your data will not be synced all the time.
Probably the best solution will be to use replica sets with hidden members.
You can create a replica set with just two members:
Primary - this will be the client server.
Secondary - hidden, with votes and priority set to 0. This will be your local instance.
Their server will always be primary (because hidden members cannot become primaries). Clients cannot see hidden members so for all intents and purposes your server will be read only.
Another upside to this is that the MongoDB replication will do all the "heavy" work of syncing the data between servers and your instance will always have the latest data.
I am trying to use the oplog monitoring class in casbah
https://github.com/mongodb/casbah/blob/master/casbah-core/src/main/scala/util/OpLog.scala
What i want to do is monitor the oplog entries at a production mongo db on
production.someserver.com
and get the entries and send them to the storage DB at
test.someotherserver.com
and replicate all the data that is in the production server to the test server. I cannot use replica sets to do this as i cannot redeploy now. I am trying to build a scala app to do this. Casbah the official scala driver for mongo as the above mentioned class which i m trying to instantiate using
val mongoColl = MongoConnection() ("test") ("test_data")
val oLog = new MongoOpLog(mongoColl)
But im not even able to instantiate it, getting an error that mongooplog is not found. Ive imported the necessary package. But even if im able to do this i have no clue on how to do what i want to do. can any one pls point me in a right direction on how to achieve this. I am pretty new to scala so a bit of detailed explanation or a link containing it would be helpful for me.
You need to have replication enabled on the server for the oplog to be created; as either a member of a replica set or in master mode for master/slave.
Otherwise, MongoDB does not waste CPU cycles and disk space maintaining an oplog. Please see the documentation on Replication for more info - http://www.mongodb.org/display/DOCS/Replication
You should really never be running any database with a single server in production, incidentally.