I am using spring batch and and my database is Couchbase. Is there any way to read documents from Couchbase in batch or in bulk mode?
Have a look at the Spring batch extensions project
https://github.com/spring-projects/spring-batch-extensions/pull/5/commits
Related
I need to read my mongo DB table data periodically and publish it into a Kafka topic using spring boot. I have created a collection in Mongo DB and inserted a few records in Mongo DB. Further, I want to read the data from Mongo DB periodically and need to publish those table data in Kafka's topic using spring boot. I'm very new to spring batch scheduler. Can you please suggest me an idea to achieve this?
Thanks in advance.
What you are talking about is more relevant to Spring Integration: https://spring.io/projects/spring-integration#overview
So, you configure a MongoDbMessageSource with a Poller to read collection periodically.
And then you have service-activator based on the KafkaProducerMessageHandler to damp data into a Kafka topic.
See more in docs:
https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/mongodb.html#mongodb
https://docs.spring.io/spring-integration/docs/5.4.0-M3/reference/html/kafka.html#kafka
Not sure though how to do that with Spring Batch...
How to use JPA-Reader to read from database using spring batch with spring boot? Also need to insert records in bulk using JPA-Writer? How to do it using Spring batch? I am new to spring batch.
In my spring batch project i am reading data from db and write data to db, when i checked the dynatrace after job completed, i am not able to see the db select and insert queries in dynatrace.
Is any configuration need to do in spring batch to get these queries in dynatrace ?
Thanks!
We are using Spring Data MongoDB to operate mongodb. I was using Spring Data JPA to manage some relationship database. The SQL can be printed into the log file (log4j or logback). But for the mongoDB operation, there is no log output. Is there any way can let our debug what document (json object) has been insert or updete into the MongoDB base on Spring Data MongoDB module?
Spring data mongodb contains the LoggingEventListener.
Unluckily, the EventListeners do not support all batch operations. If you have batch operations, you may need to add your own costum logging.
I've used Spring Batch with MySQL before and the availability of Spring Batch Admin makes the starting, stopping, restarting of Jobs a lot easier. But my current company is considering to move to MongoDB from Derby database for obvious NoSQL DB benefits and also wants to move their existing messy batch application solutions to use Spring Batch framework. They also would like to use Spring Batch's Admin for managing the jobs.
Question:
What are the tradeoffs that we will have to make for using Spring Batch with MongoDB then Spring Batch with MySQL?
After doing a bit of research I've gathered the following trade-offs for using MongoDB with Spring batch
Since MongoDB does not support transactions, Spring Batch Admin will not work since the Admin requires the meta-data schema which is not available for MongoDB.
We will not be able to Stop, Start & restart jobs.
If a Step's writer tries committing 20 documents and commit for 1 document fails, the other 19 documents will not be rolled back automatically and will have to be managed by the system.
Can you please tell me if I am right with the above and if there are any other that I have not mentioned already.