I have a Spring Batch application with two databases: one SQL DB for the Spring Batch meta data, and another which is a MongoDB where all the business data is stored. The relation DB still uses DataSourceTransactionManager.
However I dont think the Mongo writes are done within an active transaction with rollbacks. Here is the excerpt from the official Spring Batch documentation on MongoItemWriter:
A ItemWriter implementation that writes to a MongoDB store using an implementation of Spring Data's MongoOperations. Since MongoDB is not a transactional store, a best effort is made to persist written data at the last moment, yet still honor job status contracts. No attempt to roll back is made if an error occurs during writing.
However this is not the case any more; MongoDB introduced ACID transactions in version 4.
How do I go about adding transactions to my writes? I could use #Transactional on my service methods when I use ItemWriterAdapter. But still dont know what to do with MongoItemWriter... What is the right configuration here? Thank you.
I have a Spring Batch application with two databases: one SQL DB for the Spring Batch meta data, and another which is a MongoDB where all the business data is stored.
I invite you to take a look at the following posts to understand the implications of this design choice:
How to java-configure separate datasources for spring batch data and business data? Should I even do it?
How does Spring Batch transaction management work?
In your case, you have a distributed transaction across two data sources:
SQL datasource for the job repository, which is managed by a DataSourceTransactionManager
MongoDB for your step (using the MongoItemWriter), which is managed by a MongoTransactionManager
If you want technical meta-data and business data to be committed/rolled back in the scope of the same distributed transaction, you need to use a JtaTransactionManager that coordinates the DataSourceTransactionManager and MongoTransactionManager. You can find some resources about the matter here: https://stackoverflow.com/a/56547839/5019386.
BTW, there is a feature request to use MongoDB as a job repository in Spring Batch: https://github.com/spring-projects/spring-batch/issues/877. When this is implemented, you could store both business data and technical meta-data in the same datasource (so no need for a distributed transaction anymore) and you would be able to use the same MongoTransactionManager for both the job repository and your step.
Related
Can Spring Batch use no-sql database (e.g. firestore, mongodb, etc.) to store batch metadata? If yes, can you share sample?
It can, but it does not provide an implementation yet. We have a feature request for that here: https://github.com/spring-projects/spring-batch/issues/877.
That said, it is a matter of implementing a single interface: JobRepository. Otherwise, you can implement the 4 DAOs required by Spring Batch using your NoSQL database (JobInstanceDao, JobExecutionDao, StepExecutionDao, ExecutionContextDao) and use them with the provided SimpleJobRepository.
i am using Postgres as my SQL.My Springboot application uses Spring Batch for processing and insertion of data.I am auditing my code flow like say suppose one 3rd party api which i call if it fails i audit this failure event.This piece of code is in my Spring Batch Writer.I see logs of my AUDIT DTO class getting created however i dont see data in audit table.The same if i move code of auditing outside Spring Batch writer -it works.What should be done so that my audit table insertion code in Spring Batch writer works?
More details would be needed to be sure but I assume your writer writes to the 3rd party API and you write the audit log to the same DataSource that you use for the Spring Batch meta data.
Every write of a chunk that Spring Batch does in a writer is wrapped in a transaction. Such a transaction will be rolled back if you throw an exception in the writer.
You need to write the audit log outside of the transaction created by Spring Batch. For example by using Spring transaction management and starting a new transaction with propagation level REQUIRES_NEW.
My requirement is to utilize 2 different database sources(persistent) and serve to frontend. To make api response faster can I utilize in-memory database like H2 or gemfire, to store data that is known to be frequently accessed (like a cron job at some time does that) and for other calls go to the databases. Here, the challenge for me is transferring the data from persistent to in-memory as Spring needs same 2 POJO's with different annotation(For e.g #Document for mongo, for h2,gemfire #Entity). as of now it does not make sense manually go through each record from an array received from mongo and save it in in-mem.
You can utilize the Spring Framework's Cache Abstraction and different forms of caching patterns when using Apache Geode (or alternatively, VMware Tanzu GemFire). See a more general description about caching starting here.
NOTE: VMware Tanzu GemFire, up to 9.15, is built on Apache Geode and are virtually interchangeable by swapping the bits.
I also provide many Samples of the different caching patterns in action, Guide and Source included, which are reference in the relevant section in the reference documentation.
Finally, in the Spring Boot for Apache Geode project, I also have 2 test classes testing the Inline Caching Pattern, which you may refer to:
The first test class uses Apache Geode as a caching provider in Spring's Cache Abstraction to cache data from a database, HSQLDB (predecessor of H2) in this case.
The second test class is similar in that Apache Geode caches data from Apache Cassandra.
The primary, backend data store used in Inline Caching is made irrelevant since it uses the Spring Data Repository abstraction to interface with any data store (referred to as "modules" in the Spring Data portfolio; see here) supported by the SD Repository abstraction.
Caching is but 1 technique to keep relevant data in-memory in order to improve response times in a Spring (Boot) application using a backing data store, such as an RDBMS, as the primary System of Record (SOR). Of course, there are other approaches, too.
Apache Geode can even be configured for durability (persistence) and redundancy. In some cases, it might even completely replace your database for some function. So, you have a lot of options.
Hopefully this will get you started and give you more ideas.
Recently I've encountered with some problem working with microservices. My main application works with relational databases, the microservice works with Mongo DB and provides ReST API with CRUD methods for some model. CRUD methods are also implemented in the application. A call from the front-end goes to the application first, a new record is created in the relational db (only some of the fields are saved there), then the model is saved externally - in Mongo DB. In the end the transaction is committed. So if something goes wrong and the transaction is rolled back, the API call would already be executed. In the case of creation I can just delete the newly created record from Mongo DB, but in case of Edit I have no idea what to do.
One of the ideas was to overwrite the model in the Mongo DB with the record from the relational database, but in this case the data would be incosistent, as not all the fields are saved there.
Any ideas about this?
There are multiple ways of doing this:
Using a 2PC (2 phase commit): you basically request an operation in any distributed service and then confirm/rollback it afterwards.
Using sagas: With sagas, you are meant to provide some sort of "rollback operation" for operations you did in your distributed services. When you need to roll back an operation that has already been perform you call the service and indicate a rollback.
More information about sagas here: https://microservices.io/patterns/data/saga.html
I am aware of how we create POJO classes (Java) and map them to the schema of the data in MongoDB, and create a connection with spring data. But if I don't have a specific schema and I want to have MongoDB as a back end for my cache in Hazelcast, how do I do that? In my use-case specifically, I have a cache which needs to keep mongodb updated with whatever updates it comes across.
Check this out:
https://github.com/hazelcast/hazelcast-code-samples/tree/master/hazelcast-integration/mongodb
Do note that this is a sample code that is meant for referential purposes only, do not copy paste into your production system.