I'm using spring-batch to process something.
(Now I'm just using tasklet.)
Now I have to make api that shows result of batch jobs.
result is like below,
executor | job name | status | success count | fail count
ㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡㅡ
joont92 | some_job | FAIL | 100 | 20
As you guys know,
job name is JOB_INSTACE's data,
status is JOB_EXECUTION's data,
success count, fail count is STEP_EXECUTION's data.
And executor is JOB_EXECUTION_CONTEXT's data maybe.
Yeah, I need tojoin spring-batch's tables. But I don't know how I have to do.
I've found JobExplorer, JobRepository.
It seems like to select spring-batch's tables, but I'm not sure this is right way to make result api.
Is there any libraries or way to do this?
Have to use another spring's library?
Is there any libraries or way to do this?
JobExplorer is the way to go. It is a read-only version of the job repository which is perfectly fine for your use case.
Have to use another spring's library?
If you don't want to use the JobExplorer form Spring Batch, you can query the database with:
a JdbcTemplate. You can find a getting started guide about it here: https://spring.io/guides/gs/relational-data-access/
or using Spring Data
Hope this helps.
Related
In my project I am reading the data for DB table using StoredProcedure reader and calling an API to process and then saving the output using writer. I need to maintain the processing status as Processed or Error for each record that I am reading. As of now I am using the writer to update the input table column STATUS to P (Processed) or E(Error) and add logs the in case of any error to LOGS column.
Can you please suggest if this is the efficient way to maintain the processing status of each record. Does Spring batch provides any default implementation for same?
Thanks
No, Spring Batch does not provide a "default implementation" for such a requirement.
That said, a flag on each item as you did is a reasonable way to address your requirement in my opinion.
I'm trying to setup Spring Batch to move DB records from Oracle to Cassandra daily.
I know I can manually define JPA repository queries based on additional entity table (like MyBatchProgress where I store previously completed Id + date or something like that), so that the next batch job knows which entity to start with for further operations.
My question is: does Spring Batch provide something like this inbuilt (also by utilising Spring Data JPA)?
Or is this something that I have to write manually in the job reader step where I just pick up the last Id stored in my custom "progress" table?
Thanks in advance!
You can store the last ID in the execution context, which is persisted in the meta-data tables. With that in place, you can make the code that launches the job look for the last job execution, take the ID from its context and pass it as a job parameter to the next job instance.
I am currently setting up an event-store with Axon-framework in PostgreSQL (spring boot, axon-spring-boot-starter, axon-server-connector removed from the dependency).
The system loads as expected and I am able to see commands, events and event handlers working as expected.
The issue is when I want to see the contents of my events in the event table (domain_event_entry) .
I would expect that the 'payload' column in the table containing all the events that I persisted in the event store, but I am just seeing numbers: something like this:
global_index | event_identifier | metadata |payload_type |
1 | 7c23e693-558b-4013-b64f-3f272cb0102a |19435 |19436|
Also, I believe that the metadata should contain something other tha n an integer.
Is this correct? Am I missing some extra configuration?
This is because Postgres uses TOAST (The oversized attribute storage technique). How to make this better readable can be found in this blogpost
When implementing a system which creates tasks that need to be resolved by some workers, my idea would be to create a table which would have some task definition along with a status, e.g. for document review we'd have something like reviewId, documentId, reviewerId, reviewTime.
When documents are uploaded to the system we'd just store the documentId along with a generated reviewId and leave the reviewerId and reviewTime empty. When next reviewer comes along and starts the review we'd just set his id and current time to mark the job as "in progress" (I deliberately skip the case where the reviewer takes a long time, or dies during the review).
When implementing such a use case in e.g. PostgreSQL we could use the UPDATE review SET reviewerId = :reviewerId, reviewTime: reviewTime WHERE reviewId = (SELECT reviewId from review WHERE reviewId is null AND reviewTime is null FOR UPDATE SKIP LOCKED LIMIT 1) RETURNING reviewId, documentId, reviewerId, reviewTime (so basically update the first non-taken row, using SKIP LOCKED to skip any already in-processing rows).
But when moving from native solution to JDBC and beyond, I'm having troubles implementing this:
Spring Data JPA and Spring Data JDBC don't allow the #Modifying query to return anything else than void/boolean/int and force us to perform 2 queries in a single transaction - one for the first pending row, and second one with the update
one alternative would be to use a stored procedure but I really hate the idea of storing such logic so away from the code
other alternative would be to use a persistent queue and skip the database all along but this introduced additional infrastructure components that need to be maintained and learned. Any suggestions are welcome though.
Am I missing something? Is it possible to have it all or do we have to settle for multiple queries or stored procedures?
Why Spring Data doesn't support returning entity for modifying queries?
Because it seems like a rather special thing to do and Spring Data JDBC tries to focus on the essential stuff.
Is it possible to have it all or do we have to settle for multiple queries or stored procedures?
It is certainly possible to do this.
You can implement a custom method using an injected JdbcTemplate.
What I want to do is write an Order service project without using the traditional RDBMS(eg. mysql).After having read the docs from Kafka and confluent for several days, I think Kafka stream and state store could help me to implement this without using RDBMS.
But how to store the data that has a long list result in state store?Especially when I need to query result like using limit and offset?
eg:
I have a table, the columns are:
| userId | orderId | ... |
One user may have many order rows to store, so I need a query method to search the key:userId result in the state store with start and limit.But I can't find a method in the Kafka stream State Store interface.
Do I have to do this using an RDBMS? What is the standard method for me to implement an application like this?