Why is my update so slow JDBCbatchitemwriter? - postgresql

Question

This is the Step
#Bean
public Step processEodBatchUpdateActualTableStep() {
log.debug("[processEodBatchJob] Start Update Process for Actual Table");
return stepBuilderFactory.get(JobConfigurationConstants.PROCESS_EOD_FILE_UPDATE_STEP_NAME)
.<ExtensionQRMerchantTrxHistEntity, TransactionHistoryExtEntity>chunk(1000)
.reader(updateItemReader())
.processor(new ExtensionToTrxnHistExtConverter(mapper))
.writer(new UpdateActualTable(dataSource).updateActualTable())
.build();
}
This is the reader
#Bean
public JdbcCursorItemReader<ExtensionQRMerchantTrxHistEntity> updateItemReader(){
log.info("[UPDATE Reader] Read all records from temp table");
JdbcCursorItemReader<ExtensionQRMerchantTrxHistEntity> reader = new JdbcCursorItemReader<>();
reader.setSql("SELECT * FROM ext_qr_merchant_trx_hist eqmth " +
"WHERE EXISTS " +
"(SELECT 1 FROM t_trxn_detail_ext ttde WHERE eqmth.trx_ref_no = ttde.ref_no AND eqmth.trx_amt = ttde.amount " +
"AND eqmth.trx_dt = ttde.trxn_date);");
reader.setDataSource(dataSource);
reader.setFetchSize(10);
reader.setRowMapper(new RowMapper<ExtensionQRMerchantTrxHistEntity>() {
#Override
public ExtensionQRMerchantTrxHistEntity mapRow(#NonNull ResultSet rs, int rowNum) throws SQLException {
ExtensionQRMerchantTrxHistEntity entity = new ExtensionQRMerchantTrxHistEntity();
entity.setTransactionDate(rs.getTimestamp(1));
entity.setTransactionRefNo(rs.getString(2));
entity.setTransactionAmount(rs.getBigDecimal(3));
entity.setQrString(rs.getString(4));
return entity;
}
});
return reader;
}
This is the processor
#Slf4j
#RequiredArgsConstructor
public class ExtensionToTrxnHistExtConverter implements ItemProcessor<ExtensionQRMerchantTrxHistEntity, TransactionHistoryExtEntity> {
private final DuitNowRppDTOMapper mapper;
#Override
public TransactionHistoryExtEntity process(#NonNull ExtensionQRMerchantTrxHistEntity entity) throws Exception {
log.info("[Processor] Setting ExtensionQRMerchantTrxHistEntity to TransactionHistoryExtEntity");
return setTransactionHistory(entity);
}
private TransactionHistoryExtEntity setTransactionHistory(ExtensionQRMerchantTrxHistEntity tempEntity){
//Set output
TransactionHistoryExtEntity outputEntity =new TransactionHistoryExtEntity();
//Parse QR String
DuitNowRppDTO dto = mapper.mapFromQRDestination(tempEntity.getQrString());
//Set current date
Date now = new Date();
//Set field for Insert new record
UUID uuid = UUID.randomUUID();
outputEntity.setId(uuid);
outputEntity.setCreateDate(now);
outputEntity.setCreateBy(Constants.SYSTEM);
//Set field for updating record
outputEntity.setUpdateDate(now);
outputEntity.setUpdateBy(Constants.SYSTEM);
//replace field from temp table
outputEntity.setCurrencyCode(dto.getTransactionCurrencyCode());
outputEntity.setTransactionDate(tempEntity.getTransactionDate());
outputEntity.setReferenceNumber(tempEntity.getTransactionRefNo());
outputEntity.setAmount(tempEntity.getTransactionAmount());
return outputEntity;
}
}
This is the writer
#Slf4j
#RequiredArgsConstructor
public class UpdateActualTable {
private final DataSource dataSource;
public JdbcBatchItemWriter<TransactionHistoryExtEntity> updateActualTable() {
log.info("[Update] Using Batch Item Writer to UPDATE to Actual Table");
JdbcBatchItemWriter<TransactionHistoryExtEntity> itemWriter = new JdbcBatchItemWriter<>();
itemWriter.setDataSource(dataSource);
itemWriter.setSql("UPDATE t_trxn_detail_ext " +
"SET " +
"update_by = ?, update_dt = ? " +
"WHERE ref_no = ? AND amount = ? AND trxn_date = ?");
itemWriter.setItemPreparedStatementSetter((entity, preparedStatement) -> {
// insert
preparedStatement.setString(1, entity.getUpdateBy());
preparedStatement.setString(2, entity.getUpdateDate().toString());
//where
preparedStatement.setString(3, entity.getReferenceNumber());
preparedStatement.setBigDecimal(4, entity.getAmount());
preparedStatement.setString(5, entity.getTransactionDate().toString());
});
return itemWriter;
}
}
The performance of updating 100k records is slow compared to insertion of 100k records. I tried changing the update to insert statement in the writer and it manages to insert 100k records in less than 40-45 seconds. Update however, is updating 1k records out of 100k records per 2 minutes. What is causing this issue?
Does the chunk size, 1k in my case, affects the performance? I set the chunk size as a constant throughout the testing of inserting and updating using the same reader and processor.

Answer 1

I would first test the update query outside the Spring batch job (using a sql client for example) to make sure the performance hit is due to Spring Batch or not.
Typically, updates are slower than inserts (as they require an extra check for the existence of the row to update), but this could be related to missing indexes on your table.