Spring Batch MultiResourcePartitioner Performance building BATCH_STEP_EXECUTION table - spring-batch

Question

We have jobs that might process up to 20,000 files. We are using a MultiResourcePartitioner to set things up. The job does run, but we have noticed a bottleneck.
SpringBatch is creating entries in the BATCH_STEP_EXECUTION table for each file found, and will not process any files until it has created a table entry for every file. The loading of this table seems to take a very long time.
In local testing, trying to process just 1,000 files, it is taking 38-40 minutes to add the rows to 'BATCH_STEP_EXECUTION'. Once the table is loaded, the files are processed quite rapidly (usually under 1 minute).
I would hope that this is not typical behavior and that I am just missing something.
Here is how the database is set up (we really subclass the 'OracleDataSource' (we are using 'ojdbc6.jar' file to get to the class) and the db_file is a properties file to get to the url, password, etc.):
<bean id="dataSource" class="oracle.jdbc.pool.OracleDataSource" destroy-method="close">
<constructor-arg value="db_file" />
<property name="connectionCachingEnabled" value="true" />
<property name="connectionCacheProperties">
<props merge="default">
<prop key="InitialLimit">10</prop>
<prop key="MinLimit">25</prop>
<prop key="MaxLimit">50</prop>
<prop key="InactivityTimeout">1800</prop>
<prop key="AbandonedConnectionTimeout">900</prop>
<prop key="MaxStatementsLimit">20</prop>
<prop key="PropertyCheckInterval">20</prop>
</props>
</property>
</bean>
Here is the rest of the JobRepository definition:
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource" />
</bean>
<bean id="jobRepository" class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean" >
<property name="databaseType" value="oracle" />
<property name="dataSource" ref="dataSource" />
<property name="transactionManager" ref="transactionManager" />
<property name="isolationLevelForCreate" value="ISOLATION_DEFAULT"/>
</bean>
<bean id="jobExplorer" class="org.springframework.batch.core.explore.support.JobExplorerFactoryBean">
<property name="dataSource" ref="dataSource" />
</bean>
<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<bean id="jobParametersIncrementer" class="org.springframework.batch.core.launch.support.RunIdIncrementer" />
Anyone have any ideas?

Answer 1

As an FYI, SpringSource has identified this as a bug: Batch-1908.
As a workaround, we are simply lowering the number of files to process with a given run, and then increasing the number of times that the job runs in a given day.
We are using 2,000 as our file limit as it provides acceptable performance.

Answer 2

Take this as alternate approach.
For loading the table from files better to use LOADDATA .
http://infolab.stanford.edu/~ullman/fcdb/oracle/or-load.html
This will improve the performance in a better way. For me its take only 30 seconds to process a file with 1 million records