Can Spring Batch work with Amazon Redshift? - spring-batch

Question

I'm trying to use Spring Batch (4.0.1.RELEASE) working with Amazon Redshift. I got through the first major problem with Redshift's lack of support for sequences here.
However, now I've run into this when I try to run a job:
10:57:07.122 ERROR [http-nio-8080-exec-4 ] [JobLaunchingService] [] Could not start job [demoJob]
org.springframework.dao.InvalidDataAccessApiUsageException: PreparedStatementCallback; SQL [INSERT INTO BATCH_JOB_EXECUTION_CONTEXT (SHORT_CONTEXT, SERIALIZED_CONTEXT, JOB_EXECUTION_ID) VALUES(?, ?, ?)[Amazon][JDBC](10220) Driver does not support this optional feature.; nested exception is java.sql.SQLFeatureNotSupportedException: [Amazon][JDBC](10220) Driver does not support this optional feature.
This is with the Redshift 1.2.16.1027 JDBC Driver.
Is it even possible to use Redshift as the batch database? Any suggestions on how to get around this?

Answer 1

I'm not sure about you use case, if it is limitation or constrained that you need to have Spring batch only. Also, the jdbc driver, says It doesn't support the batch., then I believe there is no way around to make it work.
As a recommended approach and best practice, in Redshift instead of insert statement, COPY command should be used. Though, you could call the copy command using plain JDBC could be good idea.
You could take a look at one my answer, I have provided earlier.
I'm just copy/pasting it make it handy.
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.Statement;
import java.util.Properties;
public class RedShiftJDBC {
public static void main(String[] args) {
Connection conn = null;
Statement statement = null;
try {
//Make sure to choose appropriate Redshift Jdbc driver and its jar in classpath
Class.forName("com.amazon.redshift.jdbc42.Driver");
Properties props = new Properties();
props.setProperty("user", "username***");
props.setProperty("password", "password****");
System.out.println("\n\nconnecting to database...\n\n");
//In case you are using postgreSQL jdbc driver.
conn = DriverManager.getConnection("jdbc:redshift://********url-to-redshift.redshift.amazonaws.com:5439/example-database", props);
System.out.println("\n\nConnection made!\n\n");
statement = conn.createStatement();
String command = "COPY my_table from 's3://path/to/csv/example.csv' CREDENTIALS 'aws_access_key_id=******;aws_secret_access_key=********' CSV DELIMITER ',' ignoreheader 1";
System.out.println("\n\nExecuting...\n\n");
statement.executeUpdate(command);
//you must need to commit, if you realy want to have data copied.
conn.commit();
System.out.println("\n\nThats all copy using simple JDBC.\n\n");
statement.close();
conn.close();
} catch (Exception ex) { ex.printStackTrace(); } } }
I hope this gives you some idea. If you have specific question add a comment, I should be able to refocus the answer.

Answer 2

In order to make this work, I had to define a separate MySQL database for the Spring Batch "control" tables. That was the default (#Primary) database in the Batch application. The ItemWriters are fed with a different DataSource, the one that was pointed at Redshift.
So now I've got a DataSource for the Batch tables, one for my source db, and one for the target db. That seems to work, but I'm only using the standard DataSourceTransactionManager so it's not clear at all to me what the transactional boundaries are, if a step fails whether the databases are rolled back the same way. But I am NOT going to use XA!!

Answer 3

Amazon Redshift is not a supported database for Spring Batch. The supported databases are listed here: https://github.com/spring-projects/spring-batch/tree/master/spring-batch-core/src/main/resources/org/springframework/batch/core.