Is there any better way to load sql from file system to inject in JDBCCursorItemReader.
I want to load sql query from files instead of hardcoding in the configuration file.
//spring bean
<bean id="jdbcReader" class="com.sample.DatabaseReader">
<property name="sql" value="query.sql"/>
</bean>
and then i extended JDBCCursorItemReader
//extended cursoritemreader
class DatabaseReader extends JDBCCursorItemReader {
//Overriden method
#Override
public void setSql(String fileName) {
//file
File f = new File(fileName);
//read file from given path
String query = FileCopyUtils.copyToString(f);
//pass the query
super.setSQL(query);
}
}
Use Spring's PropertyPlaceHolder to inject the SQL directly into the reader (no need to extend our reader for this). An example would look like this:
<bean id="jdbcItemReader" class="org.springframework.batch.item.database.JdbcCursorItemReader">
<property name="dataSource" ref="dataSource" />
<property name="rowMapper" ref="myRowMapper>
<property name="sql" value="${batch.sql}"/>
</bean>
As long as you have a PropertiesPlaceholderConfigurer configured that points to the properties file that holds the batch.sql property, you should be good to go.
Related
I have a Pojo Partner:
Partner Id
List
Address Pojo :
AddressId,
Address,
City,
Country,
Pin
I want to create a Flat file in Spring Batch
- File will be
: PartnerId;AddressId;Address;City;Country;Pin
I am getting Partner Pojo with Id and List of Addresses
How can I use the FlatFileItemWriter with the PartnerPojo
My FlatFileItemWriterConfiguration configuration:
<?xml version="1.0" encoding="UTF-8"?>
<bean id="itemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" value="file:outputFile.txt" />
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter" value=";" />
<property name="fieldExtractor">
<bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names" value="partnerId, addressId, address,city,country,pin " />
</bean>
</property>
</bean>
</property>
<property name="headerCallback" ref="headerCallback" />
</bean>
I get an error on addressId
You need to flatten your data and pass the list of flat items as expected in the output file to the writer. For example:
class Partner {
int id;
List<Address> addresses;
}
class Address {
int addressId;
String address,city,country,pin;
}
// create this Pojo to encapsulate flat data (as in the expected csv)
class PartnerAddress {
int partnerId, addressId;
String address,city,country,pin;
}
An item processor would prepare the data:
class PartnerItemProcessor implements ItemProcessor<Partner, List<PartnerAddress>> {
#Override
public List<PartnerAddress> process(Partner partner) {
List<PartnerAddress> partnerAddresses = new ArrayList<>();
for (Address address : partner.getAddresses()) {
PartnerAddress partnerAddress = new PartnerAddress();
partnerAddress.setPartnerId(partner.getId());
partnerAddress.setAddressId(address.getAddressId());
partnerAddress.setAddress(address.getAddress());
partnerAddress.setCity(address.getCity());
partnerAddress.setCountry(address.getCountry());
partnerAddress.setPin(address.getPin());
partnerAddresses.add(partnerAddress);
}
return partnerAddresses;
}
}
Then the writer receives the list of PartnerAddress and write them to the flat file.
Hope this helps.
I'm trying to access the StepExecution in my RowMapper but unable to do so. I have a property set in the xml called 'prop1'. I expect this to be set but it is not setting.I also added a #BeforeStep method to the RowMapper hoping I can get the stepExecutionContext but this method is never invoked. Is there something else I need to do?
Here is my xml:
<bean id="bean1"
class="org.springframework.batch.item.database.JdbcCursorItemReader"
scope="step">
<property name="dataSource" ref="dataSource" />
<property name="sql"
value="${sql}"/>
<property name="fetchSize" value="${fetchSize}"></property>
<property name="rowMapper">
<bean id="rowMapper1" class="c.p.MyRowMapper" scope="step">
<property name="prop1" value="${prop1}"></property>
</property>
Here is my RowMapper:
public class MyRowMapper implements RowMapper<Object>{
private String prop1;
private StepExecution se;
public String getProp1() {
return stepFatpCount;
}
public void setProp1(String rop1) {
this. prop1 = prop1;
}
#BeforeStep
public void beforeStep(StepExecution stepExecution){
this.se = stepExecution;
}
}
I have some properties set in the stepExecutionContext before this step in another step and I want to use them here in the RowMapper. The same thing works in the ItemProcessor but not the RowMapper. Please let me know if I need to do something more for lazy binding or any other issue.
Thanks.
The step execution context is not shared between steps. Maybe you mean job execution context?
I suppose you could register rowMapper1 as a listener in the step (with the <listeners> tag), but if you just want to read some value from the job execution context you could use
value="#{jobExecutionContext['foobar']}"
If you do want to have injected some value from the step execution context, you just have to replace stepExecutionContext above.
I'm working on a Spring-Batch program. I unmarshalls XML files with XStreamMarshaller.
How can I make a XStreamMarshaller to skip any unknown+unannoated fields?
<bean id="merge.reader.item"
class="org.springframework.batch.item.xml.StaxEventItemReader">
<property name="fragmentRootElementName" value="xml-fragment"/>
<property name="unmarshaller" ref="merge.reader.unmarshaller"/>
</bean>
<bean id="merge.reader.unmarshaller"
class="org.springframework.oxm.xstream.XStreamMarshaller">
<property name="aliases" ref="merge.reader.binder"/>
<property name="autodetectAnnotations" value="true"/>
</bean>
<util:map id="merge.reader.binder">
<entry key="xml-fragment" value="path.to.my.Model"/>
</util:map>
public class Model {
#XStreamAlias(value = "one")
private String one;
#XStreamAlias(value = "other")
private String other;
}
The problem is that some new xml elements will be introduced in some other time.
I don't want to (actually I can't) add extra fields to my Model.
I'm answering for my own question. The solution is where #biziclop linked. (disclaimer: I also answered the same answer on that post).
public class ExtendedXStreamMarshaller extends XStreamMarshaller {
#Override
protected void configureXStream(final XStream xstream) {
super.configureXStream(xstream);
xstream.ignoreUnknownElements(); // will it blend?
}
}
I'm using Spring Batch to extract a CSV file from a DB table which has a mix of column types. The sample table SQL schema is
[product] [varchar](16) NOT NULL,
[version] [varchar](16) NOT NULL,
[life_1_dob] [date] NOT NULL,
[first_itm_ratio] [decimal](9,6) NOT NULL,
the sample Database column value for the 'first_itm_ration' field are
first_itm_ratio
1.050750
0.920000
but I would like my CSV to drop the trailing zero's from values.
first_itm_ratio
1.05075
0.92
I'd prefer not to have to define the formatting for each specific field in the table, but rather have a global object specific formatting for all columns of that data type.
My csvFileWriter bean
<bean id="csvFileWriter" class="org.springframework.batch.item.file.FlatFileItemWriter" scope="step">
<property name="resource" ref="fileResource"/>
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter">
<util:constant static-field="org.springframework.batch.item.file.transform.DelimitedLineTokenizer.DELIMITER_COMMA"/>
</property>
<property name="fieldExtractor">
<bean class="org.springframework.batch.item.file.transform.PassThroughFieldExtractor" />
</property>
</bean>
</property>
</bean>
You can
Write your own BigDecimalToStringConverter implements Converter<BigDecimal, String> to format big decimal without trailing 0's
Create a new ConversionService (MyConversionService) and register into the custom converter
Extends DelimitedLineAggregator, inject MyConversionService, override doAggregate() to format fields using injected conversion service
public class MyConversionService extends DefaultConversionService {
public MyConversionService() {
super();
addConverter(new BigDecimalToStringConverter());
}
}
public class MyFieldLineAggregator<T> extends DelimitedLineAggregator<T> {
private ConversionService cs = new MyConversionService();
public String doAggregate(Object[] fields) {
for(int i = 0;i < fields.length;i++) {
final Object o = fields[i];
if(cs.canConvert(o.getClass(), String.class)) {
fields[i] = cs.convert(o, String.class);
}
}
return super.doAggregate(fields);
}
}
I am working on setting a spring batch job that does the conventional READ > PROCESS > WRITE operation. However, I am trying to implement a listener which would capture records that are conisdered invalid during the PROCESS phase and write it out to an error log file.
My listener class uses an instance of FlatFileItemWriter to write the data. However, spring-batch is not instantiating the writer instance properly.
My listener class looks like this:
public class DTOProcessorListener extends SkipListenerSupport<AttributeReportGenerationDTO, AttributeValue> {
private static final Logger LOGGER = LoggerFactory.getLogger(DTOProcessorListener.class);
private FlatFileItemWriter<AttributeReportGenerationDTO> flatFileItemWriter;
#Override
public void onSkipInProcess(AttributeReportGenerationDTO item, Throwable t) {
try {
LOGGER.error("Record not processed for attribute value with ID : " + item.getAttributeValueId());
List<AttributeReportGenerationDTO> list = new ArrayList<AttributeReportGenerationDTO>();
list.add(item);
flatFileItemWriter.write(list);
} catch (Exception e) {
LOGGER.error("Unable to write to the error output file", e);
}
}
/**
* #param flatFileItemWriter
* the flatFileItemWriter to set
*/
public void setFlatFileItemWriter(FlatFileItemWriter<AttributeReportGenerationDTO> flatFileItemWriter) {
this.flatFileItemWriter = flatFileItemWriter;
}
}
and my job configuration XML looks like this:
<bean id="skipListener" class="something.DTOProcessorListener" scope="step">
<property name="flatFileItemWriter">
<bean id="errorItemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter" scope="step">
<property name="resource" value="file:#{jobParameters['error.filename']}" />
<property name="appendAllowed" value="true" />
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="fieldExtractor">
<bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names"
value="productTypeId, productTypeName, productId, productName, skuId, skuName, attributeValueId, attributeName, attributeValue, attributeType, nonEditableValueCheckSum, editableValueCheckSum" />
</bean>
</property>
</bean>
</property>
<property name="headerCallback">
<bean class="something.CsvHeaderImplementation">
<property name="headerString"
value="Product Type ID,Product Type,Product ID,Product Name,Sku ID,Sku Name,Attribute Value ID,Attribute Name,Attribute Value,Attribute Type,Check Sum 1,Check Sum 2" />
</bean>
</property>
</bean>
</property>
</bean>
I get the error
org.springframework.batch.item.WriterNotOpenException: Writer must be open before it can be written to
I am unable to set a stream entry in the job config as the bean for FlatFileItemWriter is internally specified (for the listener). If I create abean outside of the listener and refer to it, its returning a proxy instance of the FlatFileItemWriterClass.
Has anyone successfully wired up a writer to a flat file in the listener?
Thanks for the help
Well why don't you use the writer as a normal bean ? You could register it as a stream and to get around the step proxy you could use the PropertPlaceholderConfigurer
i created a working example under my github repo, but i think spring batch could need an improvement here, it should be easier to implement error-item logging