Testing a FieldSetMapper which reads by column names throws exception - spring-batch

The line mapping logic in the job config xml is as follows:
<property name="lineTokenizer">
<bean
class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="names" value="ID,NAME"/>
<property name="strict" value="false"/>
</bean>
</property>
<property name="fieldSetMapper">
<bean class="com.company.batch.mappers.EmpFieldSetMapper"/>
</property>
The field set mapper read logic is as follows:
fieldSet.readString("ID");
But while writing UT for EmpFieldSetMapper, I am getting below error:
java.lang.IllegalArgumentException: Cannot access columns by name without meta data
at org.springframework.batch.item.file.transform.DefaultFieldSet.indexOf(DefaultFieldSet.java:675)
at org.springframework.batch.item.file.transform.DefaultFieldSet.readString(DefaultFieldSet.java:169)
The UT is as follows:
#Before
public void setUp() throws Exception {
mapper = new EmpFieldSetMapper();
DefaultFieldSetFactory fieldSetFactory = new DefaultFieldSetFactory();
String[] inputValues = { "123", "RAJ" };
fieldSet = fieldSetFactory.create(inputValues);
}
#Test
public void testMapFieldSet() {
try {
Model model = mapper.mapFieldSet(fieldSet);
assertEquals("ID field mapping is wrong", "123", model.getId());
assertEquals("NAME field mapping is wrong", "RAJ", model.getName());
} catch (BindException e) {
fail("Exception during field set mapping");
}
}
I think I need to change the DefaultFieldSetMapper to something else but unsure about it. This issue can be resolved by replacing the column names with index but I want to retain the column names in EmpFieldSetMapper. So need suggestions.

I found the answer. Need to pass in column names as well.
String[] inputValues = { "123", "RAJ" };
String[] inputColumns = { "ID", "NAME" };
fieldSet = fieldSetFactory.create(inputValues, inputColumns);

Related

Spring Batch FlatFileItemWriter write Object with List

I have a Pojo Partner:
Partner Id
List
Address Pojo :
AddressId,
Address,
City,
Country,
Pin
I want to create a Flat file in Spring Batch
- File will be
: PartnerId;AddressId;Address;City;Country;Pin
I am getting Partner Pojo with Id and List of Addresses
How can I use the FlatFileItemWriter with the PartnerPojo
My FlatFileItemWriterConfiguration configuration:
<?xml version="1.0" encoding="UTF-8"?>
<bean id="itemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" value="file:outputFile.txt" />
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter" value=";" />
<property name="fieldExtractor">
<bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names" value="partnerId, addressId, address,city,country,pin " />
</bean>
</property>
</bean>
</property>
<property name="headerCallback" ref="headerCallback" />
</bean>
I get an error on addressId
You need to flatten your data and pass the list of flat items as expected in the output file to the writer. For example:
class Partner {
int id;
List<Address> addresses;
}
class Address {
int addressId;
String address,city,country,pin;
}
// create this Pojo to encapsulate flat data (as in the expected csv)
class PartnerAddress {
int partnerId, addressId;
String address,city,country,pin;
}
An item processor would prepare the data:
class PartnerItemProcessor implements ItemProcessor<Partner, List<PartnerAddress>> {
#Override
public List<PartnerAddress> process(Partner partner) {
List<PartnerAddress> partnerAddresses = new ArrayList<>();
for (Address address : partner.getAddresses()) {
PartnerAddress partnerAddress = new PartnerAddress();
partnerAddress.setPartnerId(partner.getId());
partnerAddress.setAddressId(address.getAddressId());
partnerAddress.setAddress(address.getAddress());
partnerAddress.setCity(address.getCity());
partnerAddress.setCountry(address.getCountry());
partnerAddress.setPin(address.getPin());
partnerAddresses.add(partnerAddress);
}
return partnerAddresses;
}
}
Then the writer receives the list of PartnerAddress and write them to the flat file.
Hope this helps.

How can I make a XStreamMarshaller skip unknown binding?

I'm working on a Spring-Batch program. I unmarshalls XML files with XStreamMarshaller.
How can I make a XStreamMarshaller to skip any unknown+unannoated fields?
<bean id="merge.reader.item"
class="org.springframework.batch.item.xml.StaxEventItemReader">
<property name="fragmentRootElementName" value="xml-fragment"/>
<property name="unmarshaller" ref="merge.reader.unmarshaller"/>
</bean>
<bean id="merge.reader.unmarshaller"
class="org.springframework.oxm.xstream.XStreamMarshaller">
<property name="aliases" ref="merge.reader.binder"/>
<property name="autodetectAnnotations" value="true"/>
</bean>
<util:map id="merge.reader.binder">
<entry key="xml-fragment" value="path.to.my.Model"/>
</util:map>
public class Model {
#XStreamAlias(value = "one")
private String one;
#XStreamAlias(value = "other")
private String other;
}
The problem is that some new xml elements will be introduced in some other time.
I don't want to (actually I can't) add extra fields to my Model.
I'm answering for my own question. The solution is where #biziclop linked. (disclaimer: I also answered the same answer on that post).
public class ExtendedXStreamMarshaller extends XStreamMarshaller {
#Override
protected void configureXStream(final XStream xstream) {
super.configureXStream(xstream);
xstream.ignoreUnknownElements(); // will it blend?
}
}

SqlPagingQueryProviderFactoryBean to support In clause

I want to write an SQL for SqlPagingQueryProviderFactoryBean. I will pass the parameter for an IN clause. I am not getting the result when I am passing it as a parameter(?). But I am getting the correct result when I am hard coding the values.
Please guide me on this.
You cannot have a single place holder and replace it with an array due to sql injection security policy, However the gettter/setters of sqlPagingQueryProvider properties selectclause, fromClause and whereCLause are String and not preparedStatement. The PreparedStatement would be constructed later by spring batch during post construct method. Hence you can send the where clause as String with values(Prepared) and pass it to job as parameter. Hence your code would something of this sort.
String topics = { "topic1", "topic2", "topic3", "topic4"};
StringBuilder str = new StringBuilder("('");
for (String topic : topics) {
str.append(topic + "','");
}
str.setLength(str.length() - 2);
str.append("')");
final JobParameters jobParameters = new JobParametersBuilder()
.addLong("time", System.nanoTime())
.addString("inputsTopics", str.toString())
.toJobParameters();
And your pagingreader bean would look like below and make sure you set scope to step
<bean id="sqlPagingReader" class="<your extended prging prvder>.KPPageingProvider" scope="step" >
<property name="dataSource" ref="dataSource" />
<property name="selectClause" value="u.topic,cu.first_name ,cu.last_name, cu.email" />
<property name="fromClause" value="ACTIVE_USER_VIEWS_BY_TOPIC u inner join cl_user cu on u.user_id=cu.id" />
<property name="whereClause" value="u.topic in #{jobParameters['inputsTopics']}" ></property>
</bean>

Query : property name in parameter

With this query, I succeed to retrieve a phone number in database:
import java.util.List;
import org.springframework.data.jpa.repository.JpaReposit ory;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.query.Param;
import com.mc.appcontacts.domain.hibernate.Contact;
public interface ContactRepository extends JpaRepository<Contact, Integer> {
#Query("SELECT c.phoneNumber from Contact c WHERE LOWER(c.name) = LOWER(:name)")
String find(#Param("name") String name);
But is it possible to specify dynamically name of the property i want to retrieve in parameter?
In all tuto i've read on the net, i learn we can pass the value of the property in parameter (In my exemple : #Param("name") String name )
but what i want to pass in parameter is the name of the property not the value !
I know the exemple below is not correct but it's to give the general idea :
#Query("SELECT c.(property) from Contact c WHERE LOWER(c.name) = LOWER(:name)")
String find(#Param("name") String name, #Param("property") String property);
With property = phoneNumber (or an other property of my table).
Thank you for your help !!
I don't understand how to do that (everything is new for me):
I have read (and try) that jpql is defined like this :
import com.mysema.query.jpa.impl.JPAQuery;
import com.mc.appcontacts.repository.ContactRepository; // managed by spring data
//jpa repository
public class ServicesC {
#Autowired
private ContactRepository repository;
#PersistenceContext // try
private EntityManager em; // try
JPAQuery query = new JPAQuery(em); // try
public Contact getOne(Integer id) {
return repository.findOne(id);
}
public String findAtt(String att, String key){
String jpql = "SELECT c." + att + " from Contact c WHERE LOWER(c.name) = LOWER(:key)"; // try
List<Contact> l = (List<Contact>) em.createQuery(jpql); // try
return "test";
}
}
But it doesn't work (i'm not surprised...) :
2014-02-24 18:18:34.567:WARN::Nested in org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'appMapping': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.mc.appcontacts.service.ServiceC com.mc.appcontacts.mvc.MappingService.service; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'Service' defined in file [C:\Professional\Workspaces\Eclipse\ContactMain\ContactCore\target\classes\com\mc\appcontacts\service\ServiceC.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.mc.appcontacts.service.ServiceC]: Constructor threw exception; nested exception is java.lang.NullPointerException:
java.lang.NullPointerException
at com.mysema.query.jpa.impl.JPAProvider.get(JPAProvider.java:72)
at com.mysema.query.jpa.impl.JPAProvider.getTemplates(JPAProvider.java:80)
at com.mysema.query.jpa.impl.JPAQuery.<init>(JPAQuery.java:46)
Must i define a second EntityManager only for jpql ? (Is it possible ? is it the right way ? I don't think so...)
I have already a EntityManager defin for Spring-data in xml file :
<tx:annotation-driven transaction-manager="transactionManager" />
<!-- Activate Spring Data JPA repository support -->
<jpa:repositories base-package="com.mc.appcontacts.repository" />
<!-- Declare a JPA entityManagerFactory -->
<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="persistenceXmlLocation" value="classpath:META-INF/contacts/hibernate/persistence.xml" />
<property name="persistenceUnitName" value="hibernatePersistenceUnit" />
<!-- <property name="dataSource" ref="dataSource" /> -->
<property name="jpaVendorAdapter" ref="hibernateVendor" />
</bean>
<!-- Specify our ORM vendor -->
<bean id="hibernateVendor" class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="showSql" value="${hibernate.showSql}" />
</bean>
<!-- Declare a transaction manager-->
<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactory" />
</bean>
Please help me ... how does it work ?
No, it's not possible to do that. You'll have to implement it by yourself by dynamically generating the JPQL query.
Using query parameyters is not an option, because query parameters can only be values to replace in a given prepared statement, and can't alter the nature of the query itself. So you'll have to do something like
String jpql = "select c." + property + " from ...";
I think for this use case of building queries dynamically your best bet would be to explore Criteria API, which is very suitable for such things. http://docs.oracle.com/javaee/6/tutorial/doc/gjitv.html

Spring Batch : PassThroughFieldExtractor with BigDecimal formatting

I'm using Spring Batch to extract a CSV file from a DB table which has a mix of column types. The sample table SQL schema is
[product] [varchar](16) NOT NULL,
[version] [varchar](16) NOT NULL,
[life_1_dob] [date] NOT NULL,
[first_itm_ratio] [decimal](9,6) NOT NULL,
the sample Database column value for the 'first_itm_ration' field are
first_itm_ratio
1.050750
0.920000
but I would like my CSV to drop the trailing zero's from values.
first_itm_ratio
1.05075
0.92
I'd prefer not to have to define the formatting for each specific field in the table, but rather have a global object specific formatting for all columns of that data type.
My csvFileWriter bean
<bean id="csvFileWriter" class="org.springframework.batch.item.file.FlatFileItemWriter" scope="step">
<property name="resource" ref="fileResource"/>
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter">
<util:constant static-field="org.springframework.batch.item.file.transform.DelimitedLineTokenizer.DELIMITER_COMMA"/>
</property>
<property name="fieldExtractor">
<bean class="org.springframework.batch.item.file.transform.PassThroughFieldExtractor" />
</property>
</bean>
</property>
</bean>
You can
Write your own BigDecimalToStringConverter implements Converter<BigDecimal, String> to format big decimal without trailing 0's
Create a new ConversionService (MyConversionService) and register into the custom converter
Extends DelimitedLineAggregator, inject MyConversionService, override doAggregate() to format fields using injected conversion service
public class MyConversionService extends DefaultConversionService {
public MyConversionService() {
super();
addConverter(new BigDecimalToStringConverter());
}
}
public class MyFieldLineAggregator<T> extends DelimitedLineAggregator<T> {
private ConversionService cs = new MyConversionService();
public String doAggregate(Object[] fields) {
for(int i = 0;i < fields.length;i++) {
final Object o = fields[i];
if(cs.canConvert(o.getClass(), String.class)) {
fields[i] = cs.convert(o, String.class);
}
}
return super.doAggregate(fields);
}
}