Spring batch write skipper item into csv file - spring-batch

I want to write skipper lines in first csv file and the result of processor in second file in one step but it not works !
My code :
// => Step cecStep1
#Bean
public Step cecStep1(StepBuilderFactory stepBuilders) throws IOException {
return stepBuilders.get("fileDecrypt")
.<CSCivique, String>chunk(100)
.reader(reader1())
.processor(processor1FileDecrypt())
.writer(writer1())
.faultTolerant()
.skip(Exception.class)
.skipLimit(100)
.listener(new MySkipListener())
.build();
}
// ##################################### Step SkipListener ###################################################
public static class MySkipListener implements SkipListener {
private BufferedWriter bw = null;
public MySkipListener(File file) throws IOException {
//this.fileWriter = new FileWriter(file);
bw= new BufferedWriter(new FileWriter(file, true));
System.out.println("MySkipListener =========> :"+file);
}
#Override
public void onSkipInRead(Throwable throwable) {
if (throwable instanceof FlatFileParseException) {
FlatFileParseException flatFileParseException = (FlatFileParseException) throwable;
System.out.println("onSkipInRead =========> :");
try {
bw.write(flatFileParseException.getInput()+"; Vérifiez les colonnes !!");
bw.newLine();
bw.flush();
// fileWriter.close();
} catch (IOException e) {
System.err.println("Unable to write skipped line to error file");
}
}
}
#Override
public void onSkipInWrite(CSCivique item, Throwable t) {
System.out.println("Item " + item + " was skipped due to: " + t.getMessage());
}
#Override
public void onSkipInProcess(CSCivique item, Throwable t) {
System.out.println("Item " + item + " was skipped due to: " + t.getMessage());
}
}
#Bean
public FlatFileItemWriter<String> writer1() {
return new FlatFileItemWriterBuilder<String>().name(greetingItemWriter)
.resource(new FileSystemResource("target/test-outputs/greetings.csv"))
.lineAggregator(new PassThroughLineAggregator<>()).build();
}
Tankyou !

In your processor, you can:
throw a skippable exception for invalid items so that the skip listener intercepts them and writes them to the specified file
let valid items go to the writer so that they are written as configured in the item writer
For example:
class MyItemProcessor implements ItemProcessor<Object, Object> {
#Override
public Object process(Object item) throws Exception {
if (shouldBeSkipped(item)) {
throw new MySkippableException();
}
// process item
return item;
}
}
Hope this helps.

Related

Spring batch write exception with file parameters

I want to write the exceptions in the same file as the item writer but do not work
// MyLitner
public static class MySkipListener implements SkipListener<CSCiviqueDTO, CSCiviqueDTO> {
private BufferedWriter bw = null;
public MySkipListener(File file) throws IOException {
bw= new BufferedWriter(new FileWriter(file, true));
}
#Override
public void onSkipInRead(Throwable throwable) {
if (throwable instanceof FlatFileParseException) {
FlatFileParseException flatFileParseException = (FlatFileParseException) throwable;
System.out.println("onSkipInRead =========> :");
try {
bw.write(flatFileParseException.getInput() + "; Step Vérifiez les colonnes !!");
bw.newLine();
bw.flush();
// fileWriter.close();
} catch (IOException e) {
System.err.println("Unable to write skipped line to error file");
}
}
}

Spring batch pass dynamic file name to onSkipInRead

I want to pass a dynamic file name to onSkipInRead.
example:
public MySkipListener(#Value("#{jobParameters['file']}") String file) throws IOException {
bw= new BufferedWriter(new FileWriter(file);
System.out.println("MySkipListener =========> :"+bw);
}
#Override
public void onSkipInRead(Throwable throwable) {
if (throwable instanceof FlatFileParseException) {
FlatFileParseException flatFileParseException = (FlatFileParseException) throwable;
try {
bw.write(flatFileParseException.getInput()+"; Step Vérifiez les colonnes !!");
bw.newLine();
bw.flush();
} catch (IOException e) {
System.err.println("Unable to write skipped line to error file"); }
}
}
Thank you.
You can create a constructor in your listener with a file and pass the file to it when you define the step. Here is an example:
class MySkipListener implements SkipListener<Integer, Integer> {
private FileWriter fileWriter;
public MySkipListener(File file) throws IOException {
this.fileWriter = new FileWriter(file);
}
// your onSkipInRead method
}
Then you pass the job parameter file to your listener when you define the step:
#Bean
#JobScope
public Step step(#Value("#{jobParameters['file']}") String file) throws IOException {
return stepBuilderFactory.get("step")
.<Integer, Integer>chunk(5)
.reader(itemReader())
.writer(itemWriter())
.listener(new MySkipListener(new File(file)))
.build();
}
Hope this helps.

Test exception of a method which contains try catch in junit

I have code snippet below.
What I want is if getNames() method catch an exception
( ex. InterruptedException ),
want to check if Got InterruptedException !!! prints out or not.
There are some examples of testing exception for a method
which throws an exception in its method ( ex. String method1() throws InterruptedException {...} ) in the Internet.
But not this case. Does anyone have some thought or idea?
public class A {
public List<String> getNames()
{
String addess = "address1";
int age = 17;
List<String> names = null;
try {
names = getSomeNames(address, sex);
}
catch (InterruptedException | ExecutionException e) {
throw new MyCustomException(e);
}
catch(Exception e) {
throw new MyCustomException(e);
}
return names;
}
List<String> getSomeNames(String address, int sex) throws InterruptedException, ExecutionException
{
// ...
// throw exceptions... at some point
//
return names;
}
}
public class MyCustomException extends Exception {
public MyCustomException(Throwable e) {
if (e.getCause() instanceof InterruptedException) {
// write log
System.out.println("Got InterruptedException !!!");
}
else if (e.getCause() instanceof ExecutionException) {
// write log
System.out.println("Got ExecutionException!!!");
}
else {
// write log
}
}
}
I tried this but the test failed and got NullPointerException in catch block.
#Test
public void testException() {
A objA = spy(new A());
try {
doThrow(MyCustomException.class).when(objA).getNames();
objA.getNnames();
}
catch (Exception e) {
System.out.println(e.getCause().toString()); // ==> throws java.lang.NullPointerException here.
}
}
There are several ways to test it.
First solution is to replace System.out with different stream and read from it later. ( I don't like this approach )
#Test
void whenSayHi_thenPrintlnCalled() throws IOException {
PrintStream normalOutput = System.out;
String result;
try (ByteArrayOutputStream baos = new ByteArrayOutputStream(); PrintStream temporalOutput = new PrintStream(baos)) {
System.setOut(temporalOutput);
ThatGuy thatGuy = new ThatGuy();
thatGuy.sayHi();
result = new String(baos.toByteArray(), StandardCharsets.UTF_8);
} finally {
System.setOut(normalOutput);
}
assertEquals("Hi", result.trim());
}
Second one is to use logger instead of just System.out. I consider this approach better not only from testing, but from code design perspective as well. Using this one you can just replace logger with Mockito.mock and user Mockito.verify to check what was called on your logger.
#Test
void whenSayHi_thenCallLogger() {
Logger logger = Mockito.mock(Logger.class);
ThatGuy thatGuy = new ThatGuy();
ReflectionTestUtils.setField(thatGuy, "logger", logger);
thatGuy.sayHiToLog();
verify(logger).error("Hi");
}
Class under testing looks like this:
class ThatGuy {
private static Logger logger = LoggerFactory.getLogger(ThatGuy.class);
void sayHi() {
System.out.println("Hi");
}
void sayHiToLog() {
logger.error("Hi");
}
}

Spring Batch ExecutionContext deserialization for PostGreSQL JOB_EXECUTION_CONTEXT failing

I am trying to create a Spring Batch POC with Java Configuration and PostGreSQL.
I have successfully created beans that would have otherwise been provided via the in memory DB using #EnableBatchProcessing and #EnableAutoConfiguration.
I am not able to get the beans (JobExplorer) to return a JobExecution list given a JobInstance bean created from the same JobExplorer bean.
The error I am getting is "Unable to deserialize the execution context" which seems to be coming from the method trying to deserialize the "SHORT_CONTEXT" field of the JOB_EXECUTION_CONTEXT table.
I have passed the created JobExplorer bean DefaultExecutionContextSerializer. Later passed a DefaultLobHandler with "wrapAsLob" set to True when I was still getting the error.
#Bean
public JobRegistry jobRegistry() {
JobRegistry jr = new MapJobRegistry();
return jr;
}
#Bean
public JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor() {
JobRegistryBeanPostProcessor jrbpp = new JobRegistryBeanPostProcessor();
jrbpp.setJobRegistry(jobRegistry());
return jrbpp;
}
#Bean
public JobOperator jobOperator() {
SimpleJobOperator sjo = new SimpleJobOperator();
sjo.setJobExplorer(jobExplorer());
sjo.setJobLauncher(jobLauncher());
sjo.setJobRegistry(jobRegistry());
sjo.setJobRepository(jobRepository());
return sjo;
}
#Bean
public JobExplorer jobExplorer() {
JobExplorerFactoryBean jefb = new JobExplorerFactoryBean();
jefb.setDataSource(dataSource());
jefb.setJdbcOperations(jdbcTemplate);
jefb.setTablePrefix("batch_");
jefb.setSerializer(new DefaultExecutionContextSerializer());
DefaultLobHandler lh = new DefaultLobHandler();
lh.setWrapAsLob(true);
jefb.setLobHandler(lh);
JobExplorer je = null;
try {
je = jefb.getObject();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return je;
}
#ConfigurationProperties(prefix = "spring.datasource")
#Bean
#Primary
public DataSource dataSource() {
return DataSourceBuilder.create().build();
}
#Bean
public JobRepository jobRepository() {
JobRepositoryFactoryBean jrfb = new JobRepositoryFactoryBean();
jrfb.setDataSource(dataSource());
jrfb.setDatabaseType("POSTGRES");
jrfb.setTransactionManager(new ResourcelessTransactionManager());
jrfb.setSerializer(new DefaultExecutionContextSerializer());
jrfb.setTablePrefix("batch_");
JobRepository jr = null;
try {
jr = (JobRepository)jrfb.getObject();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return jr;
}
Below is the get method in my rest controller where I am trying handle generate a list of failed Job executions
#Autowired
JobLauncher jobLauncher;
#Autowired
JobRegistry jobRegistry;
#Autowired
JobOperator jobOperator;
#Autowired
JobExplorer jobExplorer;
#SuppressWarnings("unchecked")
#GetMapping("batch/failedJobs")
public Map<String, List<JobExecution>> getFailedJobs() {
try {
if (jobRegistry == null || jobOperator == null || jobExplorer == null) {
System.out.println("job registry, operator or explorer is null");
} else {
Map<String, List<JobExecution>> allJobInstances = new HashMap<String, List<JobExecution>>();
// Get all jobs
jobRegistry.getJobNames().stream().forEach(jobName -> {
jobExplorer.getJobInstances(jobName, 1, 1000).forEach(l -> {
System.out.println("jobName: " + jobName + " instance: " + l);
});
jobExplorer.getJobInstances(jobName, 1, 1000).stream().forEach(jobInstance -> {
List<JobExecution> execultionList = jobExplorer.getJobExecutions(jobInstance); //Failing here
if (execultionList != null) {
System.out.println("" + execultionList);
execultionList.stream().forEach(l2 -> {
System.out.println("jobName: " + jobName + " instance: " + jobInstance
+ " jobExecution: " + l2);
});
if(allJobInstances.get(jobName) == null) {
allJobInstances.put(jobName, new ArrayList<JobExecution>());
}
allJobInstances.get(jobName).addAll((Collection<? extends JobExecution>) jobExplorer.getJobExecutions(jobInstance).stream().filter(e -> e.getStatus().equals(BatchStatus.FAILED)));
}else {
System.out.println("Could not get jobExecution for jobName " + jobName + " jobInstance: " + jobInstance);
}
});
});
return allJobInstances;
}
}catch (Exception e) {
System.out.println(e.getMessage());
logger.info(e.getMessage());
}
return null;
}
I fixed a similar issue by changing to the Jackson2 serializer:
jefb.setSerializer(new Jackson2ExecutionContextStringSerializer());
You may try it.

Javafx Task for Bluetooth data reciever

I am creating javafx application where I have this case that I need to listen for data sent over Bluetooth.
I have one fxml window on which I need to initialize Bluetooth and start listening from data.
Following is my Code for fxml controller:
//all imports
public class NewBarcodeInvoicePaneController implements Initializable{
private BluetoothController bc;
public BluetoothController getBc() {
return bc;
}
#Override
public void initialize(URL location, ResourceBundle resources) {
try {
bc = new BluetoothController();
new Thread(bc).start();
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
And BluetoothController is task where I initialize bluettoth and listen to the data
public class BluetoothController extends Task<Void> {
#Override
protected Void call() throws Exception {
LocalDevice local = null;
StreamConnectionNotifier notifier;
StreamConnection connection = null;
// setup the server to listen for connection
try {
local = LocalDevice.getLocalDevice();
try {
local.setDiscoverable(DiscoveryAgent.GIAC);
} catch (BluetoothStateException e) {
}
UUID uuid = new UUID(80087355); // "04c6093b-0000-1000-8000-00805f9b34fb"
String url = "btspp://localhost:" + uuid.toString() + ";name=RemoteBluetooth";
notifier = (StreamConnectionNotifier) Connector.open(url);
} catch (Exception e) {
e.printStackTrace();
return null;
}
try {
System.err.println("THIS IS HAPENING");
connection = notifier.acceptAndOpen();
System.err.println("HAPENING???????????????????????????");
InputStream inputStream = connection.openInputStream();
BufferedReader bReader = new BufferedReader(new InputStreamReader(inputStream));
String lineRead = bReader.readLine();
connection.close();
inputStream.close();
notifier.close();
local.setDiscoverable(DiscoveryAgent.NOT_DISCOVERABLE);
JSONParser parser = new JSONParser();
Object obj = parser.parse(lineRead);
JSONArray array = (JSONArray) obj;
array.stream().map((o) -> (String) o).forEach((stringObj) -> {
System.out.println(stringObj);
});
System.out.println("AFTER DATA RECIEVED");
} catch (Exception e) {
e.printStackTrace();
return null;
}
return null;
}
}
It Works fine if I send data over bluetooth and blocking call to notifier.acceptAndOpen() is unblocked.
My problem is when we do not pass any data and I just want to close the window I opened..
It still have blocking call open with extra thread by the task.
I tried to cancel BluetoothController task in Main controller where I open this window like following
private void openNewBarcodeInvoicePane(ActionEvent ae) {
//following are custom classes to open windows from fxml and getting controller back for further manipulation
PostoryModalWindow modalWindow = new PostoryModalWindow();
modalWindow.openNewModalPaneWithParent("New Invoice", "fxml/newbarcodeinvoicepane.fxml", ae);
//getting controller object
NewBarcodeInvoicePaneController controller = (NewBarcodeInvoicePaneController) modalWindow.getDswFromController();
controller.getWindowStage().showAndWait();
BluetoothController bc = controller.getBc();
if(bc != null){
System.err.println("CANCELLING");
bc.cancel(true);
}
}
But it doesn't throw InterrupttedExeption (In which I might have Choice to close Bluetooth thread) and after research I found that waiting on Socket doesn't work on interrupt.
Any help on this?
Thanks
Got Solution After Some Research.
I just added new task to call notifier.acceptAndOpen();
And added method to close Bluetooth notifier.
public class BluetoothController extends Task<Void> {
private final ObservableList<Item> items = FXCollections.observableArrayList();
public ObservableList<Item> getItems() {
return items;
}
StreamConnectionNotifier notifier;
#Override
protected Void call() throws Exception {
try {
BluetoothConnectionTask bct = new BluetoothConnectionTask(items);
new Thread(bct).start();
Thread.sleep(2000);
notifier = bct.getNotifier();
} catch (Exception e) {
e.printStackTrace();
return null;
}
return null;
}
public void cancelandExit() {
try {
if (notifier != null) {
notifier.close();
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
Here is new task for blocking call
public class BluetoothConnectionTask extends Task<Void>{
private StreamConnectionNotifier notifier;
private StreamConnection connection;
private ObservableList<Item> items = FXCollections.observableArrayList();
public StreamConnection getConnection() {
return connection;
}
public StreamConnectionNotifier getNotifier() {
return notifier;
}
public BluetoothConnectionTask(ObservableList<Item> is){
items = is;
}
#Override
protected Void call() throws Exception {
try {
LocalDevice local = LocalDevice.getLocalDevice();
try {
local.setDiscoverable(DiscoveryAgent.GIAC);
} catch (BluetoothStateException e) {
}
UUID uuid = new UUID(80087355); // "04c6093b-0000-1000-8000-00805f9b34fb"
String url = "btspp://localhost:" + uuid.toString() + ";name=RemoteBluetooth";
notifier = (StreamConnectionNotifier) Connector.open(url);
} catch (Exception e) {
e.printStackTrace();
return null;
}
connection = notifier.acceptAndOpen();
InputStream inputStream = connection.openInputStream();
BufferedReader bReader = new BufferedReader(new InputStreamReader(inputStream));
String lineRead = bReader.readLine();
connection.close();
inputStream.close();
notifier.close();
LocalDevice local = LocalDevice.getLocalDevice();
local.setDiscoverable(DiscoveryAgent.NOT_DISCOVERABLE);
JSONParser parser = new JSONParser();
Object obj = parser.parse(lineRead);
JSONArray array = (JSONArray) obj;
ItemDAO idao = new ItemDAO();
array.stream().map((o) -> (String) o).forEach((stringObj) -> {
String barcode = (String) stringObj;
Item i = idao.getItemByBarCode(barcode);
System.err.println("Adding Item "+i.getName());
items.add(i);
});
System.out.println("AFTER DATA RECIEVED");
return null;
}
}
Now for cancelling closing my bluetooth thread I am calling cancelandExit() after window is closed.