I want to write the exceptions in the same file as the item writer but do not work
// MyLitner
public static class MySkipListener implements SkipListener<CSCiviqueDTO, CSCiviqueDTO> {
private BufferedWriter bw = null;
public MySkipListener(File file) throws IOException {
bw= new BufferedWriter(new FileWriter(file, true));
}
#Override
public void onSkipInRead(Throwable throwable) {
if (throwable instanceof FlatFileParseException) {
FlatFileParseException flatFileParseException = (FlatFileParseException) throwable;
System.out.println("onSkipInRead =========> :");
try {
bw.write(flatFileParseException.getInput() + "; Step Vérifiez les colonnes !!");
bw.newLine();
bw.flush();
// fileWriter.close();
} catch (IOException e) {
System.err.println("Unable to write skipped line to error file");
}
}
}
Related
I want to pass a dynamic file name to onSkipInRead.
example:
public MySkipListener(#Value("#{jobParameters['file']}") String file) throws IOException {
bw= new BufferedWriter(new FileWriter(file);
System.out.println("MySkipListener =========> :"+bw);
}
#Override
public void onSkipInRead(Throwable throwable) {
if (throwable instanceof FlatFileParseException) {
FlatFileParseException flatFileParseException = (FlatFileParseException) throwable;
try {
bw.write(flatFileParseException.getInput()+"; Step Vérifiez les colonnes !!");
bw.newLine();
bw.flush();
} catch (IOException e) {
System.err.println("Unable to write skipped line to error file"); }
}
}
Thank you.
You can create a constructor in your listener with a file and pass the file to it when you define the step. Here is an example:
class MySkipListener implements SkipListener<Integer, Integer> {
private FileWriter fileWriter;
public MySkipListener(File file) throws IOException {
this.fileWriter = new FileWriter(file);
}
// your onSkipInRead method
}
Then you pass the job parameter file to your listener when you define the step:
#Bean
#JobScope
public Step step(#Value("#{jobParameters['file']}") String file) throws IOException {
return stepBuilderFactory.get("step")
.<Integer, Integer>chunk(5)
.reader(itemReader())
.writer(itemWriter())
.listener(new MySkipListener(new File(file)))
.build();
}
Hope this helps.
I want to write skipper lines in first csv file and the result of processor in second file in one step but it not works !
My code :
// => Step cecStep1
#Bean
public Step cecStep1(StepBuilderFactory stepBuilders) throws IOException {
return stepBuilders.get("fileDecrypt")
.<CSCivique, String>chunk(100)
.reader(reader1())
.processor(processor1FileDecrypt())
.writer(writer1())
.faultTolerant()
.skip(Exception.class)
.skipLimit(100)
.listener(new MySkipListener())
.build();
}
// ##################################### Step SkipListener ###################################################
public static class MySkipListener implements SkipListener {
private BufferedWriter bw = null;
public MySkipListener(File file) throws IOException {
//this.fileWriter = new FileWriter(file);
bw= new BufferedWriter(new FileWriter(file, true));
System.out.println("MySkipListener =========> :"+file);
}
#Override
public void onSkipInRead(Throwable throwable) {
if (throwable instanceof FlatFileParseException) {
FlatFileParseException flatFileParseException = (FlatFileParseException) throwable;
System.out.println("onSkipInRead =========> :");
try {
bw.write(flatFileParseException.getInput()+"; Vérifiez les colonnes !!");
bw.newLine();
bw.flush();
// fileWriter.close();
} catch (IOException e) {
System.err.println("Unable to write skipped line to error file");
}
}
}
#Override
public void onSkipInWrite(CSCivique item, Throwable t) {
System.out.println("Item " + item + " was skipped due to: " + t.getMessage());
}
#Override
public void onSkipInProcess(CSCivique item, Throwable t) {
System.out.println("Item " + item + " was skipped due to: " + t.getMessage());
}
}
#Bean
public FlatFileItemWriter<String> writer1() {
return new FlatFileItemWriterBuilder<String>().name(greetingItemWriter)
.resource(new FileSystemResource("target/test-outputs/greetings.csv"))
.lineAggregator(new PassThroughLineAggregator<>()).build();
}
Tankyou !
In your processor, you can:
throw a skippable exception for invalid items so that the skip listener intercepts them and writes them to the specified file
let valid items go to the writer so that they are written as configured in the item writer
For example:
class MyItemProcessor implements ItemProcessor<Object, Object> {
#Override
public Object process(Object item) throws Exception {
if (shouldBeSkipped(item)) {
throw new MySkippableException();
}
// process item
return item;
}
}
Hope this helps.
I have code snippet below.
What I want is if getNames() method catch an exception
( ex. InterruptedException ),
want to check if Got InterruptedException !!! prints out or not.
There are some examples of testing exception for a method
which throws an exception in its method ( ex. String method1() throws InterruptedException {...} ) in the Internet.
But not this case. Does anyone have some thought or idea?
public class A {
public List<String> getNames()
{
String addess = "address1";
int age = 17;
List<String> names = null;
try {
names = getSomeNames(address, sex);
}
catch (InterruptedException | ExecutionException e) {
throw new MyCustomException(e);
}
catch(Exception e) {
throw new MyCustomException(e);
}
return names;
}
List<String> getSomeNames(String address, int sex) throws InterruptedException, ExecutionException
{
// ...
// throw exceptions... at some point
//
return names;
}
}
public class MyCustomException extends Exception {
public MyCustomException(Throwable e) {
if (e.getCause() instanceof InterruptedException) {
// write log
System.out.println("Got InterruptedException !!!");
}
else if (e.getCause() instanceof ExecutionException) {
// write log
System.out.println("Got ExecutionException!!!");
}
else {
// write log
}
}
}
I tried this but the test failed and got NullPointerException in catch block.
#Test
public void testException() {
A objA = spy(new A());
try {
doThrow(MyCustomException.class).when(objA).getNames();
objA.getNnames();
}
catch (Exception e) {
System.out.println(e.getCause().toString()); // ==> throws java.lang.NullPointerException here.
}
}
There are several ways to test it.
First solution is to replace System.out with different stream and read from it later. ( I don't like this approach )
#Test
void whenSayHi_thenPrintlnCalled() throws IOException {
PrintStream normalOutput = System.out;
String result;
try (ByteArrayOutputStream baos = new ByteArrayOutputStream(); PrintStream temporalOutput = new PrintStream(baos)) {
System.setOut(temporalOutput);
ThatGuy thatGuy = new ThatGuy();
thatGuy.sayHi();
result = new String(baos.toByteArray(), StandardCharsets.UTF_8);
} finally {
System.setOut(normalOutput);
}
assertEquals("Hi", result.trim());
}
Second one is to use logger instead of just System.out. I consider this approach better not only from testing, but from code design perspective as well. Using this one you can just replace logger with Mockito.mock and user Mockito.verify to check what was called on your logger.
#Test
void whenSayHi_thenCallLogger() {
Logger logger = Mockito.mock(Logger.class);
ThatGuy thatGuy = new ThatGuy();
ReflectionTestUtils.setField(thatGuy, "logger", logger);
thatGuy.sayHiToLog();
verify(logger).error("Hi");
}
Class under testing looks like this:
class ThatGuy {
private static Logger logger = LoggerFactory.getLogger(ThatGuy.class);
void sayHi() {
System.out.println("Hi");
}
void sayHiToLog() {
logger.error("Hi");
}
}
I have thousands of files generated each day which I want to stream using Kafka.
When I try to read the file, each line is taken as a separate message.
I would like to know how can I make each file's content as a single message in Kafka topic and with consumer how to write each message from Kafka topic in a separate file.
You can write your own serializer/deserializer for handling files.
For example :
Producer Props :
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, org.apache.kafka.common.serialization.StringSerializer);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, YOUR_FILE_SERIALIZER_URI);
Consumer Props :
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, org.apache.kafka.common.serialization.StringDeserializer);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, YOUR_FILE_DESERIALIZER_URI);
Serializer
public class FileMapSerializer implements Serializer<Map<?,?>> {
#Override
public void close() {
}
#Override
public void configure(Map configs, boolean isKey) {
}
#Override
public byte[] serialize(String topic, Map data) {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutput out = null;
byte[] bytes = null;
try {
out = new ObjectOutputStream(bos);
out.writeObject(data);
bytes = bos.toByteArray();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (out != null) {
out.close();
}
} catch (IOException ex) {
// ignore close exception
}
try {
bos.close();
} catch (IOException ex) {
// ignore close exception
}
}
return bytes;
}
}
Deserializer
public class MapDeserializer implements Deserializer<Map> {
#Override
public void close() {
}
#Override
public void configure(Map config, boolean isKey) {
}
#Override
public Map deserialize(String topic, byte[] message) {
ByteArrayInputStream bis = new ByteArrayInputStream(message);
ObjectInput in = null;
try {
in = new ObjectInputStream(bis);
Object o = in.readObject();
if (o instanceof Map) {
return (Map) o;
} else
return new HashMap<String, String>();
} catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
bis.close();
} catch (IOException ex) {
}
try {
if (in != null) {
in.close();
}
} catch (IOException ex) {
// ignore close exception
}
}
return new HashMap<String, String>();
}
}
Compose messages in the following form
final Object kafkaMessage = new ProducerRecord<String, Map>((String) <TOPIC>,Integer.toString(messageId++), messageMap);
messageMap will contain fileName as key and the file content as value.
Value can be serializable object.
Hence each message will contain a Map with File_Name versus FileContent map.Can be single value or multiple value.
I get "Invalid thread access" in below code. I am not sure where I have written wrong code. My main intention to write the code is to just display subtask (what is happening behind the scene) so I have added subtask before method called.
#Override
public void handleEvent(Event event)
{
if((event.keyCode == SWT.CR || event.keyCode == 13 || event.type == SWT.Selection) && btnAdd.isEnabled())
{
final PreferencesMO permo = new PreferencesMO();
permo.updatePreferences();
permo.updateDocumentNumber();
final ProjectMO pmo = new ProjectMO();
final CoverSheetMO csmo = new CoverSheetMO();
final CommonError cmerror = new CommonError();
final ParameterConfigurationMO pamo = new ParameterConfigurationMO();
final SnippetNew s = new SnippetNew();
final String projName = txtpname.getText();
Display.getDefault().asyncExec(new Runnable()
{
public void run()
{
try
{
new ProgressMonitorDialog(shell).run(true, true, new IRunnableWithProgress() {
#Override
public void run(final IProgressMonitor monitor) throws InvocationTargetException,
InterruptedException
{
monitor.beginTask("Import Data", IProgressMonitor.UNKNOWN);
monitor.subTask("Connecting to databse...");
for(int i=0;i<=100;i++)
{
s.method1(i);
}
//monitor.worked(1);
try { Thread.sleep(2000); } catch (Exception e) { }
monitor.subTask("Analysing Data...");
try { Thread.sleep(2000); } catch (Exception e) { }
if(!projName.equals(""))
{
monitor.subTask("Updating coversheet ...");
try { Thread.sleep(2000); } catch (Exception e) { }
cmerror.updateCoverSheetStatusforNewProject();
monitor.subTask("Inserting Project ...");
try { Thread.sleep(2000); } catch (Exception e) { }
pmo.addProjectManager(projName,"T");
monitor.subTask("Searching Project ID ...");
try { Thread.sleep(2000); } catch (Exception e) { }
String p_id = pmo.searchprojectID(projName);
permo.insertDocumentNumber(p_id);
monitor.subTask("Inserting data into coversheet ...");
try { Thread.sleep(2000); } catch (Exception e) { }
csmo.insertCoversheet(p_id);
pamo.insertParameterConfiguration(p_id);
PlatformUI.getWorkbench().getActiveWorkbenchWindow().getShell().setText("Demo Tool - "+projName);
IWorkbenchPage page = PlatformUI.getWorkbench().getActiveWorkbenchWindow().getActivePage();
AuditLogs view = (AuditLogs) page.findView(AuditLogs.ID);
IEditorPart editorPart = page.getActiveEditor();
StackedLambdaChartInput input = new StackedLambdaChartInput();
AnalysisResult_MetricsChartInput metricsinput = new AnalysisResult_MetricsChartInput();
StackedLambdaChart_HorizantalInput stackedhorizantalinput = new StackedLambdaChart_HorizantalInput();
AnalysisResult_Metrics_HorizantalChartInput metricshorizantalinput = new AnalysisResult_Metrics_HorizantalChartInput();
BarChartInput inpuit = new BarChartInput();
BarChart_HorizantalInput barchart_horizantalinput = new BarChart_HorizantalInput();
AuditLogMO auditlog = new AuditLogMO();
monitor.subTask("Fetching audit logs to display ...");
try { Thread.sleep(2000); } catch (Exception e) { }
java.util.List<java.util.List<String>> auditlogs = auditlog.searchAuditLog(null,null);
view.table(auditlogs);
try
{
handlerService.executeCommand(AuditLogView.ID, new Event());
handlerService.executeCommand(ErrorLogView.ID, new Event());
handlerService.executeCommand(DesignHierarchyHandler.ID, new Event());
if(myeditor != null)
{
if(myeditor instanceof CoverSheet)
{
handlerService.executeCommand(CoverSheetHandler.ID, new Event());
}
else if(myeditor instanceof ParameterConfigurations)
{
handlerService.executeCommand(ParameterConfigurationHandler.ID, new Event());
}
}
}
catch (ExecutionException | NotDefinedException | NotEnabledException | PartInitException| NotHandledException e1)
{
e1.printStackTrace();
}
Constant con = new Constant();
con.createNewProject();
}
//shell.close();
monitor.done();
}
});
}
catch (InvocationTargetException | InterruptedException e)
{
e.printStackTrace();
}
}
});
}
}
Put your progress monitor as below :
Display.getDefault().asyncExec( new Runnable()
{
IWorkbenchWindow win = PlatformUI.getWorkbench().getActiveWorkbenchWindow();
new ProgressMonitorDialog(shell).run(true, true, new IRunnableWithProgress() {
#Override
public void run(final IProgressMonitor monitor) throws InvocationTargetException,
InterruptedException
{
monitor.beginTask("Import Data", IProgressMonitor.UNKNOWN);
monitor.subTask("Connecting to databse...");
for(int i=0;i<=100;i++)
}
If you want the workbench page, that also has to be called inside a UI thread like above.
You can't access UI code in the background thread used for the IRunnableWithProgress code.
So you must get the values of controls in the UI thread before you run the progress dialog.
You also can't access things like IWorkbenchPage in the background thread. If you want to update UI objects from a non-UI thread, you need to use Display.asyncExec or Display.syncExec to run the updating code in the UI thread.