(log) File watching with citrus-framework - citrus-framework

Is there a way and/or what are best practices to watch log files from the System Under Test?
My requirement is to validate presence/absence of log entries according known patterns produced by the SUT.
Thank you very much!

Well, I don't think there is a Citrus tool specifically designed for that. But I think that is a really good idea. You could open an issue and ask for this feature.
Meanwhile, here is a solution that we have used in one of our projects to check if the applicaiton log contained specific strings that were generated by our test.
sleep(2000),
echo("Searching the log..."),
new AbstractTestAction() {
#Override
public void doExecute(TestContext context) {
try {
String logfile = FileUtils.getFileContentAsString(Paths.get("target", "my-super-service.log").toAbsolutePath().normalize());
if (!logfile.contains("ExpectedException: ... | Details: BOOM!.")) {
throw new RuntimeException("Missing exceptions in log");
}
} catch (IOException e) {
throw new RuntimeException("Unable to get log");
}
}
}
OR you can replace that simple contains with a more elegant solution:
String grepResult = grepForLine(LOGFILE_PATH, ".*: SupermanMissingException.*");
if (grepResult == null) {
throw new RuntimeException("Expected error log entry not found");
}
The function goes over each line searching for a match to the regex supplied.
public String grepForLine(Path path, String regex) {
Pattern regexp = Pattern.compile(regex);
Matcher matcher = regexp.matcher("");
String msg = null;
try (
BufferedReader reader = Files.newBufferedReader(path, Charset.defaultCharset());
LineNumberReader lineReader = new LineNumberReader(reader)
) {
String line;
while ((line = lineReader.readLine()) != null) {
matcher.reset(line); //reset the input
if (matcher.find()) {
msg = "Line " + lineReader.getLineNumber() + " contains the error log: " + line;
}
}
} catch (IOException e) {
throw new RuntimeException(e);
}
return msg;
}

Related

How to convert Parquet file to Protobuf and save it HDFS/AWS S3

I have a file which is in Parquet format. I want to read it and save it in HDFS or AWS S3 in Protobuf format using spark with Scala. I am not sure of any way. Searched many blogs but could not understand anything, can anyone help?
You can use ProtoParquetReader, which is ParquetReader with ProtoReadSupport.
Something like:
try (ParquetReader reader = ProtoParquetReader.builder(path).build()
) {
while ((model = reader.read()) != null){
System.out.println("check model " + "-- " + model);
...
}
} catch (IOException e) {
e.printStackTrace();
}
In order to read from parquet you need to use the following code :
public List<Record> read(Path path) {
List<Record> records = new ArrayList<>();
ParquetReader<Record> reader = AvroParquetReader<Record>builder(path).withConf(new Configuration()).build();
for (Record value = reader.read(); value != null; value = reader.read()) {
records.add(value);
}
return records;
}
Writing to a file from parquet would be something like this. Although this is not the protobuf file this might help you get started. Have in mind that you will have issues if you end up using spark-stream with protobuf v2.6 and greater
public void write(List<Record> records, String location) throws IOException {
Path filePath = new Path(location);
try (ParquetWriter<Record> writer = AvroParquetWriter.<GenericData.Record>builder(filePath)
.withSchema(getSchema()) //
.withConf(getConf()) //
.withCompressionCodec(CompressionCodecName.SNAPPY) //
.withWriteMode(Mode.CREATE) //
.build()) {
for (Record record : records) {
writer.write(record);
}
} catch (Exception e) {
e.printStackTrace();
}
}

Apache commons net FTP clients hangs unpredictably

We tried all the solutions provided in this post (FTP client hangs) but none of them is working. We are using version 3.6 of commons net. Sometimes it hangs while uploading a file, sometimes will checking existence of a directory. Max. file size is around 400 MB. But sometime it hangs even for a small file size < 1KB. Below is the fragment of code:
public boolean uploadData(String inputFilePath, String destinationFolderName) {
if (StringUtil.isNullOrBlank(inputFilePath) || StringUtil.isNullOrBlank(destinationFolderName)) {
LOGGER.error("Invalid parameters to uploadData. Aborting...");
return false;
}
boolean result = false;
FTPSClient ftpClient = getFTPSClient();
if (ftpClient == null) {
logFTPConnectionError();
return false;
}
try {
loginToFTPServer(ftpClient);
result = uploadFileToFTPServer(ftpClient, inputFilePath, destinationFolderName);
} catch (Exception e) {
logErrorUploadingFile(inputFilePath, e);
return false;
} finally {
try {
logoutFromFTPServer(ftpClient);
} catch (Exception e) {
logErrorUploadingFile(inputFilePath, e);
result = false;
}
}
return result;
}
private FTPSClient getFTPSClient() {
FTPSClient ftpClient = null;
try {
ftpClient = new FTPSClient();
LOGGER.debug("Connecting to FTP server...");
ftpClient.setConnectTimeout(connectTimeOut);
ftpClient.connect(server);
int reply = ftpClient.getReplyCode();
if (!FTPReply.isPositiveCompletion(reply)) {
ftpClient.disconnect();
LOGGER.error("Could not connect to FTP server. Aborting.");
return null;
}
} catch (Exception e) {
LOGGER.error("Could not connect to FTP server.", e);
return null;
}
return ftpClient;
}
private void loginToFTPServer(FTPSClient ftpClient) throws Exception {
ftpClient.setDataTimeout(DATA_TIMEOUT);
ftpClient.login(ftpUserName, ftpPassword);
ftpClient.enterLocalPassiveMode();
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
LOGGER.debug("FTP Client Buffer Size Before:" + ftpClient.getBufferSize());
ftpClient.setBufferSize(BUFFER_SIZE);
LOGGER.debug("FTP Client Buffer Size After:" + ftpClient.getBufferSize());
ftpClient.execPBSZ(0);
ftpClient.execPROT("P");
ftpClient.setControlKeepAliveTimeout(300);
LOGGER.debug("Logged into FTP server.");
}
private void logoutFromFTPServer(FTPSClient ftpClient) throws Exception {
LOGGER.debug("Logging out from FTP server.");
ftpClient.logout();
ftpClient.disconnect();
LOGGER.debug("FTP server connection closed.");
}
private boolean uploadFileToFTPServer(FTPSClient ftpClient, String inputFilePath, String destinationFolderName) {
boolean result = false;
String remoteLocationFile;
File ftpFile = new File(inputFilePath);
try (InputStream inputStream = new FileInputStream(ftpFile)) {
String fileName = ftpFile.getName();
remoteLocationFile = (destinationFolderName == null || destinationFolderName.isEmpty())
? ftpFile.getName()
: destinationFolderName + File.separator + fileName;
LOGGER.info("Storing file " + ftpFile.getName() + " of size "
+ ftpFile.length() + " in folder " + remoteLocationFile);
result = ftpClient.storeFile(remoteLocationFile, inputStream);
if(result) {
LOGGER.info("Successfully stored file " + ftpFile.getName() + " in folder " + remoteLocationFile);
} else {
LOGGER.error("Unable to store file " + ftpFile.getName() + " in folder " + remoteLocationFile);
}
return result;
} catch (Exception e) {
logErrorUploadingFile(inputFilePath, e);
}
return result;
}
The application is hosted in apache tomcat 8. What could be other causes of this issue and how should we fix them? This is crucial functionality of our application and we may even consider to use alternate API if that is stable. Please suggest.
Adding ftpClient.setSoTimeout(20000); has fixed the issue.
Adding a enterLocalPassiveMode right before the retreiveFile should solve this issue.
You also need to add
ftpClient.setControlKeepAliveTimeout(300);
or Check this code which will resolve the hanging issue

Printing to console removes hyperlinks created already

I was trying to create a feature to create hyperlinks in the console while the information is added.
While debugging, I noticed that the first hyperlink is succesfully created, but when the next line is printed to console, the hyperlink dissapears.
The code which I use to create the hyperlinks is:
String hyperLinkText = "test";
myConsole.printToConsole(test);
String myFile = new File("D:/Test/testFile.txt");
URI location = myFile.toURI();
IFile[] files = project.getWorkspace().getRoot().findFilesForLocationURI(location);
HyperlinkInformation hyperlinkInformation = new HyperlinkInformation(myConsole, myConsole.getDocument().get().length(), hyperLinkText.length());
FileHyperlink fileHyperlink = new FileHyperlink(hyperlinkInformation,
files[0], 1);
try {
hyperlinkInformation.getConsole().addHyperlink(fileHyperlink,
fileHyperlink.getOffset(), fileHyperlink.getLength());
} catch (BadLocationException e) {
//
}
The function addHyperlink:
public void addHyperlink(IHyperlink hyperlink, int offset, int length) throws BadLocationException {
IDocument document = getDocument();
ConsoleHyperlinkPosition hyperlinkPosition = new ConsoleHyperlinkPosition(hyperlink, offset, length);
try {
document.addPosition(ConsoleHyperlinkPosition.HYPER_LINK_CATEGORY, hyperlinkPosition);
fConsoleManager.refresh(this);
} catch (BadPositionCategoryException e) {
ConsolePlugin.log(e);
}
}
Is this the normal behaviour or am I missing something?

I have json values , how to save into mongodb using spring mvc

I have class like "location".
"Location" class have four field.
fields are:id,city,state,country..
country is seprate class it contains 2 field , country code, country name , 2 fields must read from location class..
if i write "locationMongoRepository.save()", then it shows error as bound mismatch. please give solution for how to save in mongodb.
public void insertLocation() throws InvalidFormatException, IOException, JSONException{
FileInputStream inp;
Workbook workbook;
try {
inp = new FileInputStream( "/home/Downloads/eclipse/Workspace/Samplboot-master latest/cityListForIndia1.xlsx" );
workbook = WorkbookFactory.create( inp );
Sheet sheet = workbook.getSheetAt(0);
JSONArray json = new JSONArray();
boolean isFirstRow = true;
ArrayList<String> rowName = new ArrayList<String>();
for ( Iterator<Row> rowsIT = sheet.rowIterator(); rowsIT.hasNext(); )
{
Row row = rowsIT.next();
JSONObject jRow = new JSONObject();
if(isFirstRow)
{
for ( Iterator<Cell> cellsIT = row.cellIterator(); cellsIT.hasNext(); )
{
Cell cell = cellsIT.next();
rowName.add(cell.getStringCellValue());
}
isFirstRow = false;
}
else
{
JSONObject jRowCountry= new JSONObject();
JSONObject jRowLocation= new JSONObject();
jRowLocation.put("city", row.getCell(0));
jRowLocation.put("state", row.getCell(1));
jRowCountry.put("country",row.getCell(2) );
jRowCountry.put("countryCode", row.getCell(3) );
jRowLocation.put("country", jRowCountry);
System.out.println("Location"+jRowLocation.toString());
}
}
}
catch (InvalidFormatException e) {
// TODO Auto-generated catch block
System.out.println("Invalid Format, Only Excel files are supported");
e.printStackTrace();
} catch (IOException e) {
System.out.println("Check if the input file exists and the path is correct");
e.printStackTrace();
} catch (JSONException e) {
// TODO Auto-generated catch block
System.out.println("Unable to generate Json");
e.printStackTrace();
}
}
}
I'm using Spring Data to support working with MongoDB and it's really helpful. You should read this article to get its idea and applied to your case https://dzone.com/articles/spring-data-mongodb-hello.
P/S: In case you can't use Spring Data to work with MongoDB, please provide more detail in your code/ your exception so we can investigate it more detail.

GWT-RPC method returns empty list on success

I am creating a webpage having CellTable.I need to feed this table with data from hbase table.
I have written a method to retrieve data from hbase table and tested it.
But when I call that method as GWT asynchronous RPC method then rpc call succeeds but it returns nothing.In my case it returns empty list.The alert box show list's size as 0.
Following is the related code.
Please help.
greetingService.getDeviceIDData(new AsyncCallback<List<DeviceDriverBean>>(){
public void onFailure(Throwable caught) {
// Show the RPC error message to the user
System.out.println("RPC Call failed");
Window.alert("Data : RPC call failed");
}
public void onSuccess(List<DeviceDriverBean> result) {
//on success do something
Window.alert("Data : RPC call successful");
//deviceDataList.addAll(result);
Window.alert("Result size: " +result.size());
// Add a text column to show the driver name.
TextColumn<DeviceDriverBean> nameColumn = new TextColumn<DeviceDriverBean>() {
#Override
public String getValue(DeviceDriverBean object) {
Window.alert(object.getName());
return object.getName();
}
};
table.addColumn(nameColumn, "Name");
// Add a text column to show the device id
TextColumn<DeviceDriverBean> deviceidColumn = new TextColumn<DeviceDriverBean>() {
#Override
public String getValue(DeviceDriverBean object) {
return object.getDeviceId();
}
};
table.addColumn(deviceidColumn, "Device ID");
table.setRowCount(result.size(), true);
// more code here to add columns in celltable
// Push the data into the widget.
table.setRowData(0, result);
SimplePager pager = new SimplePager();
pager.setDisplay(table);
VerticalPanel vp = new VerticalPanel();
vp.add(table);
vp.add(pager);
// Add it to the root panel.
RootPanel.get("datagridContainer").add(vp);
}
});
Code to retrieve data from hbase (server side code)
public List<DeviceDriverBean> getDeviceIDData()
throws IllegalArgumentException {
List<DeviceDriverBean> deviceidList = new ArrayList<DeviceDriverBean>();
// Escape data from the client to avoid cross-site script
// vulnerabilities.
/*
* input = escapeHtml(input); userAgent = escapeHtml(userAgent);
*
* return "Hello, " + input + "!<br><br>I am running " + serverInfo +
* ".<br><br>It looks like you are using:<br>" + userAgent;
*/
try {
Configuration config = HbaseConnectionSingleton.getInstance()
.HbaseConnect();
HTable testTable = new HTable(config, "driver_details");
byte[] family = Bytes.toBytes("details");
Scan scan = new Scan();
int cnt = 0;
ResultScanner rs = testTable.getScanner(scan);
for (Result r = rs.next(); r != null; r = rs.next()) {
DeviceDriverBean deviceDriverBean = new DeviceDriverBean();
byte[] rowid = r.getRow(); // Category, Date, Sentiment
NavigableMap<byte[], byte[]> map = r.getFamilyMap(family);
Iterator<Entry<byte[], byte[]>> itrt = map.entrySet()
.iterator();
deviceDriverBean.setDeviceId(Bytes.toString(rowid));
while (itrt.hasNext()) {
Entry<byte[], byte[]> entry = itrt.next();
//cnt++;
//System.out.println("Count : " + cnt);
byte[] qual = entry.getKey();
byte[] val = entry.getValue();
if (Bytes.toString(qual).equalsIgnoreCase("account_number")) {
deviceDriverBean.setAccountNo(Bytes.toString(val));
} else if (Bytes.toString(qual).equalsIgnoreCase("make")) {
deviceDriverBean.setMake(Bytes.toString(val));
} else if (Bytes.toString(qual).equalsIgnoreCase("model")) {
deviceDriverBean.setModel(Bytes.toString(val));
} else if (Bytes.toString(qual).equalsIgnoreCase("driver_name")) {
deviceDriverBean.setName(Bytes.toString(val));
} else if (Bytes.toString(qual).equalsIgnoreCase("premium")) {
deviceDriverBean.setPremium(Bytes.toString(val));
} else if (Bytes.toString(qual).equalsIgnoreCase("year")) {
deviceDriverBean.setYear(Bytes.toString(val));
} else {
System.out.println("No match found");
}
/*
* System.out.println(Bytes.toString(rowid) + " " +
* Bytes.toString(qual) + " " + Bytes.toString(val));
*/
}
deviceidList.add(deviceDriverBean);
}
}
catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
catch (Exception e) {
// System.out.println("Message: "+e.getMessage());
e.printStackTrace();
}
return deviceidList;
}
Could this be lazy fetching on the server side by hbase. This means if you return the list hbase won't get a trigger to actually read the list and you will simple get an empty list. I don't know a correct solution, in the past I've seen a similar problem on GAE. This could by solved by simply asking the size of the list just before returning it to the client.
I don't have the exact answer, but I have an advise. In similar situation I put my own trace to check every step in my program.
On the server side before return put : System.out.println("size of table="+deviceidList.size());
You can put this trace in the loop for deviceidList;