Apache commons net FTP clients hangs unpredictably - apache-commons-net

We tried all the solutions provided in this post (FTP client hangs) but none of them is working. We are using version 3.6 of commons net. Sometimes it hangs while uploading a file, sometimes will checking existence of a directory. Max. file size is around 400 MB. But sometime it hangs even for a small file size < 1KB. Below is the fragment of code:
public boolean uploadData(String inputFilePath, String destinationFolderName) {
if (StringUtil.isNullOrBlank(inputFilePath) || StringUtil.isNullOrBlank(destinationFolderName)) {
LOGGER.error("Invalid parameters to uploadData. Aborting...");
return false;
}
boolean result = false;
FTPSClient ftpClient = getFTPSClient();
if (ftpClient == null) {
logFTPConnectionError();
return false;
}
try {
loginToFTPServer(ftpClient);
result = uploadFileToFTPServer(ftpClient, inputFilePath, destinationFolderName);
} catch (Exception e) {
logErrorUploadingFile(inputFilePath, e);
return false;
} finally {
try {
logoutFromFTPServer(ftpClient);
} catch (Exception e) {
logErrorUploadingFile(inputFilePath, e);
result = false;
}
}
return result;
}
private FTPSClient getFTPSClient() {
FTPSClient ftpClient = null;
try {
ftpClient = new FTPSClient();
LOGGER.debug("Connecting to FTP server...");
ftpClient.setConnectTimeout(connectTimeOut);
ftpClient.connect(server);
int reply = ftpClient.getReplyCode();
if (!FTPReply.isPositiveCompletion(reply)) {
ftpClient.disconnect();
LOGGER.error("Could not connect to FTP server. Aborting.");
return null;
}
} catch (Exception e) {
LOGGER.error("Could not connect to FTP server.", e);
return null;
}
return ftpClient;
}
private void loginToFTPServer(FTPSClient ftpClient) throws Exception {
ftpClient.setDataTimeout(DATA_TIMEOUT);
ftpClient.login(ftpUserName, ftpPassword);
ftpClient.enterLocalPassiveMode();
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
LOGGER.debug("FTP Client Buffer Size Before:" + ftpClient.getBufferSize());
ftpClient.setBufferSize(BUFFER_SIZE);
LOGGER.debug("FTP Client Buffer Size After:" + ftpClient.getBufferSize());
ftpClient.execPBSZ(0);
ftpClient.execPROT("P");
ftpClient.setControlKeepAliveTimeout(300);
LOGGER.debug("Logged into FTP server.");
}
private void logoutFromFTPServer(FTPSClient ftpClient) throws Exception {
LOGGER.debug("Logging out from FTP server.");
ftpClient.logout();
ftpClient.disconnect();
LOGGER.debug("FTP server connection closed.");
}
private boolean uploadFileToFTPServer(FTPSClient ftpClient, String inputFilePath, String destinationFolderName) {
boolean result = false;
String remoteLocationFile;
File ftpFile = new File(inputFilePath);
try (InputStream inputStream = new FileInputStream(ftpFile)) {
String fileName = ftpFile.getName();
remoteLocationFile = (destinationFolderName == null || destinationFolderName.isEmpty())
? ftpFile.getName()
: destinationFolderName + File.separator + fileName;
LOGGER.info("Storing file " + ftpFile.getName() + " of size "
+ ftpFile.length() + " in folder " + remoteLocationFile);
result = ftpClient.storeFile(remoteLocationFile, inputStream);
if(result) {
LOGGER.info("Successfully stored file " + ftpFile.getName() + " in folder " + remoteLocationFile);
} else {
LOGGER.error("Unable to store file " + ftpFile.getName() + " in folder " + remoteLocationFile);
}
return result;
} catch (Exception e) {
logErrorUploadingFile(inputFilePath, e);
}
return result;
}
The application is hosted in apache tomcat 8. What could be other causes of this issue and how should we fix them? This is crucial functionality of our application and we may even consider to use alternate API if that is stable. Please suggest.

Adding ftpClient.setSoTimeout(20000); has fixed the issue.

Adding a enterLocalPassiveMode right before the retreiveFile should solve this issue.
You also need to add
ftpClient.setControlKeepAliveTimeout(300);
or Check this code which will resolve the hanging issue

Related

(log) File watching with citrus-framework

Is there a way and/or what are best practices to watch log files from the System Under Test?
My requirement is to validate presence/absence of log entries according known patterns produced by the SUT.
Thank you very much!
Well, I don't think there is a Citrus tool specifically designed for that. But I think that is a really good idea. You could open an issue and ask for this feature.
Meanwhile, here is a solution that we have used in one of our projects to check if the applicaiton log contained specific strings that were generated by our test.
sleep(2000),
echo("Searching the log..."),
new AbstractTestAction() {
#Override
public void doExecute(TestContext context) {
try {
String logfile = FileUtils.getFileContentAsString(Paths.get("target", "my-super-service.log").toAbsolutePath().normalize());
if (!logfile.contains("ExpectedException: ... | Details: BOOM!.")) {
throw new RuntimeException("Missing exceptions in log");
}
} catch (IOException e) {
throw new RuntimeException("Unable to get log");
}
}
}
OR you can replace that simple contains with a more elegant solution:
String grepResult = grepForLine(LOGFILE_PATH, ".*: SupermanMissingException.*");
if (grepResult == null) {
throw new RuntimeException("Expected error log entry not found");
}
The function goes over each line searching for a match to the regex supplied.
public String grepForLine(Path path, String regex) {
Pattern regexp = Pattern.compile(regex);
Matcher matcher = regexp.matcher("");
String msg = null;
try (
BufferedReader reader = Files.newBufferedReader(path, Charset.defaultCharset());
LineNumberReader lineReader = new LineNumberReader(reader)
) {
String line;
while ((line = lineReader.readLine()) != null) {
matcher.reset(line); //reset the input
if (matcher.find()) {
msg = "Line " + lineReader.getLineNumber() + " contains the error log: " + line;
}
}
} catch (IOException e) {
throw new RuntimeException(e);
}
return msg;
}

Using websockets through the simple websockets for webgl asset in unity3d, can connect but can't transmit messages

Just having a problem on my mac trying to send strings over web sockets using this https://www.assetstore.unity3d.com/en/#!/content/38367
Lots of adapted code below from here mainly http://www.codepool.biz/how-to-implement-a-java-websocket-server-for-image-transmission-with-jetty.html and the web socket sharp echotest example.
I can connect but there is no sign of strings in my Jetty server console window (on a ws server running in java(eclipse)).
Iā€™m basically just trying to send a ā€œ1ā€ to my server over a websocket connection with the unity editor (5) at the moment, to prompt the server to start sending PNG files encoded as byte arrays, so I can put them back together in a C# script and apply them to a texture.
this is the script, I want to attach it to a game object like a plane or a cube and display the updating images sent over the web socket from my Jetty server, but at the moment I'm just stuck trying to send a message and see it pop up in my eclipse console window.
using UnityEngine;
using System.Collections;
using System;
public class socketTexture : MonoBehaviour {
// Use this for initialization
IEnumerator Start () {
WebSocket w = new WebSocket(new Uri("ws://192.168.0.149:8080/"));
yield return StartCoroutine(w.Connect());
Debug.Log ("Connected");
w.SendString("I'm client");
w.SendString("1");
while (true)
{
byte[] reply = w.Recv();
if (reply != null)
{
Debug.Log ("Received: "+reply);
var tex = new Texture2D(300, 300, TextureFormat.PVRTC_RGBA4, false);
// Load data into the texture and upload it to the GPU.
tex.LoadRawTextureData(reply);
tex.Apply();
// Assign texture to renderer's material.
GetComponent<Renderer>().material.mainTexture = tex;
}
if (w.Error != null)
{
Debug.LogError ("Error: "+w.Error);
break;
}
yield return 0;
}
w.Close();
}
}
...And the relevant code from the jetty server, but this works, I've tested it with some javascript and I can load the PNGs back into the browser window, so I'm definitely doing something wrong in Unity
#OnWebSocketMessage //part request from websocket client (remote browser)
public void onMessage( String message) {
System.out.println("message");
if (message.equals("1") || message.equals("2") || message.equals("3") || message.equals("4") ) {
System.out.println("Part " + message + " joined");
System.out.println( UIMain.usersPath + "/" + message + ".png" );
final String testVar = ( UIMain.usersPath + "/" + message + ".png" );
task = new FileWatcher( new File(testVar) ) {
protected void onChange( File file ) {
// here we code the action on a change
System.out.println( "File "+ file.getName() +" has changed!" );
try {
File f = new File(testVar);
BufferedImage bi = ImageIO.read(f);
ByteArrayOutputStream out = new ByteArrayOutputStream();
ImageIO.write(bi, "png", out);
ByteBuffer byteBuffer = ByteBuffer.wrap(out.toByteArray());
mSession.getRemote().sendBytes(byteBuffer);
out.close();
byteBuffer.clear();
}
catch (IOException e) {
e.printStackTrace();
}
}
};
Timer timer1 = new Timer(); {
timer1.schedule(task , new Date(), 40 );
}
}
else if (message.equals( "0")) {
zerocounter = zerocounter + 1;
if (zerocounter >= 2) {
task.cancel();
}
}
else if (message.equals( "Hi there, client here")) {
System.out.println( "Client says: " + message );
}
}
Any help would be really appreciated, been lurking on here for years, hopefully getting to the stage soon where I can help out others a bit too.
Benedict
Edit:
This is my console error message in Unity
FormatException: Invalid length. System.Convert.FromBase64String
(System.String s) (at
/Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System/Convert.cs:146)
EchoTest+c__Iterator0.MoveNext () (at
Assets/Example/EchoTest.cs:11)
I'm pretty sure the problem arises from websocket sharp for webgl. I need to send the message as a byte array.
OK So Joakim Erdfelt was right, the server was not configured to handle Byte[] messages. Here's what I added to fix it:
#OnWebSocketMessage
public void onMessage(byte[] buffer, int offset, int length) throws UnsupportedEncodingException {
System.out.println(buffer);
String sFclientOutStr = new String(buffer, "UTF-8");
sFclientOut = Integer.parseInt(sFclientOutStr);
System.out.println(sFclientOut);
if ((sFclientOut > 0) & (sFclientOut < 500)) {
System.out.println("Part " + sFclientOut + " joined");
System.out.println( UIMain.usersPath + "/" + sFclientOutStr + ".png" );
final String testVar = ( UIMain.usersPath + "/" + sFclientOutStr + ".png" );
task = new FileWatcher( new File(testVar) ) {
protected void onChange( File file ) {
// here we code the action on a change
System.out.println( "File "+ file.getName() +" has changed!" );
try {
File f = new File(testVar);
BufferedImage bi = ImageIO.read(f);
ByteArrayOutputStream out = new ByteArrayOutputStream();
ImageIO.write(bi, "png", out);
ByteBuffer byteBuffer = ByteBuffer.wrap(out.toByteArray());
mSession.getRemote().sendBytes(byteBuffer);
out.close();
byteBuffer.clear();
}
catch (IOException e) {
e.printStackTrace();
}
}
};
Timer timer1 = new Timer(); {
timer1.schedule(task , new Date(), 40 );
}
}
else if (sFclientOutStr.equals("0")) {
zerocounter = zerocounter + 1;
if (zerocounter >= 2) {
task.cancel();
}
}
else if (sFclientOutStr.equals( "I'm client")) {
System.out.println( "Client says: " + sFclientOutStr );
}
}
These links helped explain it for me http://www.programcreek.com/java-api-examples/index.php?api=org.eclipse.jetty.websocket.api.annotations.OnWebSocketMessage
http://www.eclipse.org/jetty/documentation/current/jetty-websocket-api-annotations.html

GWT-RPC method returns empty list on success

I am creating a webpage having CellTable.I need to feed this table with data from hbase table.
I have written a method to retrieve data from hbase table and tested it.
But when I call that method as GWT asynchronous RPC method then rpc call succeeds but it returns nothing.In my case it returns empty list.The alert box show list's size as 0.
Following is the related code.
Please help.
greetingService.getDeviceIDData(new AsyncCallback<List<DeviceDriverBean>>(){
public void onFailure(Throwable caught) {
// Show the RPC error message to the user
System.out.println("RPC Call failed");
Window.alert("Data : RPC call failed");
}
public void onSuccess(List<DeviceDriverBean> result) {
//on success do something
Window.alert("Data : RPC call successful");
//deviceDataList.addAll(result);
Window.alert("Result size: " +result.size());
// Add a text column to show the driver name.
TextColumn<DeviceDriverBean> nameColumn = new TextColumn<DeviceDriverBean>() {
#Override
public String getValue(DeviceDriverBean object) {
Window.alert(object.getName());
return object.getName();
}
};
table.addColumn(nameColumn, "Name");
// Add a text column to show the device id
TextColumn<DeviceDriverBean> deviceidColumn = new TextColumn<DeviceDriverBean>() {
#Override
public String getValue(DeviceDriverBean object) {
return object.getDeviceId();
}
};
table.addColumn(deviceidColumn, "Device ID");
table.setRowCount(result.size(), true);
// more code here to add columns in celltable
// Push the data into the widget.
table.setRowData(0, result);
SimplePager pager = new SimplePager();
pager.setDisplay(table);
VerticalPanel vp = new VerticalPanel();
vp.add(table);
vp.add(pager);
// Add it to the root panel.
RootPanel.get("datagridContainer").add(vp);
}
});
Code to retrieve data from hbase (server side code)
public List<DeviceDriverBean> getDeviceIDData()
throws IllegalArgumentException {
List<DeviceDriverBean> deviceidList = new ArrayList<DeviceDriverBean>();
// Escape data from the client to avoid cross-site script
// vulnerabilities.
/*
* input = escapeHtml(input); userAgent = escapeHtml(userAgent);
*
* return "Hello, " + input + "!<br><br>I am running " + serverInfo +
* ".<br><br>It looks like you are using:<br>" + userAgent;
*/
try {
Configuration config = HbaseConnectionSingleton.getInstance()
.HbaseConnect();
HTable testTable = new HTable(config, "driver_details");
byte[] family = Bytes.toBytes("details");
Scan scan = new Scan();
int cnt = 0;
ResultScanner rs = testTable.getScanner(scan);
for (Result r = rs.next(); r != null; r = rs.next()) {
DeviceDriverBean deviceDriverBean = new DeviceDriverBean();
byte[] rowid = r.getRow(); // Category, Date, Sentiment
NavigableMap<byte[], byte[]> map = r.getFamilyMap(family);
Iterator<Entry<byte[], byte[]>> itrt = map.entrySet()
.iterator();
deviceDriverBean.setDeviceId(Bytes.toString(rowid));
while (itrt.hasNext()) {
Entry<byte[], byte[]> entry = itrt.next();
//cnt++;
//System.out.println("Count : " + cnt);
byte[] qual = entry.getKey();
byte[] val = entry.getValue();
if (Bytes.toString(qual).equalsIgnoreCase("account_number")) {
deviceDriverBean.setAccountNo(Bytes.toString(val));
} else if (Bytes.toString(qual).equalsIgnoreCase("make")) {
deviceDriverBean.setMake(Bytes.toString(val));
} else if (Bytes.toString(qual).equalsIgnoreCase("model")) {
deviceDriverBean.setModel(Bytes.toString(val));
} else if (Bytes.toString(qual).equalsIgnoreCase("driver_name")) {
deviceDriverBean.setName(Bytes.toString(val));
} else if (Bytes.toString(qual).equalsIgnoreCase("premium")) {
deviceDriverBean.setPremium(Bytes.toString(val));
} else if (Bytes.toString(qual).equalsIgnoreCase("year")) {
deviceDriverBean.setYear(Bytes.toString(val));
} else {
System.out.println("No match found");
}
/*
* System.out.println(Bytes.toString(rowid) + " " +
* Bytes.toString(qual) + " " + Bytes.toString(val));
*/
}
deviceidList.add(deviceDriverBean);
}
}
catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
catch (Exception e) {
// System.out.println("Message: "+e.getMessage());
e.printStackTrace();
}
return deviceidList;
}
Could this be lazy fetching on the server side by hbase. This means if you return the list hbase won't get a trigger to actually read the list and you will simple get an empty list. I don't know a correct solution, in the past I've seen a similar problem on GAE. This could by solved by simply asking the size of the list just before returning it to the client.
I don't have the exact answer, but I have an advise. In similar situation I put my own trace to check every step in my program.
On the server side before return put : System.out.println("size of table="+deviceidList.size());
You can put this trace in the loop for deviceidList;

How to solve ConnectionNotFoundException in JavaME

I'm trying to connect to the internet via my J2ME application on my Netbeans emulator with the following function which connects to a webpage and prints out its HTML using System.out.println().
function getHTML(String url)
{
HttpConnection httpConn = null;
InputStream is = null;
OutputStream os = null;
try
{
httpConn = (HttpConnection)Connector.open(url);
int respCode = httpConn.getResponseCode();
if (respCode == httpConn.HTTP_OK)
{
StringBuffer sb = new StringBuffer();
os = httpConn.openOutputStream();
is = httpConn.openDataInputStream();
int chr;
while ((chr = is.read()) != -1)
sb.append((char) chr);
System.out.println(sb.toString());
os.close();
is.close();
}
else
{
System.out.println("Error " + respCode);
}
httpConn.close();
}
catch(IOException ioex)
{
ioex.printStackTrace();
}
}
But I've been getting the following error;
javax.microedition.io.ConnectionNotFoundException: error 10051 in socket::open
at com.sun.midp.io.j2me.socket.Protocol.open0(), bci=0
at com.sun.midp.io.j2me.socket.Protocol.connect(), bci=209
at com.sun.midp.io.j2me.socket.Protocol.open(), bci=216
at com.sun.midp.io.j2me.socket.Protocol.openPrim(), bci=4
at com.sun.midp.io.j2me.http.Protocol.createConnection(), bci=41
at com.sun.midp.io.j2me.http.Protocol.connect(), bci=41
at com.sun.midp.io.j2me.http.Protocol.streamConnect(), bci=164
at com.sun.midp.io.j2me.http.Protocol.startRequest(), bci=7
at com.sun.midp.io.j2me.http.Protocol.sendRequest(), bci=33
at com.sun.midp.io.j2me.http.Protocol.sendRequest(), bci=3
at com.sun.midp.io.j2me.http.Protocol.getResponseCode(), bci=5
I know my code isn't the problem because this used to work on my old laptop, but it hasn't worked since I installed Netbeans on my new laptop. Is it because of my internet connection, or my firewall settings, or my settings in Netbeans, or did I just not install Netbeans properly?
If you want to send some data and receive means then try the coding
try
{
httpConn = (HttpConnection)Connector.open(url);
os = httpConn.openOutputStream();
//Writing data to os
os.write(b); //Here b is a byte array
os.flush();
int respCode = httpConn.getResponseCode();
if (respCode == httpConn.HTTP_OK)
{
StringBuffer sb = new StringBuffer();
is = httpConn.openDataInputStream();
int chr;
while ((chr = is.read()) != -1)
sb.append((char) chr);
System.out.println(sb.toString());
}
else
{
System.out.println("Error " + respCode);
}
}
catch(IOException ioex)
{
ioex.printStackTrace();
}
if(os!=null) os.close();
if(is!=null) is.close();
if(httpConn!=null) httpConn.close();
os=null;is=null;httpConn=null;
}

Apache Commons FileUpload only save a part of the file

I'm using a gwt widget gwtupload.client.Uploader, and i'm trying to save the file into a blob column in a database using fileupload streaming api.
The problem is that if the file is bigger than 3k only saves 3k (well 3.25K).
Thanks for the help.
Here is the code:
try {
ServletFileUpload upload = new ServletFileUpload();
upload.setProgressListener(listener);
FileItemIterator iter = upload.getItemIterator(request);
InputStream stream = null;
while (iter.hasNext()) {
FileItemStream item = iter.next();
String name = item.getFieldName();
stream = item.openStream();
Object o = getThreadLocalRequest().getSession().getAttribute(PortailServiceIMPL.FICHIER_SESSION_STORE_KEY);
if (o != null && o instanceof Fichier) {
uploadService.sauvegarder((Fichier) o, stream);
listener.update(listener.getContentLength(),
listener.getContentLength(), 0);
} else {
throw new RuntimeException(
"Impossible de recuperer le fichier de la session.");
}
}
if (stream != null) {
stream.close();
}
} catch (SizeLimitExceededException e) {
RuntimeException ex = new UploadSizeLimitException(
e.getPermittedSize(), e.getActualSize());
listener.setException(ex);
throw ex;
} catch (UploadSizeLimitException e) {
listener.setException(e);
throw e;
} catch (UploadCanceledException e) {
listener.setException(e);
throw e;
} catch (UploadTimeoutException e) {
listener.setException(e);
throw e;
} catch (Exception e) {
logger.error("UPLOAD-SERVLET (" + request.getSession().getId()
+ ") Unexpected Exception -> " + e.getMessage() + "\n"
+ stackTraceToString(e));
e.printStackTrace();
RuntimeException ex = new UploadException(e);
listener.setException(ex);
throw ex;
}
The lina that saves the file is :
uploadService.sauvegarder((Fichier) o, stream);
And there are like 4 methods after until reach the the code to save the InputStream (the InputStream is not touched):
public void storeBlob(long id, InputStream pInputStream) throws Exception {
try {
java.sql.Connection conn = //get the connection;
PreparedStatement ps = conn.prepareStatement(SQL_STORE);
ps.setBinaryStream(1, pInputStream, pInputStream.available());
ps.setLong(2, id);
ps.executeUpdate();
ps.close();
em.getTransaction().commit();
} catch (Throwable t) {
em.getTransaction().rollback();
throw new Exception(t);
}
}
If I use the FileItemFactory it worked, but that's not what I want:
FileItemFactory factory = new DiskFileItemFactory();
ServletFileUpload upload = new ServletFileUpload(factory);
List<FileItem> uploadedItems = upload.parseRequest(request);
for (FileItem item : uploadedItems) {
InputStream stream = item.getInputStream();
Thank you for your help.
After some work around, I solved it.
in the storeBlob method i can't use the pInputStream.available(). So, this is the line i used:
ps.setBinaryStream(1, pInputStream);