below is the code when user uploads a video from mobile application to S3
def uploadVideo = Action(parse.multipartFormData) { implicit request =>
try {
var height = 0
var width = 0
request.body.files.map { mov =>
var videoName = System.currentTimeMillis() + ".mpeg"
amazonS3Client.putObject(bucketVideos, videoName, mov.ref.file)
}
val map = Map("result" -> "success")
Ok(write(map))
} catch {
case e: Exception =>
Ok(write(Map("result" -> "error")))
}
}
the above code work fine but in case user cancel while uploading of video then error occurs
[error] play - Exception caught in RequestBodyHandler
java.nio.channels.ClosedChannelException: null
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:434) ~[netty.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:129) ~[netty.jar:na]
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99) ~[netty.jar:na]
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36) ~[netty.jar:na]
at org.jboss.netty.channel.Channels.write(Channels.java:725) ~[netty.jar:na]
at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71) ~[netty.jar:na]
and this doesn't go to catch block!!
1.can this is harmfull to server or not?(because it is not needed any response if error occours)
2.if yes, how to handle?
This is all happening in Play's internals that are handling parsing the body of the Request. In fact, during the upload to your server, you haven't even reached the try block yet because the file hasn't finished uploading. Only once the upload is complete do you have the TemporaryFile available.
So no, you can't catch this error, and why would you want to? The user closed the connection. They're not even waiting for a response, so why send one? Let Play handle it.
This is also not a good way of handling an upload, though. For small files, it's passable, but if someone is proxying a huge video upload through your server to S3, it's going to:
Take almost twice is long to serve the response (which will cause the user to hang while you upload to S3).
Block one of Play's threads for handling requests for the entire time that file is uploading to S3, and given enough of these uploads (not many at all), you will no longer be able to process requests until an upload has completed.
Consider at least creating a separate ExecutionContext to use for handling uploads, or even better, look into having the user upload directly to S3 via a signed form, which would remove the need to proxy the upload at all.
Related
I create an http request using PUT to get data from the server. I got this data from the server and transform it on a PDF file.
When I run it for the first time everything runs fine, but after some calls, I start to get timeout errors.
Sometimes, I need to restart the app to be able to receive HTTP requests again.
This is the code that I use.
func callGetPdfFromEndpointUsingNSMutableURLRequest() {
if codigoBarra == "" {
messageError = "Código não localizado"
showingAlert = true
redirectToPdfView = false
showingActivityIndicator = false
return
}
let serviceRepository = ServiceRepository()
// let codigo_barra = "d152d36914313fedfbf36842a7195b723"
let json: [String: Any] = ["codigoBarra":"\(codigoBarra)"]
let request: NSMutableURLRequest = serviceRepository.clientURLRequest(endpointPesquisa, typeAutho: .basic, parms: "", body: json as Dictionary<String, AnyObject>)
print("request: \(request)")
print("request.url: \(String(describing: request.url))")
serviceRepository.put(request, retryLogin: true, completion: {isOk,msgError,httpCode,needLogin, response in
if isOk {
tratarRequisicaoPdf(response)
} else {
print("erro no request - is not ok | - httpCode: \(httpCode)")
var stringResponse:String = ""
if response != nil {
stringResponse = String(data: response as! Data, encoding: .utf8)!
} else {
stringResponse = "Sem resposta do servidor, tempo limite da solicitação foi esgotado."
}
messageError = "\(stringResponse)"
print(messageError)
showingAlert = true
redirectToPdfView = false
semaphore.signal()
}
semaphore.wait()
showingActivityIndicator = false
})
}
This error is unstable, sometimes it shows, sometimes it don't appear.
The people working on backend was not able to detect any problems.
I got the following error:
2022-05-20 15:33:15.442419-0300 CDA[2016:38068] Task <147B6F7F-E46A-47D0-A258-D6F3E5417D7E>.<1> finished with error [-1001] Error Domain=NSURLErrorDomain Code=-1001 "Esgotou-se o tempo limite da solicitação." UserInfo={_kCFStreamErrorCodeKey=-2102, NSUnderlyingError=0x7fd0e5245520 {Error Domain=kCFErrorDomainCFNetwork Code=-1001 "(null)" UserInfo={_kCFStreamErrorCodeKey=-2102, _kCFStreamErrorDomainKey=4}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask <147B6F7F-E46A-47D0-A258-D6F3E5417D7E>.<1>, _NSURLErrorRelatedURLSessionTaskErrorKey=(
"LocalDataTask <147B6F7F-E46A-47D0-A258-D6F3E5417D7E>.<1>"
), NSLocalizedDescription=Esgotou-se o tempo limite da solicitação., NSErrorFailingURLStringKey=https://myurl.sp.gov.br/gedave/api/spservicos/v1/buscaRequisicaoExame, NSErrorFailingURLKey=https://myurl.sp.gov.br/gedave/api/spservicos/v1/buscaRequisicaoExame, _kCFStreamErrorDomainKey=4}
response::: nil
erro: response is nil
httpCode: 0
What can I do to try to detect what was causing the timeout errors?
Edit:
I added the source code that can be viewed here https://swiftfiddle.com/mz2dxw6z6bda7a6t44cryncrpi.
NEW EDIT:
Answers to comments created by #Rob
Good to know about the NSUrlMutableRequest
I will try to use the 'finishAndInvalidate' in my URLSession. I didn't know about that.
My problem is unpredictable. Now I start the app and the first call got an timeout, after the second call the app works. Sometimes it starts working, but after some requests, I got a timeout
In the absence of a MCVE, there is not enough here to diagnose or reproduce the problem.
That having been said, there are some interesting clues:
You are calling wait (on a semaphore). Eliminate the DispatchSemaphore and if you want to know when the requests are done, use DispatchGroup. But when you use the dispatch group, use notify when it is done, not wait. Avoid blocking threads unnecessarily. And never block the main thread.
Your network request is performed by ServiceRepository, which you have not shared with us. But in your fiddle link, you show us unrelated URLSession code that is specifying the main queue as its delegate queue. If ServiceRepository is doing something similar, that, combined with the wait, above, could easily deadlock your code.
So, eliminate the semaphore, avoid ever calling wait (whether semaphore or dispatch group), and the deadlock risk is eliminated.
That having been said, that is only one potential timeout risk. The other scenario might be that you are simply issuing too many requests for the URLSession to run them without timing out.
If that is the case, you have a few options:
Increase the timeout threshold (either of the request or the session).
Try bumping up the timeout values and see if that mitigates the issue.
Submit uploads in just-in-time manner.
E.g., you might have a process where, rather than initiating all the uploads at once, that you issue each upload upon the completion of the prior upload. Or you can wrap the upload in a custom, asynchronous, Operation subclass (which is complicated in its own right), and then tell the operation queue to only attempt 4 at a time, or whatever. There are many techniques to tackle this, but the idea is to prevent timeouts by not even submitting the upload until you are confident that the URLSession can perform them.
Use background URLSession with file-based uploads.
If you have uploads that are so slow that they risk timing out, it begs the question of whether you really want to force the user to keep the app running in order to let the uploads finish at all. If you use a background URLSession, the background daemon will not let them timeout, but rather will upload them when it can.
Refactoring this for background URLSession is a non-trivial exercise (the first time you do it, anyway). You have to abandon completion handlers, use delegate-methods API, use file-based uploads, etc. But it is an incredibly powerful way to let uploads proceed in the background, allowing the user to leave the app, etc.
I don't see full code but you should have a look into semaphore usage - probably wrong multithread logic leads to not complete you request and your completion is hovered for a long period and causes URLDataTask to produce the timeout error
Hi I followed this excellent tutorial on how to download (and resume-download) files to an iPhone.
Everything works, except that the service-provider I am downloading the files from somehow does not allow for download-resume to work.
Apple says that a download can be resumed only if the following conditions are met (see bullets below). Therefore my dwonload.cancel(byProducingResmeData: ..) possibly returns data = nil because of one of these points :
The resource has not changed since you first requested it
The task is an HTTP or HTTPS GET request
The server provides either the ETag or Last-Modified header (or both)
in its response
The server supports byte-range requests
The temporary file hasn’t been deleted by the system in response to
disk space pressure
At least one of the above points unfortunately seems to not be fulfilled by my service-provider (definitive answer from them hanging....).
Anyway - is there another way to resume a background-downloading Task in iOS ?
If I could persistently keep the URLSessionDownloadTask - this would help. But how would I do that ?
Here is my code that normally keeps the resumeData. But unfortunately, in my case, this data is always nil.
func pauseDownload(_ file: File) {
guard let download = activeDownloads[file.previewURL] else { return }
if download.isDownloading {
download.task?.cancel(byProducingResumeData: { data in
download.resumeData = data
// data unfortunately always nil !!!!!!!!!!!!!!!!!!!!!
})
download.isDownloading = false
}
}
recently I started to build some small web processing service using akka streams. It's quite simple, I'm pulling urls from redis, then I'm downloading those urls(they are images) later I'm processing images, and pushing them to s3 and some json to redis.
I'm downloading lot of different kinds of images from multiple sites, I'm getting whole bunch of errors like 404, Unexpected disconnect , Response Content-Length 17951202 exceeds the configured limit of 8388608, EntityStreamException: Entity stream truncation and redirects. With redirects I'm invoking requestWithRedirects with address founded in location header of response.
Part responsible for downloading is pretty much like this:
override lazy val http: HttpExt = Http()
def requestWithRedirects(request: HttpRequest, retries: Int = 10)(implicit akkaSystem: ActorSystem, materializer: FlowMaterializer): Future[HttpResponse] = {
TimeoutFuture(timeout, msg = "Download timed out!") {
http.singleRequest(request)
}.flatMap {
response => handleResponse(request, response, retries)
}.recoverWith {
case e: Exception if retries > 0 =>
requestWithRedirects(request, retries = retries - 1)
}
}
TimeoutFuture is quite simple it takes future and timeout. If future takes longer than timeout it returns other future with timeout exception.
The problem I'm having is: after some time I'm getting an error:
Message: RuntimeException: Exceeded configured max-open-requests value of [128] akka.http.impl.engine.client.PoolInterfaceActor$$anonfun$receive$1.applyOrElse in PoolInterfaceActor.scala::109
akka.actor.Actor$class.aroundReceive in Actor.scala::467
akka.http.impl.engine.client.PoolInterfaceActor.akka$stream$actor$ActorSubscriber$$super$aroundReceive in PoolInterfaceActor.scala::46
akka.stream.actor.ActorSubscriber$class.aroundReceive in ActorSubscriber.scala::208
akka.http.impl.engine.client.PoolInterfaceActor.akka$stream$actor$ActorPublisher$$super$aroundReceive in PoolInterfaceActor.scala::46
akka.stream.actor.ActorPublisher$class.aroundReceive in ActorPublisher.scala::317
akka.http.impl.engine.client.PoolInterfaceActor.aroundReceive in PoolInterfaceActor.scala::46
akka.actor.ActorCell.receiveMessage in ActorCell.scala::516
akka.actor.ActorCell.invoke in ActorCell.scala::487
akka.dispatch.Mailbox.processMailbox in Mailbox.scala::238
akka.dispatch.Mailbox.run in Mailbox.scala::220
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec in AbstractDispatcher.scala::397
scala.concurrent.forkjoin.ForkJoinTask.doExec in ForkJoinTask.java::260
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask in ForkJoinPool.java::1339
scala.concurrent.forkjoin.ForkJoinPool.runWorker in ForkJoinPool.java::1979
scala.concurrent.forkjoin.ForkJoinWorkerThread.run in ForkJoinWorkerThread.java::107
I'm not sure what could be the problem but I think I have some downloads that were not finished properly and they stay in some global pool of connections after a while causing mentioned error. Any ideas what could be causing the problem? Or how to try find root of the problem: I already tested 404 responses, and Response Content-Length exceeds... errors, and they doesn't seem to be my troublemakers.
EDIT:
Most likely the problem is with my TimeoutFuture. I'm filling it with error as described here https://stackoverflow.com/a/29330010/2963977 but in my opinion future that is actually downloading an image never completes and it's taking my connection pool resources.
I wonder why those settings doesn't have any impact in my case :
akka.http.client.connecting-timeout = 1 s
akka.http.client.idle-timeout = 1 s
akka.http.host-connection-pool.idle-timeout = 1 s
EDIT2:
Apparently timeouts are not supported yet. Here is my bug report
https://github.com/akka/akka/issues/17732#issuecomment-112315953
I have a Play! Framework 2.3 project hosted on Heroku with the Postgres Addon.
It handles requests from mobile applications (Post a message).
For different reasons, I have duplicate (twice) rows (messages) in database :
the app might send the request twice in a short time ( less than 10ms )
I have multiple dynos that handle requests in parallel
Event if before writing in DB, I check the message does not exist yet. So I guess the first has still not been wrote when the second comes.
I also tried to write a message footprint in the memcache before handling the request (after form validation). But I still got twice messages sometimes.
The solutions I found are :
have a unique constraint on some database field (like a message timestamp client-side generated ?)
regularly check to remove duplicates
As I do not have means to update mobile apps I will script a regular check of duplicates.
Any other idea ?
What are the best practices to handle such concurrent requests ?
Attachement : my pseudo code
public static Result submit() {
User user = MySecured.getCurrentUser(ctx());
final Form<Message> filledForm = form(Message.class).bindFromRequest();
.... Some database pre-verification
if (filledForm.hasErrors()) {
ObjectNode error = Json.newObject();
error.put("error", filledForm.errorsAsJson());
return ok(error);
} else {
if(Cache.get(KEY_LOCK_FLASH_WRITING+filledForm.data().get("mail"))!=null){
return internalServerError();
}
//Verify this flash hasnt already been handled (requests can come twice from client)
Message sameMessage = Message.findSame(filledForm.get().mail, filledForm.get().message);
if(sameMessage!=null){
Logger.info("[Submit] message already exists" + sameMessage.id);
ObjectNode jsonResult = Json.newObject();
.... Processing a result ... no matter this does not happen.
return ok(jsonResult);
}
final Message flash = filledForm.get();
Cache.set(KEY_LOCK_FLASH_WRITING+flash.mail, "");
... some fields initializations like flash.author = new Author();
... Then some promises
return ok();
}
}
What I'm Trying To Do
I'm trying to create a solution of any kind that will run nightly on a Windows server, authenticate to a website, check a web page on the site for new links indicating a new version of a zip file, use new links (if present) to download a zip file, unzip the downloaded file to an existing folder on the server, use the unzipped contents (sql scripts, etc.) to build an instance of a database, and log everything that happens to a text file.
Forms App: The Part That Sorta Works
I created a Windows Forms app that uses a couple of WebBrowser controls, a couple of threads, and a few timers to do all that except the running nightly. It works great as a Form when I'm logged in and run it, but I need to get it (or something like it) to run on it's own like a Service or scheduled task.
My Service Attempt
So, I created a Windows Service that ticks every hour and, if the System.DateTime.Now.Hour >= 22, attempts to launch the Windows Forms app to do it's thing. When the Service attempts to launch the Form, this error occurs:
ActiveX control '8856f961-340a-11d0-a96b-00c04fd705a2' cannot be instantiated because the current thread is not in a single-threaded apartment.
which I researched and tried to resolve by either placing the [STAThread] attribute on the Main method of the Service's Program class or using some code like this in a few places including the Form constructor:
webBrowseThread = new Thread(new ThreadStart(InitializeComponent));
webBrowseThread.SetApartmentState(ApartmentState.STA);
webBrowseThread.Start();
I couldn't get either approach to work. In the latter approach, the controls on the Form (which would get initialized inside IntializeComponent) don't get initialized and I get null reference exceptions.
My Scheduled Task Attempt
So, I tried creating a nightly scheduled task using my own credentials to run the Form locally on my dev machine (just testing). It gets farther than the Service did, but gets hung up at the File Download Dialog.
Related Note: To send the key sequences to get through the File Download and File Save As dialogs, my Form actually runs a couple of vbscript files that use WScript.Shell.SendKeys. Ok, that's embarassing to admit, but I tried a few different things including SendMessage in Win32 API and referencing IWshRuntimeLibrary to use SendKeys inside my C# code. When I was researching how to get through the dialogs, the Win32 API seemed to be the recommended way to go, but I couldn't figure it out. The vbscript files was the only thing I could get to work, but I'm worried now that this may be the reason why a scheduled task won't work.
Regarding My Choice of WebBrowser Control
I have read about the System.WebClient class as an alternative to the WebBrowser control, but at a glance, it doesn't look like it has what I need to get this done. For example, I needed (or I think I needed) the WebBrowser's DocumentCompleted and FileDownload events to handle the delays in pages loading, files downloading, etc. Is there more to WebClient that I'm not seeing? Is there another class besides WebBrowser that is more Service-friendly and would do the trick?
In Summary
Geez, this is long. Sorry! It would help to even have a high level recommendation for a better way to do what I'm trying to do, because nothing I've tried has worked.
Update 10/22/09
Well, I think I'm closer, but I'm stuck again. I should end up with a decent-sized zip file with several files in it, but the zip file resulting from my code is empty. Here's my code:
// build post request
string targetHref = "http://wwwcf.nlm.nih.gov/umlslicense/kss/login.cfm";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(targetHref);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
// encoding to use
Encoding enc = Encoding.GetEncoding(1252);
// build post string containing authentication information and add to post request
string poststring = "returnUrl=" + fixCharacters(targetDownloadFileUrl);
poststring += getUsernameAndPasswordString();
poststring += "&login2.x=0&login2.y=0";
// convert to required byte array
byte[] postBytes = enc.GetBytes(poststring);
request.ContentLength = postBytes.Length;
// write post to request
Stream postStream = request.GetRequestStream();
postStream.Write(postBytes, 0, postBytes.Length);
postStream.Close();
// get response as stream
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
// writes stream to zip file
FileStream writeStream = new FileStream(fullZipFileName, FileMode.Create, FileAccess.Write);
ReadWriteStream(responseStream, writeStream);
response.Close();
responseStream.Close();
The code for ReadWriteStream looks like this.
private void ReadWriteStream(Stream readStream, Stream writeStream)
{
// taken verbatum from http://www.developerfusion.com/code/4669/save-a-stream-to-a-file/
int Length = 256;
Byte[] buffer = new Byte[Length];
int bytesRead = readStream.Read(buffer, 0, Length);
// write the required bytes
while (bytesRead > 0)
{
writeStream.Write(buffer, 0, bytesRead);
bytesRead = readStream.Read(buffer, 0, Length);
}
readStream.Close();
writeStream.Close();
}
The building of the post string is taken from my previous forms app that works. I compared the resulting values in poststring for both sets of code (my working forms app and this one) and they're identical.
I'm not even sure how to troubleshoot this further. Anyone see anything obvious as to why this isn't working?
Conclusion 10/23/09
I finally have this working. A couple of important hurdles I had to get over. I had some problems with the ReadWriteStream method code that I got online. I don't know why, but it wasn't working for me. A guy named JB in Claudio Lassala's Virtual Brown Bag meeting helped me to come up with this code which worked much better for my purposes:
private void WriteResponseStreamToFile(Stream responseStreamToRead, string zipFileFullName)
{
// responseStreamToRead will contain a zip file, write it to a file in
// the target location at zipFileFullName
FileStream fileStreamToWrite = new FileStream(zipFileFullName, FileMode.Create);
int readByte = responseStreamToRead.ReadByte();
while (readByte != -1)
{
fileStreamToWrite.WriteByte((byte)readByte);
readByte = responseStreamToRead.ReadByte();
}
fileStreamToWrite.Flush();
fileStreamToWrite.Close();
}
As Will suggested below, I did have trouble with the authentication. The following code is what worked to get around that issue. A few comments inserted addressing key issues I ran into.
string targetHref = "http://wwwcf.nlm.nih.gov/umlslicense/kss/login.cfm";
HttpWebRequest firstRequest = (HttpWebRequest)WebRequest.Create(targetHref);
firstRequest.AllowAutoRedirect = false; // this is critical, without this, NLM redirects and the whole thing breaks
// firstRequest.Proxy = new WebProxy("127.0.0.1", 8888); // not needed for production, but this helped in order to debug the http traffic using Fiddler
firstRequest.Method = "POST";
firstRequest.ContentType = "application/x-www-form-urlencoded";
// build post string containing authentication information and add to post request
StringBuilder poststring = new StringBuilder("returnUrl=" + fixCharacters(targetDownloadFileUrl));
poststring.Append(getUsernameAndPasswordString());
poststring.Append("&login2.x=0&login2.y=0");
// convert to required byte array
byte[] postBytes = Encoding.UTF8.GetBytes(poststring.ToString());
firstRequest.ContentLength = postBytes.Length;
// write post to request
Stream postStream = firstRequest.GetRequestStream();
postStream.Write(postBytes, 0, postBytes.Length); // Fiddler shows that post and response happen on this line
postStream.Close();
// get response as stream
HttpWebResponse firstResponse = (HttpWebResponse)firstRequest.GetResponse();
// create new request for new location and cookies
HttpWebRequest secondRequest = (HttpWebRequest)WebRequest.Create(firstResponse.GetResponseHeader("location"));
secondRequest.AllowAutoRedirect = false;
secondRequest.Headers.Add(HttpRequestHeader.Cookie, firstResponse.GetResponseHeader("Set-Cookie"));
// get response to second request
HttpWebResponse secondResponse = (HttpWebResponse)secondRequest.GetResponse();
// write stream to zip file
Stream responseStreamToRead = secondResponse.GetResponseStream();
WriteResponseStreamToFile(responseStreamToRead, fullZipFileName);
responseStreamToRead.Close();
sl.logScriptActivity("Downloading update.");
firstResponse.Close();
I want to underscore that setting AllowAutoRedirect to false on the first HttpWebRequest instance was critical to the whole thing working. Fiddler showed two additional requests that occurred when this was not set, and it broke the rest of the script.
You're trying to use UI controls to do something in a windows service. This will never work.
What you need to do is just use the WebRequest and WebResponse classes to download the contents of the webpage.
var request = WebRequest.Create("http://www.google.com");
var response = request.GetResponse();
var stream = response.GetResponseStream();
You can dump the contents of the stream, parse the text looking for updates, and then construct a new request for the URL of the file you want to download. That response stream will then have the file, which you can dump on the filesystem and etc etc.
Before you wonder, GetResponse will block until the response returns, and the stream will block as data is being received, so you don't need to worry about events firing when everything has been downloaded.
You definitely need to re-think your approach (as you've already begun to do) to eliminate the Forms-based application approach. The service you're describing needs to operate with no UI at all.
I'm not familiar with the details of System.WebClient, but since it
provides common methods for sending
data to and receiving data from a
resource identified by a URI,
it will probably be your answer.
At first glance, WebClient.DownloadFile(...) or WebClient.DownloadFileAsync(...) will do what you need.
The only thing I can add is that once you have scraped your screen and have the fully qualified name of the file you want to download, you could pass it along to the Windows/DOS command 'get' which will fetch files via HTTP. You can also script a command-line FTP client if desired. It's been a long time since I tried something like this in Windows, but I think you're almost there. Once you have fetched the correct file, building a batch file to do everything else should be pretty easy. If you are more comfortable with Unix, google "unix services for windows" just keep an eye on the services they start running (DHCP, etc). There are some nice utilities which will let your treat dos as a unix-like shell (ls -l, grep, etc) Finally, you could try another language like Perl or Python but I don't think that's the kind of advice you were looking for. :)