I am trying to create class that will handle multiple downloads at same time (I need to download a lot of small files) and I have problems with "disappearing" connections.
I have function addDonwload that adds url to list of urls to download, and checks if there is free download slot available. If there is one it starts download immediately. When one of downloads finishes, I pick first url form list and start new download.
I use NSURLConnection for downloading, here is some code
- (bool) TryDownload:(downloadInfo*)info
{
int index;
#synchronized(_asyncConnection)
{
index = [_asyncConnection indexOfObject:nullObject];
if(index != NSNotFound)
{
NSLog(#"downloading %# at index %i", info.url, index);
activeInfo[index] = info;
NSURLRequest *request = [NSURLRequest requestWithURL:info.url cachePolicy:NSURLRequestUseProtocolCachePolicy timeoutInterval:15];
[_asyncConnection replaceObjectAtIndex:index withObject:[[NSURLConnection alloc] initWithRequest:request delegate:self startImmediately:TRUE]];
//[[_asyncConnection objectAtIndex:i] scheduleInRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode];
return true;
}
}
return false;
}
- (void)connectionDidFinishLoading:(NSURLConnection*)connection
{
[self performSelectorOnMainThread:#selector(DownloadFinished:) withObject:connection waitUntilDone:false];
}
- (void)DownloadFinished:(id)connection
{
NSInteger index = NSNotFound;
#synchronized(_asyncConnection)
{
index = [_asyncConnection indexOfObject:(NSURLConnection*)connection];
}
[(id)activeInfo[index].delegate performSelectorInBackground:#selector(backgroundDownloadSucceededWithData:) withObject:_data[index]];
[_data[index] release];
[activeInfo[index].delegate release];
#synchronized(_asyncConnection)
{
[[_asyncConnection objectAtIndex:index] release];
[_asyncConnection replaceObjectAtIndex:index withObject:nullObject];
}
#synchronized(downloadQueue)
{
[downloadQueue removeObject:activeInfo[index]];
[self NextDownload];
}
}
- (void)NextDownload
{
NSLog(#"files remaining: %i", downloadQueue.count);
if(downloadQueue.count > 0)
{
if([self TryDownload:[downloadQueue objectAtIndex:0]])
{
[downloadQueue removeObjectAtIndex:0];
}
}
}
_asyncConnection is my array of download slots (NSURLConnections)
downloadQueue is list of urls to download
What happens is, at the beginning everything works ok, but after few downloads my connections start to disappear. Download starts but connection:didReceiveResponse: never gets called. There is one thing in output console that I don't understand I that might help a bit. Normaly there is something like
2010-01-24 21:44:17.504 appName[3057:207]
before my NSLog messages. I guess that number in square brackets is some kind of app:thread id? everything works ok while there is same number, but after some time, "NSLog(#"downloading %# at index %i", info.url, index);" messages starts having different that second number. And when that happens, I stop receiving any callbacks for that urlconnection.
This has been driving me nuts as I have strict deadlines and I can't find problem. I don't have many experiences with iphone dev and multithreaded apps. I have been trying different approaches so my code is kinda messy, but I hope you will see what I am trying to do here :)
btw is anyone of you know about existing class/lib I could use that would be helpful as well. I want parallel downloads with ability o dynamically add new files to download (so initializing downloader at the beginning with all urls is not helpful for me)
You've got a bunch of serious memory issues, and thread synchronization issues in this code.
Rather than go into them all, I'll ask the following question: You are doing this on a background thread of some kind? Why? IIRC NSURLConnection already does it's downloads on a background thread and calls your delegate on the thread that the NSURLConnection was created upon (e.g., your main thread ideally).
Suggest you step back, re-read NSURLConnection documentation and then remove your background threading code and all the complexity you've injected into this unnecessarily.
Further Suggestion: Instead of trying to maintain parallel positioning in two arrays (and some sketchy code in the above relating to that), make one array and have an object that contains both the NSURLConnection AND the object representing the result. Then you can just release the connection instance var when the connection is done. And the parent object (and thus the data) when you are done with the data.
I recommend that you take a look at this:
http://allseeing-i.com/ASIHTTPRequest/
It's a pretty sophisticated set of classes with liberal licensing terms (free too).
It may provide a lot of the functionality that you are wanting.
This snippet can be the source of the bug, you release the object pointed to by the activeInfo[index].delegate pointer right after issuing async method call on that object.
[(id)activeInfo[index].delegate performSelectorInBackground:#selector(backgroundDownloadSucceededWithData:) withObject:_data[index]];
[_data[index] release];
[activeInfo[index].delegate release];
Do you use connection:didFailWithError: ? There may be a timeout that prevents the successful download completion.
Try to get rid of the #synchronized blocks and see what happens.
The string inside the square brackets seems to be thread identifier as you guessed. So maybe you get locked in the #synchronized. Actually, I don't see a reason for switching thread - all the problematic code should run in the main thread (performSelectorOnMainThread)...
Anyhow, there is no need to use both the #synchronized and the performSelectorOnMainThread.
BTW, I didn't see the NSURLConnection *connection = [[NSURLConnection alloc] initWithRequest:request delegate:self]; line. Where do you initiate the connection?
As for the parallel downloads - I think that you can download more than one file in a time with the same code that you use here. Just create a separate connection for each download.
Consider just keeping a download queue along with a count of active connections, popping items off the top of the queue when downloads complete and a slot becomes free. You can then fire off NSURLConnection objects asynchronously and process events on the main thread.
If you find that your parallel approach prohibits doing all of the processing on the main thread, consider having intermediary manager objects between your main thread download code and NSURLConnection. Using that approach, you'd instantiate your manager and get it to use NSURLConnection synchronously on a background thread. That manager then completely deals with the downloading and passes the result back to its main thread delegate using a performSelectorOnMainThread:withObject: call. Each download is then just a case of creating a new manager object when you've a slot free and setting it going.
Related
I have four urls which consists images...
I'm downloding those images and placing them into documents folder..
here is my code..
-(void)viewDidLoad
{
NSMutableArray *myUrlsArray=[[NSMutableArray alloc]init];
[myUrlsArray addObject:#"http://blogs.sfweekly.com/thesnitch/steve_jobs3.jpg"];
[myUrlsArray addObject:#"http://www.droid-life.com/wp-content/uploads/2012/12/Steve-Jobs-Apple.jpg"];
[myUrlsArray addObject:#"http://2.bp.blogspot.com/-T6nbl0rQoME/To0X5FccuCI/AAAAAAAAEZQ/ipUU7JfEzTs/s1600/steve-jobs-in-time-magazine-front-cover.png"];
[myUrlsArray addObject:#"http://images.businessweek.com/ss/08/09/0929_most_influential/image/steve_jobs.jpg"];
[myUrlsArray addObject:#"http://cdn.ndtv.com/tech/gadget/image/steve-jobs-face.jpg"];
for (int i=0; i<myUrlsArray.count; i++)
{
[self downloadImageFromURL:[myUrlsArray objectAtIndex:i] withName:[NSString stringWithFormat:#"MyImage%i.jpeg",i]];
}
}
#pragma mark- downloading File
-(void)downloadImageFromURL:(NSString *)myURLString withName:(NSString *)fileName
{
UIImage *image = [[UIImage alloc] initWithData:[NSData dataWithContentsOfURL:[NSURL URLWithString:myURLString]]];
NSLog(#"%f,%f",image.size.width,image.size.height);
// Let's save the file into Document folder.**
NSString *documentsPath = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory,NSUserDomainMask, YES) objectAtIndex:0];
NSString *jpegPath = [NSString stringWithFormat:#"%#/%#",documentsPath,fileName];// this path if you want save reference path in sqlite
NSData *data2 = [NSData dataWithData:UIImageJPEGRepresentation(image, 1.0f)];//1.0f = 100% quality
[data2 writeToFile:jpegPath atomically:YES];
}
NOW... I need to display a UIProgressView for above downloading progress accurately.
how can i achieve this functionality...
Can any one provide some guidelines to achieve this..
Thanks in advance...
I'd suggest you use some asynchronous downloading technique (either AFNetworking, SDWebImage, or roll your own with delegate-based NSURLSession) rather than dataWithContentsOfURL so that (a) you don't block the main queue; and (b) you can get progress updates as the downloads proceed.
I'd also suggest creating a NSProgress for each download. When your delegate method gets updates about how many bytes have been downloaded, update the NSProgress object.
You then can associate each NSProgress with a observedProgress for a UIProgressView, and when you update your NSProgress, the UI can be updated automatically.
Or, if you and a single UIProgressView to show the aggregate progress of all of the NSProgress for each download, you can create a parent NSProgress, establish each download's NSProgress as a child of the parent NSProgress, and then, as each download updates its respective NSProgress, this will automatically trigger the calculation of the parent NSProgress. And again, you can tie that parent NSProgress to a master UIProgressView, and you'll automatically update the UI with the total progress, just by having each download update its individual NSProgress.
There is a trick, though, insofar as some web services will not inform you of the number of bytes to be expected. They'll report an "expected number of bytes" of NSURLResponseUnknownLength, i.e. -1! (There are logical reasons why it does that which are probably beyond the scope of this question.) That obviously makes it hard to calculate what percentage has been downloaded.
In that case, there are a few approaches:
You can throw up your hands and just use an indeterminate progress indicator;
You can try changing the request such that web service will report meaningful "expected number of bytes" values (e.g. https://stackoverflow.com/a/22352294/1271826); or
You can use an "estimated download size" to estimate the percentage completion. For example, if you know your images are, on average, 100kb each, you can do something like the following to update the NSProgress associated with a particular download:
if (totalBytesExpectedToWrite >= totalBytesWritten) {
self.progress.totalUnitCount = totalBytesExpectedToWrite;
} else {
if (totalBytesWritten <= 0) {
self.progress.totalUnitCount = kDefaultImageSize;
} else {
double written = (double)totalBytesWritten;
double percent = tanh(written / (double)kDefaultImageSize);
self.progress.totalUnitCount = written / percent;
}
}
self.progress.completedUnitCount = totalBytesWritten;
This is a bit of sleight of hand that uses the tanh function to return a "percent complete" value that smoothly and asymptotically approaches 100%, using the kDefaultImageSize as the basis for the estimation.
It's not perfect, but it yields a pretty decent proxy for percent completion.
Your call to dataWithContentsOfURL is synchronous, meaning you don't get updates as the download is in process.
You can use a library like AFNetworking (https://github.com/AFNetworking/AFNetworking) which has callbacks to the progress of the download.
Actually a better solution is to use SDWebImage manager which will load the images in the background for you and cache them. Then the next time you use that image it will check the cache. Google it.
That way the user also doesn't have to sit around and wait while you're downloading stuff..
Then look at this other question that has some ideas on how to do a status:
How to show an activity indicator in SDWebImage
Do not use dataWithContentsOfURL, you are blocking the main thread until the data arrives.
Instead create your own connection with NSURLConnection and start listening to your delegate.
connection:(NSURLConnection *)connection didReceiveResponse:(NSURLResponse *)response: get the total data size with [response expectedContentLength].
connection:(NSURLConnection *)connection didReceiveData:(NSData *)data: This is where you do your calculations and update your UIProgressView. Something like, loadedBytes/total data size.
Good luck.
folks! I'm implementing a shared cache in my app. The idea is to get the cached data from the web in the background and then update the cache and the UI with the newly retrieved data. The trick is of course to ensure thread-safety, since the main thread will be continuously using the cache. I don't want to modify the cache in any fashion while someone else might be using it.
It's my understanding that using #synchronized to lock access to a shared resource is not the most elegant approach in ObjectiveC due to it trapping to the kernel and thus being rather sluggish. I keep reading that using GCD instead is a great alternative (let's ignore its cousin NSOperation for now), and I'd like to figure out what a good pattern for my situation would be. Here's some sample code:
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
// download the data in a background thread
dispatch_async(queue, ^{
CacheData *data = [Downloader getLatestData];
// use the downloaded data in the main thread
dispatch_sync(dispatch_get_main_queue(), ^{
[AppCache updateCache:data];
[[NSNotificationCenter defaultCenter] postNotificationName:#"CacheUpdated" object:nil];
});
});
Would this actually do what I think it does, and if so, is this the cleanest approach as of today of handling this kind of situation? There's a blog post that's quite close to what I'm talking about, but I wanted to double-check with you as well.
I'm thinking that as long as I only ever access shared the shared resource on the same thread/queue (main in my case) and only ever update UI on main, then I will effectively achieve thread-safety. Is that correct?
Thanks!
Yes.
Other considerations aside, instead of shunting read/write work onto the main thread consider using a private dispatch queue.
dispatch_queue_t readwritequeue;
readwritequeue = dispatch_queue_create("com.myApp.cacheAccessQueue", NULL);
Then update your AppCache class:
- (void)updateCache:(id)data {
dispatch_sync(readwritequeue, ^{ ... code to set data ... });
}
- (id)fetchData:... {
__block id data = nil;
dispatch_sync(readwritequeue, ^{ data = ... code to fetch data ...});
return data;
}
Then update your original code:
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
// download the data in a background thread
dispatch_async(queue, ^{
CacheData *data = [Downloader getLatestData];
**[AppCache updateCache:data];**
// use the downloaded data in the main thread
dispatch_async(dispatch_get_main_queue(), ^{
[[NSNotificationCenter defaultCenter] postNotificationName:#"CacheUpdated" object:nil];
});
});
If you ask 100 developers here was is the most elegant way to do this, you will get at least 100 different answers (maybe more!)
What I do, and what is working well for me, is to have a singleton class doing my image management. I use Core Data, and save thumbnails directly in the store, but use the file system and a URL to it in Core Data for "large" files. Core Data is setup to use the new block based interface so it can do all its work on a private thread managed by itself.
Possible image URLS get registered with a tag on the main thread. Other classes can ask for the image for that tag. If the image is not there, nil is returned, but this class sets a fetchingFlag, uses a concurrent NSOperation coupled to a NSURLConnection to fetch the image, when it gets it it messages the singleton on its thread with the received image data, and the method getting that message uses '[moc performBlock:...]' (no wait) to process it.
When images are finally added to the repository, the moc dispatches a notification on the main queue with the received image tag. Classes that wanted the image can listen for this, and when they get it (on the main thread) they can then ask the moc for the image again, which is obviously there.
I am writing an application that periodically fetches data from a web server using ASI HTTP and then processes that data to display something relevant to the user on the UI. The data is retrieved from different requests on a single server. The data itself needs to be processed in a specific order. One of the blocks of data is much bigger than the other ones.
In order not to lock the UI while the data is being processed, I have tried to use the NSOperationQueue to run the data processing on different threads. This works fine about 90% of the times. However, in the remaining 10% of the time, the biggest block of data is being processed on the main thread, which cause the UI to block for 1-2 seconds. The application contains two MKMapViews in different tabs. When both MKMapViews tabs are loaded the percentage of time the biggest block of data is being processed on the main thread increases above 50% (which seems to point to the assumption that this happens when there is more concurrent activity).
Is there a way to prevent the NSOperationQueue to run code on the main thread?
I have tried to play with the NSOperationQueue –setMaxConcurrentOperationCount:, increasing and decreasing it but there was no real change on the issue.
This is the code that starts the periodic refresh:
- (void)refreshAll{
// Create Operations
ServerRefreshOperation * smallDataProcessor1Op = [[ServerRefreshOperation alloc] initWithDelegate:_smallDataProcessor1];
ServerRefreshOperation * smallDataProcessor2Op = [[ServerRefreshOperation alloc] initWithDelegate:_smallDataProcessor2];
ServerRefreshOperation * smallDataProcessor3Op = [[ServerRefreshOperation alloc] initWithDelegate:_smallDataProcessor3];
ServerRefreshOperation * smallDataProcessor4Op = [[ServerRefreshOperation alloc] initWithDelegate:_smallDataProcessor4];
ServerRefreshOperation * smallDataProcessor5Op = [[ServerRefreshOperation alloc] initWithDelegate:_smallDataProcessor5];
ServerRefreshOperation * hugeDataProcessorOp = [[ServerRefreshOperation alloc] initWithDelegate:_hugeDataProcessor];
// Create dependency graph (for response processing)
[HugeDataProcessorOp addDependency:smallDataProcessor4Op.operation];
[smallDataProcessor5Op addDependency:smallDataProcessor4Op.operation];
[smallDataProcessor4Op addDependency:smallDataProcessor3Op.operation];
[smallDataProcessor4Op addDependency:smallDataProcessor2Op.operation];
[smallDataProcessor4Op addDependency:smallDataProcessor1Op.operation];
// Start be sending all requests to server (startAsynchronous directly calls the ASIHTTPRequest startAsynchronous method)
[smallDataProcessor1Op startAsynchronous];
[smallDataProcessor2Op startAsynchronous];
[smallDataProcessor3Op startAsynchronous];
[smallDataProcessor4Op startAsynchronous];
[smallDataProcessor5Op startAsynchronous];
[hugeDataProcessorOp startAsynchronous];
}
This is the code that sets the ASI HTTP completion block that starts the data processing:
[_request setCompletionBlock:^{
[self.delegate setResponseString:_request.responseString];
[[MyModel queue] addOperation:operation]; // operation is a NSInvocationOperation that calls the delegate parse method
}];
I have added this block of code in all NSInvocationOperation Invoked method at the entry point:
if([NSThread isMainThread]){
NSLog(#"****************************Running <operation x> on Main thread");
}
The line is printed every time the UI freezes. This shows that the whole operation is run on the main thread. It is actually always the hugeDataProcessorOp that is run on the main thread. I assume that this is because it is the operation that always receives its answer last from the server.
After much investigation in my own code, I can confirm that this was a coding error.
There was an old call remaining that did not go through the NSInvocationOperation but was calling the selector that NSInvocationOperation should have called directly (therefore not using the concurrent NSOperationQueue.
This means that the NSOperationQueue DOES NOT use the main thread (except if it is the one retrieved by +mainQueue).
Override isConcurrent on your NSOperation and return YES. According to the documentation this will cause your NSOperation to be run asynchronously.
I am trying to download files from remote, and I now can monitor every single files download success status
[operation setCompletionBlockWithSuccess:^(AFHTTPRequestOperation *operation, id responseObject) {
But any way to monitor the whole process of downloading? How should I know all downloads are finished?
And I tried start download request with
[afhttpClient enqueueBatchOfHTTPRequestOperations:operationArray
progressBlock:progressBlock
completionBlock:completionBlock];
seems not work, so what the difference between above code and start download request with [operation start] in a loop?
#mattt (if you can see this)
You most probably have an NSArray of URL objects that you use to download images. What you need to do is to create an integer value equal to the count of your URL objects. Each time you successfully download an image or absolutely fail to download it (for instance after few timeouts or upon receiving 404 HTTP status code) you need to decrement that integer (note that it should be an atomic property, since blocks are being executed on different threads). Once the count reaches zero - all requests are finished. You can also use that integer to update a progress bar or simply notify user that "#/15 images are downloaded". Let me know if you need any other clarifications.
And unfortunately I have not worked with AFHTTPClient, so I can't tell you the difference between the two operations precisely, but contextually, first one executes all the requests almost at the same time asynchronously and the latter one uses consecutive approach, where second request will only be launched upon completion of the first one.
You can create NSOperationQueue and put all AFHTTPRequestOperation into it.
To observe the "operations" by using KVO. When the count go to zero, that is the time to say all operations completed.
If what you want is a constant progress update for each operation with bytes downloaded and total expected then I can highly recommend Peter Steinberger's AFDownloadRequestOperation.
This class derives from AFHTTPRequestOperation and adds a progressiveDownloadProgressBlock per-operation rather than just a per-operation-completion progress at the operation queue level, which is what I think you're looking for. Another great bonus is that it makes resumable/partial downloads much more accessible than in the core AFNetworking implementation.
it's this easy to use (example from the GitHub project's README.md):
[pdfRequest setProgressiveDownloadProgressBlock:^(NSInteger bytesRead,
long long totalBytesRead, long long totalBytesExpected,
long long totalBytesReadForFile, long long totalBytesExpectedToReadForFile)
{
self.downloadProgress = totalBytesReadForFile/(float)totalBytesExpectedToReadForFile;
}];
I use this in a few enterprise iOS projects to download multi-gigabyte files and I can tell you that it works great with the 1.0.1 release of AFNetworking.
Hope that helps…
I am working with network request - response in Objective-C. There is something with asynchronous model that I don't understand.
In summary, I have a view that will show my statuses from 2 social networks: Twitter and Facebook. When I clicked refresh, it will call a model manager. That model manager will call 2 service helpers to request for latest items. When 2 service helpers receive data, it will pass back to model manager and this model will add all data into a sorted array.
What I don't understand here is that : when response from social networks come back, how many threads will handle the response. From my understanding about multithreading and networking (in Java), there must have 2 threads handle 2 responses and those 2 threads will execute the code to add the responses to the array. So, it can have race condition and the program can go wrong right? Is it the correct working model of iphone objective-C? Or they do it in a different way that it will never have race condition and we don't have to care about locking, synchronize?
Here is my example code:
ModelManager.m
- (void)updateMyItems:(NSArray *)items {
self.helpers = [self authenticatedHelpersForAction:NCHelperActionGetMyItems];
for (id<NCHelper> helper in self.helpers) {
[helper updateMyItems:items]; // NETWORK request here
}
}
- (void)helper:(id <NCHelper>)helper didReturnItems:(NSArray *)items {
[self helperDidFinishGettingMyItems:items callback:#selector(model:didGetMyItems:)];
break;
}
}
// some private attributes
int *_currentSocialNetworkItemsCount = 0; // to count the number of items of a social network
- (void)helperDidFinishGettingMyItems:(NSArray *)items {
for (Item *item in items) {
_currentSocialNetworkItemsCount ++;
}
NSLog(#"count: %d", _currentSocialNetworkItemsCount);
_currentSocialNetworkItemsCount = 0;
}
I want to ask if there is a case that the method helperDidFinishGettingMyItems is called concurrently. That means, for example, faceboook returns 10 items, twitter returns 10 items, will the output of count will ever be larger than 10?
And if there is only one single thread, how can the thread finishes parsing 1 response and jump to the other response because, IMO, thread is only executed sequently, block of code by block of code
Yes, there is probably a thread per network request. The trick is to handle the response on the main thread. You should have something like this:
- (void)helper:(id <NCHelper>)helper didReturnItems:(NSArray *)items;
{
[self performSelectorOnMainThread:#selector(helperDidFinishGettingMyItems:)
withObject:items
waitUntilDone:NO];
}
Putting the response back onto the main thread will avoid a whole bunch of multithreading problems.
Also, the output of count will never be larger than 10. It's just that multiple threads may be running helperDidFinishGettingMyItems: at the same time. They won't automatically combine the two arrays.
The counter could possibly be more than 10, because multiple threads could be increasing that ivar at the same time.