So I'm using a very simple CDN service. You point to your website and if you call it through their HostName they'll cache it for you after the first call.
I use this for all my static content, like JavaScript files and images.
This all works perfect - and I like that it has very little maintenance or setup cost.
Problem starts when rolling out new versions of JavaScript files. New JavaScript files automatically get a new hash if the files changes.
Because roll out over multiple instances is not simultaneously a problem occurs though. I tried to model it in this diagram:
In words:
Request hits server with new version
Requests Js file with new version hash
CDN detects correctly that the file is not cached
CDN requests the original file with the new hash from the load balancer
loadbalancer serves request of CDN to a random server - accidently serving from a server with the old version
CDN caches old version with the new hash
everyone gets served old versions from the CDN
There are some ways I know how to fix this - i.e. manually uploading files to a seperate storage with the hash baked in, etc.
But this needs extra code and has more "moving parts" that makes maintenance more complicated.
I would prefer to have something that works as seamlessly as the normal CDN behavior.
I guess this is a common problem for sites that are running on multiple instances, but I can't find a lot of information about this.
What is the common way to solve this?
Edit
I think another solution would be to somehow force the CDN to go to the same instance for the .js file as the original html file - but how?
Here are a few ideas from my solutions in the past, though the CDN you are using will rule out some of these:
Exclude .js files from the CDN Caching Service, prevents it being cached in the first place.
Poke the CDN with a request to invalidate the cache for a specific file at the time of release.
In your build/deploy script, change the name of the .js file and reference the new file in your HTML.
Use query parameters after the .js file name, which are ignored but cached under a different address reference, e.g. /mysite/myscript.js?build1234
The problem with this kind of issues is that the cache control resides on the browser side, so you cannot do too much form the server side.
The most common way I know is basically the one you mention about adding some hash to the file names or the URLs you use to get them.
The thing is that you should not do this manually. You should use some web application builder, like Webpack, to automate this process and it will depend on the technologies you are using. I saw this for the first time using GWT 13 years ago, and all the last projects I worked with, using AngularJS or React, had been integrated with builders that does what you need automatically.
Once it's implemented, your users will get the last version, and resources will be cached correctly to speed up your site.
If you can also automate the full pipeline to remove the old resources from the CDN once the expiration configured on them have been reached, you touched the sky.
I fixed this in the end by only referencing to the CDN version after a few minutes of runtime.
So if the runtime is less then 5 minutes it refers to:
/scripts/example.js?v=351
After 5 minutes it refers to the CDN version:
https://cdn.example.com/scripts/example.js?v=351
After 5 minutes we are pretty sure that all instances are running the new version, so that we don't accidently cache an old version with the new hash.
The downside is that on very busy moments you don't have the advantage of the CDN if you would redeploy, but I haven't seen a better alternative yet.
Related
I have an application in the store that has a bunch of UITableViews, each has a bunch of items that when selected, load a UIWebView that loads a local HTML file. The reason I do this is so it allows for viewing when there is no internet access, however it makes it tedious to update.
How would I go about both keeping local files, and keeping web files, so that If I update one on the web it would let the user know there is an 'update' and it can download and overwrite the old file with the new. Is this possible? Like I would make myself a backend to edit my HTML, and have a connection to the app so whenever I clicked save the app would know to ask the user to update the files so I don't have to submit to apple for such small updates.
But I still need to allow offline viewing in case they didn't have internet access.
What technologies or techniques would I have to research to accomplish this?
Thanks
what you're looking for is an html5 concept called application cache.. it directly addresses your problem. With app cache you can cache static content at the client side, which they can view even if they're offline, yet they can update it when the content on the server changes. If you look around you'll see people use this for the iPhone as well.
update:
What way would I go about rewriting files on the phone from ones on a server?
you basically update the cache to do that. from the docs:
The application cache automatically updates only if the manifest file
changes. It does not automatically update if resources listed in the
manifest file change. The manifest file is considered unchanged if it
is byte-for-byte the same; therefore, changing the modification date
of a manifest file also does not trigger an update. If this is not
sufficient for your application, you can update the application cache
explicitly using JavaScript.
I encourage you to do some reading on html5 local storage just to get the concept of local cache and manifest files etc, then just follow through the instructions on the apple docs.. It's not that difficult.
You can cache the content as abbood said but that won't allow you to provide initial offline content.
Another approach would be to use NSURLProtocol, which allows you to swizzle a request. For example, if you have a request for "http://google.com", you would be able to either change the URL or load your own content (say from a local directory).
I am building an internal iOS application (so - it won't ever be in the app store), and I need to keep a directory of content synchronized between a server and each of the instances of the iOS application. This would be easy enough if I just wanted to delete and re-download this content each time, but I would rather use something similar to rsync to only download the elements that have changed.
I haven't found any good way to utilize rsync. I considered looking at Objective-Git as a possibility here, but at a quick glance it looked like there is still a lot of the support for remote repositories that isn't supported yet.
As a final note, while this won't be in the app store, I will not be jailbreaking these devices and I would prefer to not rely on any private API's (although if there was an elegant solution that utilized private API's I might consider it).
Thoughts?
ADDITIONAL NOTE: This needs to be an isolated solution. I won't be relying on outside services (like Dropbox, Box.net, etc...). This needs to work solely between the device and the server (which is on a local network with the device).
Use HTTP to list the contents of each folder on the server.
Compare last modification time of each file with those on the device, and identify added/removed files.
Get added and modified files, remove deleted files.
It sounds like you're maybe asking for a library that already does this, but if you don't find one it's obviously moderately easy to write this from the ground up using stat(2) on the server and the same or a higher-level equivalent on the iOS devices. Have the iPhone send a tree of files with their modification date to the server and get back a list of insert/delete/update operations to do with the url (or whatever) for each one so you can do them incrementally on a background thread. Have the information from the server for new/updated files include the mod date that the server has so you can set it to be the same on the iOS device and send that when asking the server for the status of each file (kind of hack using the file system to store that, but it works).
Why not just set up a RESTful interface and do it across HTTP; that way you could query the modification times easily enough to determine whether client or server files need to be updated. You might also want to keep track of what files on the client have been synced, so you can easily know which files to add or delete. This can be done with a simple .sync file or using a plist / sqlite / etc.
If you'll consider FTP, there are some pretty advanced client libraries available.
For example, the iOS Chilkat bundle includes an FTP client library that supports synchronization in both directions. It's not free, but it's pretty cheap -- and you get a ton of other stuff that will likely prove useful someday. Here's an example of iOS pulling down all additions and changes (mode 2):
http://www.example-code.com/ios/ftp_syncLocalTree.asp
One caveat -- judging solely from the example, it doesn't appear to synchronize deletions. If this is a requirement, you could do it yourself without too much effort immediately following a sync.
acrosync (see https://acrosync.com/library.html) seems like a good fit given the initial question, however I haven't used it myself yet.
I am often working with websites to which I have no code access but for which I need to test javascripts, ajax calls and other resources in locally modified versions.
As an example I needed to test a new creative code originally coming from a DoubleClick server. Instead of working directly on DoubleClick and loose a lot of time waiting for my changes to take effect I needed to manipulate a local copy of that javascript resource.
So I changed my /etc/hosts to point the DoubleClick server to localhost where I run a local web server. This way I was able to test a local script instead of the original.
Of course this redirected all resources and not only the one i was interested in what often results in either weird behaviour as resources become unavailable or a lot of effort on my part to make them available.
Is there a way to replace a specific URL by redirecting it to a local file/URL and let all the others from the same origin be untampered?
I'm in the process of building a simple update server for an application. The parts of the application being updated are configuration files; the most up-to-date copies of these files exist on the update server and these files can be edited by the individual managing the application (the "application manager") at any time. However, I don't want the application to be able to download one of these files while the file is being edited by the application manager; this would obviously cause consistency issues. How can I prevent these files from being accessed in an inconsistent state? Alternatively, would a solution be to provide a checksum along with the file that the application could use to determine if the file was received in a consistent state?
EDIT: I've seen this post concerning access restrictions using .htcaccess and think it could be of use. However, I want the application manager to do as little thinking as possible; having them forget to re-allow connections might be problematic. That being said, they're going to have to do some work at some point; maybe this is the way I should go?
I am trying to build an iPhone web application using ASP.NET. The page is dynamically rendered once for each visitor. At this point the page can be bookmarked and it will never change again for that visitor. For this reason it should be cached locally from that point on so the application will run if referenced from the bookmark even if no network connection is available. No matter what I try the phone continues to request the page from the server forcing a re-render or it fails if the phone is offline.
Louis Gerbarg suggested in this post that I use HTML5 Cache Manifest to get this working however following the w3.org docs does not appear to work for the iPhone. Does anyone have a good example where application cache is working?
The cache manifest file has to be served with a 'text/cache-manifest' mime-type. This is absolutely critical, it will not work without it. If you navigate to the url of your manifest file, it should trigger a download...
Also, I've found that putting the manifest location in the tag as an absolute location, as well as all the entries in the manifest file to be more effective.
I answered your previous question related to this, but it was not clear from that question that you were trying to cache dynamic content. The cache manifest is for getting static content you want for offline web apps to work.
I am not sure you can do what you want. Do you want the app to be able to function offline, or are you just trying to peg something in the cache because it is slow to download? Unless you are actually constructing an offline webapp (which the user will add to as a bookmark or an app in the Spring Board) then your page can (and must necessarily) be evicted from local storage at the browsers discretion, regardless of how loose a cache policy you set on the page.
You should use the Safari Javascript Database API which should work for iPhone and Safari 3.1. It works great for local caching and data storage:
http://developer.apple.com/documentation/iPhone/Conceptual/SafariJSDatabaseGuide/
It could be to do with the size of the output.
I can't talk from any serious experience in tweaking things specifically for an iphone, but there is an intersting read from the YUI team here: http://yuiblog.com/blog/2008/02/06/iphone-cacheability/, which indicates that the largest unzipped cache file that can be held in an iphone is 25k, and that for optimal caching, as many components as possible should be <25k.
That may be the cause of your problems, but that's only a guess.