My group is getting into web services for the first time. What is the usual way to control versioning issues with web services and clients? We're generating our services and clients using in IBM's RAD using the wizards it presents. The services and clients work well enough, but we're wondering how managing these will become a task going forward.
-If a service changes, and it's interface doesn't change I presume older clients would never know the difference and would work fine.
-If a service changes to add a new property to a parameter, would older clients work with it if they didn't need to set that value?
-How do people handle version control of web services as they grow in number and the number of apps using their clients grows? What is best practice on this sort of thing?
There are two types of versioning that we use. One that is visible to clients (public) and one that isn't (private). This comes from (what you already touched upon) whether clients are affected or not.
If clients are affect by, for example change in XSD Schema definitions or functionality of the WS changes in such way that clients must also modify their end, we change the public version. We increment the version number in WS context root, which means that it will have different URL than the previous version. Also make sure that the code archive, in our case they are war, have also the public version, as not to overwrite the previous deployment.
For example our WS called foo is in public version 2. Its URL is http://ourserver:8000/foo_2 and war file is called foo_2. We modified our XSD Schema, so the clients must react to the change. We update the version and now URL is http://ourserver:8000/foo_3 and war file is called foo_3. The previous version still exists, while clients can slowly transition to the new version.
If the change does not force any action from clients, then we call this private versioning. This usually shows up, in combination with public version, as a part of a project name. Using previous example, we have a WS foo, with private version 5 and public 2. Our project for this service is called WS_foo_2_5. We now change the order in which we store incoming data. This does not effect the clients, so we change the private version. In effect we have project WS_foo_2_6. We create a code archive from it called foo_2 and deploy it with URL set to http://ourserver:8000/foo_2. This way we modified the version, without changing anything from clients POV.
Related
The server holds logic, iOS/Android App holds UI. Common case.
How do I suppose to deploy new features in this case with continuous deployment methodology?
I assume that server-side deploy looks like that:
I'm triggering new feature deployment, load balancer starts redirecting 1% of all users to the server instance with the new feature. If everything goes smoothly, then load balancer starts redirecting 10%, 30%, etc up to 100%.
The same can be done for client apps, using, say, Codepush.
So, if I'll deploy server without an app, then there will be no new features usage and therefore no problems with new deployment for sure.
So, probably I have to deploy app first and put some kind of server version checker, so if the server has api for this new feature, the UI for this feature is being shown, and if the app is connected to the wrong server, the new UI is hidden.
That's seems primitive. I need to persist socket connection to the same server to avoid hitting the wrong server, right? And what if instance/zone/region will go down and the user will be suddenly redirected to another sone/region and new server will not have the new feature api? Probably, my assumption is wrong.
So, how do I suppose to deploy new features in this case with continuous deployment methodology?
I would say that your question is more of version compatibility nature of server/client API than CD. We have a similar requirement where a server and the clients communicate and both are constantly enhanced with features. I don't know your production software architecture which might change the needs accordingly but I'll try to come up with some ideas.
I'm going to describe two cases which might apply for you.
First case:
The thing is easier when you do not face the situation that new client versions need to communicate with old server versions. The new server version is deployed first and old clients simply do not use the new feature, as you've already pointed out. In this situation my recommendation is to deploy the server app first and then start to roll out the new client apps. If that's possible I would do that. It applies only when the new feature doesn't force you to break the API.
Second case:
In the case that new client app versions need to talk to an old server app, which I would try to avoid at all costs, the new client needs some switch inside to deactivate feature e.g. B when it's talking to an old server that doesn't support this feature. An API version counter could be the solution. But it requires the client to be able to distinct between server versions. In REST you often see the .../v1/.. inside the URL but could be solved differently as well. Hopefully the API provides some mechanism to get the version the server speaks.
We faced both cases at the same time, the protocol changed over the time including breaking changes, so we needed to implement an API version negotiation mechanism.
I am designing REST APIs for some resources in my system.
System allows users to upload files.
There are 3 kind of resources:
Post/GET a file (data file).
Get Config files (meta file of file format (format is system specific - like how CSV file in my system should be , how Json file should be etc).
Get configfiles of server resources and permissions.
I am thinking of some url of the form:
host/api/v1/files.
host/api/v1/config/files.
host/api/v1/config/server.
Does host/api/v1/config/files, host/api/v1/config/server
makes more sense or host/api/v1/files/config, host/api/v1/server/config
makes more sense?
Also when version of config of datafiles changes to v2 does it make sense to change version of server config files also to v2 - despite being the fact they are unrelated and don't change together?
Or can I broadly classify files and config of files under same category as
/host/api/files/v1/data
/host/api/files/v1/config
and server config in another category as
/host/api/server/v1/config
Then both can change independently no need to migrate server/v1/config to v2.
Does host/api/v1/config/files, host/api/v1/config/server makes more sense or host/api/v1/files/config, host/api/v1/server/config makes more sense?
The latter. Think about expanding the service in the future, what if e.g. you want to add a method for getting/adding server users. Its URL should be host/api/v1/server/config.
Regarding the versioning, service should be observed as a whole. If you update one part to new version, whole service goes to a new version. If some parts are completely independent, consider making two services.
IBM states the following:
It can be argued that, when the interface to a service changes in a
non-backwards-compatible way, in reality an entirely new service has
been created. In such a case, unless implementations of the first
interface continue to exist, the preexisting service is, in effect,
discontinued. From the client's perspective, a service is no more than
an interface and some non-functional qualities (such as trust and QoS
attributes) that it may claim to exhibit; thus, if the interface to a
service changes in a non-backwards-compatible way, it no longer
represents an instance of the original service, but is rather a
completely new service.
Situation
We're using socket.io for mobile-server communications. Since we can't force-upgrade users' devices, if we want to upgrade to version 1 (non-back-compatible), we have to handle both versions on the server for a while.
Question
What are the options?
My current favourite is to wrap both the old version and the new version in a multiplexer. It detects the version of the incoming request based on headers and query parameters and thereby knows which functions to invoke.
Another (shittier) option is to wrap the new version in a module that can translate the old version of the protocol into the new version (and back again) when necessary. This suffers from a serious drawback. It would be time-consuming and uncertain work to ensure I've properly determined and handled all the tiny differences. Some differences might take some serious massaging.
(In case you're curious or it's helpful to know, we're doing this in Go.)
It appears that you could run two separate versions of socket.io on the server. Since the two versions don't have unique module filenames you would probably need to load one version from a different path. And, then obviously when loading the modules and initializing them you'd assign them to differently named variables. For example:
var io_old = require('old/socket.io');
var io = require('socket.io);
Once you have the two versions loaded on the server, I think there are two different approaches for how they could be run.
1) Use a different port for each version. The older version would use the default port 80 (no configuration change required for that) which is shared with the node.js web server. The newer version would be run on a different port (say port 3000). You would then initialize each version of socket.io to its own port. Your newer version clients would then connect to the the port the newer version was running on.
For the old socket.io server running on port 80, you would use whatever initialization you already have which probably hooks into your existing http server.
For the new socket.io server running on some other port, you would initialize it separately like this:
var io_old = require('old/socket.io')(server);
var io = require('socket.io')(3000);
Then, in the new version client, you would specify port 3000 when connecting.
var socket = io("http://yourdomain.com:3000");
2) Use a different HTTP request path for each version. By default, each socket.io connection starts with an HTTP request that looks like this: http://yourdomain.com/socket.io?EIO=xx&transport=xxx?t=xxx. But, the /socket.io portion of that request is configurable and two separate versions of socket.io could each be using a different path name. On the server, the .listen() method that starts socket.io listening takes an optional options object which can be configured with a custom path as in path: "/socket.io-v2". And similarly, the .connect() method in the client also accepts that options object. It's kind of hard to find the documentation for this option because it's actually an engine.io option (which socket.io uses), but socket.io passes the options through to engine.io.
I have not tried either of these myself, but I've studied how socket.io connections are initiated from client and server and it looks like the underlying engine supports this capability and I can see no reason why it should not work.
Here's how you'd change the path on the server:
var io = require('socket.io')(server, {path: "/socket.io.v1"});
Then, in the client code for the new version, you'd connect like this:
var socket = io({path: "/socket.io.v1"});
This would then result in the initial connection request being made to an HTTP URL like this:
http://yourdomain.com/socket.io.v1?EIO=xx&transport=xxx?t=xxx
Which would be handled by a different request handler on your HTTP server, thus separating the two version.
FYI, it is also possible that the EIO=3 query parameter in the socket.io connection URL is actually an engine.io version number and that can also be used to discern client version and "do the right thing" based on that value. I have not found any documentation on how that works and could not even find where that query parameter was looked at in the engine.io or socket.io source code so that would take more investigation as a another possibility.
I don't really have an immediate solution for this, but I have some kind of advice. I guess you could use it to save a lot of time.
first of all Im working in a startup which uses socketIo for almost
everything
We knew that this problem would happen so our initial design was to
make everything pluggable which means that we can swap out socketio for
sockjs and it will still work.
The way its done is by defining the common set of APIs which rarely change
in a system. We call it managers. The managers can just expose the API which the rest of the devs need to use without messing up anything. It speeds up a lot.
The manager implementation changes in the background but still the APIs are the same, so the engineers working on the core can confidently make changes.
Seems like you have a tight dependency in your code. Or may be not. I'm not so sure. Try following this principle if you haven't.
We're going to go the route of keeping both the 0.9.x version and the current version as separate libraries on the server. Eventually, when the pool of clients has more-or-less all updated, we'll just pull the plug on the 0.9.x version.
The way we'll manage the two versions is by wrapping the socket.io services in a package that will determine which wrapped socket.io version to pass the request off to. This determination will depend on features of the request, such as custom headers (that can be added to the newer clients) as well as query parameters and other headers utilized exclusively by one version or the other.
Since we're using Go, there's so far no universally agreed upon way to manage dependencies, let alone a way that can respect version differences. Assuming the back-compat branch of the repo wasn't broken (which it is), we'd have two options. The first would be to fork the repo and make the back-compat version the master. We'd then import it as if it had nothing to do with the other one. The second option would be to use gopkg.in to pretend the separate branches were separate repos.
In either event, we could import the two branches/repos like so
import (
socketioV0 "github.com/path/to/older/version"
socketioV1 "github.com/path/to/current/version"
)
And then refer to them in the code using their import names socketioV0 and socketioV1.
We have an app that is written in PHP. The front end uses javascript heavily. Generally, for normal applications that require page reloads, continuous deployment is not really an issue, because:
The app can be deployed with build tags: myapp-4-3-2013-b1, myapp-4-3-2013-b2, etc.
When the user loads a page (we are using the front controller pattern), we can inject the buildtag and the files are loaded from the app directory with the correct build tag.
We do not need to keep the older builds around for too long because as the older requests finish, they will move to the newer build tags.
The issue with database and user data being incompatible is not very high as we move people to the newer builds after their requests finish (more on this later).
Now, the problem with our app is that it uses AJAX heavily for smooth page loads. In addition, because there is no page refresh at all when people navigate through the application, people can keep their unsaved data in a their current browser session and revisit it as long as the browser has not been refreshed.
This leads to bigger problems if we want to achieve continuous deployment:
We can keep the user's buildtag in their session (set when they make the first request) and only switch to newer buildtags after the logout and login again. This is obviously bad, because if things like the database schema changes or the format of files to be written to disk changes in a newer build, there is no way to reconcile this.
We force all new requests to a newer build tag, but there is a possibility we change client side javascript and will break a lot of things if we force everyone with a session onto the new build tags immediately.
Obviously, the above won't occur with every build we push and hopefully will not happen a lot, but we want to build a fool proof process so that every build which passes our tests can be deployed. At the same time, we want to make sure that every deployed and test passing build does not inadvertently break in clients with running sessions cause a whole bunch of problems.
I have done some investigation and what google does (at least in google groups) is that they push a message out to the clients to refresh the application (browser window). However, in their case, all unsaved client side data (like unsaved message, etc) would be lost.
Given that applications that uses AJAX and local data are very common these days, what are some more intelligent ways of handling this that will provide minimal disruption to users/clients?
Let me preface this that I haven't ever thought of continuous deployment before reading your post, but it does sound like quite a good idea! I've got a few examples where this would be nice.
My thoughts on solving your problem though would be to go for your first suggestion (which is cleaner), and get around the database schema changes like this:
Implement an API service layer in your application that handles the database or file access, which is outside of your build tag environment. For example, you'd have myapp-4-3-2013-b1, and db-services folders.
db-services would provide any interaction with the database with a series of versioned services. For example, registerNewUser2() or processOrder3().
When you needed to change the database schema, you'd provide a new version of that service and upgrade your build tag environment to look at the new version. You'd also provide a legacy service that handles the old schema to new schema upgrade.
For example, say you registered new users like this:
registerNewUser2(username, password, fullname) {
writeToDB(username, password, fullname);
}
And you needed to update the schema to add the user's date of birth:
registerNewUser3(username, password, fullname, dateofbirth) {
writeToDB(username, password, fullname, dateofbirth);
}
registerNewUser2(username, password, fullname) {
registerNewUser3(username, password, fullname, NULL);
}
The new build tag will be changed to call registerNewUser3(), while the previous build tag is still using registerNewUser2().
So the old build tag will continue to work, just that any new users registered will have a NULL date of birth. When an updated build tag is used, the date of birth is written to the database correctly.
You would need to update db-services immediately, as soon as you roll out the new build tag - or even before you roll out the build tag I guess.
Once you're sure that everyone is using the new version, you can just delete registerNewUser2() from the next version of db-services.
This will be quite complicated to make sure that you are correctly handling the conversion between old API and new API calls, but might be feasible if you're already handling continuous deployment.
We have an internal web system that handles the majority of our companies business. Hundreds of users use it throughout the day, it's very high priority and must always be running. We're looking at moving to ASP.NET MVC 2; at the moment we use web forms. The beauty of using web forms is we can instantaneously release a single web page as opposed to deploying the entire application.
I'm interested to know how others are deploying their applications whilst still making them accessible to the user. Using the deployment tool in Visual Studio would supposedly cause a halt. I'm looking for a method that's super quick.
If you had high priority bug fixes for example, would it be wise to perhaps mix web forms with MVC and instead replace the view with a code-behind web form until you make the next proper release which isn't a web form?
I've also seen other solutions on the same server of having the same web application run side-by-side and either change the root directory in IIS or change the web.config to point to a different folder, but the problem with this is that you have to do an entire build and deploy even if it were for a simple bug fix.
EDIT: To elaborate, how do you deploy the application without causing any disruption to users.
How is everyone else doing it?
I guess you can run the MVC application uncompiled also? and just replace .cs/views and such on the run.
A websetup uninstall/install is very quick, but it kills the application pool.. which might cause problem. Depending on how your site is built.
The smoothest way is to run it on two servers and store the sessions in sql server or shared state. Then you can just bring S1 down and patch it => bring s1 back up again and bring S2 down => patch S2 and then bring it up again. Al thought this might not work if you make any major changes to the session parts of the code.
Have multiple instances of your website running on multiple servers. The best way to do it is to have a production environment, a test environment, and a developement environment. You can create test cases and run the load every time you have a new build, if can get through all the tests, move the version into production ;).
You could have two physical servers each running IIS and hosting a copy of the site. OR you could run two copies of the site under different IIS endpoints on the SAME server.
Either way you cut it you are going to need at least two copies of the site in production.
I call this an A<->B switch method.
Firstly, have each production site on a different IP address. In your company's DNS, add an entry set to one of the IPs and give it a really short TTL. Then you can update site B and also pre-test/warm-up the site by hitting the IP address. When it's ready to go, get your DNS switched to the new site B. Once your TTL has expired you can take down site A and update it.
Using a shared session state will help to minimise the transition of users between sites.