I am working on a website of 3,000+ pages that is updated on a daily basis. It's already built on an open source CMS. However, we cannot simply continue to apply hot fixes on a regular basis. We need to replace the entire system and I anticipate the need to replace the entire system on a 1-2 year basis. We don't have the staff to work on a replacement system while the other is being worked on, as it results in duplicate effort. We also cannot have a "code freeze" while we work on the new site.
So, this amounts to changing the tire while driving. Or fixing the wings while flying. Or all sorts of analogies.
This brings me to a concept called "continuous migration." I read this article here: https://www.acquia.com/blog/dont-wait-migrate-drupal-continuous-migration
The writer's suggestion is to use a CDN like Fastly. The idea is that a CDN allows you to switch between a legacy system and a new system on a URL basis. This idea, in theory, sounds like a great idea that would work. This article claims that you can do this with Varnish but Fastly makes the job easier. I don't work much with Varnish, so I can't really verify its claims.
I also don't know if this is a good idea or if there are better alternatives. I looked at Fastly's pricing scheme, and I simply cannot translate what it means to a specific price point. I don't understand these cryptic cloud-service pricing plans, they don't make sense to me. I don't know what kind of bandwidth the website uses. Another agency manages the website's servers.
Can someone help me understand whether or not using an online CDN would be better over using something like Varnish? Is there free or cheaper solutions? Can someone tell me what this amounts to, approximately, on a monthly or annual basis? Any other, better ways to roll out a new website on a phased basis for a large website?
Thanks!
I think I do not have the exact answers to your question but may be my answer helps a little bit.
I don't think that the CDN gives you an advantage. It is that you have more than one system.
Changes to the code
In professional environments I'm used to have three different CMS installations. The fist is the development system, usually on my PC. That system is used to develop the extensions, fix bugs and so on supported by unit-tests. The code is committed to a revision control system (like SVN, CVS or Git). A continuous integration system checks the commits to the RCS. When feature is implemented (or some bugs are fixed) a named tag will be created. Then this tagged version is installed on a test-system where developers, customers and users can test the implementation. After a successful test exactly this tagged version will be installed on the production system.
A first sight this looks time consuming. But it isn't because most of the steps can be automated. And the biggest advantage is that the customer can test the change on a test system. And it is very unlikely that an error occurs only on your production system. (A precondition is that your systems are build on a similar/equal environment. )
Changes to the content
If your code changes the way your content is processed it is an advantage when your
CMS has strong workflow support. Than you can easily add a step to your workflow
which desides if the content is old and has to be migrated for the current document.
This way you have a continuous migration of the content.
HTH
Varnish is a cache rather than a CDN. It intercepts page requests and delivers a cached version if one exists.
A CDN will serve up contents (images, JS, other resources etc) from an off-server location, typically in the cloud.
The cloud-based solutions pricing is often very cryptic as it's quite complicated technology.
I would be careful with continuous migration. I've done both methods in the past (continuous and full migrations) and I have to say, continuous is a pain. It means double the admin time for everything, and assumes your requirements are the same at all points in time.
Unfortunately, I would say you're better with a proper rebuilt on a 1-2 year basis than a continuous migration, but obviously you know best about that.
I would suggest you maybe also consider a hybrid approach? Build yourself an export tool to keep all of your content in a transferrable state like CSV/XML/JSON so you can just import into a new system when ready. This means you can incorporate new build requests when you need them in a new system (what's the point in a new system if it does exactly the same as the old one) and you get to keep all your content. Plus you don't need to build and maintain two CMS' all the time.
Background: I'm using jira for bug tracking, and git for source control. I've got a complete end-to-end system comprising of an iOS front end, and a Java/Tomcat back end that provides web services and a GUI. Right now I've got a single git repository holding all the software and a single jira project tracking issues for the whole system.
Now that the software is live, I'm finding that changes are being made to either the iOS application or the server, but generally not both. The version numbers of the two components have diverged somewhat.
It's probably too late for this project, but in future:
Should I pursue the path of having all related components in a single source repository and tracked using a single bug-tracking project; or
Should each component be in a separate repository and be managed by a separate bug-tracking project?
I can see pro's and con's for both approaches, and I can also see that the answer could easily be "it depends".
Which way would you lean, and why?
I'd go with distinct source repositories for a few reasons
The developers working on the two are likely to have distinct
skill sets. Also, you may have management reasons for wanting to
segregate who sees what.
They should not be tightly tied at a protocol level - different versions need to interact.
The first point becomes even more important when you do another front end
The second reason is my main one.
However, I'd go with a common bug database. Defects/features may need changes on both ends. Also, it is extremely likely you will have bugs that are believed to be in one component but actually end up fixed in the other. If you try to migrate across databases, information will get lost. I've seen that too many times.
I am curious as to how other development teams spec out new features. The team I have just moved up to lead has no real specification process. I have just implemented a proper development process with CI, auto deployment and logging all bugs using Trac and I am now moving on to deal with changes.
I have a list of about 20 changes to our product to have done over the next 2 months. Normally I would just spec out each change going into detail of what should be done but I am curious as to how other teams handle this. Any suggestions?
I think we had a successful approach in my last job as we delivered the project on time and with only a couple of issues found in production. However, there were only 3 people working on the product, so I'm not entirely sure how it would scale to larger teams.
We wrote specs upfront for the whole product but without going into too much detail and with an emphasis on the UI. This was a means for us to get a feel for what had to be done and for the scope of the project.
When we started implementing things, we had to work everything out in a lot more detail (and inevitably had to do some things differently from the spec). To that end, we got together and worked out the best approach to implementing each feature (sometimes with prototypes). We didn't update the original spec but we did make notes after the meetings as it's very easy to forget the details afterwards.
So in summary, my approach is to treat specs as an exploratory tool and to work out finer details during implementation. Depending on the project, it may also be a good idea to keep the original spec up to date as the application evolves (which we didn't need to do this time).
Good question but its can be subjective. I guess it depends on the strategy of the product, if its to be deployed to multiple clients in the same way or to a single client on a bespoke project, the impact, dependency these changes have on the system and each other and the priority these changes need to be made.
I would look at the priority and the dependency, that will naturally start grouping things?
I've been tasked with setting up a version control for our web developers. The software, which was chosen for me because we already have other non-web developers using it, is Serena PVCS.
I'm having a hard time trying to decide how to set it up so I'm going to describe how development happens in our system, and hopefully it will generate some discussion on how best to do it.
We have 3 servers, Development, UAT/Staging, and Production. The web developers only have access to write and test their code on the Development server. Once they write the code, they must go through a certification process to get the code moved to UAT/Staging, then after the code is tested thoroughly there, it gets moved to Production.
It seems like making the Developers use version control for their code on Development which they are constantly changing and testing would be an annoyance. Normally only one developer works on a module at a time so there isn't much, if any, risk of over-writing other people's work.
My thought was to have them only use version control when they are ready to go to UAT/Staging. This allows them to develop and test without constantly checking in their code.
The certification group could then use the version control to help see what changes had been made to the module and to make sure they were always getting the latest revision from the developer to put up on UAT/Staging (now we rely on the developer zip'ing up their changed files and uploading them via a web request system).
This would take care of the file side of development, but leaves the whole database side out of version control. That's something else that I need to consider...
Any thoughts or ideas would be greatly appreciated. Thanks.
I would not treat source control as annoyance. See Nicks answer for the reasons.
If I were You, I would not decide this on my own, because it is not a
matter of setting up a version control software on some server but
a matter of changing and improving development procedures.
In Your case, it might be worth explaining and discussing release branches
with Your developers and with quality assurance.
This means that Your developers decide which feature to include into a release
and while the staging crew is busy on testing the "staging" branch of the source,
Your developers can already work on the next release without interfering with the staging team.
You can also think about feature branches, which means that there is a new branch for every specific new feature of the web site. Those branches are merged back, if the feature is implemented.
But again: Make sure, that Your teams agreed to the new development process. Otherwise, You waste Your time by setting up a version control system.
The process should at least include:
When to commit.
When to branch/merge.
What/When to tag.
The overall work flow.
I have used Serena, and it is indeed an annoyance. In addition to the unpleasantness of the workflow overhead Serena puts on top of the check in-check out process, it is a real pain with regard to doing anything besides the simplest of tasks.
In Serena ChangeMan, all code on local machines is managed through a central server. This is a really bad design. This means a lot of day-to-day branch maintenance work that would ordinarily be done by developers has to go through whomever has administrator privileges, making that person 1) a bottleneck and 2) embittered because they have a soul-sucking job.
The centralized management also strictly limits what developers are able to do with the code on their own machine. For example, if you want to create a second copy of the code locally on your box, just to do a quick test or whatever, you have to get the administrator to set up a second repository on your box. When you limit developers like this, you limit the productivity and creativity of your team.
Also, the tools are bad and the user interface is horrendous. And you will never be able to find developers who are already trained to use it, because its too obscure.
So, if another team says you have to use Serena, push back. That product is terrible.
Using source control isn't any annoyance, it's a tool. Having the benefits of branching and tagging is invaluable when working with new APIs and libraries.
And just a side note, a couple of months back one of the dev's machine's failed and lost all his newest source, we asked when the last time he committed code to the source control and it was 2 months. Sometimes just having it to back up stuff when you reach milestones is nice.
I usually commit to source control a couple of times a week, depending if I've hit a good stopping point and I'm about to move on to something different or bigger.
Following on from the last two good points I would also ask your other non-web developers what developmet process they are using so you won't have to create a new one. They would also have encountered many of he problems that occur in your environment, both technical using the same OS and setup and managerial.
I recently joined the IT department of a big insurance company. Although the department's title is "IT", a lot of code gets written here; Java, JSP, JavaScript, COBOL and even some C++ from what I've heard. All the programs that allow insurers, brokers and the rest of the tie-wearing, white-collar workers to issue new contracts and deal with clients runs on the code produced by this department. I've been told that the department is pretty good by the parent company's standards and that we've even received an internal award or two. We're 17 people in the department, split in smaller groups of 2 or 3. As you might've guessed from the COBOL part further up, the average age is over 40 years (as a point of reference I'm 29 yo).
Right now, there is no version control system in place (there exists a general backup scheme though). When needed, files are passed around through shared folders. Usually there's one person in every group responsible for copying the "final" version of the files back to the production server. I find this absurd and even a bit dangerous.
How may I try to convince management that we should implement a VCS scheme in our department? I've never deployed a VCS myself but every other place I've worked at had one. I think I'll hit a "we've been OK until now, why bother" wall from the first step, coupled with the age of most of my co-workers that will feel this step is an unnecessary hurdle.
I know the basic advantages of VCS (traceability, granular backups, accountability etc). I'm looking to back my case with realistic cases and examples of real added value over the implementation costs, not just a "but-but-but, we must have a VCS you fools!" :-)
You don't necessarily need their permission.
Install svn on your machine, start using it, and then start convincing your fellow team members to use it too.
Then watch and see what happens.
Edits
The basic idea of this is that it's easier to show than to tell.
It's a great idea to support your ideas with a working implementation/solution.
Of course, if you succeed, and they want the system used department/company wide, you must be prepared to support the transition, know how the software is to be installed and used.
Going ahead and using something accepted in the industry is faster than having discussions on what system should be used.
There is a good change that this will get you noticed. You may also get your peers respect and support.
As suggested, the same approach can be made on other areas:
issue/bug tracking systems
quality tools
time tracking
continuous integration
a wiki for knowledge base, HOWTO's, guidelines, tutorials, presentations, screencasts
different IDEs and tools
build tools
automated deployment
various scripts that would save your team time
.. any item that will visibly add quality to your work, but doesn't (yet) disrupt existing methodologies and practices.
Joel Spolsky has an excellent article: Getting Things Done When You're Only a Grunt
Quote
Nobody on your team wants to use
source control? Create your own CVS
repository, on your own hard drive if
necessary. Even without cooperation,
you can check your code in
independently from everybody else's.
Then when they have problems that
source control can solve (someone
accidentally types rm * ~ instead of
rm *~), they'll come to you for help.
Eventually, people will realize that
they can have their own checkouts,
too.
Management? I will put bold the expressions and words you should use:
Your should display some examples how a VCS will prevent losing money to the company if some error/bugs or disaster happens. It will be faster to solve all problems, so maintaning the systens won't be so lazy and people become more productive.
You should also mention that implementing a VCS has no costs.
VCS will also give advantages for backup all the existing code. So, all the code will be safe.
My opinion on how to go about doing this, is that you should try to convince your fellow developers first. The way I see it, there are two ways this might go about:
You give the right arguments to the other developers (possibly only the head developers will suffice), they like the idea, and the suggest it to management. Management is easy to convince at that point, so everyone is happy.
You give the right arguments to management, who get all excited (great!) and mandate that version control has to be installed and used by everyone. Here's the thing: If at this point the other developers are not sold to the idea already, then (a) they might be hostile to an idea that management is forcing upon them, and (b) they might not like you for being the cause of it all.
So what are good arguments to convince fellow developers? As someone who uses subversion (which is the one I recommend in this case, by the way) even for his solo projects, here's a few advantages I can think of:
Using version control forces people to think of code modification in terms of a series of small, self-contained changes. This is an extremely beneficial way of working: where otherwise people would be inclined to make lots of changes all over the place, leaving the code in a mess, version control kinda forces them to change the code in bite-size, easy-to-swallow bits, keeping the code compiling at all times, easing the cost of integration with other modules, etc.
Version control makes it very easy to see what has changed in the code each time. This might sound trivial, but when you start modifying code it's easy to lose track after a while. But with VC it's all an "svn diff" (or equivalent) away, always.
Version control makes it very easy to see who has changed the code each time. So that, for example, when something breaks, you know who to blame. (It's not an accident that the subversion command which shows who last changed each line is called "svn blame".)
Version control makes it very easy to see why a piece of code was changed. Commit messages, if used properly, essentially provide continual documentation of the ongoing development process. Documentation that otherwise wouldn't be written.
Version control makes it very easy to track down regressions and see where they appeared. In the easy case, you just track down commit messages and spot the culprit. In the average case, you have to consult the diffs too. In the hard case, you have to do regression testing of previous versions using what amounts to binary search, which is still better than the no-VC case, where you simply have no clue.
This list is not exhaustive, of course, but these are the main benefits that come to mind right now. Obviously, as others have already mentioned, it's easier to show all this to your colleagues than to describe it to them, and setting it up for yourself first (but importing everyone's code, mind) sounds like a great idea.
As Joel points out on one of his articles, start using your own one man version control system and market its benefit on every opportunity you get. Show them the benefits of traceability, granular backups etc from your single instance. People will start realizing its benefits irrespective of their age.
I agree with the answer that are referring to the Joel Guerrilla article.
Install/Use some thing with a low overhead. Hg (Mercurial) is easy in a mixed eniroment and is good because you can bail out and use something else in an easy way.
You must share your things without making a fuzz about it. When someone needs your code, export it and use the "standard" corporate method (shared folder or whatever)
When you get code, always import it into a repos, if you think it is a new commit of a repos you already have, try to get it into that one.
Sooner or later you will have a code for several project and hopefully some commits on some repos. Then you can expose those with the mercurials webserver interface (hg serve -p XXXX).
When the times comes when someone don't know why something suddenly don't works as it should be and is trying to figure out why becase it was working last monday ... and you know that you have that code in a repos step up and ask if you can be of any assistance. Get the falty code, commit it into your repos and expose with hg serve. Look at it in the browser.
My point is that you must prove with real cases to your colleges that this stuff has a value.
If the haven't figured it out by themselves after some many years you have a mountain to climb but it can be done. You must be patience though. It could very well take a year to convert one man (old dog). If you have any younger coworkers try to do this together, the more code you can get hold on the better.
I would point out the hazards of not having one - lost code, developers over writing each other changes, ability to rollback problems, etc.
Also since Subversion and some others are free, point out there is no real cost to purchase, jsut the time to implement.
The biggest issue you will have as the new guy is that you will be seen as rockign the boat, if they had no issues to date they will be hard to convince. Perhaps start using it locally jsut for yourself and maybe they will like what they see and start to adopt it.
I would try small steps, maybe ask the others if they ever used one, point out the benefits, when an issue arises that a system would have prevented or aided in point it out delicately.
From a purely business perspective, and depending on the size and nature of your parent company an IT auditor may consider your lack of a VCS a finding (i.e. something that needs fixing). I believe you could improve your pitch to management by telling them that any CVS is a great way of showing that your department respects its resources and works in a structured way and efficient way, something auditors always like to see.
I don't know how your corporate culture works but I'd be careful about rolling out your own CVS since if it does see use it suddenly becomes your responsibility when things go wrong, even if you were not at fault. To cover your ass (and keep the aforementioned auditors even happier) I'd roll the system out with a full set of written procedures for its use and maintenance.
Finally while I myself am a big fan of initiative at any level of the enterprise don't expect people to remember to say thanks when they figure out how great it is. Some might, but for the most part you're doing this to make your own life easier and for your own karma.
Remember, there are plenty of version control systems that are absolutely free. And the amount of time spent installing and maintaining a version control system should be somewhere near 0 (they shouldn't require any maintenance). There isn't even a space penalty for most systems, as they can compress things internally.
You have listed some advantages, and there are others. But more importantly, I can't think of a single disadvantage.
I would also recommend starting with implementing VCS (Version Control System) for yourself first. I'd recommend using one of distributed VCS (Git, Mercurial, Bazaar) rather than centralized Subversion, because it would be easier to create central repository (or repositories) by cloning than moving your Subversion repository to central place. Distributed SCM can be also used in a smaller group to exchange ideas.
A few advantages of (modern) version control systems:
You can always revert (go back) to last working version of your code (provided that you follow some sane version control conventions, like at least tagging only tested code). With code shared via folders it might turn out that no one version works, backup copies were deleted to save space, and recovering code from backup is tedious / was never tested.
You can switch between working on some new feature (some experimental work), and working on urgent fix in currently deployed version (maintenace work) thanks to branches (and stash / shelve for uncommitted work).
If you follow good practices for version control (small and often commits, changes being about one single thing, writing good commit message describing change and whys of change) you would have much, much easier finding bugs, be it by bisecting history to find which change introduced bug, or by using version control system to look up who was responsible for given area of code (annotate / blame).
Start talking to the other developers about problems thay have had in the past as a way to get to know the system and how it evolved (sneaky, sneaky, sneaky, but hey this information will probably come in handy at some poitn even aside from the version control issue). You are bound to sooner or later find some wonderful examples of things that have already happened which would have been far less painful if you had version control. Use these examples when you present the idea to management.
I agree with the idea that you can probably start using your own version control and eventually will be able to help thm out of a fix, but I'd bet money they have been in some of those fixes already and if they already remember how painful the problme was before, it will help sell the new idea.
Look for another job.
Seriously.
There are way better jobs out there that don't require you to teach the existing staff.
Ones where you could go into work and just, y'know, work.
Also, keep in mind that 30 isn't far off. That's the age at which most people
stop suffering fools gladly.
Just a heads up.
EDIT
It's been suggested that quitting a bad job is for quitters.
Maybe so, but keep in mind that you're supposed to
put your employer to the Joel test before you accept the job, not after.