We are a small team of 4 developers working on a web application. We use trac+svn on a shared server for version control and ticketing and we are happy and satisfied with this. The same shared server also hosts our web application. The application itself is a Perl CGI application that uses CGI::Application and a moderate number of standard (CPAN) and custom Perl modules that are installed in the usual (/usr/lib/perl...) and a few unusual locations (/home/user/lib/perl..). While the broad details might be irrelevant, the most important point is that the location/layout of libraries on our development machines is different from that on the production (shared) server. We have to live with this as a given. The library layout is identical on all development machines though.
Here is a typical, but clearly sub-optimal work-cycle that my colleagues and I follow:
Code and test on development machines
Checkout/Commit/Update our code onto the SVN
Periodically "svn export" onto the appropriate DocumentRoot of the server
Hand edit the exported tree to set the library includes match the library layout on the server
Test application on live server, raise tickets for each other
Go to 1
Clearly there must be a better way and would appreciate hearing from others who might be handling this better than we are. For example is there a way to svn export and fix the library locations in an automated way? Or is there some completely different way to handle this situation than we have been doing so far.
Thank you for your attention
You should have scripts that do this for you that can be run from a local box. Mine always look something like:
$> checkout from source or copy from working
$> run sed/perl -pi/copy to convert configs to the production values
(ie cp production.config myconfig)
$> upload to web server (rsync/ssh/ftp/etc)
$> ssh $SERVER migrate_db, set permissions, run unit tests, etc
The last one requires ssh access which I always look for but everything else can be done locally. You'd usually have a set of dev configs and a set of production configs (or a script to convert from dev to production
One step uploads are always a really good idea.
Keep a config file (e.g. config.pl) that stores all the system-dependant paths and variables. Then set the svn:ignore property on this file so that it is never commited. This will allow you to easily keep a local configuration script per system that is separate from the commited tree.
If you don't have the possibility of mirroring your development server in production, why can't you mirror your production server in development? That might take some reconfiguring, but what's the risk? Everything's checked into svn.
But maybe that really, really isn't an option for you. My preference for deploying web applications is to do an svn checkout and then run a symlinking script. The idea is that you write a system of rules that logically maps the contents of one folder to the contents of another. Of course, if you drop folder symlinks in your document root, you have to tell Apache to follow them.
Frankly, the absolute safest scenario would be to set up a virtual machine that you can configure exactly like your production machine. This way you can actually test the contents of your deployment script and submit tickets. Then, when the problem is discovered, you modify the script to make it more likely that development deployment will follow the new and improved procedure.
And, as a side note: I much prefer to use svn checkouts in place rather than svn exports. It shouldn't be hard (especially if you use a deployment script) to make sure that apache or whatever your web server is doesn't have permission on the .svn folders. Ideally, anything you can do to make an svn rollback a one-line command is absolutely key.
If this is a Linux box you could write a cron job to take care of this for you. You could use sed/awk to replace the needed strings in the code and svn export works fine from a cron job. You would need to maintain the script but it seems faster than doing it by hand every time.
For the hand-editing part, I would have a separate branch in Subversion for the local modifications you need. The developers commit into trunk and when you need to deploy, use 'svn merge' or svnmerge.py to merge changes from trunk to the branch.
After creating the branch the first time, make your local modifications in there.
On the servers, have the directories in DocumentRoot and /usr/lib/perl and /home/user/lib/perl be Subversion working copies checked out from the branch.
Do not use svn export, just have a checkout so you can 'cd /usr/lib/perl; svn up'.
The one thing to be careful about is not exposing your .svn directories in DocumentRoot, use this:
# Prevent any access to .svn directories.
<DirectoryMatch "^/.*/\.svn/">
Options None
AllowOverride None
Order allow,deny
Deny from all
</DirectoryMatch>
Having working copies deployed in DocumentRoot is also nice, if you need to rollback a change, just 'svn up -r PREV'.
Codesion offers enterprise grade subversion and trac on demand. In addition, we now have the ability to one-click publish / deploy your code via ftp, scp, rsync and many other methods. This will be an easy and quick way for you to accomplish what you are trying to do.
See our Codesion Publisher features:
https://help.codesion.com/View.jsp?procId=01fabe5e83381dda4edda959b97b2c5b
Related
I am trying a lot and i am not bale to get how this version control work in my scenario
I have the VPS server where i host php sites. Users have home directories in /home/users.
Currently users edit files via FTP and i have no control what they do. I want to setup version control system on VPS i don't know hoe to start . I mean
I will explain what i want , i may be wrong but please correct me.
How can i install VCS on my VPS server so that all directories in /home/users are version controlled. I don't know if its possible or not. I want that final saving place or repo should be /home/user/public_html so that when user commit then my live site should change. Now i don't know if VCS works that way or not.
Now how will my client computers connect that VCS server
Is it possible to have version control for one user i mean /home/user1/public_html and not for others
Now users will still have FTP details , can't they change files via FTP even if i use VCS
Please clear my doubts , i really want to learn VCS systems
Yes, it should be feasible. Expect to be storing some extra data as the whole history will be stored plus separate copy of the current version for the stored.
You have to decide which version control system you want to use. The most common options are:
Subversion
Git
Mercurial
Bazaar
If you or your users already have experience with one, than it's probably best choice.
You want to:
Install the version control system of choice and create a post-commit hook to check out each version into the target directories.
Clients will commit into the respository. All the systems support access through restricted ssh (users log in using public key and the key is set in .ssh/authorized_keys to only allow one particular command). Some also have HTTP(s)-based method (special Apache module for Subversion, CGI script for Mercurial, Bazaar and Git).
Yes; the hook script will check out what you tell it to. You can implement it to checkout for all users, listed users, users in a group, whatever you need.
Turn the FTP server off.
Usually the workflow is that you have a repository with all the revisions and changes. This uses a special format, there is no point in directly accessing these files. The repo is typically accessed thru WebDAV interface (running as an apache module), or running a standalone server (with it's own protocol).
Users commit their changes to the repo, then can export the latest revision (or one of their choice) to their publicly accessible *public_html* directory. This involves them interacting with the VCS and knowing (and caring) about it.
A simpler setup can be that the *public_html* contains a working copy and they interact with it thru conventional FTP. (You have to make sure that the VCS's files for example the .svn folders can not be accessed by the general public). This way you can expose the VCS functions (basically commit and rollback) to your users thru a web interface (you write a small PHP script that does the commit and update for your them).
Incremental backups: a completely different story
As I understood you probably need something more like incremental backups, for example rsync. Each time a user closes an FTP connection you can initialize an rsync backup. It has flexible options, you can have all the changes for the last X days, or last X FTP sessions, so the user could roll back after an accidental upload. (It can be used with a remote or local storage for backups).
VCS (Version Control System) is just a class of software: You need to select one before you can implement it. In your case you probably want subversion, or one of the DVCS (Distributed Version control system) (git or mercurial).
It sounds like what you want is some kind of automated deployment system for your websites, which is certainly possible.
Disabling ftp is easy: simply stop the ftp server from running: ftp is insecure and the servers are often dangerous themselves.
Have a look at how Branchable works. They have specific web framework (ikiwiki), but the underlying principle of keeping the web sites in version control (git) is the same and all the software they use is open-source including the scripts that bind it all together, so you can look how it works.
I am coding a website using the Codeigniter PHP framework.
I am using mercurial for version control.
I have 3 systems I work with. I do my coding on a Windows 7 machine using Netbeans 6.9.1. I am occasionally making commits, and pushing to a repository at Bitbucket.org, purely for the purposes of backup and version control.
I have a "beta" website (on a shared Linux box with it's own dedicated IP address) that I upload to using FTP, where I can test that everything is working as intended on an actual site running Linux.
Once I'm happy with that, I upload to my "live" site, which is on it's own dedicated server. Again I'm just using FTP to upload the files from my development server.
I realize that this is all kinds of wrong. For one thing I have to go in and change some things on the beta and live machines so that they're referring to the correct domain name, instead of localhost. For another, I'm not making use of mercurial at all to help with this. I assume instead of uploading from FTP, I could be using mercurial to "grab" a particular revision that I've marked as ready to deploy. I also think I could possibly be doing something in Netbeans differently to make the process easier.
What I want to do is have some very smoothe way to control all this, and hopefully one that knows how to deal with the issue of a slightly different configuration setup for the beta and live sites from the localhost.
Is there a standard way to do what I'm looking for? I've seen references to some third party apps for "continuous integration" but I'm not sure I need anything like that.
I'm a little lost as to what would be the SIMPLEST thing for me to do that would make my life easier....any help greatly appreciated :) Thanks!
It depends on how different the setup for each site is, and if there are secrets involved, which should not be visible on a public place (I assume you use a public bitbucket repository).
If the changes are not sensitive, then you can add two additional branches for your test and production servers, where only the configuration changes are applied. Every time you change something in default and deploy it to test, you would simply merge default on top of test, and mercurial fill in the different configuration settings in the process. Then the server deployment wold be a call to hg archive within the correct branch.
A typical change history would look like this:
O----o-o-o-o-o-o-o-o---o default
\ \ \
T1--------T2-----------T3 test
\ \
P1---------------------P2 production
where in T1 and P1 the parameters for test and production are filled in. You also can use this branch setup to mature the development of your site, where you hack in default, and only propagate stable changes into test and production.
If the changes are sensitive, you can create a non-versionized deploy script (or better a versionized deployment script and a not versionized configuration file), which patches the output of hg archive.
You should use deployment scripts anyway, which handles the packaging of the product and deploy an the target in an automated and standardized way. Within this script you can also embed information about the source revision into the final archive.
Note that this model works fine for an environment, where no changes are made on the server. If you do changes to the product on the server, you need to copy the files from the server back into your development environment(at the correct revision), to check what was changed on the server. When you want to make changes also on the server, you might want to install mercurial also there.
Tipically a deploy in production does not involve just a mere source code update (build) but requires a lot of other important tasks like, for example:
Db scripts
Configuration files (differents from test\production)
Batch to schedule
Executables to move to the correct path
Etc. etc.
In our company we just send an email to a "Release email address" describing the tasks in order, which changeset need to be published (TFS), which SP need to be updated, db scripts and so on.
I believe there's not a magic tool that does these tasks automagically in order, rollback included; but probably there's something better than email that helps to keep track of releases in production.
Do you have any tools to suggest or practices to share?
When multiple tasks are required to support a full project deployment (and that's frequently the case, in my experience), I'd suggest using a build/deployment tool. I've used Ant in the past with great success, but know others who swear by Capistrano, Maven and others.
Using Ant, I wrote a script that would:
Pull the specific revision I wanted from my VCS
Create a tarball of the target directory on the remote machine (in case a rollback was required)
Create a MySQL dump file of the database (also for rollback purposes)
Delete the remote directory and SSH the new content just pulled from the VCS
Perform various other logistical operations (setting file perms, ownership, etc.)
Create a release branch on the VCS itself
Create a tag with the appropriate version information so I always had a snapshot of the code base at that moment of deployment.
Hope that helps some. I've written a few blog posts about this that may (or may not) be useful. They're dated now, but the general information should still be solid enough.
Introductory thoughts
Details of how I use Ant for deploying--including scripts
You might be interested in the Team Foundation Build Recipes Website, that showcases some build scripts developed using SDC Tasks Library and the MSBuildTasks library
How about something like SVN? You can put all of your code in a repository, then when you are ready to release from production bring your stuff over from test. Then you'll have very specific revisions with information on what happened. SVN keeps track of all of it.
I have a CakePhp Website that is currently live. I would like to keep working on the site, without impacting the deployed site.
What is the best way to keep a production version separate from a deployed version, and then merging the two when appropriate?
Currently, I am using Git for version control.
Thanks!
First thing, get to know a version control system Subversion, Git, Bazaar, Mercurial are some examples. They are a safety net that can save your bacon because they save EVERY change to EVERY file in your fileset.
Then, typically I have a local development server and also a subdomain (staging.example.com) on the production server. I then do my heavy development on the local development server. Then I use SVN to archive all my site changes. Then, using a shell account on the production server I check out the new version of the software to the staging subdomain. If it works ok there, I can then update the live site using just a single SVN check out.
I've also heard of people placing a symbolic link in the location where the site root should be (/var/www/public_html) that points to the live directory (/var/www/site_ver_01234) , then set up the new version in a parallel directory (/var/www/site_ver_23456). Finally, just recreate the symbolic link pointing to the new version's directory. The switch is instantaneous and transparent. I'm sorry I'm not more clear on this method though, I read about it a while back but never tried it myself though.
I've also looked at Bazaar (another version control system) that has a plugin that automatically ftps any changed files to a given server every time a version is checked in.
The general idea, first of all, is to use a version control system. Using this, you're developing your site on your local machine or with several people, having a central repository somewhere.
When you're happy with a certain revision and would like to deploy it, you "tag" it. That means you freeze the state of that revision and separate it from the continually evolving "trunk". What that means specifically depends on your version control system.
You then take that tagged revision and copy it to the live server. Possibly you may copy it to a "staging server" before to test it in another environment. This copying can be as simple as overwriting all existing files using FTP, or it can involve automated deployment systems which will take care of the details for you and allow you to roll back an unsuccessful deployment. If a database is involved as well, you're probably also looking at database schema migration scripts that need to be run.
Each of these steps can be done in many different ways, and you'll have to figure out what's the best approach for you. If you're not doing so already, start using a version control system such as SVN or git. Do it now! Then you might want to google or search on SO about different techniques to tag and branch using that system. For serious deployment, start with a keyword like Capistrano or one of its PHP clones.
currently my work-flow is as follows:
Locally on a machine I maintain a git repo on each website I am working on, when the time comes to publish something I compress the folder and upload this single file to the production server via ssh then I decompress, test the changes a move the changes to the live folder and I get rid of the .git folder.
I was wondering if the use of a git repo on the live server was a good idea, seems to be at first but it can be problematic if a change doesn't look the same on on the production server in comparison to the local development machine... this could start a fire...
What about creating a bare repo on some folder on production server then clone from there to the public folder thus pushing updates from local machine to the bare repo and pulling from the bare on the public folder of the production server... may anyone plese provide some feedback.
Later I read about capistrano http://capify.org but I have no experience w/ this software...
In your experience what is the best practice/methodology to accomplish a website deployment/updates?
Thanks in advance and for your feedback.
I don't think that our method can be called best practice, but it has served us well.
We have several large databases for our application (20gb+), so maintaining local copies on each developers computer has never really been an option, and even though we don't develop against the live database, we do need to do the development against a database that is as close to the real thing as possible.
As a consequence we use a central web server as well, and keep a development branch of our subversion trunk on it. Generally we don't work on the same part of the system at once, but when we do need to do that, or someone is making a lot of substantial changes, we branch the trunk and create a new vhost on the dev server.
We also have a checkout of the code on the production servers, so after we're finished testing we simply do a svn update on the production servers. We've implemented a script that executes the update command on all servers using ssh. This is extremely convinient, since our code base is large and takes a lot of time to upload. Subversion will only copy the files that actually have been changed, so it's a lot faster.
This has worked really well for us, and the only thing to watch out for is making changes on the production servers directly (which of course is a no-no from the beginning) since it might cause conflicts when updating.
I never thought about having a repository copy on the server. After reading it, I thought it might be cool... However, updating the files directly in the live environment without testing is not a great idea.
You should always update a secondary environment matching exactly the live one (webserver + DB version, if any) and test there. If everything goes well, then put the live site under maintenance, update files, and go live again.
So I wouldn't make the live site a copy of the repository, but you could do so with the test env. You'll save SSH + compressing time, plus you can check out any specific revision you'd like to test.
Capistrano is great. The default recipes The documentation is spotty, but the mailing list is active, and getting it set up is pretty easy. Are you running Rails? It has some neat built-in stuff for Rails apps, but is also used fairly frequently with other types of webapps.
There's also Webistrano, which is based on Capistrano but has a web front-end. Haven't used it myself. Another deployment system that seems to be gaining some traction, at least among Rails users, is Vlad the Deployer.