I am after some advice / guidance on how to deal with a source control issue. Historically we have everything under SVN in the one dev branch and a release branch. We have one section that contains a lot of the content files as well as images and CSS files. We do make changes to the files in these folders sometimes, normally the CSS files. Our client also has access to these files as they do part of their CSS and general content management them selves.
We have now taken this folder out of the main dev branch and moved it into its own "resource" branch. This has taken it from over 3GB to under 1GB in the dev branch.
We are in the process of setting up TeamCity and getting as much as possible automated. Our issue is how best to deal with the clients making changes and pulling them back down into our dev enviornment, also how to push our changes up without conflicting with their changes. Obviously this can be done manually but we want to get away from this time consuming model and automate as much as possible.
We are happy to except that we can make the assumption that the clients changes take precedent over ours. I have got as far (in thinking about it, not implementation), to say that we have another branch "dev-resources" and we do a nightly pull from the clients resources branch and merge them together. Not 100% sure what the best way to automate this is, for example conflicts, making it choose theirs and then flagging this up so we know there was an issue, or getting it to fail at the point of a conflict.
The next issue is how to push our changes up to the clients resources branch.
I cannot see any process that would be fully automated due to conflict issues but if we can automate 90% and can make it visible when there is an issue that would be great.
Thanks
Related
I have always worked alone rather than in a Dev team, so this has never been an issue
I am about to take on someone else to code with so some kind of versioning control is required
I have been looking at GitHub and integrating it with Netbeans
I can make changes and commit them no problem.
I was expecting to be able to ‘check out’ a file or files which would prevent the other user(s) from editing those files while I was editing until I commit the file and check it in
Is this the normal procedure and I am missing something, or is my understanding of GitHub incorrect?
it does help with some general understanding, but is there anything to stop one of us editing a file the other one is currently editing?
No: with a decentralized version control system, there is no (optimist or pessimist) locking mechanism possible or desirable.
The reconciliation is done when you are pushing your local work to the common remote repository: if other commits have already been pushed, you will have to pull first, merge their work to your branch (or rebase your branch on top of their work), test locally and, if everything is still working, try and push again.
Minimizing conflict and avoiding multiple team members are working on the same set of file remains an organization and communication issue.
In the Boomla Control Panel, I can fork websites and then merge (publish) them, which is a very good workflow. But how can I accept modifications from other users? They can clone my website, but how can I import their code with their modifications to automatically merge into my upstream?
Unfortunately there is no support for this yet.
The best you can do is to merge them manually by copying the files across branches. The upside is that in Boomla you have access to both branches simultaneously (not only the one checked out), thus it's fairly trivial to move the files that changed. You'd have to write a little script to show the diff between the branches, but that is fairly simple given the Boomla Filesystem.
That's definitely on the roadmap, we are just not there yet.
The issue that I am trying to figure out is whether it is best to keep the buildfile together with the source code, i.e. trunk and branches, or in some separate location (obviously still under SCM).
The question: (to keep in mind while reading the rest of the text) Buildfile that ensures branch re-buildability at any time (at expense of maintenance), or one that ensures latest bugfixes/improvements (to build process) are quickly used by multiple branches and different projects (at expense of backward compatibility with older branches)
It doesn't matter what we are building, or what technologies are used, but just for the sake of completeness: making a mobile application, building/packaging with ANT, using SVN for SCM.
Buildfile = instructions for your builder/compiler/packager to compile and package an application from source.
Buildfile with code
This is what we have right now. Ant's build.xml is stored alongside the main code in SVN. A number of other supporting "packaging" files (Apple's provisioning profiles and certs) are stored there as well.
Pros:
Single checkout from DEV perspective. When developers checkout the trunk or one of its branches, the buildfile is right there. They don't need to search for it elsewhere. A simple ant build after checkout is all they need.
When changes to the build/packaging process are done on trunk that require some reorganization in code (different file locations, support for compiler constants, etc), I need not worry about breaking existing branches, since each branch gets its own revision.
Cons:
When changes to the build/packaging process are done on trunk that improve the process and fix bugs, I now need to worry about merging those changes to all active branches, which means having to keep track of all dev/feature branches in addition to release branches.
No reusability. A technologically identical project, that only requires a few switches/property changes to the buildfile should be able to use identical buildfile. But because they are spread across multiple project locations (in addition to multiple branches, as from the point above), it becomes a nightmare to do a generic improvement that affects all those locations. Mainly due to the fact that no matter what, these files end up with little "patch-works" here and there, and eventually with conflicting merges and ever-so-slightly different processes that cannot be resolved without putting one of the projects on hold and modifying that process to "catch up" with the other.
Buildfile separate from code
To address the cons of the previous scenario in regards to re-using a single file and avoid a plethora of small fixes all over the place, I was thinking of keeping the build file separate. Shareable between the trunk, branches and other similar projects.
Pros:
Single file to modify, improve and bugfix, reusable by multiple other projects.
Cons:
No "single checkout" for DEVs (but it can be solved with svn externals or other linking solution
Breaking old/existing builds. Since there is only one version of the file now, introducing an improvement that requires code restructuring would make it incompatible with older branches. When that older branch needs to be rebuild (urgent fix to already released software), the build file will no longer work. Yes, it's solvable by getting a previous revision of the file, however:
It is not directly obvious which previous revision had worked with this branch
The older revision may be missing some other critical bugfixes to the build process.
Toss up question
So for me it is a toss up between making my life easier and only maintaining one file for bugfixes and improvements, and thus ensuring that projects use identical processes, latest bugfixes to the build process, etc. Or making developer's life easier by providing a single point of checkout, and ensuring branch "stability/re-buildability" because the buildfile that's checked in with the branch is guaranteed to work with that branch.
Is there a proper way for this? What is the proper way for this? Am I approaching this wrong?
I would suggest tagging your code with a version number every time you build. IF you ever want to roll back to a revision , you can always create a new branch from the tag of that revision and use the build.xml from that revision
You can also publish all your artefacts into a revision named directory. If you want to rollback you can just use the artefacts from this directory instead of going through the build process again .
Your build file should always be part of your code . That way a developer can check out his/her code into an IDE and start building from there for his local testing.
If you have env related configurations that are different per environment you should separate it out into a deployment configuration that is used when the code is deployed.
If you want to reuse your build files across projects , you can create a master build file with macros that is fetched prior to starting a build. The only thing you need to do is override the macros in your local project if you want to override the default behaviour
I am starting to get a reputation at work as the "guy who breaks the builds".
The problem is not that I am writing dodgy code, but when it comes to checking my fixes back into source control, it all goes wrong.
I am regularly doing stupid things like :
forgetting to add new files
accidentally checking in code for a half fixed bug along with another bug fix
forgetting to save the files in VS before checking them in
I need to develop some habits / tools to stop this.
What do you regularly do to ensure the code you check in is correct and is what needs to go in?
Edit
I forgot to mention that things can get pretty chaotic in this place. I quite often have two or three things that Im working on in the same code base at any one time. When I check in I will only really want to check in one of those things.
A few suggestions:
try work on one issue at a time. It's easy to make unrelated changes to the codebase that then end up being committed as one big chunk with a poor log message. Git is excels here since you can so easily move switch branches, and stash and cherry pick changes.
run the status command before a commit to see which files you've touched and if you've created new files that need to be added to version control.
run the diff command to see what you've actually changed. Often times you find that you've left in some debug logging that should be taken out or made some unnecessary change that is just cluttering up the diff. Try to make your diffs as small and clean as possible.
make sure your working copy builds with your changes in it
update before checking in and make sure that your working copy builds with other peoples changes in it
run what ever smoke test suite you might have to make sure that your changes work correctly
make small and frequent commits. It's a lot easier to figure out what has broken the build when the breaking commit is small.
Other things that the team can do is setup a continuous integration server like David M suggested so that the broken build is discovered as soon as possibly and automatically.
I usually always do a Get Latest before, then build. If build is good then I check in my code.
Here is what I have been doing. I have used ClearCase and CVS in the past for source control, and most recently I have been using Subversion and Visual Studio 2008 as my IDE.
Make my code changes and build on the local machine.
Make sure they do, in fact, fix the bug in question.
Run an SVN update on the local machine and repeat steps 1 and 2.
Run through the automated unit tests to verify that they pass.
If an automated smoke test is available, which automatically tests a lot of the system's capabilities, run it. Verify that the results are correct.
Then go to the build machine and run the build script.
If the project's configuration has changed, this could definitely break a build. Perform an SVN update on the build machine, whether the build script does that or not. Open the build machine's copy of the IDE, and do a complete rebuild. This will show you whether the build box has any problems that you have taken care of on your machine but not on the build box.
The suggestions to keep separate branches for each issue are also very good, if you can keep track of all of the issues you are working on.
First, use multiple working copies (a.k.a sandboxes) - one per issue. So, if you've been working on some complex feature for a while, and you need to deal with a quick bug fix on the same project, check out a new clean working copy and do the bug fix there. With independent working copies for each issue there is no confusion about which changes to commit from the working copy to the reposistory.
Second, before committing changes, always perform the following three steps:
Buld the software.
Run a smoke test (does it start and run without crashing).
Inspect the changes you're checking in by diffing your changes against the baseline.
These should be repeated after any merge operations (e.g. after an SVN update).
At my workplace, the safety net for this is peer review. That is, get someone else to build, run, and reproduce your solution on their machine, on their view.
I cannot recommend this enough. It has caught so many omissions, would-be problems, and other accidental pieces of junk to make it a valuable part of the process. Not to mention that the mere knowledge that you have to place your work in front of someone else before having it go on to the main branch means that you raise your own quality standards.
In the past I have used branching in Clear Case to help with this issue. The process I used is below. I've never used SorceDepot so I do not know how this can be adapted to work with it.
Create a branch for the bug fix
Code all changes on the branch
Code Review
Merge to stable branch in a different view (the different view is important)
BEFORE checking in: compile, test, and run
Check in code to stable branch
By creating the branch and then merging the changes to a different view (I use Merge Manager to do the merge) any files that were not included or checked in immediately cause issues. This way everything gets tested as it will be when checked in on the stable branch.
The best thing to avoid your problems, is to use hooks, that are provided in most SCMs (they are for sure in SVN and Mercurial, and I believe they must be in other advanced SCMs). Attach unit tests to the hook and make it run every time someone checks code in - exactly before it is checked in. This way you will achieve two things:
code in SCM repo will always pass the tests,
you won't make most simple mistakes, because they should be easily detectable, if you have decent test suite.
I like having Tortoise plugins for Windows Explorer. The file icons are all badged with committed, modified or not added icons making it very easy to see what status the files are in. I also enable the meta data for Modified so I can sort changed files in the list (Details) view, where they bubble to the top so I can see them.
I bet there is a Tortoise* plugin for your SCM, I saw one for Mercurial and SVN (and CVS, ugh). I really wish Mac OS X's Finder would accept plugins like Tortoise, its so much easier than having to pop open a dedicated app most of the time.
Get someone else to go through "every" change "before" you check in the code.
I've tried using source control for a couple projects but still don't really understand it. For these projects, we've used TortoiseSVN and have only had one line of revisions. (No trunk, branch, or any of that.) If there is a recommended way to set up source control systems, what are they? What are the reasons and benifits for setting it up that way? What is the underlying differences between the workings of a centralized and distributed source control system?
Think of source control as a giant "Undo" button for your source code. Every time you check in, you're adding a point to which you can roll back. Even if you don't use branching/merging, this feature alone can be very valuable.
Additionally, by having one 'authoritative' version of the source control, it becomes much easier to back up.
Centralized vs. distributed... the difference is really that in distributed, there isn't necessarily one 'authoritative' version of the source control, although in practice people usually still do have the master tree.
The big advantage to distributed source control is two-fold:
When you use distributed source control, you have the whole source tree on your local machine. You can commit, create branches, and work pretty much as though you were all alone, and then when you're ready to push up your changes, you can promote them from your machine to the master copy. If you're working "offline" a lot, this can be a huge benefit.
You don't have to ask anybody's permission to become a distributor of the source control. If person A is running the project, but person B and C want to make changes, and share those changes with each other, it becomes much easier with distributed source control.
I recommend checking out the following from Eric Sink:
http://www.ericsink.com/scm/source_control.html
Having some sort of revision control system in place is probably the most important tool a programmer has for reviewing code changes and understanding who did what to whom. Even for single person projects, it is invaluable to be able to diff current code against previous known working version to understand what might have gone wrong due to a change.
Here are two articles that are very helpful for understanding the basics. Beyond being informative, Sink's company sells a great source control product called Vault that is free for single users (I am not affiliated in any way with that company).
http://www.ericsink.com/scm/source_control.html
http://betterexplained.com/articles/a-visual-guide-to-version-control/
Vault info at www.vault.com.
Even if you don't branch, you may find it useful to use tags to mark releases.
Imagine that you rolled out a new version of your software yesterday and have started making major changes for the next version. A user calls you to report a serious bug in yesterday's release. You can't just fix it and copy over the changes from your development trunk because the changes you've just made the whole thing unstable.
If you had tagged the release, you could check out a working copy of it and use it to fix the bug.
Then, you might choose to create a branch at the tag and check the bug fix into it. That way, you can fix more bugs on that release while you continue to upgrade the trunk. You can also merge those fixes into the trunk so that they'll be present in the next release.
The common standard for setting up Subversion is to have three folders under the root of your repository: trunk, branches and tags. The trunk folder holds your current "main" line of development. For many shops and situations, this is all they ever use... just a single working repository of code.
The tags folder takes it one step further and allows you to "checkpoint" your code at certain points in time. For example, when you release a new build or sometimes even when you simply make a new build, you "tag" a copy into this folder. This just allows you to know exactly what your code looked like at that point in time.
The branches folder holds different kinds of branches that you might need in special situations. Sometimes a branch is a place to work on experimental feature or features that might take a long time to get stable (therefore you don't want to introduce them into your main line just yet). Other times, a branch might represent the "production" copy of your code which can be edited and deployed independently from your main line of code which contains changes intended for a future release.
Anyway, this is just one aspect of how to set up your system, but I think giving some thought to this structure is important.