I've been working with CircleCI, TravisCI and Azure DevOps for quite some time now.
While it's great to have a managed server that integrates beautifully with all of the external services you can think of (VCS, CD pipeline, app stores etc.), one of my greatest pains is testing for CI. When setting up such systems, I spend the majority of my days tweaking and configuring YAML files, Bash scripts and other parts of the CI process to work faster and smarter. However, this process is slow: it takes 1-5 minutes to even get the first error message that something is wrong, not to mention sometimes 1+ hours for final builds.
You can only do so much locally - mock environment variables, use exactly the right container and add credentials for the relevant services. But it's not quite there, just yet. I envision a full-service solution for this, one that takes care of setting up the environment exactly the same as in Azure or Travis, including getting data from Git based on the specific push and the like.
I would be able to pay good money for that - this will save me hours every week of waiting for the CI agents to spin and finish small ls, pwd and echo statements just to figure out where the heck I am.
Is there something like that out there?
Edit: This is a good example of a "duck-tape" solution that could be of great use, but is not full-fledged. I'm looking for something like this, but more robust.
Edit 2: This is a bad example of a solution that requires me to do the most crucial part - dependencies - manually.
Edit 3: This has a lot of interesting options, specifically using Chef and travis-cookbooks or vagrant. This is the most promising direction so far, albeit it's for Travis alone, seemingly. Maybe set up vagrant boxes for each type of agent you have?
Related
I am looking for some advice on the best practice for versioning software.
Background
Build automation with gradle.
Continuous integration with Jenkins
CVS as SCM
Semantic Versioning
Sonatype Nexus inhouse repo
Question
Lets say I make a change to come code. An automated CI job will pull it in and run some tests against it. If these tests should pass, should Jenkins update the version of the code and push it to nexus? Should it be pushed up as a "SNAPSHOT"? Should it be pushed up to nexus at all, or instead just left in the repository until I want to do a release?
Thanks in advance
I know you said you are using CVS, but first of all, have you checked the git-flow methodology?
http://nvie.com/posts/a-successful-git-branching-model/
I have little experience with CVS, but it can be applied to it, and a good versioning and CI procedure begins with having well defined branches, basically at least one for the latest release, and one for the latest in-development version.
With this you can tell the CI application what it is working with.
Didn't have time for a more detailed answer before, so I will extend now, trying to give a general answer.
Branches
Once you have clearly defined branches you can control your work flow. For example, it is usual having a 'master' and a 'develop' branches, where the master will contain the latest release, and develop will contain the next release.
This means you can always point to the latest release of the code, it is in the master branch, while the next version is in the develop branch. Of course, this can be more detailed, such as tagging the master branch for the various releases, or having an additional branch for each main feature, but it is enough having these two.
Following this, if you need to change something to the code, you edit the develop branch, and make sure it is all correct, then keep making changes until you are happy with the current version, and move this code to the master.
Tests
Now, how to make sure all is correct and valid? By including tests in your project. There is a lot which can be read about testing, but let's keep it simple. There are two main types of tests:
White box tests, where you know the insides of the code, and prepare the tests for the specific implementation, making sure it is built as you want
Black box tests, where you don't know how the code is implemented (or at least, you act as if you didn't), and prepare more generic tests, meant to make sure it works as expected
Now, going to the next step, you won't hear much about these two tests, and instead people will talk about the following ones:
Unit tests, where you test the smallest piece of code possible
Integration test, where you connect several pieces of code and test them
"The smallest piece of code possible" has a lot of different meanings, depending on the person and project. But keeping with the simplification, if you can't make a white box test of it, then you are creating an integration test.
Integration tests include things like database access, running servers, and take a long time. At least much longer than unit testing. Also, integration tests due to their complexity may require setting up a specific environment.
This means that while unit tests can be run locally with ease, integration tests may be so slow that people dislike running them, or may just be impossible to run in your machine.
So what do you do? Easy, separate the tests, so unit tests can be run locally after each change, while integration test are run (after unit tests) by the CI server after each commit.
Additional tests
Just as a comment, don't stop at this simplified vision of tests. There are several ways of handling tests, and some tests I wouldn't fit into unit or integration tests. For example it is always a good idea validating code style rules, or you can make a test which just deploys the project into a server, to make sure it doesn't break.
CI
Your CI server should be listening to commits, and if correctly configured it will know when this commit comes from a development version, a release or anything else. Allowing you to customise the process as you wish.
First of all it should run all the tests. No excuses, and don't worry if it takes two hours, it should run all the tests, as this is your shield against future problems.
If there are errors, then the CI server will stop and send a warning. Fix the code and start again. If all tests passed, then congratulations.
Now it is the time to deploy.
Deploying should be taken with care, always. The latest version available in the dependencies repository should, always, be the most current one.
It is nice having a script to deploy the releases into the repository after a commit, but unless you have some short of final validation, a manual human-controlled one at it, you may end releasing a bad version.
Of course, you may ignore this for development versions, as long as they are segregated from the actual releases, but otherwise it is best handling the final deployment by hand.
Well, it may be with a script or whatever you prefer, but it should be you who begins the deployment of releases, not the machine.
CI customisation
Having a CI server allows for much more than just testing and building.
Do you like reports? Then generate a test coverage report, a quality metrics one, or whatever you prefer. You are using a external service for this? Then send him the files and let it work.
Your project contains files to generate documentation? Build it and store it somewhere.
I have an interesting dev requirement that i've been trying to find a decent solution to for some time. I thought I might look here for suggestions.
I have 4 server locations on a network drive: 2 for development, 2 for production. each pair is intended to be identical, though because we're manually managing files gaps often surface. Developing in this context goes like this:
Make changes on one dev server.
Copy the updated files to the other dev server.
preview in browser.
repeat as necessary until ready to go live.
copy files from one dev server to one prod server, then the other.
review and test and hope to got you remember all the dependancies.
Realize that something is missing in a stylesheet somewhere because a colleague missed a server 6 months ago.
commence swearing. lather. rinse. repeat.
The closest thing to a solution I've been able to get to thus far involved using the local/remote server setup with autosave in dreamweaver with mappings to the two dev servers. But all this stuff being over a network makes things slow. And dreamweaver is far from my favorite editor.
I'd love to be able to find some way of making this better. In my freelance stuff I've been getting into things like NPM and Grunt, and I'm convinced I might be able to do something with something along the lines of task automation as is seen there, but I'm not experienced enough with these yet to know if that's a direction I should be considering.
Any suggestions would be greatly appreciated.
You can use any source control system you like to achieve what you want. Git seems to be the most popular choice those days, but even CVS would do the trick
Our development departments grows and I want to force a stable master/trunk.
Up to now every developer can commit into the master/trunk. In future developers should commit into a staging area, and if all tests pass the code gets moved to the trunk automatically. If the test fails, the developer gets a mail with the failed tests.
We have several repositories: One for the core product, several plugins and a repository for every customer.
Up to now we run SVN and git, but switching all repos to git could be done, if necessary.
Which software could help us to get this done?
There a some articles on the web which explain how to use gerrit and jenkins to force a stable branch.
I am unsure if I need both, or if it is better to use something else.
Environment: We are 10 developers, and use python and django.
Question: Which tool can help me to force a stable master branch?
Update
I was on holiday, and now the bounty has expired. I am sorry. Thank you for your answers.
Question: Which tool can help me to force a stable master branch?
Having been researching this particular aspect of CI quasi-pathologically since our ~20 person PHP/ZF1-based dev team made the switch from SVN to Git over the winter (and I became the de-facto git mess-fixer), I can't help but share my experience with this particular aspect of continuous integration.
While obviously, having a "critical mass of unit tests running" in combination with a slew of conditionally parameterized Jenkins jobs, triggering infinitely more conditionally parameterized jobs, covering every imaginable circumstance would (perhaps) be the best and most proper way to move towards a Continuous Integration/Delivery/Deployment model, the meatspace resources required for such a migration are not insignificant.
Some questions:
Does your team have some kind of VCS workflow or, minimally, rules defined?
What percentage would you say, roughly, of your codebase is under some kind of behavioral (eg. selenium), functional or unit testing?
Does your team ( / senior devs ) actually have the time / interest to get the most out of gerrit's peer-based code review functionality?
On average, how many times do you deploy to production in any given day / week / month?
If the answers to more than one of these questions are 'no', 'none', or 'very little/few', then I'd perhaps consider investing in some whiteboard time to think through your team's general workflow before throwing Jenkins into the mix.
Also, git-hooks. Seriously.
However, if you're super keen on having a CI/Jenkins server, or you have all those basics covered already, then I'd point you to this truly remarkable gem of a blog post:
http://twasink.net/2011/09/16/making-parallel-branches-meet-regularly-with-git-and-jenkins/
And it's equally savvy cousin:
http://twasink.net/2011/09/20/git-feature-branches-and-jenkins-or-how-i-learned-to-stop-worrying-about-broken-builds/
Oh, and of course, the very necessary devopsreactions tumblr.
There a some articles on the web which explain how to use gerrit and jenkins to force a stable branch.
I am unsure if I need both, or if it is better to use something else.
gerrit is for coding review
Jenkins is a job scheduler that can run any job you want, including one:
compiling everything
launching sole unit test.
In each case, the idea is to do some guarded commit, ie pushing to an intermediate repo (gerrit, or one monitored by Jenkins), and only push to the final repo if the intermediate process (review or automatic build/test) passed successfully.
By adding intermediate repos, you can easily force one unique branch on the final "blessed" repo to which those intermediate referential will push to if the commits are deemed worthy.
It sounds like you are looking to establish a standard CI capability. You will need the following essential tools:
Source Version Control : SVN, git (You are already covered here)
CI server : Jenkins (you will need to build and run tests with each
check in, and report results. Jenkins is the defacto standard tool
used for this)
Testing : PyUnit
Artifact Repository : you will need a mechanism for organizing and
archiving the increments created with each build. This could be a
simple home grown directory based system. I have also used Archiva,
but there are other tools.
There are many additional tools that might be useful depending on your development process:
Code review : If you want to make code review a formal gate in your
process, Gerrit is a good tool.
Code coverage analysis : I've used EMMA in the past for Java. I am
sure that are some good tools for Python coverage.
Many others : a library of Jenkin's plugins that provide a variety of
useful tools is available to you. Taking some time to review
available plugins will definitely be time well spent.
In my experience, establishing the right cultural is as important as finding the right tooling.
Testing : one of the 10 principles of CI is "self testing builds". In
other words, you must have a critical mass of unit tests running.
Developers must become test infected. Unit testing must become a
natural, highly value part of each developers individual development
process. In my experience, establishing a culture of test infection
is the hardest part of deploying CI.
Frequent check-in : Developers and managers must organize there work
in a way that allows for frequent small check-ins. CI calls for daily
checkins. This is sometimes a difficult habit to establish.
Responsiveness to feedback : CI is about immediate feedback. The
developers must be conditioned to response to the immediate feedback.
If unit tests fail, the build it broken. Within 15 minutes of a CI
build breaking, the developer responsible should either have a fix
checked in, or have the original, bad check-in backed out.
I am pretty comfortable with the producing web apps now. I am using a NodeJs stack on the back-end and usually have a fair amount of Javascript on the front end. Where I really lack understanding is the deployment process.
What is a typical deployment process?
From what I have gathered in my reading a deployment/build process can include several tasks:
Running through unit-test suites
Concatenating script and CSS files
Version numbering your app
Tracing module dependencies (node_modules)
Pushing it to a remote repo (GitHub)
Instructing 'staging' servers to pull down the latest repo
Instructing 'production' server to pull down the latest repo
This has all left me a little overwhelmed. I don't know whether I should be going into this level of detail for my own projects, it seems like a lot of work! I am using Sublime Text 2 IDE and it seems to have a Build Script process, is this suitable? How does one coordinate all these separate tasks? I'm imagining ideally they would all run at the flick of a switch.
Sorry so many questions, but I need to know how people learnt similar principles. Some of my requirements may be specific to NodeJS but I'm sure processes are similar no matter what choice of stack you are developing in.
First off, let's split the job in two: front-end and back-end stuff. For both, you really want some kind of bulid system, but their goals and scope are vastly different.
For the front-end, you want your source to be as small as possible; concatenate/minify JavaScript, CSS and images. A colleague of mine has written a "compiler", Assetgraph, to do this for you. It has a somewhat seep learning-curve, but it does wonders for your code (our dev-builds are usually ~20 megs, production ~500 k).
As to the back-end, you want contained, easily managed bundles of some sort. We re-package our stuff into debian-packages. As long as the makefile is wired up correctly, you get a lot of the boring build- and deploy-time stuff for free. Here's my (pre-NPM 1.0) Debianizing node programs. I've seen other ways to do this in NPM and on Github, but I haven't looked into them, so I can't speak on their quality.
For testing/pusing around/deploying, we use a rather convoluted combination of Debian package-archives, git-hooks, Jenkins-servers and what not. While I highly recommend using the platforms' native package-manager for rolling out stuff, it can be a bit too much. All in all, we usually deploy staging either automatically (on each git push), or semi-automatic for unstable codebases. Production deployments are always done explicitly.
For the assets I use asereje https://github.com/masylum/asereje
I recently documented my nodejs deployment process in a blog post:
http://pau.calepin.co/how-to-deploy-a-nodejs-application-with-monit-nginx-and-bouncy.html
A build script sounds like a good idea indeed.
What should that build script do?
make sure all the test pass, else exit immediately
concatenate your javascript and css files into one single js/css file and minify them also
increment the version number (unless you decide to set that up yourself manually)
push to the remote repo (which instructs the staging and production servers through a git hook)
This is at least my opinion.
Other resources:
http://howtonode.org/deploying-node-with-spark
http://howtonode.org/deploying-node-upstart-monit
http://dailyjs.com/2010/03/15/hosting-nodejs-apps/
How to deploy node app depencies
I can think of plenty of good reasons to using it; however, what are the downsides to it?
(Apart from buying another server)
What are some advantages to using a daily build instead of it?
(It's worth noting that by "continuous integration" I mean automated integration with an automated build process and automatically runs tests and automatically detects failure of each piece.
It's also worth noting that "continuous integration" just means to a trunk or test server. It does not mean "push every change live".
There are plenty of ways to do continuous integration wrong.)
I can't think of any reason not to do continuous integration testing. I guess I'm assuming that "continuous integration" includes testing. Just because it compiles doesn't mean it works.
If your build and/or tests take a long time then continuous integration can get expensive. In that case, run the tests obviously related to your change before the commit (coverage analysis tools, like Devel::CoverX::Covered can help discover what tests go with what code), do your integration testing after the commit using something like SVN::Notify, and alert the developers if it fails. Archive the test results using something like Smolder. That allows developers to work quickly without having to sit around watching test suites run, while still catching mistakes early.
That said, with a little work you can often you can speed up your build and test process. Many times slow tests are the result of each test having to do too much setup and teardown pointing at a system that's far too coupled requiring the whole system to be setup just to test a small piece.
Decoupling often helps, breaking out sub-systems into independent projects. The smaller scope makes for easier understanding and faster builds and tests. Each commit can do a full build and test without inconveniencing the programmer. Then all the sub-projects can be collected together to do integration testing.
One of the major advantages of running the test suite on every commit, even if it's after the commit, is you know just what broke the build. Rather than "something we did yesterday broke the build", or worse "four things we did yesterday broke the build in different ways and now we have to untangle it" it's "revision 1234 broke the build". You only have to examine that one revision to find the problem.
The advantage of doing a daily build is that at least you know there's a complete, clean build and test run happening every day. But you should be doing that anyway.
I don't think there are any downsides to it. But for the sake of the argument, here is Eric Minick's article on UrbanCode ("It's about tests not builds.") He criticises the tools that are based on Martin Fowler's work saying that they don't let enough time for tests.
"To be truly successful in CI, Fowler asserts that the build should be self-testing and that these tests include both unit and end-to-end testing. At the same time, the build should be very fast - ideally less than ten minutes - because it should run on every commit. If there are a significant number of end-to-end tests, executing them at build time while keeping the whole process under ten minutes is unrealistic.
Add in the demand for a build on every commit, and the requirements start to feel improbable. The options are either slower feedback or the removal of some tests."
James Shore had a great series of blog entries on the dangers of thinking that using a CI tool like CruiseControl meant you were doing continuous integration:
Why I Don't like CruiseControl
Continuous Integration is an Attitude not a Tool
Continuous Integration on a Dollar a Day
One danger of setting up a CI server is goal displacement, thinking that the important thing is to "keep the build passing" as opposed to "ensuring we have high quality software". So people stop caring about how long the tests take to run. Then they take too long to run all of them before checkin. Then the build keeps breaking. Then the build is always broken. So people comment out the tests to make the build pass. And the quality of the software goes down, but hey, the build is passing...
There are generally two cases where I've seen continuous integration not really make sense. Keep in mind I am a big advocate of CI and try to use it when I can.
The first one is when the roi just doesn't make sense. I currently develop several small internal apps. The applications are normally very trivial and the whole lifecycle of the development is about a week or two. To properly setup everything for CI would probably double that and I probably would never see that investment back again. You can argue that I'll get it back in maintenance, but these apps are as likely to be discarded as they are updated. Keep in mind that your job is probably to ship software, not reach 100% code coverage.
The other scenario that I have heard mentioned is that CI doesn't make sense if you're not going to do anything with the results. For example, if your software has to be sent to QA, and the QA staff can only really look at a new version every couple of days, it makes no sense to have builds every few hours. If other developers aren't going to look at code metrics and try to improve them, it makes no sense to track them. Granted this is not the fault of CI not being a good technique, it is a lack of your team willing to embrace CI. Nevertheless, implementing a CI system in such a scenario doesn't make sense.
When starting, it takes a while to set everything up.
If you add tests, coverage, static code inspections, duplicate search, documentation build and deploys, it can take a long time (weeks) to get it right. After that, maintaining the build can be a problem.
e.g, if you add tests to solution, you can have the build detect them automatically based on some criteria or you have to manualy update build settings. Auto detection is much harder to get right. Same for coverage. Same of documentation generation...
The only good reason not to do continuous integration comes when you've gotten your project working to the point where your integration tests hadn't identified any defect in a good long while and they're taking too much time to run every time you do a build. In other words: you've done enough continuous integration that you've proven to yourself that you no longer need it.