First I should disclose I only manage vendor relationships and lack deep technical knowledge.
I just had a conference call with one of our sub-contractors who has asked me to sign off on a PO for 4 different servers (one for Jenkins another for Github, a third for Puppet and a fourth as a test box).
The technical architect seems quite defensive when I ask him questions. I know its not my job to question his ability but I do have a budget to manage and I am concerned they have over- engineered this (or at least the 3 products they have mentioned seem to overlap alot).
Would someone be so kind as to clearly explain to me the role each one plays.
I would also appreciate a second opinion as to whether they really need 4 servers and whether some of the technologies mentioned could co-exist on the same servers. (i.e could Jenkins, Github and Puppet all live on a single Ubuntu server ?).
The 4 tools do not overlap. They interact and complement altogether.
GIT is the source control tool. You store all the history of programming there. It's the dev blackbox.
Jenkins is the continuous integration tool. It will use GIT to get the latest version of the code ( or the testing version or the pre-production version ) to test it against the test patterns you defined.
Puppet seems to be a server administration toolbox.
Honestly, it depends on your project. If it's a huge app that requires heavy building cycles, then jenkins will be better off on its own server, so that people can still work normally with other services.
I believe Jenkins/Puppet could be on the same server. Or so could be Pupper and GIT ( git is very low resource ).
The testbox sounds ok. But I think that the GIT box could also be served as a testbox.
I think you could cut that server needs by 2. But again, it depends on the size of your project. If it's a big project, play it fair and let them do it. Even if the server split is a little strange, it could be necessary.
But frankly I think you don't really need that.
Related
our small team of 2 people recently got upgraded to 5, which means we should introduce a bit more infrastructure around our project in order to work efficiently together. Its a university research project.
How much administrative effort is it to run
Gitlab
Jenkins
A Release Server
on a rented machine, in comparison to a SaaS solution e.g github with Travis ?
Unfortunately nobody in our (quite academic) team has practical experience with that. I know the setup can be done fast, but how time consuming is it to keep the stuff running. Are there other concerns we might be missing?
Of course we would like to mainly work on the project itself, but since the toolstack keeps growing by time we are not sure if a SaaS solution is what we need.
With these inputs be ready to spend from one to several weeks of one from the five.
Instead you might want to check which leading hosting companies provide free services for open source or educational projects.
I've been developing a web app locally on my local MAMP computer for the last few months. Now I am ready to launch it while continuing to add enhancements/fixes. So, I am wondering what is a good way to implement a development AND production server in order to efficiently manage updates, prevent overwrites, and seamlessly add other developers into the workflow. I also want something that has a minimal learning curve for me. Personally, for whatever reason, I've never been able to fully grasp version control systems like Git or SVN so I am hoping for an easier solution until I am able to invest more info the business.
As I see it, the options that I have are:
Spend more time learning Git before launching. And hoping that I don't break anything while further developing my app.
Buy two hosting accounts. One for Dev and one for Prod, where only I can do the deployments into Prod. I suppose I'd have to keep track of all files we've modified in a spreadsheet that are deemed ready for deployment.
Editing right on the FTP (no Dev server).
Are there any other options that you can recommend? I've heard that there are some new types of Web Hosting companies that can do the heavy lifting...
While personally, I have had good experiences using svn/git for multi-developer websites, I can understand your reticence to start relying on something you are not entirely familiar with. Unfortunately, I do believe that is your best option, but failing that, you might try using subdomains. My former employer would create test area on the disk and point beta.thedomainname.com at it. When bug fixes or upgrades were complete and verified to be working in the beta directory, the entire directory would be copied over to the live domain. Not the most elegant solution, but it worked. It certainly is cheaper than buying two hosting accounts.
So I've been a developer on a one man team for a few years now. I came to recognize the benefit of separating all of the design, html, and css work into the hands of the more gifted individuals, so I started a business, made up of other long-time 1 man teams who were limited to static websites (that looked gorgeous). You can imagine what comes next.
We've been using Zend Framework (PHP applications for most applications, which separates most of the view code from business and data layers, but occasionally one of the designers or front-end guys has to tinker with a model or view script, and I'm starting to have coordination nightmares for who is updating what files and when so we don't overwrite anyone's important changes in the dev environment. (obviously waiting for stability before production deployment)
I work with NetBeans on my local, while the others use dreamweaver. What I'm looking for is a tool that can manage file versions on the dev server, allowing check in/check out, so nobody is working on the same file at the same time. (or better yet, merges changes)
Any suggestions on a good tool for this? Would a CI tool be better or a VCS tool? The only other option I can think of is to work directly on a development server, deploy to a simprod environment, than after testing, move to prod, but I like being able to develop behind a firewall before putting something on the internet. Any recommendations would be helpful.
Thank you.
You are looking for git. Perfect for managing versions and branches, deploying to the dev or prod servers, pretty much anything you can throw at it.
I am pretty comfortable with the producing web apps now. I am using a NodeJs stack on the back-end and usually have a fair amount of Javascript on the front end. Where I really lack understanding is the deployment process.
What is a typical deployment process?
From what I have gathered in my reading a deployment/build process can include several tasks:
Running through unit-test suites
Concatenating script and CSS files
Version numbering your app
Tracing module dependencies (node_modules)
Pushing it to a remote repo (GitHub)
Instructing 'staging' servers to pull down the latest repo
Instructing 'production' server to pull down the latest repo
This has all left me a little overwhelmed. I don't know whether I should be going into this level of detail for my own projects, it seems like a lot of work! I am using Sublime Text 2 IDE and it seems to have a Build Script process, is this suitable? How does one coordinate all these separate tasks? I'm imagining ideally they would all run at the flick of a switch.
Sorry so many questions, but I need to know how people learnt similar principles. Some of my requirements may be specific to NodeJS but I'm sure processes are similar no matter what choice of stack you are developing in.
First off, let's split the job in two: front-end and back-end stuff. For both, you really want some kind of bulid system, but their goals and scope are vastly different.
For the front-end, you want your source to be as small as possible; concatenate/minify JavaScript, CSS and images. A colleague of mine has written a "compiler", Assetgraph, to do this for you. It has a somewhat seep learning-curve, but it does wonders for your code (our dev-builds are usually ~20 megs, production ~500 k).
As to the back-end, you want contained, easily managed bundles of some sort. We re-package our stuff into debian-packages. As long as the makefile is wired up correctly, you get a lot of the boring build- and deploy-time stuff for free. Here's my (pre-NPM 1.0) Debianizing node programs. I've seen other ways to do this in NPM and on Github, but I haven't looked into them, so I can't speak on their quality.
For testing/pusing around/deploying, we use a rather convoluted combination of Debian package-archives, git-hooks, Jenkins-servers and what not. While I highly recommend using the platforms' native package-manager for rolling out stuff, it can be a bit too much. All in all, we usually deploy staging either automatically (on each git push), or semi-automatic for unstable codebases. Production deployments are always done explicitly.
For the assets I use asereje https://github.com/masylum/asereje
I recently documented my nodejs deployment process in a blog post:
http://pau.calepin.co/how-to-deploy-a-nodejs-application-with-monit-nginx-and-bouncy.html
A build script sounds like a good idea indeed.
What should that build script do?
make sure all the test pass, else exit immediately
concatenate your javascript and css files into one single js/css file and minify them also
increment the version number (unless you decide to set that up yourself manually)
push to the remote repo (which instructs the staging and production servers through a git hook)
This is at least my opinion.
Other resources:
http://howtonode.org/deploying-node-with-spark
http://howtonode.org/deploying-node-upstart-monit
http://dailyjs.com/2010/03/15/hosting-nodejs-apps/
How to deploy node app depencies
I've been tasked with setting up a version control for our web developers. The software, which was chosen for me because we already have other non-web developers using it, is Serena PVCS.
I'm having a hard time trying to decide how to set it up so I'm going to describe how development happens in our system, and hopefully it will generate some discussion on how best to do it.
We have 3 servers, Development, UAT/Staging, and Production. The web developers only have access to write and test their code on the Development server. Once they write the code, they must go through a certification process to get the code moved to UAT/Staging, then after the code is tested thoroughly there, it gets moved to Production.
It seems like making the Developers use version control for their code on Development which they are constantly changing and testing would be an annoyance. Normally only one developer works on a module at a time so there isn't much, if any, risk of over-writing other people's work.
My thought was to have them only use version control when they are ready to go to UAT/Staging. This allows them to develop and test without constantly checking in their code.
The certification group could then use the version control to help see what changes had been made to the module and to make sure they were always getting the latest revision from the developer to put up on UAT/Staging (now we rely on the developer zip'ing up their changed files and uploading them via a web request system).
This would take care of the file side of development, but leaves the whole database side out of version control. That's something else that I need to consider...
Any thoughts or ideas would be greatly appreciated. Thanks.
I would not treat source control as annoyance. See Nicks answer for the reasons.
If I were You, I would not decide this on my own, because it is not a
matter of setting up a version control software on some server but
a matter of changing and improving development procedures.
In Your case, it might be worth explaining and discussing release branches
with Your developers and with quality assurance.
This means that Your developers decide which feature to include into a release
and while the staging crew is busy on testing the "staging" branch of the source,
Your developers can already work on the next release without interfering with the staging team.
You can also think about feature branches, which means that there is a new branch for every specific new feature of the web site. Those branches are merged back, if the feature is implemented.
But again: Make sure, that Your teams agreed to the new development process. Otherwise, You waste Your time by setting up a version control system.
The process should at least include:
When to commit.
When to branch/merge.
What/When to tag.
The overall work flow.
I have used Serena, and it is indeed an annoyance. In addition to the unpleasantness of the workflow overhead Serena puts on top of the check in-check out process, it is a real pain with regard to doing anything besides the simplest of tasks.
In Serena ChangeMan, all code on local machines is managed through a central server. This is a really bad design. This means a lot of day-to-day branch maintenance work that would ordinarily be done by developers has to go through whomever has administrator privileges, making that person 1) a bottleneck and 2) embittered because they have a soul-sucking job.
The centralized management also strictly limits what developers are able to do with the code on their own machine. For example, if you want to create a second copy of the code locally on your box, just to do a quick test or whatever, you have to get the administrator to set up a second repository on your box. When you limit developers like this, you limit the productivity and creativity of your team.
Also, the tools are bad and the user interface is horrendous. And you will never be able to find developers who are already trained to use it, because its too obscure.
So, if another team says you have to use Serena, push back. That product is terrible.
Using source control isn't any annoyance, it's a tool. Having the benefits of branching and tagging is invaluable when working with new APIs and libraries.
And just a side note, a couple of months back one of the dev's machine's failed and lost all his newest source, we asked when the last time he committed code to the source control and it was 2 months. Sometimes just having it to back up stuff when you reach milestones is nice.
I usually commit to source control a couple of times a week, depending if I've hit a good stopping point and I'm about to move on to something different or bigger.
Following on from the last two good points I would also ask your other non-web developers what developmet process they are using so you won't have to create a new one. They would also have encountered many of he problems that occur in your environment, both technical using the same OS and setup and managerial.