I have an interesting dev requirement that i've been trying to find a decent solution to for some time. I thought I might look here for suggestions.
I have 4 server locations on a network drive: 2 for development, 2 for production. each pair is intended to be identical, though because we're manually managing files gaps often surface. Developing in this context goes like this:
Make changes on one dev server.
Copy the updated files to the other dev server.
preview in browser.
repeat as necessary until ready to go live.
copy files from one dev server to one prod server, then the other.
review and test and hope to got you remember all the dependancies.
Realize that something is missing in a stylesheet somewhere because a colleague missed a server 6 months ago.
commence swearing. lather. rinse. repeat.
The closest thing to a solution I've been able to get to thus far involved using the local/remote server setup with autosave in dreamweaver with mappings to the two dev servers. But all this stuff being over a network makes things slow. And dreamweaver is far from my favorite editor.
I'd love to be able to find some way of making this better. In my freelance stuff I've been getting into things like NPM and Grunt, and I'm convinced I might be able to do something with something along the lines of task automation as is seen there, but I'm not experienced enough with these yet to know if that's a direction I should be considering.
Any suggestions would be greatly appreciated.
You can use any source control system you like to achieve what you want. Git seems to be the most popular choice those days, but even CVS would do the trick
Related
I've been developing a web app locally on my local MAMP computer for the last few months. Now I am ready to launch it while continuing to add enhancements/fixes. So, I am wondering what is a good way to implement a development AND production server in order to efficiently manage updates, prevent overwrites, and seamlessly add other developers into the workflow. I also want something that has a minimal learning curve for me. Personally, for whatever reason, I've never been able to fully grasp version control systems like Git or SVN so I am hoping for an easier solution until I am able to invest more info the business.
As I see it, the options that I have are:
Spend more time learning Git before launching. And hoping that I don't break anything while further developing my app.
Buy two hosting accounts. One for Dev and one for Prod, where only I can do the deployments into Prod. I suppose I'd have to keep track of all files we've modified in a spreadsheet that are deemed ready for deployment.
Editing right on the FTP (no Dev server).
Are there any other options that you can recommend? I've heard that there are some new types of Web Hosting companies that can do the heavy lifting...
While personally, I have had good experiences using svn/git for multi-developer websites, I can understand your reticence to start relying on something you are not entirely familiar with. Unfortunately, I do believe that is your best option, but failing that, you might try using subdomains. My former employer would create test area on the disk and point beta.thedomainname.com at it. When bug fixes or upgrades were complete and verified to be working in the beta directory, the entire directory would be copied over to the live domain. Not the most elegant solution, but it worked. It certainly is cheaper than buying two hosting accounts.
So I've been a developer on a one man team for a few years now. I came to recognize the benefit of separating all of the design, html, and css work into the hands of the more gifted individuals, so I started a business, made up of other long-time 1 man teams who were limited to static websites (that looked gorgeous). You can imagine what comes next.
We've been using Zend Framework (PHP applications for most applications, which separates most of the view code from business and data layers, but occasionally one of the designers or front-end guys has to tinker with a model or view script, and I'm starting to have coordination nightmares for who is updating what files and when so we don't overwrite anyone's important changes in the dev environment. (obviously waiting for stability before production deployment)
I work with NetBeans on my local, while the others use dreamweaver. What I'm looking for is a tool that can manage file versions on the dev server, allowing check in/check out, so nobody is working on the same file at the same time. (or better yet, merges changes)
Any suggestions on a good tool for this? Would a CI tool be better or a VCS tool? The only other option I can think of is to work directly on a development server, deploy to a simprod environment, than after testing, move to prod, but I like being able to develop behind a firewall before putting something on the internet. Any recommendations would be helpful.
Thank you.
You are looking for git. Perfect for managing versions and branches, deploying to the dev or prod servers, pretty much anything you can throw at it.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
Web developing isn't what it used to be. It used to consist of hacking together a few PHP scripts (I have nothing against PHP, actually it's currently my main programming language), uploading them via FTP to some webhost and that was that. Today, things are more complicated. As I can see by looking at a number of professional and modern websites (SO being the main one, I consider SO being a great example of good practice in web developing, even if it's made with ASP.NET and hosted on Windows), developing a website is much more than that:
The website code is actually in a repository (that little svn revision in the footer makes my nerdy feelings tingle);
Static files (CSS, JavaScript, images) are stored on a separate domain;
Ok, these were my observations. Now for my questions:
What do you do with JavaScript and CSS files? Do you just not keep them under version control? That would seem stupid. Do you create a separate repository for them?
How do you set up the repository? Do you just create one in the root of the web server? Or do you create some sort of post-commit trigger that copies the latest files to their appropriate destinations?
What happens if you have multiple machines running the website and want to push some changes to all of them?
Every such project has to have configuration files. These differ from the local repository to the remote one. For example, on my development machine I have no MySQL root password, while on the production server I certainly have a password. This password would be stored in a config file, amongst other such things, which would be completely different on my machine and on the server. Maybe they are different between production machines, too (like I said earlier, maybe the website runs on multiple machines for load balancing). How do I handle that?
I'm looking to start a new web project using:
Python + SQLAlchemy + Werkzeug + Jinja2
Apache httpd + modwsgi
MySQL
Mercurial
What I'd like is some best practice advice on using the aforementioned tools and answers to my questions above.
You're right, things can get complicated when trying to deploy a scalable website. Here are what I've found to be a few good guidelines (disclaimer: I'm a rails engineer):
Most of the decisions regarding file structure for your code repository are largely based upon the convention of the language, framework and platform you choose to implement. Many of the questions you brought up (JS, CSS, assets, production vs development) is handled with Rails. However, that may differ from PHP to Python to whichever other language you want to use. I've found you should do some research about what language you're choosing to use, and try to find a way to fit the convention of that community. This will help you when you're trying to find help on an obstacle later. Your code will be organized like their code, and you'll be able to get answers more easily.
I would version control everything that isn't very substantial in size. The only problem I've found with VC is when your repo gets large. Apart from that I've never regretted keeping a version of previous code.
For deployment to multiple servers, there are many scripts that can help you accomplish what you need to do. For Ruby/Rails, the most widely used tool is Capistrano. There are comparable resources for other languages as well. Basically you just need to configure what your server setup is like, and then write or look to open source for a set of scripts that can deploy/rollback/manipulate your codebase to the servers you've outlined in your config file.
Development vs Production is an important distinction to make. While you can operate without that distinction, it becomes cumbersome quickly when you're having to patch up code all over your repository. If I were you, I'd write some code that is run at the beginning of every request that determines what environment you're running in. Then you have that knowledge available to you as you process that request. This information can be used when you specify which configuration you want to use when you connect to your db, all the way to showing debug information in the browser only on development. It comes in handy.
Being RESTful often dictates much of your design with regards to how your site's pages are discovered. Trying to keep your code within the restful framework helps you remember where your code is located, keeps your routing predictable, keeps your code from becoming too coupled, and follows a convention that is becoming more and more accepted. There are obviously other conventions that can accomplish these same goals, but I've had a great experience using REST and it's improved my code substantially.
All that being said. I've found that while you can have good intentions to make a pristine codebase that can scale infinitely and is nice and clean, it rarely turns out this way. If I were you, I'd do a small amount of research on what you feel the most comfortable with and what will help make your life easier, and go with that.
Hopefully that helps!
While I have little experience working with the tools you've mentioned, except for MySQL, I can give you a few fairly standard answers for the questions you posted.
1) Depends on the details, but most often you keep them in the same repository but in a separate folder.
2) Just because something is commited to the repository doesn't mean that it's ready to go live - it's quite often an intermediary build that could be riddled with bugs. A publish is done manually, with an export from the repository. Setting up the webserver in the same folder as a svn checkout is a huge nono as the .svn folder contains quite a bit of sensitive information, such as how to push changes to the svn server.
3) You use some sort of NAS or SAN solution, or simply a network share on one of the servers, and read all your data from there. That way, when you push information to one place, it's accessible by all servers. If your network is slow, you set up scripts that pushes the files out to all the servers automatically from a single location. If you use a multi-server environment in ASP.NET, don't forget to update the machine key in the config files or your shared encrypted caches, like the viewstate, won't work across servers. Having a session store in a database is also a good idea.
4) I've got a post build step that only triggers on publish that replaces my database connectionstrings with production ones, and also changes my Production app config value from false to true in the published web.config/app.config files. I can't see any case where you'd want different config files for different servers serving the same content.
If something is unclear, just comment and I'll try to clarify.
Good luck! // Eric Johansson
I think you are mixing 2 different aspects, source control and deployment. Just because you have all your files in a single repository doesnt mean they have to be deployed that way. Its also arguable whether you should be deploying directly using source control or instead using a build/deploy script which could handle any number of configurations.
Also hosting static files on a seperate domain only really becomes worthwhile on high traffic websites. Are you sure you aren't prematurely optimising?
I need to create some way to get a local WAR file deployed on a Linux server. What I have been doing until now is the following process:
Upload WAR using WinSCP.
SSH into server using PuTTY.
Move/Rename/Delete certain files folders to prepare for WAR explosion.
Explode WAR.
Send email notifying users of restart.
Stop Tomcat server.
Use tail to make sure server stopped correctly.
Change symlink to point to exploded WAR.
Start Tomcat.
Use tail to make sure server started correctly.
Send email notifying users of completed restart.
This stuff is all relatively straightforward. And I'm sure there are a million and one different ways to do it. Id like to hear about some options. My first thought was a Bash script. I have very little experience with scripting in general but thought this would be a good way to learn. I would also be interested in doing this with Ruby/Python or something current like this as I have little to no experience with these languages. I think as a young developer, I should definitely get some sort of scripting language under my belt. I may also be interested in some sort of software solution that could do this stuff for me, although I think scripting would be a better way to go for the sake of ease and customizability (I might have just made that word up).
Some actual questions for those that made it this far. What language would you recommend to automate the process I've listed above? Would this be a good opportunity for me to learn Bash/Ruby/Python/something else, or should I simply take the 10 minutes to do this by hand 2-3 times a week? I would think the answer to this is obviously no. Can I automate these things from my computer, or will I need to setup the scripts to run within the Linux server? Is the email something I can automate or am I better off doing that part myself?
More questions will almost certainly come up as I do this so thanks to all in advance.
UPDATE
I should mention, I am using Maven to build the WAR. So if I can do all of this with Maven please let me know.
This might be too heavy duty for your needs, but have you looked at build automation tools such as CruiseControl or Hudson? You might also want to look at Integrity, which is more lightweight and written in Ruby (instead of Java like the other two I mentioned). These tools can do everything you said you needed in your question plus way, way more.
Edit
Since you want this to be more of a learning exercise in scripting languages than a practical solution, here's an idea for you. Instead of manually uploading your WAR each time to your server, set up a Mercurial repository on your server and create a hook (see here, here, and especially here) that executes a Ruby (or ant, or maven) script each time a changeset is pushed from a remote computer (i.e. your local workstation). You would write the script so it does all the action items in your list above. That way, you will get to learn three new things: a distributed version control paradigm, how to customize said tool, and how to write Ruby scripts to interact with your operating system (since your actions are very filesystem heavy).
The most common in my experience is ant, it's worth learning, it's all pretty simple, and very usefull.
You should definately automate it, and you should aim to have it happen in 1 step.
What are you using to build the WAR file itself? There's some advantage to using the same tool for build and deployment. On several projects I've used Ant to build a Java project and deploy it to the servers.
My company has 3 developers. Me, another guy, and a VP dev. I really want to implement source control, especially since our code seems to randomly change on it's own. We tend to develop on the server, live, etc.
I'm fine with having a copy of our database on my machine to work against, if necessary, as is the other guy. The VP dev doesn't want it. How can I work with him to change his mind, or make it work for him?
You have to make him think it's his idea.
Point out that with source control you not only have a built-in backup of everything, but you also have the previous versions - let him realize how much of a good thing that is.
Install SVN and tell the one that opposes it that "everybody does it" :)
And seriously - source control is a MUST even for a single developer, let alone for three.
As for the DB server - you can use one development server (it can be a regular machine). It is of course no problem if you use each a local copy, but you must have a strong database schema generation/synchronization tools.
You should have source control. There isn't much excuse for not having it. Source control will protect you against changes that will cause problems in your code. I would recommend putting the db schema and data (sample set) in your version control. This will allow independent changes to the db without screwing up what your users see live on the site.
Note that you're not really asking about source control here, but about where your development dataset resides. Local databases per-developer are best, if possible, but failing that, a reasonable alternative is to just have a virtual machine containing your source control server and a development database.
Putting things under source control is really easy - literally, 10 minutes from now you could have your source under source control. Rather than try and persuade him the benefits I would just go ahead and do it anyway.
Start simply by putting a copy of your source under source control - even if he doesn't use it just merge the changes from live into your source control repository on a regular basis. At least that way you have a revision history (and if you are him are the only people changing the source, it means that any changes you didnt make, he must have made)
With luck, slowly over time he will begin to see the benefits (him: oh no - everything just broke! you: Don't worry, I'll just look and see what has changed since the last working copy...)
It sounds like you need to convince him that it is
Necessary to solve a problem,
an appropriate solution (does exactly what you need) and
easy to use.
It sounds like you have the information to demonstrate #1: the last time the code, "changed on its own," on the server and you lost someone's work or mixed results poorly. Bam, there's your "problem." #3 is the next more difficult: you need to pick an SCM with a good set of tools and do a demo. The TortoiseX line of products (TortoiseHg, TortoiseSVN) are great for this, because they make it non-scary.
Item 2 is the hardest: to demonstrate that this is the appropriate solution. Perhaps, to convince him of this, you might refer to anecdotes of other programmers or by looking at Github, where you can look back at previous versions of a product. I'm clutching at straws, here, because I feel like his argument will be, "Ach, and that's when it's a huge headache, is when things break. It won't be worth it."
Obviously there are a large number of ways to deal with people (and for the most part you have a "people") problem.
The first thing I'd do is find out why he's so against source control. Often times people who don't like source control either don't like:
The extra work of committing
Don't always work next to an internet
See no extra value in it
There are different solutions to each of these problems. Obviously the third one is tricky, so I'll handle it last.
If they don't like the extra work of committing, some chron scripts will help them (or windows scheduler). Something that regularly commits in the background, or recursively goes through his files and adds them for the next commit. This will mean you'll do a little more work on your end to clean up extra files and deal with broken builds, but its a step. Alternatively if he's emailing you the code, a script that commits the emails works as well.
If he's not always working next to internet access, consider a system like GIT. The advantage of GIT (over something like SVN) is that it utilizes a pull model instead of a push. As a result you pull updates from other GIT users instead of pushing commits. If you are working on a plane and don't have internet access, this is a valuable feature.
Finally, demonstrating the importance of the system is tough. The best example is almost always: "My machine burned down." I suppose you could nuke his box, but for the moment let's look at ways that don't piss off your boss.
A good way to demonstrate the importance of a repository is a Daily Build. Having a daily build means you can readily integrate features and find bugs faster. Setting up a repository with a daily build will significantly improve your work conditions, and its likely to make a good impression.
These are just a few of the reasons that people don't like source control, but the key idea is finding what his reason is and adapting to it.