In our small office we're setting up mercurial - our first time using a "real" version control system. We've got three servers - a live server, a staging server and a development server.
We've also got three relatively large web sites - one for visitors, one for users and an intranet site, for the office staff.
The three web sites share some code. (for instance - a php class library, some commonly used code snippets, etc.)
Before version control, we just used symbolic links to link to the shared libraries. For example: each site had a symbolic link to an "ObjectClasses" directory - any changes made to a file in ObjectClasses would be instantly available to all the sites. You'd just upload the changed file to staging and to live, and you were done.
But... Mercurial doesn't follow symbolic links. So I've set up a subrepository for the shared libraries in the three sites on the three servers (actually 'four' servers if you count the fact that there are two programmers with two separate clones of the repository on the development server).
So there are 12 working copies of the shared object library.
So here's the question:
Is there any way to simplify the above set up?
Here's an example of what our workflow will be and it seems too complicated - but maybe this is what it's like using version control and we just need to get used to it:
Programmer A makes a change to Object Foo in the subrepo in Site 1. He wants to make this available everywhere, so he commits it, then pushes it to the staging server. I set up hooks on the staging server to automatically propogate the changes to the three sites, on the staging server, and again to the three sites on the live server. That takes care of the 6 working copies on the staging and live servers. So far, so good.
but what about the development server, where there may be work-in-progress on these files?
Programmer A now needs to manually pull the shared subrepo to Sites 2 and 3 on the development server. He also needs to tell Programmer B to manually pull the shared subrepo on Sites 1, 2 and 3 on his copy of the site on the development server. What if he's editing Object Foo on Site 1 and making different edits to Object Foo on Site 2. He's going to have to resolve two separate conflicts.
We make changes to the objects relatively frequently. This is going to drive us nuts. I really love the idea of version control - but after two weeks of wrestling with trying to find the best setup, the old sloppy way of having one copy of the shared files and calling out "hey - ya working on that file, I wanna make a change" is looking pretty good right now.
Is there really no simpler way to set this up?
Without more information about the specific web platform and technologies you're using (e.g., .NET, LAMP, ColdFusion, etc.), this answer may be inadequate, but let me take a stab nevertheless. First, if I understand you correctly, it's your working paradigm that's the problem. You're having developers make changes to files and then push them to three different sites. I suggest separating the development concerns altogether from the build/deploy concerns.
It sounds like you're using subrepositories in Mercurial to handle shared code--which is smart, by the way--so that's good. That takes care of sharing code across multiple projects. But rather than have each programmer pushing stuff to a given server after he updates it, have the programmers instead be pushing to some other "staging" repository. You could have one for each of your servers if you wish, though I think it probably makes more sense to keep all development in a single staging or "master" repository which is then used to build/deploy to your staging and/or live server.
If you wish to automate this process, there are a number of tools that can do this. I usually prefer NAnt with CruiseControl for build integration, but then my work is mostly .NET which makes it a great fit. If you can provide more specifics I can provide more details if you like, but I think the main problem for you to overcome is the way you're handling the workflow. Use Mercurial to keep multiple developers happy pulling/pushing from a single repository and then worry about deploying to your servers for testing as a separate step.
Related
We're planning to select DNN+2sxc for a project for our team.
Normally when it comes to a CMS, I usually fly solo, but in a corporate .Net or Java environment it’s team collaboration, source control, Azure, deployments etc.
With our upcoming project we’re taking one of our main sites (C#/asp.net/razor) and converting it to DNN.
However, I’m currently unsure as to how to approach a CMS in a team development environment?
So in the development phase, we'll have some guys doing styling, others creating 2sxc reusable content templates and others building the actual pages. All at the same time, on the same website. In terms of Git/Visual Studio I'm not sure how it will actually work with relation to the DB especially. This question obviously applies to all CMSes (not just DNN) in a shared development environment.
What is the best practice to do this?
So I prefer to do most development locally, in my own instance (local IIS and local DB), with each individual project (module,theme/skin) in a separate repository. This makes the risk of me breaking someone else, or someone else causing me pain, minimal.
You can use a tool like Polydeploy to automate the deployment from the repository check ins into that upper environment. Requiring that individuals check code into the repository when ready to deploy to a test/uat/prod type environment.
Where it gets tricky is content for sure, I would typically do that in a test/uat environment that will ultimately be pushed to production once it is finalized.
I NEVER source control the DNN instance itself, that's just asking for pain.
This can be quite challenging, especially since some parts are user-data (which shouldn't be re-deployed on development) and other parts are dev.
There is a minimal guide to this here: https://docs.2sxc.org/abyss/enterprise-development/index.html
So I've been a developer on a one man team for a few years now. I came to recognize the benefit of separating all of the design, html, and css work into the hands of the more gifted individuals, so I started a business, made up of other long-time 1 man teams who were limited to static websites (that looked gorgeous). You can imagine what comes next.
We've been using Zend Framework (PHP applications for most applications, which separates most of the view code from business and data layers, but occasionally one of the designers or front-end guys has to tinker with a model or view script, and I'm starting to have coordination nightmares for who is updating what files and when so we don't overwrite anyone's important changes in the dev environment. (obviously waiting for stability before production deployment)
I work with NetBeans on my local, while the others use dreamweaver. What I'm looking for is a tool that can manage file versions on the dev server, allowing check in/check out, so nobody is working on the same file at the same time. (or better yet, merges changes)
Any suggestions on a good tool for this? Would a CI tool be better or a VCS tool? The only other option I can think of is to work directly on a development server, deploy to a simprod environment, than after testing, move to prod, but I like being able to develop behind a firewall before putting something on the internet. Any recommendations would be helpful.
Thank you.
You are looking for git. Perfect for managing versions and branches, deploying to the dev or prod servers, pretty much anything you can throw at it.
I'm looking for a best practice scenario on managing multiple "sites" in mercurial. Since I'm likely to have multiple sites in a web root, all of which are different - but somewhat similar (as they are 'customizations' of a root app)
Should I
A) make a single repository of the wwwroot folder (catching all changes across all sites)
B) make EACH sits folder a different repository
this issue is that each site needs a distinct physical directory, due to vhost pointing for development, and a current need to have "some" physical file difference cross site.
What's the best practice here? I'm leaning towards separate repositories for each directory. which will make tracking any branching and merging for that ONE site cleaner....
It depends on how your software is structured, and how independent the different sites are. The best situation is when you can use your core code like a library, which lives in its own directory, and there is no need in the different sites to change even a single file of core. Then you have the free choice if you want to develop the core along with the different sites in a single repo, or to seperate core from sites. When core and the different sites are dependent on each other, you very probably have to deal with all of them in a sigle repo.
Since in my experience development work better when the different parts are independend of each other I strongly recommend to bring the core stuff into something which can be included into the sites by a directory inclusion.
The next point is how are the different sites developed. If they share lots of code, they can be developed as different branches. But there are two disadvantages of this scheme:
the different sites are normally not visible to the developer, since there is typically only one checked out
The developer has to take great care where to create changes, so that only the wanted changes are going into other branches, not something which is special to a single branch only
You might consider to move common parts of different sites into core if they share lots of common code.
Another situation is if they all have nothing in common, since then things are much better. Then you need to decide if you want them to reside in different repos, or as different directories in a single repos. When these different sites are somehow related to each other (say that they are all of the same company), then it might be better to put them into a common repo, as different subdirectories. When they are unrelated to each other (every site belongs to a different customer, and changes on these sites are not created in synch to each other), than one repo per site is better.
When you have the one repo per site approach, it might also be good if you first create a template site, which includes the core component and basic configuration, and derive your site-repos as clones from this template. Then when you change something in the core which also affects the sites, you do these changes in the template, and merge these changes afterwards into the sites repos (you only need to take care to NOT do this change in one of the site-repos, since when you merge from sites to template you might get stuff from the specific site into the template, which you don't want to be in the template).
So I suggest
develop core as a single independent product
choose the correct development model for your sites
all in one repo, with branches, when there is much code-exchange is goin on between different sites
but better refactor the sites to not share code, since the branches approach has drawbacks
all in one repo, no branches but different folders, if there is no code exchange between different sites
one repo for each site if they are completely independent.
I think, you have to try Mercurial Queues with one repo. I.e
you store "base" site in repository
all site-specific changes separated into the set of MQ-patches (one? patch per site)
you can create "push-only" repos in sites, add they to [paths] section of "working" repo and push changes or use export-copy technique
and after applying the site-patch to codebase you'll get ready to use code for each and every site
Our repositories are getting huge because there's tons of media we have ( hundreds of 1 MB jpegs, hundreds of PDFs, etc ).
Our developers who check out these repositories have to wait an abnormally long time because of this for certain repos.
Has anyone else had this dilemma before? Am I going about it the right way by separating code from media? Here are some issues/worries I had:
If I migrate these into a media server then I'm afraid it might be a pain for the developer to use. Instead of making updates to one server he/she will have to now update two servers if they are doing both programming logic and media updates.
If I migrate these into a media server, I'll still have to revision control the media, no? So the developer would have to commit code updates and commit media updates.
How would the developer test locally? I could make my site use absolute urls, eg src="http://media.domain.com/site/blah/image.gif", but this wouldn't work locally. I assume I'd have to change my site templating to decide whether it's local/development or production and based on that, change the BASE_URL.
Is it worth all the trouble to do this? We deal with about 100-150 sites, not a dozen or so major sites and so we have around 100-150 repositories. We won't have the time or resources to change existing sites, and we can only implement this on brand new sites.
I would still have to keep scripts that generate media ( pdf generators ) and the generated media on the code repository, right? It would be a huge pain to update all those pdf generators to POST files to external media servers, and an extra pain taking caching into account.
I'd appreciate any insight into the questions I have regarding managing media and code.
First, yes, separating media and generated content (like the generated pdf) from the source control is a good idea.
That is because of:
disk space and checkout time (as you describe in your question)
the lack of CVS feature actually used by this kind of file (no diff, no merge, only label and branches)
That said, any transition of this kind is costly to put in place.
You need to separate the release management process (generate the right files at the right places) from the development process (getting from one or two referential the right material to develop/update your projects)
Binaries fall generally into two categories:
non-generated binaries:
They are best kept in an artifact repository (like Nexus for instance), under a label that would match the label used for the text sources in a VCS
generated binaries (like your pdf):
ideally, they shouldn't be kept in any repository, but only generated during the release management phase in order to be deployed.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
Web developing isn't what it used to be. It used to consist of hacking together a few PHP scripts (I have nothing against PHP, actually it's currently my main programming language), uploading them via FTP to some webhost and that was that. Today, things are more complicated. As I can see by looking at a number of professional and modern websites (SO being the main one, I consider SO being a great example of good practice in web developing, even if it's made with ASP.NET and hosted on Windows), developing a website is much more than that:
The website code is actually in a repository (that little svn revision in the footer makes my nerdy feelings tingle);
Static files (CSS, JavaScript, images) are stored on a separate domain;
Ok, these were my observations. Now for my questions:
What do you do with JavaScript and CSS files? Do you just not keep them under version control? That would seem stupid. Do you create a separate repository for them?
How do you set up the repository? Do you just create one in the root of the web server? Or do you create some sort of post-commit trigger that copies the latest files to their appropriate destinations?
What happens if you have multiple machines running the website and want to push some changes to all of them?
Every such project has to have configuration files. These differ from the local repository to the remote one. For example, on my development machine I have no MySQL root password, while on the production server I certainly have a password. This password would be stored in a config file, amongst other such things, which would be completely different on my machine and on the server. Maybe they are different between production machines, too (like I said earlier, maybe the website runs on multiple machines for load balancing). How do I handle that?
I'm looking to start a new web project using:
Python + SQLAlchemy + Werkzeug + Jinja2
Apache httpd + modwsgi
MySQL
Mercurial
What I'd like is some best practice advice on using the aforementioned tools and answers to my questions above.
You're right, things can get complicated when trying to deploy a scalable website. Here are what I've found to be a few good guidelines (disclaimer: I'm a rails engineer):
Most of the decisions regarding file structure for your code repository are largely based upon the convention of the language, framework and platform you choose to implement. Many of the questions you brought up (JS, CSS, assets, production vs development) is handled with Rails. However, that may differ from PHP to Python to whichever other language you want to use. I've found you should do some research about what language you're choosing to use, and try to find a way to fit the convention of that community. This will help you when you're trying to find help on an obstacle later. Your code will be organized like their code, and you'll be able to get answers more easily.
I would version control everything that isn't very substantial in size. The only problem I've found with VC is when your repo gets large. Apart from that I've never regretted keeping a version of previous code.
For deployment to multiple servers, there are many scripts that can help you accomplish what you need to do. For Ruby/Rails, the most widely used tool is Capistrano. There are comparable resources for other languages as well. Basically you just need to configure what your server setup is like, and then write or look to open source for a set of scripts that can deploy/rollback/manipulate your codebase to the servers you've outlined in your config file.
Development vs Production is an important distinction to make. While you can operate without that distinction, it becomes cumbersome quickly when you're having to patch up code all over your repository. If I were you, I'd write some code that is run at the beginning of every request that determines what environment you're running in. Then you have that knowledge available to you as you process that request. This information can be used when you specify which configuration you want to use when you connect to your db, all the way to showing debug information in the browser only on development. It comes in handy.
Being RESTful often dictates much of your design with regards to how your site's pages are discovered. Trying to keep your code within the restful framework helps you remember where your code is located, keeps your routing predictable, keeps your code from becoming too coupled, and follows a convention that is becoming more and more accepted. There are obviously other conventions that can accomplish these same goals, but I've had a great experience using REST and it's improved my code substantially.
All that being said. I've found that while you can have good intentions to make a pristine codebase that can scale infinitely and is nice and clean, it rarely turns out this way. If I were you, I'd do a small amount of research on what you feel the most comfortable with and what will help make your life easier, and go with that.
Hopefully that helps!
While I have little experience working with the tools you've mentioned, except for MySQL, I can give you a few fairly standard answers for the questions you posted.
1) Depends on the details, but most often you keep them in the same repository but in a separate folder.
2) Just because something is commited to the repository doesn't mean that it's ready to go live - it's quite often an intermediary build that could be riddled with bugs. A publish is done manually, with an export from the repository. Setting up the webserver in the same folder as a svn checkout is a huge nono as the .svn folder contains quite a bit of sensitive information, such as how to push changes to the svn server.
3) You use some sort of NAS or SAN solution, or simply a network share on one of the servers, and read all your data from there. That way, when you push information to one place, it's accessible by all servers. If your network is slow, you set up scripts that pushes the files out to all the servers automatically from a single location. If you use a multi-server environment in ASP.NET, don't forget to update the machine key in the config files or your shared encrypted caches, like the viewstate, won't work across servers. Having a session store in a database is also a good idea.
4) I've got a post build step that only triggers on publish that replaces my database connectionstrings with production ones, and also changes my Production app config value from false to true in the published web.config/app.config files. I can't see any case where you'd want different config files for different servers serving the same content.
If something is unclear, just comment and I'll try to clarify.
Good luck! // Eric Johansson
I think you are mixing 2 different aspects, source control and deployment. Just because you have all your files in a single repository doesnt mean they have to be deployed that way. Its also arguable whether you should be deploying directly using source control or instead using a build/deploy script which could handle any number of configurations.
Also hosting static files on a seperate domain only really becomes worthwhile on high traffic websites. Are you sure you aren't prematurely optimising?