I'll admit this is an odd question as I'm writing, but I'm curious if such a solution exists or not. Basically, I want to run database-style migrations on files that live on the operating system itself, and I'm wondering if a solution exists or not.
The background of this is that I have a virtual machine appliance which I'm providing to customers deployed behind corporate firewalls, and thus, I don't have much access to it. However, I need to provide continuous updates not only for the software running on the machine (running in Docker), but sometimes, for some supporting files as well.
Since this is a product, it means I need to have the capability to roll-back a migration to a certain version if something goes wrong. git cannot work for this purpose since putting entire OS folders into a Git repo seems like a terrible idea.
Am I thinking about this wrong? Is such a solution like this reasonable?
Related
I would like to develop and deploy an Erlang/OTP application into production on a VPS.
I am pretty familiar with developing Erlang code on a local machine and my question is about deployment.
Basically, I would like to know what steps I should take in order to move Erlang code from a local machine to a production server and make it run, i.e. be available for users.
Note: I have read some documentation about Erlang and command line, Erlang code module, Erlang releases, but I am still not sure how to pursue the required task.
However, I guess that deploying an Erlang-based software on a server is a bit more tricky than doing sudo tasksel for LAMP.
I plan to have an Erlang/OTP application which has Mochiweb, CouchDB (couchbeam) and boss_db as dependencies.
So, my newbie questions about deploying all that stuff on a production server are the following:
I plan to use Ubuntu Server 12.04; is there any better choice for a Linux distro to use for Erlang/OTP in production?
How all the code should be organized? Should I put my application into a /home/myapp/ dir and then put all the dependencies into /home/myapp/deps? Or should I put all dependencies into /usr/local/lib/erlang/lib? (returned by code:get_path()). Should I somehow update the dependencies regularly or should I freeze them?
How do I make the whole application start once the server starts? Should it be some kind of bash script or anything else?
I know that Erlang allows hot code upgrades, but how should I organize that? On Rails I could update the code with git, does anything similar exist in the Erlang world?
There are two types of dependencies: Internal and External. If you want to do it the right way(tm), it takes a bit of time getting to work:
External dependencies:
Taking the latter first, an external dependency is some other thing that has to run before your application can run. For instance a PostgreSQL database, or a Riak cluster. For those, you usually just use the usual stuff in Ubuntu for making it start up properly. I've had good experience with using monit for these tasks:
http://mmonit.com/monit/
Internal Dependencies:
For internal dependencies, you need to arrange your program into applications inside the Erlang VM. These have dependencies on each other, like the external dependencies. Your main application may need a logger running before it should start, for instance. Then you create a release. A release copies the Erlang binaries and necessary libraries/beams/applications into a release directory, forming a self-contained Erlang system. It contains a boot-script which tells how to start up the applications in the right order and keep them running. So you can tar-ball up this release, copy it to the server and then start it. There are some basics covered here:
http://learnyousomeerlang.com/release-is-the-word
but do also read the chapters before it on applications. You can also get rebar to call reltool for you to build a release. This is what I usually do.
Hot upgrades:
Handling hot upgrades in production can be done in a couple of ways. You can move the beam to the machine and then deploy it, take the shell and then call l(Module) to load it into the running system. This works for smaller fixes. For large systematic upgrades you can do a release-upgrade which will upgrade the running system on the fly without stopping service. But if your system is mostly shared nothing, it is usually not worth it. Instead, you can have multiple machines and upgrade them in sequence.
For instance, you can upgrade a machine and then use a system like HAProxy to send 2% of all requests to the new system. Then systematically turn up the request load weight.
While #I GIVE CRAP ANSWERS gave a pretty thorough summary, I feel compelled to throw in the use of sync, which helps to automate the hot-recompiling and reloading of modules.
The simple way is you specify sync as a rebar dependency, then when you're getting ready to deploy an upgrade, you can run sync:go() on the Erlang node. This starts the sync engine, which watches for filesystem changes. Then you can use git to push to your server. Sync will notice the files change, recompile them, and load the new beams automatically.
Then, you can run sync:stop() right away to tell the system to stop watching for filesystem changes (it's generally not recommended to keep sync running on a live server, just to prevent accidental recompiling if, for whatever reason, a source file changes and it's unintentional.
I have an application running on a solaris machine with configuration modifications and deletions of configuration handled by multiple persons in the team....I would like to streamline this process to ensure no loss of configuration and make it easier to identify when/where the changes are done and by whom and retreive the configuration files as necessary.
I looked at svn and other repositories but they all seem to work with a repository stored on some machine and all the chnages have to be made then and there....
I am hoping to find a solution where a service would be running on solaris in teh background and monitors the changes and automatically creates the necessary versioning.
Am I asking for something that doesn't exist? or are they better approaches to solving this issue?
Thank you.....
A couple thoughts. If you mean system configuration then you can use SMF (Service Management Facility). This provides a service for managing the configuration, inter dependencies, and the running of services. Tracking of changes is easy but I'm not so sure if versioning would be easy to manage.
The next to last paragraph makes me think you want to allow the user to make ad hoc changes and have some daemon monitor and store those. That could probably be done with FAM and maybe use of ZFS versioning but that seems like overkill.
As far as a better approach. I would tend to use ZFS snapshots or a like mechanism to checkpoint my configuration. And then I would use SMF to manage my configuration. Maintaining versions of the configuration of individual subsystems doesn't quite do it for me. It is the configuration of the whole box that is interesting.
We will be embarking on an Application developement project (.NET 3.5) for a large organization. As we started thinking about the upgrades we would be giving across the machines, we are looking at options like ClickOnce.
What we need is a push model, as long as the client machine is connected to the network, the server can send updates. I believe ClickOnce is a pull model(although by specifying minimum version we can kind of push). Also ClickOnce downloads complete files only, it cannot download the change (byte difference) among the files.
Can anyone point me to a better tool that can be used here. Also better strategies, if any, are welcome, we are in a very early stage of the project.
I don't have a definitive answer on better options, but I've used ClickOnce and can offer some advice.
There are several update options with ClickOnce (before starting, after starting, check every time, check every X Hours/Days/Weeks, etc). You can also throw those out and write code to check for updates. It's not a "push" from the server, but your client could poll for updates which would be the next best thing. Just remember, the application is going to have to restart after the update to see changes.
ClickOnce only downloads changed files. However, the progress dialog always shows the entire size of the application even if it's only downloading a single file. Everyone worries about that, but it's just a bug with the progress dialog.
Finally, I'm a big fan of keeping it simple. It's really easy to over-think these things and create a monstrosity that was never needed. We went through something similar at my company. We were so worried about users downloading unnecessary bytes, we broke our apps up into more, smaller assemblies. It turned into a nightmare; apps were harder to maintain and performed worse on the client. We finally undid it all and wasted weeks just to end up where we started.
I'm not saying you don't need the features you're asking for, I don't know your scenario. Just educate yourself first and know what you're getting yourself into.
We use clickonce at my company (about few hundred users for the app geographically dispersed). By specifying the minimum version we can make sure that every app installation gets updated after deployment automatically. You are right that clickonce downloads full files only but only files that have changed since previous version. If that is still a concern you can break your application into more smaller assemblies. I think you can also use netmodules but then Visual Studio has not built in support for that.
In general clickonce has worked good for us.
I am just in the process of implementing such a service on top of my distributed application platform. In essence I have developed a "push" model for corporates that follows these basic principles:
Software upgrades are "managed" from the server, NOT from the client, which is in line with the deployment of corporate software as opposed to user software (this is a very important point)
Software upgrades can be customised per client application on the server, i.e. the server can deploy unique configurations to every client if required
Software upgrades can be deployed to clients at different times, or all at the same time, or any combination of the two
The software upgrade version can be specified per client, i.e. different versions can be deployed to different clients as required
All software upgrades for all clients can be "managed" from a single server, i.e. the software upgrading "service" is consistent across any application, and all applications can utilise the software upgrading "service"
Clients can implement a software upgrade policy of automatic (application restarts as soon as the upgrade has been downloaded and available at the client), manual (application needs to be "sent" a custom "force upgrade"
message"), or on restart (application upgrades on shutdown if an upgrade has been downloaded and is available)
All auto-upgrading functionality is transparent to any running applications as this is all performed in autonomous background threads and all inter-process communication and file transfer is handled by my framework
In essence this now allows me (or will allow me when I have tidied a few things up and thoroughly tested the implementation) to manage the version of any application developed by me from a central server after it has been initially installed, without any client intervention.
What are the recommendations for including your compiler, libraries, and other tools in your source control system itself?
In the past, I've run into issues where, although we had all the source code, building an old version of the product was an exercise in scurrying around trying to get the exact correct configuration of Visual Studio, InstallShield and other tools (including the correct patch version) used to build the product. On my next project, I'd like to avoid this by checking these build tools into source control, and then build using them. This would also simplify things in terms of setting up a new build machine -- 1) install our source control tool, 2) point at the right branch, and 3) build -- that's it.
Options I've considered include:
Copying the install CD ISO to source control - although this provides the backup we need if we have to go back to an older version, it isn't a good option for "live" use (each build would need to start with an install step, which could easily turn a 1 hour build into 3 hours).
Installing the software to source control. ClearCase maps your branch to a drive letter; we could install the software under this drive. This doesn't take into account non-file part of installing your tools, like registry settings.
Installing all the software and setting up the build process inside a virtual machine, storing the virtual machine in source control, and figuring out how to get the VM to do a build on boot. While we capture the state of the "build machine" with ease, we get the overhead of a VM, and it doesn't help with the "make the same tools available to developers issue."
It seems such a basic idea of configuration management, but I've been unable to track down any resources for how to do this. What are the suggestions?
I think the VM is your best solution. We always used dedicated build machines to get consistency. In the old COM DLL Hell days, there were dependencies on (COMCAT.DLL, anyone) on non-development software installed (Office). Your first two options don't solve anything that has shared COM components. If you don't have any shared components issue, maybe they will work.
There is no reason the developers couldn't take a copy of the same VM to be able to debug in a clean environment. Your issues would be more complex if there are a lot of physical layers in your architecture, like mail server, database server, etc.
This is something that is very specific to your environment. That's why you won't see a guide to handle all situations. All the different shops I've worked for have handled this differently. I can only give you my opinion on what I think has worked best for me.
Put everything needed to build the
application on a new workstation
under source control.
Keep large
applications out of source control,
stuff like IDEs, SDKs, and database
engines. Keep these in a directory as ISO files.
Maintain a text document, with the source code, that has a list of the ISO files that will be needed to build the app.
I would definitely consider the legal/licensing issues surrounding the idea. Would it be permissible according to the various licenses of your toolchain?
Have you considered ghosting a fresh development machine that is able to build the release, if you don't like the idea of a VM image? Of course, keeping that ghosted image running as hardware changes might be more trouble than it's worth...
Just a note on the versionning of libraries in your version control system:
it is a good solution but it implies packaging (i.e. reducing the number of files of that library to a minimum)
it does not solves the 'configuration aspect' (that is "what specific set of libraries does my '3.2' projects need ?").
Do not forget that set will evolves with each new version of your project. UCM and its 'composite baseline' might give the beginning of an answer for that.
The packaging aspect (minimum number of files) is important because:
you do not want to access your libraries through the network (like though dynamic view), because the compilation times are much longer than when you use local accessed library files.
you do want to get those library on your disk, meaning snapshot view, meaning downloading those files... and this is where you might appreciate the packaging of your libraries: the less files you have to download, the better you are ;)
My organisation has a "read-only" filesystem, where everything is put into releases and versions. Releaselinks (essentially symlinks) point to the version being used by your project. When a new version comes along it is just added to the filesystem and you can swing your symlink to it. There is full audit history of the symlinks, and you can create new symlinks for different versions.
This approach works great on Linux, but it doesn't work so well for Windows apps that tend to like to use things local to the machine such as the registry to store things like configuration.
Are you using a continuous integration (CI) tool like NAnt to do your builds?
As a .Net example, you can specify specific frameworks for each build.
Perhaps the popular CI tool for whatever you're developing in has options that will allow you to avoid storing several IDEs in your version control system.
In many cases, you can force your build to use compilers and libraries checked into your source control rather than relying on global machine settings that won't be repeatable in the future. For example, with the C# compiler, you can use the /nostdlib switch and manually /reference all libraries to point to versions checked in to source control. And of course check the compilers themselves into source control as well.
Following up on my own question, I came across this posting referenced in the answer to another question. Although more of a discussion of the issue than an aswer, it does mention the VM idea.
As for "figuring out how to build on boot": I've developed using a build farm system custom-created very quickly by one sysadmin and one developer. Build slaves query a taskmaster for suitable queued build requests. It's pretty nice.
A request is 'suitable' for a slave if its toolchain requirements match the toolchain versions on the slave - including what OS, since the product is multi-platform and a build can include automated tests. Normally this is "the current state of the art", but doesn't have to be.
When a slave is ready to build, it just starts polling the taskmaster, telling it what it's got installed. It doesn't have to know in advance what it's expected to build. It fetches a build request, which tells it to check certain tags out of SVN, then run a script from one of those tags to take it from there. Developers don't have to know how many build slaves are available, what they're called, or whether they're busy, just how to add a request to the build queue. The build queue itself is a fairly simple web app. All very modular.
Slaves needn't be VMs, but usually are. The number of slaves (and the physical machines they're running on) can be scaled to satisfy demand. Slaves can obviously be added to the system any time, or nuked if the toolchain crashes. That'ss actually the main point of this scheme, rather than your problem with archiving the state of the toolchain, but I think it's applicable.
Depending how often you need an old toolchain, you might want the build queue to be capable of starting VMs as needed, since otherwise someone who wants to recreate an old build has to also arrange for a suitable slave to appear. Not that this is necessarily difficult - it might just be a question of starting the right VM on a machine of their choosing.
We recently had a project where we released beta of a big web app on our client's server. Our client requested us to do bug fixes as they come, and we tried to do it same way. Normally while building an app on our prototype server is way easier, as I just have to issue simple 'svn up' command which takes a second.
But on production environment, we do not have any version control tool available. Is it possible to automate the patching work, so that we need not to login to ftp and upload each a every file one by one?
Its very difficult to work this way. As I'm having this problem, its for sure that some of you have already solved the problem. Please share your solutions.
Looking forward to your replies... Thanks a lot for reading guys.
Depending on the tools available on the server, you could either do a svn diff -r x:y where x is the revision you last updated too and y the last revision you want to update to (probably the last revision on your repository) to generate a patch and then apply the patch with the patch command.
If rsync is available on the production platform, and you can use it (though ssh for instance) you could set up a production ready tree, rsync it on the production server, and when an update comes in, svn update your production tree, and rsync it again.
What is stopping you from installing a Subversion client on the production server?
[EDIT] So someone doesn't allow you to install the software you need on the server. The question is: What is more important? A stable production server or an arbitrary policy? If the someone doesn't listen to arguments, go to your computer, start MS Word and write this letter:
"I hereby refuse to accept any responsibility for the stability of our production system based on the fact that [insert name here] refuses to equip me with the tools to make sure that the production system contains all the necessary files and data after an installation."
Sign this, have your boss sign it and then send a copy to [insert name here]. All of a sudden, any problem that might arise after an installation will be on his turf. Or to put it more clearly: He will be responsible for any mistake you might make.
Now, all you have to do is wait. :)
Depends on the programming environment you use. In Smalltalk and the web application server like Aida/Web we can upgrade the live web applications on the fly, without stopping it.
The server is connected to the SCM of choice like Monticello for Squeak Smalltalk or Store for VisualWorks. New versions are then manually or automatically loaded to the server's Smalltalk image.