What are the best practices for deploying a Perl application? Assume that you are deploying onto a vanilla box with little CPAN module installation. What are the ideal build, deploy methods? Module::Build, ExtUtils::MakeMaker, other? I am looking for some best practice ideas from those who have done this repeatedly for large scale applications.
The application is deploying onto a server. It's not CPAN or a script. It's actually a PSGI web application. That is, a ton of Perl packages.
I currently have a deployment script that uses Net::SSH::Expect to SSH into new servers, install some tools and configure the server, then pull down the desired application branch from source control. This feels right, but is this best practice?
The next step is building the application. What are the best practices for tracking and managing dependencies, installing those dependencies from CPAN, and ensuring the application is ready to run?
Thanks
The company that I work at currently build RPMs for each and every CPAN & Internal dependency of an application (quite a lot of packages!) that install into the system site_perl directory. This has a number of problems:
It is time consuming to keep building RPMs as versions get bumped across the CPAN.
Tying yourself to the system perl means that you are at the mercy of your distribution to make or break your perl ( in Centos 5 we have a max perl version of 5.8.8 ! ).
If you have multiple applications deployed to the same host, having a single perl library for all applications means that upgrading dependencies can be dangerous without retesting every application of the host. We deploy quite a lot of separate distributions all with varying degrees of maintenance attention, so this is a big deal for us.
We are moving away from building RPMs for every dependency and instead planning to use carton [1] to build a completely self contained perl library for every application we deploy. We're building these libraries into system packages, but you could just as easily tarball them up and manually copy them places if you don't want to deal with a package manager.
The problem with carton is that you'll need to setup an internal CPAN mirror that you can install your internal dependencies to if your application depends on modules that aren't on the CPAN. If you don't want to deal with that, you could always just manually install libs you need into local::lib [2] or perlbrew [3] and package the resulting libraries up for deployment to your production boxes.
With all of the prescribed solutions, be very careful of XS perl libs. You'll need to build your cartons/local:libs/perlbrews on the same architecture as the host you're deploying to and make sure your productions boxes have the same binary dependencies as what you used to build.
To answer the update to your question about whether it is best practice to source checkout and install onto you production host; I personally don't think that it is a good idea. The reasons why I believe that it is risky lays in the fact that it is hard to be completely sure that the set of libraries that you install exactly lines up to the libraries that you tested against, so deployments have the potential to be unpredictable. This issue can be exasperated by webapps as you are very likely to have the same code deployed to multiple production boxes that can get out of synch, also. While the perl community does a wonderful job of trying to release good quality code that is backwards compatible, when things go wrong it is normally quite an effort to figure things out. This is why carton is being developed, as this creates a cache of all the distribution tarballs that you need to install frozen at specific versions so that you can predictably deploy your code. All of that said though; if you are happy to accept that risk and fix things when they break then locally installing should be fine for you. However, at the very minimum I would strongly suggest installing to a local::lib so that you can back up the old local lib before installing updates so you have a rollback point if things get messed up.
Carton
local::lib
perlbrew
If it has some significant CPAN dependencies, then you might want to either write a small script that uses CPAN::Shell to install the necessary modules or edit the Makefile.PL of your application so that it reflects the necessary dependencies in the BUILD_REQUIRES portion of the file.
You may take a look at sparrowdo a perl6 configuration management tool, it comes with some handy plugins related to perl5 deployment, like installing cpan packages or deploying psgi application.
Update: this link https://dev.to/melezhik/deploying-perl5-application-by-sparrowdo-9mb could be useful.
Disclosure - I am the tool author.
Related
our Linux Systems have no Internet Access.
Only Windows Clients have Internet Access.
I want to build a offline CPAN Repository.
I can't download each Module an picking all the dependencies.
Is there a way to download the Modules automated with dependencies to a Windows System?
Even a whole download of the CPAN Repository would be a solution for me.
You can create a local mirror of CPAN with minicpan. It's intended to distribute a copy of CPAN (or a subset of it) on media so you have it in case there is no internet available, e.g. a USB drive that you can use on a laptop while you are on an airplane so you can still install a dependency, even if it's outdated a bit.
Check out CPAN::Mini and the minicpan utility on CPAN as well as this guide on how to set up a mirror on perl.org. There is a guide on blogs.perl.org as well. Furthermore, this guide in German is very comprehensive and has a list of related material at the bottom.
In fact, there is a whole tag minicpan here on Stack Overflow. One of the very helpful reads (though not a full duplicate) is ysth's answer here.
My collegue and I develop a small Python application. We use Vagrant to set up development environments.
Suppose my collegue introduces a new feature into the application. Feature's implementation requires a new python dependency (3rd party package) and the dependency itself needs some system libraries. If I do not read through all pulled commits carefully I can miss, that some systems libraries have to be installed prior running the project.
Of course we update Vagrantfile to install such non-python dependencies during provisioning, so if someone clones project's repository and issues vagrant up he will get a fully working development environment, but what shoud I do to automate updates in my existing environment?
How should we indicate, that a new dependency (python or non-python) was added and we need to install it by firing a specific command?
UPD I can try and run the application and if I encounter any errors it is a sign to reprovision my vagrant box, but it seems tedious to me to test a feature by hands and run provisioning scripts later
I ran into this with Ruby as well. We used Bundler, which is a dependency management system for Ruby. If I pulled in new code, ran it and got funky exceptions saying that a certain dependency was missing, I just knew it was time for a bundle install from the command line. The solution to your problem is the same. If you run the code and get errors saying a dependency is missing, your default response to that exception should be to vagrant up on the command line, and try again.
Barring that, sending an email to your teammates with instructions about the new or updated dependency is a good way to go, especially if a vagrant up is insufficient to resolve the missing or incorrect dependency.
I have two CentOS platforms. Both run "CentOS release 5.10 (Final)". One is a "real" machine and the other is a VM. Both are 64 bit. Call the real machine Prod and the VM Spare.
When I got this gig I was told that the two machines were identical. Spare is supposed to be a hot spare for Prod. It is now obvious that is not true. The two machines have different yum repo lists. There are duplicate looking install packages from different channels. Prod looks like a server. Spare looks like it had been somebody's desktop with Evolution, OpenOffice and other desktop cruft.
Prod and Spare have similar applications installed but found in different repos so the available yum update levels are different.
I have tried disabling the non-standard repos and uninstalling the non-standard packages. This has led to tears as removing X-Windows, for example, has led to the removal of hundreds of dependant modules that in turn have dependants which, in the end, made Spare deaf, blind and mute. Blessedly we had a copy of the VM.
My latest idea is to migrate both machines to the latest stable CentOS level and basically have a do-over. The downside (I think) is the downtime to the production machine and unknown custom software vs new package level issues.
My basic question is, what is the best way to make the platforms as identical as possible, and minimize (or better yet negate) downtime.
How should we maintain packages and other installs across them into the future? I am aware of Puppet, Chef and CFEngine but have not used them before. Are these the way to go for the future? Something else?
This is not really a programming related question (You might have better luck at https://serverfault.com/)
Your question is quite broad, but essentially you want two machines that are as identical as possible, one production, one VM, correct?
Two get machines in a consistent state, you'll need a configuration tool of some sort. Ansible is probably the easiest to get setup and get cracking with. At it's most basic setup, is basically nice wrappers around SSH. With this you can create consistent, and easily track changes to servers as they happen.
To have a VM you can easily provision, I recommend reading up on Vagrant and Packer. Vagrant to easily create a VM that accurately reflects your production environment, packer so you can repeatedly create an image in various platforms. In an ideal case, you can take the configuration tool and use it to provision your VM, meaning you can test your production changes on a VM first.
In general, having repeatable automated configuration you can easily test, I'd also recommend reading up on the concept of DevOps
I wonder if there are any options in Perl to build a local CPAN repository including the modules I want and then redistribute it with Perl distribution and then do not need to access CPAN at all.
If yes, could some me show an example?
I looked on CPAN and found a mincpan but it seems that minicpan bring all the mirror of CPAN. If it's possible to bring only a specific subset of modules using minicpan and in case I have a repository is it possible to copy it to another OS with the same type and install the relevant modules there with no headaches?
See Pinto (manages a local cpan-like repository) or Carton (can bundle up dependencies and provide them as needed, but you must run your application under carton after deployment.)
Alternatively, instead of a local minimal CPAN distribution, you can bundle requirements with your module, if you use the Module::Install installer.
The Pinto tools only work on Unix-like machines. However, Windows users can install modules from the repository as long as they can read the filesystem (like with NFS) or reach the host via HTTP (using pintod). So it is possible to use a Pinto repository with Windows, so long as you have one Unix-like machine to create and manage the repository.
Stratopan provides Pinto repositories hosted in the cloud. With Stratopan, you don't have to install any tools and everything can be managed through the browser. You can then install modules anywhere that has internet access using the standard tools. Stratopan doesn't yet support all the features Pinto has, but it is the most hassle-free solution for creating a private CPAN.
Disclaimer: I operate Stratopan.
This is addressed by How to mirror CPAN
I need to rejig some VERY old Windows code that uses Perl to talk to MQ. Specifically, I need to be able to install Perl's MQClient::MQSeries, MQSeries::QueueManager, MQSeries::Queue and MQSeries::Message modules.
When I fire up Strawberry Perl, go into CPAN and try to install them, I can see that there's several MQ client DLLs that are required for these Perl modules to build. However, they're not on my system, even after downloading and installing the current MQ Client from IBM.
It used to be that downloading and installing the MQ Client from IBM gave you the option to install a whole bunch of development libraries (presumably including the bits necessary to install the above Perl libraries), but it seems that's no longer the case. For example, there's no MQM.DLL file anywhere, which is one file that the Perl libraries seem to need to build correctly.
Suspect I've probably just not installed the correct MQ Client package to get this stuff, but have no idea where to find it (Google is no help). Any suggestions?
MQSeries 1.29 was released in 16 Jun 2009, so I guess you can just ask it's maintainer.
You can also ask in newsgroup news://news.software.ibm.com/ibm.software.websphere.mq , or ask IBM's support.
where'dja get the MQ download?
maybe you might try mqseries.net - it's a hotbed of MQ activity.
I'd suggest downloading the WMQ Client installation. Depending on which one you want, v7.0 is SupportPac MQC7 and v7.1 is SupportPac MQC71. This will get you the client libraries with which to build the Per module. If you need the server libs, the Evaluation download of WMQ Server v7.1 is here: