I develop and maintain a bioinformatics application suite of 50+ scripts and its deployment process is a mess:
Entire suite is in one big git repository. It has lots of CPAN dependencies, and dozens of internal modules as well.
Development platform is Linux.
Deployment platforms are Windows (20+ users), Mac (10+), Linux (2-3). Most are not 'power users'.
For windows, I have one installer (made with NSIS) for strawberry perl + required modules (ie, I installed strawberry on a windows box, installed all modules and zipped c:\strawberry), and another installer for the suite-- I did this b/c the suite is updated a lot more than the list of required modules.
For Mac, I bundle perl 5.14, all required cpan modules, and the application suite into a double-clickable installer. I don't use the system perl b/c it tends to be out of date. I bundle everything together unlike on windows b/c I suck at mac.
For Linux, I handle their installations manually since there are only a few of them, and they use different distros.
This is obviously a mess that grew organically over several generation of developers. Ideally I would like to create cpan-installable distributions out of the internal libraries and various groups of related scripts, and use module dependencies to let cpan install them for me.
But I'm not sure what the best approach for this is, b/c I'd still need distribute perl itself, would have to write some sort of non-command-line interface to CPAN, control the exact versions of 3rd party CPAN modules, point it by default to my "DarkPan" where I would store our modules, how I would push updates, etc. etc.
I don't think I can use PerlApp or Par since afaik those are for bundling single scripts, not an entire suite of them.
Any advice highly appreciated.
Besides the 3 platforms mentioned (more, if you count the Linux variants), you really have a couple different problems:
Deployment of a standard known-good Perl executable and libraries (CPAN modules).
Deployment of your Perl scripts and modules.
Once upon a time, I supported a large Solaris Perl installation. I tried for a while to stand up a Linux Perl installation "side-by-side", re-using the same CPAN modules. Didn't work. The big problem for me is that a fair number of Perl modules require compilation, which means they target a specific platform. I ended up just with 2 installs, and always remembered to install a new CPAN module in both areas.
We're now 100% Windows, so I don't have the same issue. However, we do run Perl off a shared network drive. All the users map this drive, and run a Registry script that associates .PL files with the network install of Perl. (See my answer to this other Perl question.)
So, besides the mapped drive and the Registry script, users don't need to install anything. Even the CPAN modules are picked up from the network. This solves item #1 (for Windows only users).
Same thing holds true for item #2: the scripts are stored on a network drive (same one) and the users run another Registry script to include the scripts folder in their search PATH. We edit our scripts in one area, and have a "Check-In 'n Release" ("CINR") that we use to, well, check-in and release scripts to the area the users point to. The users can double-click the scripts in Explorer, run them in DOS, or even better yet get them included in the contextual-menu in Explorer, etc. (Actually, we use a .NET application to map the drive and make all these settings for the user, but it can be done much simpler.)
So, how does this help with the other platforms, Linux and Mac? As I ran into with my Solaris/Linux experiment, I think you're stuck with different Perl installation for all 3 platforms, although you should be able to reach the same network drive for your Perl scripts and modules.
The Perl installation might even be OK on a network drive for the Linux users. It's probably easier for them than the Windows users. Mac users are tough. I administer a home Mac network, and I think network drives are very difficult to do in Mac OS X compared to other OSes. It should be as easy as in Linux since so much is the same, but there are very strange problems (for me) mapping NFS and SMB drives. AFP drives are a little easier for the user to map manually, but not so easy to map programmatically.
My Mac recommendation is to try using Platypus. It's definitely great at bundling scripts into a double-clickable application, although your interface options are limited to output only (no user input allowed during execution that I can tell). Not sure if you could put an entire Perl installation into the Platypus app or not, but if you could the paths figured out, you might be able to.
Good luck!
You may wish to check out CAVA packager. It can deal with multiple scripts in a single package.
Related
I am trying to install a few Perl modules, one of them being Time::Format. My corporate group policy does not allow me to use CPAN (since it uses FTP) to install modules. Tried using makefile.pl, but nmake.exe (I am using windows cygwin) seems to be missing too. The policy does not allow to download the nmake.exe executable too. PPM isn't available too.
How do I go about installing module manually? Is there a way where I could manually copy files from this module on to individual folders in my Perl directory? If yes, which files, go where?
You might be able to get around the FTP restriction by using a minicpan. Basically this lets you build your own local CPAN mirror, for example on a USB key drive. It's intended to carry a snapshot of CPAN with you, e.g. when you are in transatlantic flights and just really need that module you didn't install.
However, those restrictions are probably there for a reason. Downloading and installing/copying stuff from the internet might be against the corporate policy just the same as it's an external source. You should check that. On the other hand, those policies are often intended for the average office user that is not a developer, so you might be able to talk to them and explain why you and your team should be excepted from this policy, and how reusing code from CPAN would save your company a lot of time, which equals money.
Being very new to Perl but not to dynamic languages, I'm a bit surprised at how not straight forward the manage of modules is.
Sure, cpan X does theoretically work, but I'm working on the same project from three different machines and OSs (at work, at home, testing in an external environment).
At work (Windows 7) I have problem using cpan because of our firewall that makes ftp unusable
At home (Mac OS X) it does work
In the external environment (Linux CentOs) it worked after hours because I don't have root access and I had to configure cpan to operate as a non-root user
I've tried on another server where I have an access. If the previous external environment is a VPS and so I have a shell access, this other one is a cheap shared hosting where I have no way to install new modules other than the ones pre-installed
At the moment I still can't install Template under Windows. I've seen that as an alternative I could compile it and I've also tried ActiveState's PPM but the module is not existent there.
Now, my perplexity is about Perl being a dynamic language. I've had all these kind of problems while working, for example, with C where I had to compile all the libraries for all the platform, but I thought that with Perl the approach would have been very similar to Python's or PHP's where in 90% of the cases copying the module in a directory and importing it simply works.
So, my question: if Perl's modules are written in Perl, why the copy/paste approach will not work? If some (or some part) of the modules have to be compiled, how to see in CPAN if a module is Perl-only or it relies upon compiled libraries? Isn't there a way to download the module (tar, zip...) and use cpan to deploy it? This would solve my problem under Windows.
Now, Perl is a dynamic language, but that doesn't imply that everything that people write is portable across platforms. That's not the fault of the language. It's not even the fault of the programmer. Some things, like Win32::OLE shouldn't work on Unix. :)
Other dynamic languages will have some of the same problems. If you have to compile C code, you won't be able to merely copy files to another machine. Some distributions configure the code slightly differently depending on your operating system, etc.
Even if you could copy files, you have to ensure that you copy all of the files that you need. Do you know everything that you need for a particular module? Remember, many of them have dependencies.
Most of the problems you're having aren't anything to do with the language. You're having trouble with the tools. If you want a zero conf CPAN tool that makes all the decisions for you, try cpanminus. It's mostly the same thing that you'd get out of cpan (although different code), but it makes all of the decisions for you. It doesn't run any of the distribution tests, and it installs into your user directory. When you need something that gives you control, come back to cpan.
In the external environment (Linux CentOs) it worked after hours because I don't have root access and I had to configure cpan to operate as a non-root user
This is one of those times when it helps to know The Trick. In this case local::lib, which lets you configure a non-root install area and all the ENV variables in about three minutes.
if perl's modules are written in perl, why the copy/past approach will not work?
Some are written in Pure Perl, but many are written partially in C (using Perl's XS API) for efficiency.
Sometimes you end up with situations like JSON::XS, JSON::PP and JSON::Any to autoselect the best one that is installed.
Isn't there a way to download the module (tar, zip...) and use cpan to deploy it?
The cpan program is all about getting things from the Internet. You can download the package (there will be a link along the lines of "Download: CGI.pm-3.49.tar.gz" on the right hand side of the CPAN page), untar it, then
perl Makefile.PL
make
make install
You would probably be better off configuring your cpan installation to use only HTTP sources (in the urllist config option). Possibly going to far as to create a mini CPAN mirror inside your network.
I need to install my Perl-based software on networked machines which aren't connected to the internet. Therefore, I would like to download specific versions and/or latest versions of the Perl modules and I would also like to know if there is an install procedure required for these modules.
Background:
The machines aren't connected to the internet for security reasons and its deemed unnecessary also.
I would place the downloaded modules on a machine that I call the 'install server' and it contains my Perl based software and would also contain the local copies of the Perl modules.
I call a machine that I want to install my Perl-based software on, the 'target machine', also not connected to the internet. There can be several target machines, each can run this software that I want to install. I log onto the target machine and run an install script which would connect to the install machine via the local network to obtain the Perl-based software and dependent Perl modules and installs them.
So I need to know:
How/Where to get specific versions of Perl modules, e.g. CGI.pm etc
How to install these Perl modules. Is it a case of just placing them in a directory somewhere, e.g. a library path and making sure that this directory path is in the #INC library path environmental variable, if it is not already?
I would prefer not to have to do anything like make install etc. as part of installing the modules. I would like to modules to be pre-compiled or prepared as necessary so it is as simple as possible to install them. I want to avoid additional dependencies like make and its configuration, and having to parse its output to check whether it was successful.
Please help me by asking the above specific questions as I am not able to change the concept of 'install machine' and 'target machine' which aren't connected to the internet - I have to provide a solution that works within this arrangement.
The usual way to solve "I want to install stuff from CPAN but without network" problems is to use a minicpan as David Dorward wrote in his answer. But since you're going one step further, saying that you'd rather not do any real installation on the client (target) machines at all, and that you want to use precompiled modules if possible, I urge you to check out PAR and specifically PAR::Repository (server) and PAR::Repository::Client.
Since this approach needs some research before you're up to speed, I wouldn't suggest it for "I just need Foo.pm" like problems. Once you're talking about a handful of dependencies and at least a handful of clients, then it becomes a more appropriate solution.
For an outline of how it works, check out the slides of my talk at YAPC::EU 2008. It also hints at solutions to the bootstrapping problem of making the PAR::Repository::Client module available on the clients (hint: PAR can generate self-contained executables).
You can create a MiniCPAN that has just the latest versions of everything from CPAN. You can insert additional, non-public modules into it with CPAN::Mini::Inject. If you need to greater control over versions (i.e. not choosing the latest versions), you might want to create a DPAN.
With any of these solutions, you can configure your CPAN client to pull from your local source. That could be a directory you know ahead of time or something that you figure out dynamically, like a CD or a thumb-drive. It's just a matter of setting up the configuration correctly.
You might be able to get away with creating operating-system packages for most of your work, but that still means you have to compile them at least the first time.
1) How/Where to get specific versions of Perl modules, e.g. CGI.pm etc
http://search.cpan.org/
If you don't want the latest version, you can get an earlier version by following the link in the breadcrumbs.
http://img.skitch.com/20091209-bu7kt3bj65374k7iijfnhrue2y.png
2) How to install these Perl modules. Is it a case of just placing them
in a directory somewhere, e.g. a library path and making sure that this
directory path is in the #INC library path environmental variable, if
it is not already?
That sometimes work, but you really should go through the perl Makefile.PL && make && make test && make install process.
Doing this would require that you manually chase all the dependencies though. You would probably be better off with something like minicpan.
I have several scripts that I supply to users as tools for their engineering projects. These scripts contain a lot of duplicated code. I want to extract this code and maintain it in a set of modules. However, in order for this to work, the users would have to install these modules. I don't want to have to tell my users to "make install", etc., as I'm sure none of them would have the patience for that.
I understand that one option is to package everything together with PAR, but ideally the user would be able to open up && edit these scripts if they need to, just like they can now. They also need to be able to move them to whatever directory they want, and I don't want them to have to move a bunch of library files as well.
Is it possible to make a double-click file that installs some bundled Perl modules?
I distribute my script as modules, and then use the normal CPAN toolchain to install them. See my articles Scripts as Modules and Automating Script Distributions with scriptdist. Once you have them in a conventional module distribution, you can install them from their current directory with cpan:
% cpan . # install from distribution in the current directory
Depending on how complex your situation is, you might want to create a DPAN, which is a private version of CPAN that your CPAN tools can draw from. You put all of your private modules there, point your CPAN tools at it, and everything happens as it does with a real CPAN mirror. You never have to share your work though.
yeah package with either PAR or Shipwright (not sure about binaries). Also use scandeps.pl along the way.
If the users are using systems with a packaging system (dpkg, cygwin, etc.), consider using that.
If you don't mind spending some green, one of the better bet is Perl Dev Kit from Activestate.
From their own description of the product,
Develop and deploy your Perl programs
to anyone on any platform with
PerlApp's new cross-platform
wrapping.
Deliver code as
executables or as Windows Services,
ActiveX components, .NET assemblies
or in the System Tray.
Easily create
MSI files using Perl code.
You may also consider using sparrow - scripts distribution system. Sparrow plays nice with Perl as it writen on it. It supports CPAN modules dependencies via carton tool.
PS Disclaimer - I am the tool author
It seems that most of the installers for Perl are centered around installing Perl modules, not applications. Things like ExtUtils::MakeMaker and Module::Build are very well suited for modules, but require some additional work for Web Apps.
Ideally it would be nice to be able to do the following after checking out the source from the repository:
Have missing dependencies detected
Download and install dependencies from CPAN
Run a command to "Build" the source into a final state (perform any source parsing or configuration necessary for the local environment).
Run a command to install the built files into the appropriate locations. Not only the perl modules, but also things like template (.tt) files, and CGI scripts, JS and image files that should be web-accessible.
Make sure proper permissions are set on installed files (and SELinux context if necessary).
Right now we have a system based on Module::Build that does most of this. The work was done by done by my co-worker who was learning to use Module::Build at the time, and we'd like some advice on generalizing our solution, since it's fairly app-specific right now. In particular, our system requires us to install dependencies by hand (although it does detect them).
Is there any particular system you've used that's been particularly successful? Do you have to write an installer based on Module::Build or ExtUtils::MakeMaker that's particular to your application, or is something more general available?
EDIT: To answer brian's questions below:
We can log into the machines
We do not have root access to the machines
The machines are all (ostensibly) identical builds of RHEL5 with SELinux enabled
Currently, the people installing the machines are only programmers from our group, and our source is not available to the general public. However, it's conceivable our source could eventually be installed on someone else's machines in our organization, to be installed by their programmers or systems people.
We install by checking out from the repository, though we'd like to have the option of using a distributed archive (see above).
The answer suggesting RPM is definitely a good one. Using your system's package manager can definitely make your life easier. However, it might mean you also need to package up a bunch of other Perl modules.
You might also take a look at Shipwright. This is a Perl-based tool for packaging up an app and all its Perl module dependencies. It's early days yet, but it looks promising.
As far as installing dependencies, it wouldn't be hard to simply package up a bunch of tarballs and then have you Module::Build-based solution install them. You should take a look at pip, which makes installing a module from a tarball quite trivial. You could package this with your code base and simply call it from your own installer to handle the deps.
I question whether relying on CPAN is a good idea. The CPAN shell always fetches the latest version of a distro, rather than a specific version. If you're interested in ensuring repeatable installs, it's not the right tool.
What are your limitations for installing web apps? Can you log into the machine? Are all of the machines running the same thing? Are the people installing the web apps co-workers or random people from the general public? Are the people installing this sysadmins, programmers, web managers, or something else? Do you install by distributed an archive or checking out from source control?
For most of my stuff, which involves sysadmins familiar with Perl installing in control environments, I just use MakeMaker. It's easy to get it to do all the things you listed if you know a little about MakeMaker. If you want to know more about that, ask a another question. ;) Module::Build is just as easy, though, and the way to go if you don't already like using MakeMaker.
Module::Build would be a good way to go to handle lots of different situations if the people are moderately clueful about the command line and installing software. You'll have a lot of flexibility with Module::Build, but also a bit more work. And, the cpan tool (which comes with Perl), can install from the current directory and handle dependencies for you. Just tell it to install the current directory:
$ cpan .
If you only have to install on a single platorm, you'll probably have an easier time making a package in the native format. You could even have Module::Build make that package for you so the developers have the flexibility of Module::Build, but the installers have the ease of the native process. Sticking with Module::Build also means that you could create different packages for different platforms from a single build tool.
If the people installing the web application really have no idea about command lines, CPAN, and other things, you'll probably want to use a packager and installer that doesn't scare them or make them think about what is going on, and can accurately report problems to you automatically.
As Dave points out, using a real CPAN mirror always gets you the latest version of a module, but you can also make your own "fake" CPAN mirror with exactly the distributions you want and have the normal CPAN tools install from that. For our customers, we make "CPAN on a CD" (although thumb drives are good now too). With a simple "run me" script everything gets installed in exactly the versions they need. See, for instance, my Making my own CPAN talk if you're interested in that. Again, consider the audience when you think about that. It's not something you'd hand to the general public.
Good luck, :)
I'd recommend seriously considering a package system such as RPM to do this. Even if you're running on Windows I'd consider RPM and cygwin to do the installation. You could even set up a yum or apt repository to deliver the packages to remote systems.
If you're looking for a general installer for customers running any number of OSes and distros, then the problem becomes much harder.
Take a look at PAR.
Jonathan Rockway as a small section on using this with Catalyst in his book.