Installing an perl based web-app in extremely restricted environment - perl

Because i have a long series of comments with #ikegami, I cleaning up the question, in a hope it will be more understandable. Unfortunately, english isn't my "main" language. :(
Let say, having an environment where:
no development tools are installed (no make, nor gcc or like)
perl is installed with its core packages, nothing more
no outgoing network access is allowed - e.g. the user couldn't use curl nor cpan to download/install perl dependencies
the user even doesn't have admin (root) rights
but want install and evaluate some perl based web-app, let call it as MyApp
The MyApp
doesn't uses any XS-based module. (at least, I hope - in the development me using plenv and cpanm, so never checked the installed dependencies in depth)
it is an pure PSGI app, the simple plackup app.psgi works OK
the app uses some data-files which should be included in the "deployment".
The main question is: how to prepare the MyApp, and the all used CPAN-modules, to be easily installed in such restricted environment?
The goal is:
i don't need save my efforts and my time
but i want save the user's time and want minimize the needed actions on his side, so the installation (deployment) should be simple-as-possible.
E.g. how to get an running web-app to the user's machine with minimum possible (his) steps.
- the simplest thing is could be something as:
- copy one file (zip, or tarbal)
- unpack it
- from the terminal execute some run.pl in the unpacked directory.
To get the above simple installation, my idea was the following:
1.) Create an tarball, and after the unpacking will contain 3 folders and 1 perl-script, let say:
myapp_repo/
myapp_repo/distlib #will contain all MyApp's perl modules also ALL used CPAN modules and their dependecies
myapp_repo/datafiles #will contain app-specific data files and such
myapp_repo/install.pl
myall_repo/lib #will contain modules directly used by the `install.pl`
2.) I will develop an install.pl script, and it will be used as the installer-tool, like
perl install.pl new /path/to/app_root
and it will (should):
create the all needed directories under the /path/to/app_root (especially the lib where the will install the perl modules)
will call "local" cpanm internally (from the myapp_repo/lib) to install the app's perl modules and their CPAN dependencies using only distribution files from the distlib.
will generate and install the needed runtime script and the app.psgi into the /path/to/app_root/bin
will install the needed data-files for the app.
3.) So, after this the user should be able to simply run:
/path/to/app_root/bin/plackup /path/to/app_root/bin/app.psgi
In short, the user should use:
the system-wide perl and the system-wide perl-core modules
and any other
runtime perl-scripts (like plackup)
and the required CPAN-modules
should be installed to an self-contained directory tree using only files (no net-access).
E.g. the install.pl should somewhat call internally the cpanm to achieve (as equivalent) for the following cpanm command
cpanm --mirror file://path/to/myapp_repo/distlib --mirror-only My::App
which, should install My::App and all dependencies without network access using only the files from the myapp_repo/distlib
Some questions:
Is possible to use cpanm (called as an locally installed module) without the make?
For creating the myapp_repo/distlib, me thinking about using Pinto. Is it the right tool for achieve the above?
forgot me something? or with other words:
Is the above an viable (read: working) way?
are are any other tools, which i could/should to use for simplifying the creation of such distribution tarball?
#ikegami suggesting some method:
- "install everything" in one fresh-directory on my machine
- transfer this self-contained directory to the target machine
It sound very good, because this directory could contain all the needed app-specific data-files too, unfortunately, I don't understand the details how his solution should be done.
The FatPacked solution looks interesting too - need learn about it.

Don't write your own make or installer. Just copy it make from a different machine (which is basically what apt/yum/etc do anyway, and which you'd have to do even if you wrote your own). You'd be able to use cpan in 5 minutes!
Also, that should allow you to install gcc if you need it (e.g. to install an XS module), although it doesn't sound like you do. If you do install gcc, I'd install my own perl to avoid having to deal with PERL5LIB.
Tools such as minicpan will allow you to install any module from CPAN without internet access. Of course, you can keep using the command you are already using it if mirrors the packages you need.
The above explains how to simply and quickly setup a machine so it can use cpan and thus install any module easily.
If you just want to install a specific module and its dependencies, you can completely avoid using cpan on the target machine. First, you need a fresh install of Perl (preferable of the same version as the one on the target system). Then, simply install the module to a fresh dir on your machine, and transfer that dir to the target machine. That's it; nothing else needs to be done. This even works for XS modules if the two machine are similar enough.
This is what ppm (ActiveState's Perl package manager) does.
Unfortunately, while this solution is almost as simple as the one above, it's not nearly as flexible, it doesn't run the test suite of the modules being installed, etc. It does have the advantage of not requiring the transfer of any binary (if you're not installing any XS modules).

Related

Bundling up a perl script with its dependencies?

I have a perl script that I've put together to do some monitoring and graphing.
It works nicely on my dev system, where I have carte-blanch to install my own modules from CPAN.
What I'm looking at doing is bundling it up to deploy onto another system. But here's the catch - this other system is 'standalone' and has no network connection. (And I have change control paperwork to fill in, indicating what I'm installing).
As a result, I'd really like a nice easy way to figure out:
- What modules my scripts are making use of. (Including dependencies)
- how to easily grab them (cpan get probably)
- Is there an easy way to tell what external binaries I'm using? (I'm using for sure ssh and rrdtool - the former is definitely installed, the latter probably not).
I have a few thoughts on how to do this, but it strikes me as something that should be smoother.
I may also need to deploy a new perl, so I'm pondering whether I'm better off 'installing' the modules with system perl (probably 5.8.8 on RHEL5), or just 'packaging' the whole thing in a directory of it's own with a standalone perl instance.
Use pp to package your script and all dependant modules and libraries into a stand alone executable.
pp -x yourscript.pl -o outputfilename
See the documentation for examples of how to link to external shared objects (etc) if required. With pp you don't need perl on the target system where outputfilename will run.
Revisiting this, as the need hasn't really gone away. I have moved towards using docker - this is an 'image' and 'container' system for app deployment, which amongst other things, allows you to 'package' an application.
You create a Dockerfile - which is analagous to a Makefile - that runs through the steps to install perl + dependencies (either via a package manager, or from CPAN).
Once it has, you have a self contained, runnable 'image' that you can clone and create an instance of (a "container" in docker parlance).
It's also quite useful - even if you don't then deploy via container - to figure out what the dependencies of this application/packages were. The version in the container has everything locally installed that it needed, because it was a clean build.
When you have a system where you can't control the Perl installation (and the install is a really, really old version of Perl like 5.8.8 which is missing many nice improvements like state variables, autodie, say, and switch), you should look into Perlbrew.
Perlbrew allows you to install a user version of Perl. (In fact, it allows you to install multiple versions of Perl), and allows you to switch between the Perlbrew install and the officially installed version. It makes doing everything in Perl much, much easier.
You will have freer access to install new Perl modules, and you can do that task yourself rather than wait for your IT department to do it for you.
I ended up using it on one of our systems where the primitive version of Perl just wasn't doing what my version of Perl did. I originally asked our IT to upgrade, but they really messed up the upgrade. After going back and forth, I simply asked if I could install Perlbrew.
Which is an important point. Always ask permission. A lot of time, the IT department is more than happy to oblige. They're not Perl people, and CPAN is a world they don't want to deal with. Being able to get out of having to answer your beck and call about installing this or that Perl module is a great relief.

How to make a Dist::Zilla based Perl module (or app) install files into /etc/?

I maintain multiple Perl written (unix-ish) applications whose current installation process consists of a manually written Makefile and installs configuration files into /etc/.
I'd really like to switch their development to use Dist::Zilla, but so far I haven't found any Dist::Zilla plugin or feature which allows me to put given files into /etc/ when the make install (or ./Build install in case of using Module::Build instead of ExtUtils::MakeMaker) is run by the local administrator who's installing my application.
With pure ExtUtils::MakeMaker, I could define additional make targets in MY::postamble and the let the install target depend on one of them via the depend { install => … } attribute. Doing something similar, but via dzil build, would probably suffice, but I'd appreciate a more obvious way.
One orthogonal approach would be to make the application not to require the files under /etc/ to exist, but for just switching to Dist::Zilla that seems to much change in the actual code despite I only want to change the build system for now.
For the curious: the two applications I currently have in mind for switching to Dist::Zilla are xen-tools and unburden-home-dir.
The best thing to do is to avoid installing files into /etc from any Perl distribution. You cannot ensure that the cpan client (or the installing user) has permissions to install there, and there can be multiple Perls installed on a system, so each one of them would clobber the /etc files of another install. You can't really prevent the file from being overwritten by a subsequent install, so you shouldn't put config data there that you don't want to lose.
You could put the config file in /etc/, if the application knows to look for it there, but you should allow for that path to be customized (say on a test system, look for the file in the local directory, or in a user's home directory).
For installing read-only module-specific data, the best practice in Perl is to install into a Perl-install-specific location, and the module to do that is File::ShareDir::Install. You can use it from Dist::Zilla using the [ShareDir] plugin, Dist::Zilla::Plugin::ShareDir. It is even included in the [#Basic] plugin bundle, so if you use [#Basic] in your dist.ini, you don't need to do anything at all, other than drop your data files into the share/ directory in your distribution repository.
To access the contents of the sharedir from code, use File::ShareDir.
For porting a complex module installer to Dist::Zilla, I recommend my plugins MakeMaker::Custom or ModuleBuild::Custom, depending on which installer you prefer. These allow you to keep your existing Makefile.PL or Build.PL and just have Dist::Zilla plug in necessary bits like the dependencies.

Is it possible to install perl prerequisites before distribution testing and how?

I try to build a Perl distribution for a home-made module, from the Module::Starter base. Every test pass on my machine, but when I upload it to CPAN to get some more universal tests from cpantesters.org, some test failed on other architectures or OS, but I can't understand why. I can see in test reports that some of my prerequisites are not installed before testing but I would like it to.
I've tried to list these dependencies into the Makefile.PL PREREQ_PM hash and then in the TEST_REQUIRES hash, but it didn't changed a lot of results.
Then, when I've removed the dependencies from my local machine and tried to install my module using Cpanm, it downloads dependencies first, test passed and install has been a success.
This is my first try for a module, so I think I am missing something, maybe I am too used of the Cpanm magic. Thanks for any help.
The problem is something different. Andreas' smoker very probably built the dependency App::Ack (which looks in the fail reports like being absent) successfully. But here come at least two problems:
When a distribution gets tested, then its dependencies may or may not be installed already. However, it's guaranteed that all dependent modules are made available through the PERL5LIB environment variable, so make test usually works (To be more specific, if the install Module command is used in the CPAN shell, then all dependencies are installed immediately. If the test Module command is used, then dependencies are only built, but not installed. The CPAN user can do the installation later using install_tested). So it may be that App::Ack is not installed here, just built. Especially this means that the ack script is not installed in the final location.
Even if it is installed, many smoke testers or users who have multiple perls installed in parallel use a non-standard directory for this perl. So ack wouldn't be installed in /usr/bin or /usr/local/bin, but in the bin directory belonging to this perl. This directory may or may not be in the user's PATH at all. So you cannot assume that can_run("ack") works here. A workaround here is to add $Config{scriptdir} temporarily to $ENV{PATH}. Another solution would be to use the App module instead of the script, if it's possible. Unfortunately it looks like ack can only be called as a script.
If you look at a sample fail report, then you can see that App::Ack was installed (it appears in the PREREQUISITES section both under requires and build_requires, you can also see which App::Ack version is installed in the "HAVE" column). You can also see the user's PATH (in the ENVIRONMENT section). And you may guess about the scriptdir for this perl, it's usually the same directory where the perl binary itself is installed, and the path to current perl is visible in $^X (under "Perl special variables").
If you want to reproduce the behavior, then you need to deinstall ack from your machine, build a custom perl using ./configure.gnu --prefix=/path/to/custom/perl-5.X.Y, and use this perl for tests.

How to install Perl offline

I have a Linux server that has no access to the internet (access is prevented by a firewall). I would like to install a new Perl. What are my options and what is the best way to do this? The system Perl (included in OS installation) must remain unchanged.
I have been using perlbrew and I think it is the best way to do an online installation. But all the steps involved in perlbrew seem to require internet access: you download it from the net, it downloads new Perl versions from the net etc. and I haven't found a glue how to make it work offline.
If perlbrew is out of question I could build Perl from source into a custom location on the server. I assume that this could end up being complicated, time-consuming and error-prone. And every time I update Perl I have make a new build manually.
There can also be other ways to install that I'm not currently aware of. And of course I could stick with the system Perl but it is an outdated version and I'm already using the new syntax features. Or I could start negotiations to change the firewall policy to allow internet access for perlbrew.
But all the steps involved in perlbrew seem to require internet access
Not if properly configured.
To install perlbrew itself off-line, install the App-perlbrew dist. Following its dependencies manually is a chore, so instead prepare a MiniCPAN mirror (with -p to include Perl dists), take it over to the target machine and configure CPAN to use the local mirror. Run cpan App::perlbrew to install.
After perlbrew is installed, run its mirror command to configure a CPAN mirror into $PERLBREWROOT/Config.pm. Edit this file to change it to the local MiniCPAN mirror. Drop Perl dist tarballs into $PERLBREWROOT/dists/.
Be aware that compiling Perl requires a working C compiler toolchain, and optionally the development files for libdb (BerkeleyDB) and gdbm. (Read the INSTALL file once over, even though perlbrew's autoconfiguration and Perl's configure.SH defaults hide these details from you.)
The compiler toolchain is probably much more difficult to procure off-line, unless the OS installation has already been used before for compiling other C stuff.
There's nothing that special about perlbrew. If you aren't going to use it to download the Perl sources, it's not saving you that much. Once you have the Perl sources, you just need to configure and install it:
% ./Configure -des -Dprefix=/path/to/installation
% make install
Once done, everything for that Perl is under that installation path.
I dislike perlbrew mostly because it hides from people how amazingly simple this task is so they feel like they can't do it on their own.
Have you considered attacking it from a different direction? Keeping this up-to-date is going to be a pain if you have to request internet access each time. Likewise, if you've missed out/misconfigured any packages in your CPAN mirror it's difficult to correct once you're actually trying to use them.
Perhaps just build a small VM with a cut-down linux + perl + modules. Keep that up-to-date at your end and just take the whole lot in on a USB stick. You'd have a known-working easy-to-setup installation.
What I personally do is using git checkout when I'm offline (and not on vacation). Once you have the whole git work directory, it's trivial to build any released version by checking out the tags:
git checkout v5.17.4
git clean -f # cleanup previously compiled .o files etc
sh ./Configure ...
Depending on how you can transfer files to your host, this can be handy, since you you can also setup a private git repo there so other computer can git push new commits to there.

How do I install Perl modules on machines without an Internet connection?

I need to install my Perl-based software on networked machines which aren't connected to the internet. Therefore, I would like to download specific versions and/or latest versions of the Perl modules and I would also like to know if there is an install procedure required for these modules.
Background:
The machines aren't connected to the internet for security reasons and its deemed unnecessary also.
I would place the downloaded modules on a machine that I call the 'install server' and it contains my Perl based software and would also contain the local copies of the Perl modules.
I call a machine that I want to install my Perl-based software on, the 'target machine', also not connected to the internet. There can be several target machines, each can run this software that I want to install. I log onto the target machine and run an install script which would connect to the install machine via the local network to obtain the Perl-based software and dependent Perl modules and installs them.
So I need to know:
How/Where to get specific versions of Perl modules, e.g. CGI.pm etc
How to install these Perl modules. Is it a case of just placing them in a directory somewhere, e.g. a library path and making sure that this directory path is in the #INC library path environmental variable, if it is not already?
I would prefer not to have to do anything like make install etc. as part of installing the modules. I would like to modules to be pre-compiled or prepared as necessary so it is as simple as possible to install them. I want to avoid additional dependencies like make and its configuration, and having to parse its output to check whether it was successful.
Please help me by asking the above specific questions as I am not able to change the concept of 'install machine' and 'target machine' which aren't connected to the internet - I have to provide a solution that works within this arrangement.
The usual way to solve "I want to install stuff from CPAN but without network" problems is to use a minicpan as David Dorward wrote in his answer. But since you're going one step further, saying that you'd rather not do any real installation on the client (target) machines at all, and that you want to use precompiled modules if possible, I urge you to check out PAR and specifically PAR::Repository (server) and PAR::Repository::Client.
Since this approach needs some research before you're up to speed, I wouldn't suggest it for "I just need Foo.pm" like problems. Once you're talking about a handful of dependencies and at least a handful of clients, then it becomes a more appropriate solution.
For an outline of how it works, check out the slides of my talk at YAPC::EU 2008. It also hints at solutions to the bootstrapping problem of making the PAR::Repository::Client module available on the clients (hint: PAR can generate self-contained executables).
You can create a MiniCPAN that has just the latest versions of everything from CPAN. You can insert additional, non-public modules into it with CPAN::Mini::Inject. If you need to greater control over versions (i.e. not choosing the latest versions), you might want to create a DPAN.
With any of these solutions, you can configure your CPAN client to pull from your local source. That could be a directory you know ahead of time or something that you figure out dynamically, like a CD or a thumb-drive. It's just a matter of setting up the configuration correctly.
You might be able to get away with creating operating-system packages for most of your work, but that still means you have to compile them at least the first time.
1) How/Where to get specific versions of Perl modules, e.g. CGI.pm etc
http://search.cpan.org/
If you don't want the latest version, you can get an earlier version by following the link in the breadcrumbs.
http://img.skitch.com/20091209-bu7kt3bj65374k7iijfnhrue2y.png
2) How to install these Perl modules. Is it a case of just placing them
in a directory somewhere, e.g. a library path and making sure that this
directory path is in the #INC library path environmental variable, if
it is not already?
That sometimes work, but you really should go through the perl Makefile.PL && make && make test && make install process.
Doing this would require that you manually chase all the dependencies though. You would probably be better off with something like minicpan.