Is there a way to use app-specific mini-CPANs (to ensure specific module versions for different apps and make an install work without net access). The 'standard' mini-cpan way of things relies on setting 'o conf urllist' with a file url in your CPAN/Config.pm
That's fine with one single mini-cpan. However, I need to automate having a CPAN install from a different directory for each app (local to the app), as each app has different version requirements.
The following almost works, but then has only a partially working shell, and still fetches from the web instead of using the mini-cpan from $file_url location:
use CPAN;
use FindBin qw($Bin);
my $file_url="file:///$Bin/../cpan";
CPAN::Shell->o(qw( conf urllist ),$file_url);
CPAN::shell();
You load a different configuration file for each application. The latest cpan command has a -j switch to do just that:
$ cpan -j some/app/Config.pm -i Foo::Bar
This latest feature isn't included in the CPAN.pm distribution yet since it's experimental. I've been using it for several months, however, as part of my DPAN work.
If you don't like any of that, you just have to provide your application-specific CPAN::Config module somewhere Perl will find it before it finds your personal or site-wide version. Copy your current CPAN/Config.pm into your application modules directory and ensure that the directory is first in #INC (perhaps using the PERL5LIB environment variable). CPAN.pm should find that one first and use it. It only uses the first one it finds. This is handy when the configuration is fixed, although it's a bit flexible since you can run code in the module. It's configuration as Perl code.
If that is not good enough for you, you can override CPAN::HandleConfig() in your application so CPAN.pm doesn't load any files. You then set the values yourself. This is especially handy when you have a lot of application logic to resolve to figure out some of the configurations values. The CPAN.pm configuration is just the %CPAN::Config hash, so you just have to set the right values for the appropriate keys.
In any case, you probably want to set the "connect_to_internet_ok" configuration to a false value and clean out your "urllist" to have only the local minicpans.
Related
I love the idea of local::lib. At least, I find it preferable to the default system-wide installs that cpan/perl defualts to in most every way. However, I'm having difficulties with modules that install XS and distribution upgrades to newer Perls. The instructions for local::lib call for adding this line to your .bashrc (or like file)
eval "$(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)"
That executes arbitrary shell code, which is fine. You can see the code it executes by running
perl -I$HOME/perl5/lib/perl5 -Mlocal::lib
In my case it returns this code,
PATH="/home/ecarroll/perl5/bin${PATH:+:${PATH}}"; export PATH;
PERL5LIB="/home/ecarroll/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}"; export PERL5LIB;
PERL_LOCAL_LIB_ROOT="/home/ecarroll/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"; export PERL_LOCAL_LIB_ROOT;
PERL_MB_OPT="--install_base \"/home/ecarroll/perl5\""; export PERL_MB_OPT;
PERL_MM_OPT="INSTALL_BASE=/home/ecarroll/perl5"; export PERL_MM_OPT;
In the above, we're setting the default options for the two most widely used module build systems,
MB is for Module::Build
MM is for ExtUtils::MakeMaker
We're telling those system to install to /home/ecarroll/perl5. The rest of the command sets
PERL5LIB which is used in setting #INC. The #INC variable in Perl is used to resolve calls with use.
and, PATH which prepends the bin/ directory that local::lib installs to.
So basically, install perl modules in your home directory, and look first for files installed in your home directory.
My question is what happens when you install something that has XS? In my case, I have an XS file,
/home/ecarroll/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Class/Load/XS/XS.so
This was installed, with local::lib using an earlier version of Perl. I've since ran a Debian dist-upgrade and got a newer version of Perl (v5.36). Now a lot of utilities produce an obtuse error about this, including ones installed through the distro itself like in my case Dist::Zilla dzil which I installed with Debian's libdist-zilla-perl package.
$ dzil
XS.c: loadable library and perl binaries are mismatched (got first handshake key 0xeb00080, needed 0xeb80080)
Which is all a result of this module's XS,
perl -MClass::Load::XS -e1
XS.c: loadable library and perl binaries are mismatched (got first handshake key 0xeb00080, needed 0xeb80080)
This seems like very routine use of local::lib:
Am I using local::lib in the way intended? What else should I be doing?
How is local::lib supposed to handle XS?
Is there a way to make local::lib support multiple versions of Perl? My distro reads and writes the XS it manages to /usr/share/perl/$PERL_VERSION. This means a dist-upgrade leaves all the XS from the old version behind. It would be nice to have local::lib install not to /home/ecarroll/perl5/lib/perl5/ but instead to /home/ecarroll/perl5/lib/perl5.30/? Note the added minor version. Then at least a dist-upgrade wouldn't break the distribution.
Perl doesn't maintain ABI compatibility between versions. (5.34.x -> 5.36.y, for example.) After installing a new version of Perl, you will need to re-install modules that install themselves in arch dirs (XS modules and a few others). Might be easiest and safest to remove (rename) the local lib directory (/home/ecarroll/perl5) and start from scratch.
That's why I avoid using anything but distro-provided modules with distro-provided perl. The package manager ensures you always have compatible builds of the modules.
If that's not good enough, I recommend installing your own builds of perl instead of using local::lib. Tools like perlbrew makes this easy. You'll still have to re-install all the modules when you install a new perl, but you'll get the cleaner "Can't locate" error message if you forget one.
I rarely use local::lib, but I'm also a minimalist when it comes to administration. The more layers you have, the worse things get. As you see, you're own answer (now deleted) proposes even more complexity to deal with complexity that isn't likely appropriate for your situation.
But, local::lib has a way to set the locations of everything. You are trying to use the same location for everything instead of changing the locations based on what you are trying to do. However, even though local::lib lets you do that, you're likely to forget to switch things up.
My solution is simpler. I merely install a perl for whatever I'm doing (or reuse a suitable perl). I don't change the default locations at all. I never have to manage that. I simply call the perl I want:
$ ls -d /usr/local/perls/*
/usr/local/perls/perl-5.10.1 /usr/local/perls/perl-5.24.3 /usr/local/perls/perl-5.32.0
/usr/local/perls/perl-5.12.5 /usr/local/perls/perl-5.26.1 /usr/local/perls/perl-5.32.1
/usr/local/perls/perl-5.14.4 /usr/local/perls/perl-5.26.2 /usr/local/perls/perl-5.34.0
/usr/local/perls/perl-5.16.3 /usr/local/perls/perl-5.28.0 /usr/local/perls/perl-5.34.1
/usr/local/perls/perl-5.18.4 /usr/local/perls/perl-5.30.0 /usr/local/perls/perl-5.35.11
/usr/local/perls/perl-5.20.3 /usr/local/perls/perl-5.30.1 /usr/local/perls/perl-5.36.0
/usr/local/perls/perl-5.22.4 /usr/local/perls/perl-5.30.2 /usr/local/perls/perl-5.8.9
I have various symlinks and stable path adjustments to I use the tool name and the version tag I gave it:
% cpan5.36.0 ...
% cpan5.32.1 ...
People use all sorts of tools such as perlbrew, plenv, and now some other tool that you think might solve it. It's a big red flag if you have to constantly create new tools to deal with the tools that you are using. I'd rather spend my time doing something else.
I stand by my other answer—this is too much complexity and you should reduce that. But, lets work with that constraint. You want to use local::lib.
The problem is that you are trying to force the same set of modules on any particular perl, no matter how it was compiled. As ikegami explained, you can't assume that two different perls, or the same library compiled with different perls, are compatible. The version of the perls does not matter. Two different perl5.36, or any version, can be incompatible with each other.
But local::lib gives you what you need to set the appropriate values. The module itself can supply some starter values:
$ perl -I$HOME/perl5/lib/perl5 -Mlocal::lib
Attempting to create directory /Users/brian/perl5
PATH="/Users/brian/perl5/bin${PATH:+:${PATH}}"; export PATH;
PERL5LIB="/Users/brian/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}"; export PERL5LIB;
PERL_LOCAL_LIB_ROOT="/Users/brian/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"; export PERL_LOCAL_LIB_ROOT;
PERL_MB_OPT="--install_base \"/Users/brian/perl5\""; export PERL_MB_OPT;
PERL_MM_OPT="INSTALL_BASE=/Users/brian/perl5"; export PERL_MM_OPT;
But, you don't need to use those values everywhere. You can have as many sets of those variables as you like. You can have a different set for every project, every perl, or however you want to organize it. The only trick is how you select the right set.
The synopsis of local::lib indicates that it expects you to set the appropriate value for your project:
use FindBin;
use local::lib "$FindBin::Bin/../support"; # app-local support library
Knowing that, you can have your per-version libraries by adding $^V in the path you give to local::lib (still with the binary incompatibility problems):
use FindBin;
use local::lib "/my/modules/$^V";
Because i have a long series of comments with #ikegami, I cleaning up the question, in a hope it will be more understandable. Unfortunately, english isn't my "main" language. :(
Let say, having an environment where:
no development tools are installed (no make, nor gcc or like)
perl is installed with its core packages, nothing more
no outgoing network access is allowed - e.g. the user couldn't use curl nor cpan to download/install perl dependencies
the user even doesn't have admin (root) rights
but want install and evaluate some perl based web-app, let call it as MyApp
The MyApp
doesn't uses any XS-based module. (at least, I hope - in the development me using plenv and cpanm, so never checked the installed dependencies in depth)
it is an pure PSGI app, the simple plackup app.psgi works OK
the app uses some data-files which should be included in the "deployment".
The main question is: how to prepare the MyApp, and the all used CPAN-modules, to be easily installed in such restricted environment?
The goal is:
i don't need save my efforts and my time
but i want save the user's time and want minimize the needed actions on his side, so the installation (deployment) should be simple-as-possible.
E.g. how to get an running web-app to the user's machine with minimum possible (his) steps.
- the simplest thing is could be something as:
- copy one file (zip, or tarbal)
- unpack it
- from the terminal execute some run.pl in the unpacked directory.
To get the above simple installation, my idea was the following:
1.) Create an tarball, and after the unpacking will contain 3 folders and 1 perl-script, let say:
myapp_repo/
myapp_repo/distlib #will contain all MyApp's perl modules also ALL used CPAN modules and their dependecies
myapp_repo/datafiles #will contain app-specific data files and such
myapp_repo/install.pl
myall_repo/lib #will contain modules directly used by the `install.pl`
2.) I will develop an install.pl script, and it will be used as the installer-tool, like
perl install.pl new /path/to/app_root
and it will (should):
create the all needed directories under the /path/to/app_root (especially the lib where the will install the perl modules)
will call "local" cpanm internally (from the myapp_repo/lib) to install the app's perl modules and their CPAN dependencies using only distribution files from the distlib.
will generate and install the needed runtime script and the app.psgi into the /path/to/app_root/bin
will install the needed data-files for the app.
3.) So, after this the user should be able to simply run:
/path/to/app_root/bin/plackup /path/to/app_root/bin/app.psgi
In short, the user should use:
the system-wide perl and the system-wide perl-core modules
and any other
runtime perl-scripts (like plackup)
and the required CPAN-modules
should be installed to an self-contained directory tree using only files (no net-access).
E.g. the install.pl should somewhat call internally the cpanm to achieve (as equivalent) for the following cpanm command
cpanm --mirror file://path/to/myapp_repo/distlib --mirror-only My::App
which, should install My::App and all dependencies without network access using only the files from the myapp_repo/distlib
Some questions:
Is possible to use cpanm (called as an locally installed module) without the make?
For creating the myapp_repo/distlib, me thinking about using Pinto. Is it the right tool for achieve the above?
forgot me something? or with other words:
Is the above an viable (read: working) way?
are are any other tools, which i could/should to use for simplifying the creation of such distribution tarball?
#ikegami suggesting some method:
- "install everything" in one fresh-directory on my machine
- transfer this self-contained directory to the target machine
It sound very good, because this directory could contain all the needed app-specific data-files too, unfortunately, I don't understand the details how his solution should be done.
The FatPacked solution looks interesting too - need learn about it.
Don't write your own make or installer. Just copy it make from a different machine (which is basically what apt/yum/etc do anyway, and which you'd have to do even if you wrote your own). You'd be able to use cpan in 5 minutes!
Also, that should allow you to install gcc if you need it (e.g. to install an XS module), although it doesn't sound like you do. If you do install gcc, I'd install my own perl to avoid having to deal with PERL5LIB.
Tools such as minicpan will allow you to install any module from CPAN without internet access. Of course, you can keep using the command you are already using it if mirrors the packages you need.
The above explains how to simply and quickly setup a machine so it can use cpan and thus install any module easily.
If you just want to install a specific module and its dependencies, you can completely avoid using cpan on the target machine. First, you need a fresh install of Perl (preferable of the same version as the one on the target system). Then, simply install the module to a fresh dir on your machine, and transfer that dir to the target machine. That's it; nothing else needs to be done. This even works for XS modules if the two machine are similar enough.
This is what ppm (ActiveState's Perl package manager) does.
Unfortunately, while this solution is almost as simple as the one above, it's not nearly as flexible, it doesn't run the test suite of the modules being installed, etc. It does have the advantage of not requiring the transfer of any binary (if you're not installing any XS modules).
I maintain multiple Perl written (unix-ish) applications whose current installation process consists of a manually written Makefile and installs configuration files into /etc/.
I'd really like to switch their development to use Dist::Zilla, but so far I haven't found any Dist::Zilla plugin or feature which allows me to put given files into /etc/ when the make install (or ./Build install in case of using Module::Build instead of ExtUtils::MakeMaker) is run by the local administrator who's installing my application.
With pure ExtUtils::MakeMaker, I could define additional make targets in MY::postamble and the let the install target depend on one of them via the depend { install => … } attribute. Doing something similar, but via dzil build, would probably suffice, but I'd appreciate a more obvious way.
One orthogonal approach would be to make the application not to require the files under /etc/ to exist, but for just switching to Dist::Zilla that seems to much change in the actual code despite I only want to change the build system for now.
For the curious: the two applications I currently have in mind for switching to Dist::Zilla are xen-tools and unburden-home-dir.
The best thing to do is to avoid installing files into /etc from any Perl distribution. You cannot ensure that the cpan client (or the installing user) has permissions to install there, and there can be multiple Perls installed on a system, so each one of them would clobber the /etc files of another install. You can't really prevent the file from being overwritten by a subsequent install, so you shouldn't put config data there that you don't want to lose.
You could put the config file in /etc/, if the application knows to look for it there, but you should allow for that path to be customized (say on a test system, look for the file in the local directory, or in a user's home directory).
For installing read-only module-specific data, the best practice in Perl is to install into a Perl-install-specific location, and the module to do that is File::ShareDir::Install. You can use it from Dist::Zilla using the [ShareDir] plugin, Dist::Zilla::Plugin::ShareDir. It is even included in the [#Basic] plugin bundle, so if you use [#Basic] in your dist.ini, you don't need to do anything at all, other than drop your data files into the share/ directory in your distribution repository.
To access the contents of the sharedir from code, use File::ShareDir.
For porting a complex module installer to Dist::Zilla, I recommend my plugins MakeMaker::Custom or ModuleBuild::Custom, depending on which installer you prefer. These allow you to keep your existing Makefile.PL or Build.PL and just have Dist::Zilla plug in necessary bits like the dependencies.
I need help: is there a way to fetch a file from the remote server using only core modules of perl 5.8.8? File::Fetch became core module only from 5.9.
This comes up all the time. Take a look at the classic yes, even you can use CPAN. If you have the ability to create and run a Perl script, then you also have the ability to put a module in your local directory and use it. The requirement to use only core modules is entirely artificial.
In your case, LWP::Simple's getstore() function will do what you want. While it is technically not core, LWP::Simple is included by default with many Perl distributions. You may well already have it.
Update: so, you want to do this on 1000 servers? No need to manually install the module on each server. Use CPAN programmatically to download and install the module(s) you need (some tweaking will be needed to get CPAN to install it locally rather than in the root module library). Also Leon Timmermans's suggestion of fatpacking the module is another option.
If you really don't want to do it this way, then basically the answer is no: there is no simple way to fetch a remote file via HTTP using neither the appropriate modules, nor a system command (I didn't consider writing your own HTTP client to be a simple method, but that's fine if it works for you).
The only other potential solution I see would be a different approach to your problem, such as:
Using a script in a single location to get the file, then distribute
it to all 1000 servers via FTP.
Or, putting the file on an FTP server, then using a simple Perl
script on each server to fetch it via FTP.
As Dan already said, yes, even you can use CPAN. One approach his link doesn't mention is writing it as a normal CPAN-using distribution, and then fatpack it. Fatpacker combines a script with all its (pure-perl) dependencies, creating a single easy to distribute file.
You could use:
my $wgetoutput = `wget "$myFileToGet"`;
Stuff in backticks (`) will be given to the default shell, so you can call whatever you want (and are allowed) there.
Caveat: $myFileToGet could have stuff like "&& rm -rf *" in it, so dont forget to sanitize!
I'm using DBI and DBD::SQLite, and now I'd like to use the R*Tree feature of SQLite. Since this feature is not compiled by DBD::SQLite by default, I have to add a -DSQLITE_ENABLE_RTREE=1 to the #CC_DEFINE variable in DBD::SQLite's Makefile.PL. If I do a 'perl Makefile.PL && make && make install', everything works fine locally on my machine, but this ultimately needs to be deployable/distributable to end users.
What should I do in a case like this? Should I copy the source, grep the source, and create a DBD::SQLite::WithRTree? Create a private version of DBD::SQLite 1.31.1 (Where 1.31 is the current version of DBD::SQLite)? Perhaps a better way altogether?
All other distributions in the project are deployed/distributed via a non-public CPAN::Mini mirror + CPAN::Mini::Inject.
I have to add a '-DSQLITE_ENABLE_RTREE=1' to the #CC_DEFINE variable in DBD::SQLite's Makefile.PL
You're doing this wrong, perl Makefile.PL DEFINE='-DSQLITE_ENABLE_RTREE=1' works. This is documented in ExtUtils::MakeMaker. Now that you know that, a simple solution involving Distroprefs will likely fall in place.
For similar problems, I have installed the modified distribution in a separate directory (without changing any module names), and using use lib qw(the/special/directory) or setting $PERL5LIB for scripts that need to use the enhanced module.
Tweaking the name of the module would also do the job, but that sound like a lot more work to make and test.
You can do this:
cpan
o conf makepl_arg "DEFINE='-DSQLITE_ENABLE_RTREE=1'"
o conf commit
CPAN will then permanently add that DEFINE to the front of all your Makefile.PL calls.
So, it should just be
cpan DBD::SQLite
And your makefile options should get stuffed onto your compile lines