I'm trying to build DBD::Pg on a linux host via the Makefile.PL; my requirements are such that I must be able to dynamically link against perl, but statically link against libpq.so (since it may not be available on all boxes).
Is there an easy way to do this? I have tried changing the link options in the LIBS directive of the Makefile.PL but MakeMaker ignores my options.
IMO you've mis-specified your requirements.
You do not need to statically link to libpq just because it might not be available on all systems.
What you should generally do instead is dynamically link to libq and either set LD_LIBRARY_PATH in a wrapper script or use rpath linking so that libpq can be found.
Be aware though that whether statically or dynamically linking, if some other module loads a libpq into the same Perl you'll either get two incompatible libpqs linked into the same executable (boom) or one of the modules using a libpq other than the one it was compiled against (also boom). If you use rpath linking, ld.so's awareness of link scopes might let you get away with it, but setting LD_LIBRARY_PATH will almost certainly cause issues.
You might want to look into using rpath with $ORIGIN.
Unfortunately, trying to do static linking of libpq is not likely to solve your problem at large.
libpq itself is likely to depend on libc (glibc). If you link it statically, but other modules dynamically, it means you will have 2 copies of libc: one inside libpq, another referenced by Perl itself and loaded dynamically. This is very dangerous situation, especially if some procedure allocates memory using malloc and passes pointer back to caller. If you have memory allocated by malloc from one copy of libc, but freeed by another copy, your program (and Perl) will definitely crash.
In other words, if you want to go static, you must go all way through - everything, 100% must be compiled statically, so only one copy of libc used by your application. And the opposite is true - if you are dynamic, everything should be dynamic, as to only ever use one copy of libc. These rules don't apply only if your libraries do not use anything from libc (not even sprintf).
Even if you succeed at static libpq compilation and it will work (not very likely), what if DBI is not installed? I have seen enough Linux boxes where DBI is not present by default. Do you compile DBI statically then as well? What if Perl is not present (as unlikely this is on Linux), or if it is very old?
Proper solution is to install it using native OS package manager:
sudo apt-get install libdbd-pg-perl # Ubuntu/Debian
sudo yum install perl-DBD-Pg # Redhat/Fedora
If you do not have root on hosts in question, maybe you should consider using perlbrew - install your own Perl in home directory. With this, you should be able to compile your own copy of libpq and link it with Perl provided by perlbrew dynamically.
Related
I love the idea of local::lib. At least, I find it preferable to the default system-wide installs that cpan/perl defualts to in most every way. However, I'm having difficulties with modules that install XS and distribution upgrades to newer Perls. The instructions for local::lib call for adding this line to your .bashrc (or like file)
eval "$(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)"
That executes arbitrary shell code, which is fine. You can see the code it executes by running
perl -I$HOME/perl5/lib/perl5 -Mlocal::lib
In my case it returns this code,
PATH="/home/ecarroll/perl5/bin${PATH:+:${PATH}}"; export PATH;
PERL5LIB="/home/ecarroll/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}"; export PERL5LIB;
PERL_LOCAL_LIB_ROOT="/home/ecarroll/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"; export PERL_LOCAL_LIB_ROOT;
PERL_MB_OPT="--install_base \"/home/ecarroll/perl5\""; export PERL_MB_OPT;
PERL_MM_OPT="INSTALL_BASE=/home/ecarroll/perl5"; export PERL_MM_OPT;
In the above, we're setting the default options for the two most widely used module build systems,
MB is for Module::Build
MM is for ExtUtils::MakeMaker
We're telling those system to install to /home/ecarroll/perl5. The rest of the command sets
PERL5LIB which is used in setting #INC. The #INC variable in Perl is used to resolve calls with use.
and, PATH which prepends the bin/ directory that local::lib installs to.
So basically, install perl modules in your home directory, and look first for files installed in your home directory.
My question is what happens when you install something that has XS? In my case, I have an XS file,
/home/ecarroll/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Class/Load/XS/XS.so
This was installed, with local::lib using an earlier version of Perl. I've since ran a Debian dist-upgrade and got a newer version of Perl (v5.36). Now a lot of utilities produce an obtuse error about this, including ones installed through the distro itself like in my case Dist::Zilla dzil which I installed with Debian's libdist-zilla-perl package.
$ dzil
XS.c: loadable library and perl binaries are mismatched (got first handshake key 0xeb00080, needed 0xeb80080)
Which is all a result of this module's XS,
perl -MClass::Load::XS -e1
XS.c: loadable library and perl binaries are mismatched (got first handshake key 0xeb00080, needed 0xeb80080)
This seems like very routine use of local::lib:
Am I using local::lib in the way intended? What else should I be doing?
How is local::lib supposed to handle XS?
Is there a way to make local::lib support multiple versions of Perl? My distro reads and writes the XS it manages to /usr/share/perl/$PERL_VERSION. This means a dist-upgrade leaves all the XS from the old version behind. It would be nice to have local::lib install not to /home/ecarroll/perl5/lib/perl5/ but instead to /home/ecarroll/perl5/lib/perl5.30/? Note the added minor version. Then at least a dist-upgrade wouldn't break the distribution.
Perl doesn't maintain ABI compatibility between versions. (5.34.x -> 5.36.y, for example.) After installing a new version of Perl, you will need to re-install modules that install themselves in arch dirs (XS modules and a few others). Might be easiest and safest to remove (rename) the local lib directory (/home/ecarroll/perl5) and start from scratch.
That's why I avoid using anything but distro-provided modules with distro-provided perl. The package manager ensures you always have compatible builds of the modules.
If that's not good enough, I recommend installing your own builds of perl instead of using local::lib. Tools like perlbrew makes this easy. You'll still have to re-install all the modules when you install a new perl, but you'll get the cleaner "Can't locate" error message if you forget one.
I rarely use local::lib, but I'm also a minimalist when it comes to administration. The more layers you have, the worse things get. As you see, you're own answer (now deleted) proposes even more complexity to deal with complexity that isn't likely appropriate for your situation.
But, local::lib has a way to set the locations of everything. You are trying to use the same location for everything instead of changing the locations based on what you are trying to do. However, even though local::lib lets you do that, you're likely to forget to switch things up.
My solution is simpler. I merely install a perl for whatever I'm doing (or reuse a suitable perl). I don't change the default locations at all. I never have to manage that. I simply call the perl I want:
$ ls -d /usr/local/perls/*
/usr/local/perls/perl-5.10.1 /usr/local/perls/perl-5.24.3 /usr/local/perls/perl-5.32.0
/usr/local/perls/perl-5.12.5 /usr/local/perls/perl-5.26.1 /usr/local/perls/perl-5.32.1
/usr/local/perls/perl-5.14.4 /usr/local/perls/perl-5.26.2 /usr/local/perls/perl-5.34.0
/usr/local/perls/perl-5.16.3 /usr/local/perls/perl-5.28.0 /usr/local/perls/perl-5.34.1
/usr/local/perls/perl-5.18.4 /usr/local/perls/perl-5.30.0 /usr/local/perls/perl-5.35.11
/usr/local/perls/perl-5.20.3 /usr/local/perls/perl-5.30.1 /usr/local/perls/perl-5.36.0
/usr/local/perls/perl-5.22.4 /usr/local/perls/perl-5.30.2 /usr/local/perls/perl-5.8.9
I have various symlinks and stable path adjustments to I use the tool name and the version tag I gave it:
% cpan5.36.0 ...
% cpan5.32.1 ...
People use all sorts of tools such as perlbrew, plenv, and now some other tool that you think might solve it. It's a big red flag if you have to constantly create new tools to deal with the tools that you are using. I'd rather spend my time doing something else.
I stand by my other answer—this is too much complexity and you should reduce that. But, lets work with that constraint. You want to use local::lib.
The problem is that you are trying to force the same set of modules on any particular perl, no matter how it was compiled. As ikegami explained, you can't assume that two different perls, or the same library compiled with different perls, are compatible. The version of the perls does not matter. Two different perl5.36, or any version, can be incompatible with each other.
But local::lib gives you what you need to set the appropriate values. The module itself can supply some starter values:
$ perl -I$HOME/perl5/lib/perl5 -Mlocal::lib
Attempting to create directory /Users/brian/perl5
PATH="/Users/brian/perl5/bin${PATH:+:${PATH}}"; export PATH;
PERL5LIB="/Users/brian/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}"; export PERL5LIB;
PERL_LOCAL_LIB_ROOT="/Users/brian/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"; export PERL_LOCAL_LIB_ROOT;
PERL_MB_OPT="--install_base \"/Users/brian/perl5\""; export PERL_MB_OPT;
PERL_MM_OPT="INSTALL_BASE=/Users/brian/perl5"; export PERL_MM_OPT;
But, you don't need to use those values everywhere. You can have as many sets of those variables as you like. You can have a different set for every project, every perl, or however you want to organize it. The only trick is how you select the right set.
The synopsis of local::lib indicates that it expects you to set the appropriate value for your project:
use FindBin;
use local::lib "$FindBin::Bin/../support"; # app-local support library
Knowing that, you can have your per-version libraries by adding $^V in the path you give to local::lib (still with the binary incompatibility problems):
use FindBin;
use local::lib "/my/modules/$^V";
Because i have a long series of comments with #ikegami, I cleaning up the question, in a hope it will be more understandable. Unfortunately, english isn't my "main" language. :(
Let say, having an environment where:
no development tools are installed (no make, nor gcc or like)
perl is installed with its core packages, nothing more
no outgoing network access is allowed - e.g. the user couldn't use curl nor cpan to download/install perl dependencies
the user even doesn't have admin (root) rights
but want install and evaluate some perl based web-app, let call it as MyApp
The MyApp
doesn't uses any XS-based module. (at least, I hope - in the development me using plenv and cpanm, so never checked the installed dependencies in depth)
it is an pure PSGI app, the simple plackup app.psgi works OK
the app uses some data-files which should be included in the "deployment".
The main question is: how to prepare the MyApp, and the all used CPAN-modules, to be easily installed in such restricted environment?
The goal is:
i don't need save my efforts and my time
but i want save the user's time and want minimize the needed actions on his side, so the installation (deployment) should be simple-as-possible.
E.g. how to get an running web-app to the user's machine with minimum possible (his) steps.
- the simplest thing is could be something as:
- copy one file (zip, or tarbal)
- unpack it
- from the terminal execute some run.pl in the unpacked directory.
To get the above simple installation, my idea was the following:
1.) Create an tarball, and after the unpacking will contain 3 folders and 1 perl-script, let say:
myapp_repo/
myapp_repo/distlib #will contain all MyApp's perl modules also ALL used CPAN modules and their dependecies
myapp_repo/datafiles #will contain app-specific data files and such
myapp_repo/install.pl
myall_repo/lib #will contain modules directly used by the `install.pl`
2.) I will develop an install.pl script, and it will be used as the installer-tool, like
perl install.pl new /path/to/app_root
and it will (should):
create the all needed directories under the /path/to/app_root (especially the lib where the will install the perl modules)
will call "local" cpanm internally (from the myapp_repo/lib) to install the app's perl modules and their CPAN dependencies using only distribution files from the distlib.
will generate and install the needed runtime script and the app.psgi into the /path/to/app_root/bin
will install the needed data-files for the app.
3.) So, after this the user should be able to simply run:
/path/to/app_root/bin/plackup /path/to/app_root/bin/app.psgi
In short, the user should use:
the system-wide perl and the system-wide perl-core modules
and any other
runtime perl-scripts (like plackup)
and the required CPAN-modules
should be installed to an self-contained directory tree using only files (no net-access).
E.g. the install.pl should somewhat call internally the cpanm to achieve (as equivalent) for the following cpanm command
cpanm --mirror file://path/to/myapp_repo/distlib --mirror-only My::App
which, should install My::App and all dependencies without network access using only the files from the myapp_repo/distlib
Some questions:
Is possible to use cpanm (called as an locally installed module) without the make?
For creating the myapp_repo/distlib, me thinking about using Pinto. Is it the right tool for achieve the above?
forgot me something? or with other words:
Is the above an viable (read: working) way?
are are any other tools, which i could/should to use for simplifying the creation of such distribution tarball?
#ikegami suggesting some method:
- "install everything" in one fresh-directory on my machine
- transfer this self-contained directory to the target machine
It sound very good, because this directory could contain all the needed app-specific data-files too, unfortunately, I don't understand the details how his solution should be done.
The FatPacked solution looks interesting too - need learn about it.
Don't write your own make or installer. Just copy it make from a different machine (which is basically what apt/yum/etc do anyway, and which you'd have to do even if you wrote your own). You'd be able to use cpan in 5 minutes!
Also, that should allow you to install gcc if you need it (e.g. to install an XS module), although it doesn't sound like you do. If you do install gcc, I'd install my own perl to avoid having to deal with PERL5LIB.
Tools such as minicpan will allow you to install any module from CPAN without internet access. Of course, you can keep using the command you are already using it if mirrors the packages you need.
The above explains how to simply and quickly setup a machine so it can use cpan and thus install any module easily.
If you just want to install a specific module and its dependencies, you can completely avoid using cpan on the target machine. First, you need a fresh install of Perl (preferable of the same version as the one on the target system). Then, simply install the module to a fresh dir on your machine, and transfer that dir to the target machine. That's it; nothing else needs to be done. This even works for XS modules if the two machine are similar enough.
This is what ppm (ActiveState's Perl package manager) does.
Unfortunately, while this solution is almost as simple as the one above, it's not nearly as flexible, it doesn't run the test suite of the modules being installed, etc. It does have the advantage of not requiring the transfer of any binary (if you're not installing any XS modules).
I try to build a Perl distribution for a home-made module, from the Module::Starter base. Every test pass on my machine, but when I upload it to CPAN to get some more universal tests from cpantesters.org, some test failed on other architectures or OS, but I can't understand why. I can see in test reports that some of my prerequisites are not installed before testing but I would like it to.
I've tried to list these dependencies into the Makefile.PL PREREQ_PM hash and then in the TEST_REQUIRES hash, but it didn't changed a lot of results.
Then, when I've removed the dependencies from my local machine and tried to install my module using Cpanm, it downloads dependencies first, test passed and install has been a success.
This is my first try for a module, so I think I am missing something, maybe I am too used of the Cpanm magic. Thanks for any help.
The problem is something different. Andreas' smoker very probably built the dependency App::Ack (which looks in the fail reports like being absent) successfully. But here come at least two problems:
When a distribution gets tested, then its dependencies may or may not be installed already. However, it's guaranteed that all dependent modules are made available through the PERL5LIB environment variable, so make test usually works (To be more specific, if the install Module command is used in the CPAN shell, then all dependencies are installed immediately. If the test Module command is used, then dependencies are only built, but not installed. The CPAN user can do the installation later using install_tested). So it may be that App::Ack is not installed here, just built. Especially this means that the ack script is not installed in the final location.
Even if it is installed, many smoke testers or users who have multiple perls installed in parallel use a non-standard directory for this perl. So ack wouldn't be installed in /usr/bin or /usr/local/bin, but in the bin directory belonging to this perl. This directory may or may not be in the user's PATH at all. So you cannot assume that can_run("ack") works here. A workaround here is to add $Config{scriptdir} temporarily to $ENV{PATH}. Another solution would be to use the App module instead of the script, if it's possible. Unfortunately it looks like ack can only be called as a script.
If you look at a sample fail report, then you can see that App::Ack was installed (it appears in the PREREQUISITES section both under requires and build_requires, you can also see which App::Ack version is installed in the "HAVE" column). You can also see the user's PATH (in the ENVIRONMENT section). And you may guess about the scriptdir for this perl, it's usually the same directory where the perl binary itself is installed, and the path to current perl is visible in $^X (under "Perl special variables").
If you want to reproduce the behavior, then you need to deinstall ack from your machine, build a custom perl using ./configure.gnu --prefix=/path/to/custom/perl-5.X.Y, and use this perl for tests.
Being very new to Perl but not to dynamic languages, I'm a bit surprised at how not straight forward the manage of modules is.
Sure, cpan X does theoretically work, but I'm working on the same project from three different machines and OSs (at work, at home, testing in an external environment).
At work (Windows 7) I have problem using cpan because of our firewall that makes ftp unusable
At home (Mac OS X) it does work
In the external environment (Linux CentOs) it worked after hours because I don't have root access and I had to configure cpan to operate as a non-root user
I've tried on another server where I have an access. If the previous external environment is a VPS and so I have a shell access, this other one is a cheap shared hosting where I have no way to install new modules other than the ones pre-installed
At the moment I still can't install Template under Windows. I've seen that as an alternative I could compile it and I've also tried ActiveState's PPM but the module is not existent there.
Now, my perplexity is about Perl being a dynamic language. I've had all these kind of problems while working, for example, with C where I had to compile all the libraries for all the platform, but I thought that with Perl the approach would have been very similar to Python's or PHP's where in 90% of the cases copying the module in a directory and importing it simply works.
So, my question: if Perl's modules are written in Perl, why the copy/paste approach will not work? If some (or some part) of the modules have to be compiled, how to see in CPAN if a module is Perl-only or it relies upon compiled libraries? Isn't there a way to download the module (tar, zip...) and use cpan to deploy it? This would solve my problem under Windows.
Now, Perl is a dynamic language, but that doesn't imply that everything that people write is portable across platforms. That's not the fault of the language. It's not even the fault of the programmer. Some things, like Win32::OLE shouldn't work on Unix. :)
Other dynamic languages will have some of the same problems. If you have to compile C code, you won't be able to merely copy files to another machine. Some distributions configure the code slightly differently depending on your operating system, etc.
Even if you could copy files, you have to ensure that you copy all of the files that you need. Do you know everything that you need for a particular module? Remember, many of them have dependencies.
Most of the problems you're having aren't anything to do with the language. You're having trouble with the tools. If you want a zero conf CPAN tool that makes all the decisions for you, try cpanminus. It's mostly the same thing that you'd get out of cpan (although different code), but it makes all of the decisions for you. It doesn't run any of the distribution tests, and it installs into your user directory. When you need something that gives you control, come back to cpan.
In the external environment (Linux CentOs) it worked after hours because I don't have root access and I had to configure cpan to operate as a non-root user
This is one of those times when it helps to know The Trick. In this case local::lib, which lets you configure a non-root install area and all the ENV variables in about three minutes.
if perl's modules are written in perl, why the copy/past approach will not work?
Some are written in Pure Perl, but many are written partially in C (using Perl's XS API) for efficiency.
Sometimes you end up with situations like JSON::XS, JSON::PP and JSON::Any to autoselect the best one that is installed.
Isn't there a way to download the module (tar, zip...) and use cpan to deploy it?
The cpan program is all about getting things from the Internet. You can download the package (there will be a link along the lines of "Download: CGI.pm-3.49.tar.gz" on the right hand side of the CPAN page), untar it, then
perl Makefile.PL
make
make install
You would probably be better off configuring your cpan installation to use only HTTP sources (in the urllist config option). Possibly going to far as to create a mini CPAN mirror inside your network.
I need to install my Perl-based software on networked machines which aren't connected to the internet. Therefore, I would like to download specific versions and/or latest versions of the Perl modules and I would also like to know if there is an install procedure required for these modules.
Background:
The machines aren't connected to the internet for security reasons and its deemed unnecessary also.
I would place the downloaded modules on a machine that I call the 'install server' and it contains my Perl based software and would also contain the local copies of the Perl modules.
I call a machine that I want to install my Perl-based software on, the 'target machine', also not connected to the internet. There can be several target machines, each can run this software that I want to install. I log onto the target machine and run an install script which would connect to the install machine via the local network to obtain the Perl-based software and dependent Perl modules and installs them.
So I need to know:
How/Where to get specific versions of Perl modules, e.g. CGI.pm etc
How to install these Perl modules. Is it a case of just placing them in a directory somewhere, e.g. a library path and making sure that this directory path is in the #INC library path environmental variable, if it is not already?
I would prefer not to have to do anything like make install etc. as part of installing the modules. I would like to modules to be pre-compiled or prepared as necessary so it is as simple as possible to install them. I want to avoid additional dependencies like make and its configuration, and having to parse its output to check whether it was successful.
Please help me by asking the above specific questions as I am not able to change the concept of 'install machine' and 'target machine' which aren't connected to the internet - I have to provide a solution that works within this arrangement.
The usual way to solve "I want to install stuff from CPAN but without network" problems is to use a minicpan as David Dorward wrote in his answer. But since you're going one step further, saying that you'd rather not do any real installation on the client (target) machines at all, and that you want to use precompiled modules if possible, I urge you to check out PAR and specifically PAR::Repository (server) and PAR::Repository::Client.
Since this approach needs some research before you're up to speed, I wouldn't suggest it for "I just need Foo.pm" like problems. Once you're talking about a handful of dependencies and at least a handful of clients, then it becomes a more appropriate solution.
For an outline of how it works, check out the slides of my talk at YAPC::EU 2008. It also hints at solutions to the bootstrapping problem of making the PAR::Repository::Client module available on the clients (hint: PAR can generate self-contained executables).
You can create a MiniCPAN that has just the latest versions of everything from CPAN. You can insert additional, non-public modules into it with CPAN::Mini::Inject. If you need to greater control over versions (i.e. not choosing the latest versions), you might want to create a DPAN.
With any of these solutions, you can configure your CPAN client to pull from your local source. That could be a directory you know ahead of time or something that you figure out dynamically, like a CD or a thumb-drive. It's just a matter of setting up the configuration correctly.
You might be able to get away with creating operating-system packages for most of your work, but that still means you have to compile them at least the first time.
1) How/Where to get specific versions of Perl modules, e.g. CGI.pm etc
http://search.cpan.org/
If you don't want the latest version, you can get an earlier version by following the link in the breadcrumbs.
http://img.skitch.com/20091209-bu7kt3bj65374k7iijfnhrue2y.png
2) How to install these Perl modules. Is it a case of just placing them
in a directory somewhere, e.g. a library path and making sure that this
directory path is in the #INC library path environmental variable, if
it is not already?
That sometimes work, but you really should go through the perl Makefile.PL && make && make test && make install process.
Doing this would require that you manually chase all the dependencies though. You would probably be better off with something like minicpan.