Why do official perl docker images have two version of perl? - perl

I'm working on a legacy product which uses the Docker perl:5.10-threaded image and ran into an issue trying to debug things when I discovered there are two version of perl - one in /usr/local/bin/perl and one in /usr/bin/perl. In this particular image, they are actually different versions
/usr/local/bin/perl -> 5.10.1
/usr/bin/perl -> 5.20.2
The issue it was causing is that each has a different #INC path.
$ /usr/local/bin/perl -V
[snip]
/usr/local/lib/perl5/5.10.1/x86_64-linux-thread-multi
/usr/local/lib/perl5/5.10.1
/usr/local/lib/perl5/site_perl/5.10.1/x86_64-linux-thread-multi
/usr/local/lib/perl5/site_perl/5.10.1
$ /usr/bin/perl -V
[snip]
/etc/perl
/usr/local/lib/x86_64-linux-gnu/perl/5.20.2
/usr/local/share/perl/5.20.2
/usr/lib/x86_64-linux-gnu/perl5/5.20
/usr/share/perl5
/usr/lib/x86_64-linux-gnu/perl/5.20
/usr/share/perl/5.20
/usr/local/lib/site_perl
.
The latest versions, like perl:5.30-threaded also have two versions and different #INC paths as well.
/usr/local/bin/perl -> 5.30.0
/usr/bin/perl -> 5.28.1
It turns out my legacy app uses #!/usr/bin/perl, which was very confusing to me when perl foo.pl didn't work the same as foo.pl (it complained about missing libraries).
To add a bit more color, this legacy app also installs a bunch of perl libraries via apt-get. E.g.,
libcpanel-json-xs-perl libxml-libxml-perl libcgi-pm-perl
libtie-ixhash-perl
libswitch-perl libmime-lite-perl liblist-moreutils-perl
libdate-calc-perl libnet-sftp-foreign-perl
libxml-libxslt-perl liburi-escape-xs-perl
libdatetime-perl
These seem to install things accessible by /usr/bin/perl, which is perhaps why the app uses it in the shebang lines.
The other thing this app does is install some cpanm items and then copy them over, so they are accessible to /usr/bin/perl
RUN cpanm XML::XML2JSON
RUN cpanm JSON
RUN cp /usr/local/lib/perl5/site_perl/5.10.1/XML/XML2JSON.pm /usr/lib/x86_64-linux-gnu/perl5/5.20/XML
RUN cp /usr/local/lib/perl5/site_perl/5.10.1/JSON.pm /usr/lib/x86_64-linux-gnu/perl5/5.20
This seems like a hack to me, but I'm new to perl development, so who knows?
All this leads up to my questions:
why are there two version of perl in the Docker images?
are apt-get and cpanm incompatible with each other?
is there a better way to install these perl libs?

Perl is an essential part of many Linux distributions, and has to come pre-installed. The system perl that is used by the operating system is usually installed as /usr/bin/perl. Modules for it are managed through the package manager (e.g. apt) and not via cpan/cpanm. If you were to install modules for the system perl yourself, this might conflict with modules expected by the operating system. Worse, installing the wrong module version could break parts of the OS. Similarly, replacing the system perl is a bad idea. That's why those Docker images install the different perl alongside.
For your apps, you should avoid the system perl. If you want to install extra modules for use with the system perl, consider using local::lib. In some cases you might install dependencies such as C libraries or external tools via apt, but you wouldn't use apt-provided Perl modules.
Unless you are targeting a specific operating system, do not hardcode the #!/usr/bin/perl shebang. Instead, prefer #!/usr/bin/env perl so that the script will use the perl that is first in the PATH. Alternatively, use wrapper scripts to explicitly invoke the correct perl installation. For example:
#!/bin/sh
exec /usr/local/bin/perl -I/path/to/extra/modules /path/to/my/script "$#"
Note that you cannot share modules between different versions of Perl. During installation of XS modules they are compiled for a specific version, and will fail to load with a different Perl version. For your local perl, just install dependencies via cpanm and ignore modules that were installed for the system perl.

Related

Using local Perl module instead of one installed by CPAN

I have found it necessary to expand upon a CPAN module. (Unicode::CharName goes up to Unicode 4.1; I need some characters from Unicode 5.0 & 5.1).
I've made the changes needed and have my own CharName.pm module.
I now would like to use it with my various Perls. I currently use:
Strawberry Perl for Windows
git for Windows MINGW64; My .bashrc sets
$PATH to Strawberry perl and $PERL5LIB=/c/Strawberry/perl/vendor/lib:/c/Strawberry/perl/site/lib
WSL Ubuntu
Where should I put my version of Unicode::CharName, so that it over-rides the ones installed by CPAN?
I don't want to have to change any scripts that currently
use Unicode::CharName;
Using cpanm you could download the module, patch it, and install it as normal:
$ cpanm --look Unicode::CharName
# new shell opens
$ patch lib/Unicode/CharName.pm custom.patch # or whatever process
$ perl Makefile.PL
$ make install
$ exit
You can also install it to a local::lib to avoid overwriting it globally, by adding the -l local/ option to the cpanm command. Then you can add the absolute path of this local::lib to your PERL5LIB or via -I or use lib. If you specified /path/to/local for the -l option, it would be /path/to/local/lib/perl5.
Manually copying files rather than going through the normal installation process is likely to lead to problems. Many distributions depend on the installation process to build the modules correctly. Also, you will need to install the module separately for each Perl you want to use it for; installed Perl modules are not generally cross-compatible between Perl versions or architectures. (A strictly simple pure-Perl module can be an exception to these rules, but the only module I feel comfortable abusing this way is App::cpanminus, because it was designed to do this.)

install different version of perl module using perl brew

we are using perlbrew for maintaining the various version of perl.
Along , now we are facing challenges as we need to update the scripts to the latest CPAN module. But we want still some the script to keep make use of old perl module till the compatibility issue is resolved.
Is there any away to install the lastest version of module with impacting the older version
perlbrew keeps each version separate, include its CPAN modules. If you run:
perl -E'say for #INC'
you will see the Perl version in the paths. If you switch the Perl, you will see different paths for #INC.
You can still use PERL5LIB with perlbrew-installed perl. But since you're already using perlbrew, why not just install a second build of that version of Perl for use with your newer scripts? Upgrading modules for one perl install won't affect others.
In some environments, Carton maybe the tool you are looking for. It requires a cpanfile where the dependencies are specified, so each script should be handled as an independent application in order to isolate each of them

How do I install Perl libraries in Cygwin?

I'm a C/C++/Java/Unix geek by trade, but now I have to write a Perl program in Windows. So I've got Cygwin, Eclipse with EPIC installed, and simple test programs run. I do not have a Unix machine available to me that I can run Eclipse on.
But I need to use Net::TCP::Server and I can't figure out how to install it. It doesn't appear to be part of the Cygwin Perl install, and I don't want to spend 5 days learning the world of Perl and CPAN and Perl configuration. I just want to write my program.
Any simple ways of installing a Perl module in Cygwin? I'm thinking something like apt-get install would be nice, but I expect that's too much to hope for.
Thanks
$ perl -MCPAN -e shell
cpan shell -- CPAN exploration and modules installation (v1.9402)
Enter 'h' for help.
cpan[1]> install Net::TCP::Server
And it's instructive to list the configuration with the o conf command.
Seeing that some of the info here is a bit outdated and too complicated, I'd rather suggest the following. There are a few different Perl package managers in use. They are all installed with cpan (which is already part of the Cygwin Perl installation), like this:
# Install ppm (outdated)
cpan PPM
# Install cpanp (still used)
cpan CPANPLUS
# Install cpanm (most recent)
cpan App::cpanminus
Then you can install any Perl package you like, as for example in the OP, using cpanm:
cpanm Net::TCP::Server
Sometimes (as noted above) Cygwin may fail certain tests. For example, when using IPv6 on a machine only configured with IPv4, or when your windows firewall is blocking some tests, etc. To attempt to install anyway, try to use the force flag; -f.
cpanm -f Net::TCP::Server
I'm a C/C++/java unix geek by trade, but now I have to write a perl program in windows. So I've got cygwin, eclipse with EPIC installed, and simple test programs run. I do not have a unix machine available to me that I can run eclipse on.
You should be able to run Eclipse with EPIC right under Windows without Cygwin. I like Cygwin for many things, but it isn't exactly a very stable platform. Eclipse runs as a Java program, so all you have to do is make sure Java is installed on your PC. There is even a pre-built Eclipse package.
You can also get a decent Perl that runs right under Windows. Either ActivePerl or Strawberry Perl. Either one will work although many Unix heads prefer Strawberry Perl because it comes with the same MIGW environment that Cygwin has and many feel it's more compatible with the Unix Perl brethren. However, ActiveState Perl can use the Perl Package Manager (PPM) that contains pre-built modules. Unfortunately, the module you want isn't available via PPM.
But I need to use Net::TCP::Server and I can't figure out how to install it. It doesn't appear to be part of the cygwin perl install, and I don't want to spend 5 days learning the world of perl and cpan and perl configuration. I just want to write my program.
Installing from CPAN is really quite simple. The first time you run CPAN, it should pretty much configure itself. Then, to do an install, you run cpan install Net::TCP::Server and that should be it. However, I've tried on several Mac, Unix, and Linux machines, and I can't get it to work. So, this isn't a Windows problem as much as a problem with this module. It is fairly old, and might not work well in newer versions of Perl. You might have to force the install over test's objections.
Maybe you can use one of the already installed IO modules that come with Perl instead. That'll save you a boatload of trouble because the required modules are part of Perl's existing package.
Despite Cygwin's "problems," I use it regularly whenever I have to use Windows. I would recommend first installing a separate installation of Perl using perlbrew so that you won't interfere with Cygwin's copy of Perl in case something bad happens since Cygwin does not enforce root-user policy. In cygwin shell, type
\curl -L http://install.perlbrew.pl | bash
This should walk through the installation for perlbrew and set it up in one of your executable path. Next type
perlbrew init
perlbrew install --force stable
perlbrew switch stable
Wait a bit while a mint Perl is compiled. For unknown reason, Perl can only pass 99.23% of the core module tests on Cygwin (at least on my machine),hence the --force flag). My experience is that it mostly have something to do with handling of device files, like ports and pipes. I am unaware of people trying to resolve the issue as it seems like a Cygwin problem. Although it has not presented much problem for me with general system and web programming tasks. The module testing routines will fail if any problem exists so I am not fretting over it.
Next step is to install cpanm (cpanminus), type
perlbrew install-cpanm
From here on out, to install any library from CPAN, just type
cpanm [library::name1] [library::name2]
cpanm makes it trivial to install any Perl modules. You can even install from your local directory instead of CPAN.
Your mileage may vary, but I had a lot of trouble until I realized that Strawberry perl had a lot of bin folders in my PATH, and when I changed my .bashrc to export only a very simple PATH=/bin:/usr/bin:/usr/local/bin, Cygwin's perl installation's cpan started working beautifully. I used local:lib as Cygwin doesn't support sudo. Before it got into a bad loop saying "Press SPACE and ENTER to exit Patch" over and over.

Cant run some perl scripts from Textmate (Can't locate in #INC error)

I primarily use Textmate for perl hacking on OSX Snow Leopard. It was all fine till some days ago when I installed Macports (I've heard a rumor that macports installs some version of perl in /opt/local). Now, for any new package that I install from CPAN, and try to use if from Textmate, it gives me an error like this when I hit Cmd+R to run it (I had installed HTML::TreeBuilder from CPAN moments before):
Can't locate HTML/TreeBuilder.pm in #INC (#INC contains: /Applications/TextMate.app/Contents/SharedSupport/Bundles/Perl.tmbundle/Support /Library/Perl/Updates/5.10.0/darwin-thread-multi-2level /Library/Perl/Updates/5.10.0 /System/Library/Perl/5.10.0/darwin-thread-multi-2level /System/Library/Perl/5.10.0 /Library/Perl/5.10.0/darwin-thread-multi-2level /Library/Perl/5.10.0 /Network/Library/Perl/5.10.0/darwin-thread-multi-2level /Network/Library/Perl/5.10.0 /Network/Library/Perl /System/Library/Perl/Extras/5.10.0/darwin-thread-multi-2level /System/Library/Perl/Extras/5.10.0 .) at /workspace/PerlScripts/test.pl line 2.
However it runs just fine when I run it from command line. If you will notice, textmate is using Perl version 5.10.0. When I run a "Perl -v" from command line, it shows me the version as 5.12.3 (see a copy paste from terminal below)
This is perl 5, version 12, subversion 3 (v5.12.3) built for darwin-multi-2level
The small perl code that I used is this:
#!/usr/bin/env perl <br/>
use HTML::TreeBuilder; <br/>
print $HTML::TreeBuilder::VERSION."\n";<br/>
print "$INC{'HTML/TreeBuilder.pm'}\n";<br>
Then I ran "which perl" from command line and it returned this:
/opt/local/bin/perl
then I ran "print system("which perl");" from Textmate and it returned
/usr/bin/perl
Now I clearly have two versions of perl sitting in my system. 5.10.0 that Textmate is trying to refer to and which cannot see my CPAN installs. And 5.12.3 that looks to be the default version of perl when I run it from command line and the one which my CPAN installs to. I tried to replace this:
#! /usr/bin/perl -w
with this (had read on google that this will force the system default perl to be picked up in textmate):
#!/usr/bin/env perl
No dice yet. Textmate still gives me the same error as I mentioned at the start of the question. So, is there a way to get it to work with Textmate? And is there a way to make textmate point to version 5.12.3?
You need to set the relevant environment variables for OS X. At the moment you probably have them set just for your shell. These will probably be PATH and PERL5LIB.
mkdir ~/.MacOSX
touch ~/.MacOSX/environment.plist
defaults write ~/.MacOSX/environment VARIABLE_NAME VARIABLE_VALUE
You'll need to log out and back in again to activate it.
Make sure that TextMate has your $PATH variables
in Textmate go to Preferences, then the Advanced Tab.
Under Shell Variables, make sure the path to your preferred perl (and cpan) is there something like:
/opt/local/bin:/opt/local/sbin:/usr/local/mysql/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
Your inc path can be gotten from:
perl -e 'print #INC,"\n"'
Or from perl -V.
It is higly probable that you installed HTML::Treebuilder in a version dependent path component, so 5.12 can findit, whereas 5.10 does not.
If this is the case, you may either install the relevant modules with 5.10 OR try some tricks with lib.pm:
use lib qw(/usr/perl/site_perl/5.12);
This kind of things are best left to your package manager though, and I do not advise you do this at home unless you are totally sure of what you are doing, as you might introduce subtle bugs due to version incompatibilities (or it could just blow up, but that's easier) .

perlbrew and local::lib at the same time?

So far I have been using the system perl (on Ubuntu 10.10) and I was using local::lib to install CPAN modules in my private directory ~/perl5
As I am trying to use perlbrew it seems that they don't know about each other. I installed perl-5.12.3 using perlbrew but when I switch to it using perlbrew use perl-5.12.3 I still see the PERL5LIB and PERL_MM_OPT set by local::lib.
That's not good:
$ cpan XML::Simple
/home/gabor/perl5/perlbrew/perls/perl-5.12.3/bin/perl: symbol lookup error: /home/gabor/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Cwd/Cwd.so: undefined symbol: Perl_Gthr_key_ptr
while
$ which cpan
/home/gabor/perl5/perlbrew/perls/perl-5.12.3/bin/cpan
so it is using the right version of the cpan client but dues to the PERL5LIB environment variable it picks up the modules from the wrong place.
Does perlbrew have some compability mode or do I need to turn off PERL5LIB and PERL_MM_OPT manually?
Since I started using perlbrew I stopped using local::lib for the shell use, because now that I have my own perl that i have write permissions to everything, just installing to site_perl is much more straightforward, and that allows me to have different versions of modules for each perl.
I still use local::lib (or more specifically, cpanm's -l or -L options that automatically sets up local::lib directory) to keep application specific dependencies inside an application directory.
local::lib was not designed to work with multiple versions of Perl installed at the same time. Pure-Perl modules aren't usually a problem, but XS modules aren't compatible across major releases.
You can continue to use local::lib for pure-Perl modules (so you don't have to install them for every version of Perl you have brewed up, but XS modules should be installed into the perlbrew-created directories. You don't need to clear PERL5LIB (and you shouldn't, as XS modules might have pure-Perl dependencies that are installed there), but you will need to clear PERL_MB_OPT and PERL_MM_OPT when installing XS modules to keep them from installing into the local::lib directory.
If you need to continue using local::lib for XS modules for the system Perl, then I suggest creating a second local::lib environment for that (perhaps in ~/perl5sys). It might be easier to use perlbrew to install a copy of the same version of Perl as the system Perl, and then use that instead of the system Perl.
You can clean out the XS modules in your existing local::lib by removing the /home/gabor/perl5/lib/perl5/x86_64-linux-gnu-thread-multi directory.
It is possible, but not comfortable. If this is a single-user setup, you might be better off not using local::lib and just letting perlbrew manage the modules for you. Also if it's a multi-user setup on a homogenous network, where everyone has the same machine and OS, then you can just set PERLBREW_ROOT to e.g. /net/share/perlbrew and then your installed perls (and their modules) can be shared. As noted in the other answers, this will be a problem if you try to mix machines (and possibly also problematic if you have different operating systems).
On a very diverse network, we prefer to keep everything separate. You can simply setup your local::lib to be a function of your current perl and your current platform, e.g.
distro=lsb_release -d|cut -f2|tr ' ' '-'
arch=`uname -m`
platform="$distro-$arch"
export PERLBREW_ROOT=/net/share/perlbrew/$platform
# You will have to first do 'perlbrew init' (just once for all users)
# In this case you don't need (and shouldn't have) a ~/.perlbrew
source $PERLBREW_ROOT/etc/bashrc
perl5base=/net/share/perl
# When $PERLBREW_PERL is not defined, local::lib puts modules in $perl5base/$platform
perl5=$perl5base/$platform/$PERLBREW_PERL
# We also found that we needed to clean PERL5LIB in between
export PERL5LIB=`echo -n $PERL5LIB|sed "s|${perl5base}[^:]*||g"`
export PATH=`echo -n $PATH|sed "s|${perl5base}[^:]*||g"`
# Setup local lib, relative to the perl being used
lib=$perl5/lib/perl5
mkdir -p $lib
eval $(perl -I"$lib" -Mlocal::lib="$perl5")
This is not our exact script, in particular you would need to check that these directories all exist first. You need to run perlbrew init once per platform and you need to bootstrap local::lib each time as well.
I don't recommend this approach, but provide as an example of one way to make this work, which it does for our mixed network (even on Mac OS). Leaving local::lib out and just using perlbrew (ignoring the system perl), would be a cleaner approach.
perlbrew is quite happy to use local::lib and has been for some time -- there are even special options for it -- after running perlbrew install ..., you can create a new local-lib directory with perlbrew lib create ...
for example:
perlbrew install -j 9 --as 34.0 5.34.0
chmod -R a-w $HOME/perl5/perlbrew/perls/34.0
perlbrew lib create 34.0#std
perlbrew switch 34.0#std
This installs a new 5.34.0 build, locks down its modules so you can't change them, then creates a local-lib directory. This install can be used with perlbrew use 34.0#std -- you can create a new set of module installations by perlbrew use 34.0; perlbrew lib create 34.0#other_install, to use side-by-side with the existing one without having to build a new perl again.
As miyagawa said, it might not be necessary to use local::lib if you use the Perls installed by Perlbrew exclusively.
But if you still want to be able to switch back and forth between your brewed Perls and the system Perl, there is a script called Perlswitcher for this. It's not pretty but it works. All you need to do is download the script, you could save it as ~/perl5/userperls/bashrc and source it.
It provides two commands. perlswitch allows you to switch to a Perl that was installed by Perlbrew or to the system Perl. perlinfo tells you which Perl is currently being used. You can then use cpanm, which will install packages to your local lib when using the system Perl or directly into the site Perl when using a custom Perl.
After switching to a custom Perl using perlswitch, perlbrew list will also know which Perl is being used:
$ perlswitch perl-5.18.4
Setting new perl /var/www/perl5/perlbrew/perls/perl-5.18.4/bin/perl...
Using user perl (site_perl) instead of local::lib
$ perlbrew list
perl-5.16.3
* perl-5.18.4
perl-5.20.2