I'm writing a CI script to build a Perl library located at GitHub.
Now I do it this way:
cpanp i Module::Install # and other configure dependencies
perl Makefile.PL
make dist
cpanp i Foo-Bar-6.66.tar.gz
I want to create a PAR packages for all the dependencies I build. That's why I use cpanp.
Unlike cpanm ., cpanp i . is buggy and uses the name of the directory to name the distribution, which is incorrect. That's why I use this make dist step.
I can use something else as long as my PARs are built, of course. There is https://metacpan.org/pod/PAR::Dist#blib_to_par function which can in principle be called during any other build process.
The problem with my current approach is that:
the list of configure dependencies is maintained manually. I'd like to use some tool which does it automatically
on Windows make may be called gmake or dmake depending on perl version. I want to use an existing autodetection logic instead of rolling my own
Finally, I'd like autodetection of Makefile.PL vs Build.PL so I can copy-paste the code for different distributions
My question is: Is there any way to build local sources of a distribution either by using cpanp (this way would solve my problem) or in any other way which works for different versions of Windows Perl and builds PAR distributions of dependencies?
I managed to find a solution myself.
CPANPLUS comes with another tool besides cpanp named cpan2dist. cpan2dist --archive Foo-Bar-6.66.tar.gz works as intended - it builds distributions of dependencies and the current tarball, and autodetects the necessary build tool the way cpanp does.
To determine the real make you can use CPANPLUS::Config->new->get_prog('make'). It again uses the same autodetection logic as CPANPLUS so it's pretty robust.
Related
I love the idea of local::lib. At least, I find it preferable to the default system-wide installs that cpan/perl defualts to in most every way. However, I'm having difficulties with modules that install XS and distribution upgrades to newer Perls. The instructions for local::lib call for adding this line to your .bashrc (or like file)
eval "$(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)"
That executes arbitrary shell code, which is fine. You can see the code it executes by running
perl -I$HOME/perl5/lib/perl5 -Mlocal::lib
In my case it returns this code,
PATH="/home/ecarroll/perl5/bin${PATH:+:${PATH}}"; export PATH;
PERL5LIB="/home/ecarroll/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}"; export PERL5LIB;
PERL_LOCAL_LIB_ROOT="/home/ecarroll/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"; export PERL_LOCAL_LIB_ROOT;
PERL_MB_OPT="--install_base \"/home/ecarroll/perl5\""; export PERL_MB_OPT;
PERL_MM_OPT="INSTALL_BASE=/home/ecarroll/perl5"; export PERL_MM_OPT;
In the above, we're setting the default options for the two most widely used module build systems,
MB is for Module::Build
MM is for ExtUtils::MakeMaker
We're telling those system to install to /home/ecarroll/perl5. The rest of the command sets
PERL5LIB which is used in setting #INC. The #INC variable in Perl is used to resolve calls with use.
and, PATH which prepends the bin/ directory that local::lib installs to.
So basically, install perl modules in your home directory, and look first for files installed in your home directory.
My question is what happens when you install something that has XS? In my case, I have an XS file,
/home/ecarroll/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Class/Load/XS/XS.so
This was installed, with local::lib using an earlier version of Perl. I've since ran a Debian dist-upgrade and got a newer version of Perl (v5.36). Now a lot of utilities produce an obtuse error about this, including ones installed through the distro itself like in my case Dist::Zilla dzil which I installed with Debian's libdist-zilla-perl package.
$ dzil
XS.c: loadable library and perl binaries are mismatched (got first handshake key 0xeb00080, needed 0xeb80080)
Which is all a result of this module's XS,
perl -MClass::Load::XS -e1
XS.c: loadable library and perl binaries are mismatched (got first handshake key 0xeb00080, needed 0xeb80080)
This seems like very routine use of local::lib:
Am I using local::lib in the way intended? What else should I be doing?
How is local::lib supposed to handle XS?
Is there a way to make local::lib support multiple versions of Perl? My distro reads and writes the XS it manages to /usr/share/perl/$PERL_VERSION. This means a dist-upgrade leaves all the XS from the old version behind. It would be nice to have local::lib install not to /home/ecarroll/perl5/lib/perl5/ but instead to /home/ecarroll/perl5/lib/perl5.30/? Note the added minor version. Then at least a dist-upgrade wouldn't break the distribution.
Perl doesn't maintain ABI compatibility between versions. (5.34.x -> 5.36.y, for example.) After installing a new version of Perl, you will need to re-install modules that install themselves in arch dirs (XS modules and a few others). Might be easiest and safest to remove (rename) the local lib directory (/home/ecarroll/perl5) and start from scratch.
That's why I avoid using anything but distro-provided modules with distro-provided perl. The package manager ensures you always have compatible builds of the modules.
If that's not good enough, I recommend installing your own builds of perl instead of using local::lib. Tools like perlbrew makes this easy. You'll still have to re-install all the modules when you install a new perl, but you'll get the cleaner "Can't locate" error message if you forget one.
I rarely use local::lib, but I'm also a minimalist when it comes to administration. The more layers you have, the worse things get. As you see, you're own answer (now deleted) proposes even more complexity to deal with complexity that isn't likely appropriate for your situation.
But, local::lib has a way to set the locations of everything. You are trying to use the same location for everything instead of changing the locations based on what you are trying to do. However, even though local::lib lets you do that, you're likely to forget to switch things up.
My solution is simpler. I merely install a perl for whatever I'm doing (or reuse a suitable perl). I don't change the default locations at all. I never have to manage that. I simply call the perl I want:
$ ls -d /usr/local/perls/*
/usr/local/perls/perl-5.10.1 /usr/local/perls/perl-5.24.3 /usr/local/perls/perl-5.32.0
/usr/local/perls/perl-5.12.5 /usr/local/perls/perl-5.26.1 /usr/local/perls/perl-5.32.1
/usr/local/perls/perl-5.14.4 /usr/local/perls/perl-5.26.2 /usr/local/perls/perl-5.34.0
/usr/local/perls/perl-5.16.3 /usr/local/perls/perl-5.28.0 /usr/local/perls/perl-5.34.1
/usr/local/perls/perl-5.18.4 /usr/local/perls/perl-5.30.0 /usr/local/perls/perl-5.35.11
/usr/local/perls/perl-5.20.3 /usr/local/perls/perl-5.30.1 /usr/local/perls/perl-5.36.0
/usr/local/perls/perl-5.22.4 /usr/local/perls/perl-5.30.2 /usr/local/perls/perl-5.8.9
I have various symlinks and stable path adjustments to I use the tool name and the version tag I gave it:
% cpan5.36.0 ...
% cpan5.32.1 ...
People use all sorts of tools such as perlbrew, plenv, and now some other tool that you think might solve it. It's a big red flag if you have to constantly create new tools to deal with the tools that you are using. I'd rather spend my time doing something else.
I stand by my other answer—this is too much complexity and you should reduce that. But, lets work with that constraint. You want to use local::lib.
The problem is that you are trying to force the same set of modules on any particular perl, no matter how it was compiled. As ikegami explained, you can't assume that two different perls, or the same library compiled with different perls, are compatible. The version of the perls does not matter. Two different perl5.36, or any version, can be incompatible with each other.
But local::lib gives you what you need to set the appropriate values. The module itself can supply some starter values:
$ perl -I$HOME/perl5/lib/perl5 -Mlocal::lib
Attempting to create directory /Users/brian/perl5
PATH="/Users/brian/perl5/bin${PATH:+:${PATH}}"; export PATH;
PERL5LIB="/Users/brian/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}"; export PERL5LIB;
PERL_LOCAL_LIB_ROOT="/Users/brian/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"; export PERL_LOCAL_LIB_ROOT;
PERL_MB_OPT="--install_base \"/Users/brian/perl5\""; export PERL_MB_OPT;
PERL_MM_OPT="INSTALL_BASE=/Users/brian/perl5"; export PERL_MM_OPT;
But, you don't need to use those values everywhere. You can have as many sets of those variables as you like. You can have a different set for every project, every perl, or however you want to organize it. The only trick is how you select the right set.
The synopsis of local::lib indicates that it expects you to set the appropriate value for your project:
use FindBin;
use local::lib "$FindBin::Bin/../support"; # app-local support library
Knowing that, you can have your per-version libraries by adding $^V in the path you give to local::lib (still with the binary incompatibility problems):
use FindBin;
use local::lib "/my/modules/$^V";
Because i have a long series of comments with #ikegami, I cleaning up the question, in a hope it will be more understandable. Unfortunately, english isn't my "main" language. :(
Let say, having an environment where:
no development tools are installed (no make, nor gcc or like)
perl is installed with its core packages, nothing more
no outgoing network access is allowed - e.g. the user couldn't use curl nor cpan to download/install perl dependencies
the user even doesn't have admin (root) rights
but want install and evaluate some perl based web-app, let call it as MyApp
The MyApp
doesn't uses any XS-based module. (at least, I hope - in the development me using plenv and cpanm, so never checked the installed dependencies in depth)
it is an pure PSGI app, the simple plackup app.psgi works OK
the app uses some data-files which should be included in the "deployment".
The main question is: how to prepare the MyApp, and the all used CPAN-modules, to be easily installed in such restricted environment?
The goal is:
i don't need save my efforts and my time
but i want save the user's time and want minimize the needed actions on his side, so the installation (deployment) should be simple-as-possible.
E.g. how to get an running web-app to the user's machine with minimum possible (his) steps.
- the simplest thing is could be something as:
- copy one file (zip, or tarbal)
- unpack it
- from the terminal execute some run.pl in the unpacked directory.
To get the above simple installation, my idea was the following:
1.) Create an tarball, and after the unpacking will contain 3 folders and 1 perl-script, let say:
myapp_repo/
myapp_repo/distlib #will contain all MyApp's perl modules also ALL used CPAN modules and their dependecies
myapp_repo/datafiles #will contain app-specific data files and such
myapp_repo/install.pl
myall_repo/lib #will contain modules directly used by the `install.pl`
2.) I will develop an install.pl script, and it will be used as the installer-tool, like
perl install.pl new /path/to/app_root
and it will (should):
create the all needed directories under the /path/to/app_root (especially the lib where the will install the perl modules)
will call "local" cpanm internally (from the myapp_repo/lib) to install the app's perl modules and their CPAN dependencies using only distribution files from the distlib.
will generate and install the needed runtime script and the app.psgi into the /path/to/app_root/bin
will install the needed data-files for the app.
3.) So, after this the user should be able to simply run:
/path/to/app_root/bin/plackup /path/to/app_root/bin/app.psgi
In short, the user should use:
the system-wide perl and the system-wide perl-core modules
and any other
runtime perl-scripts (like plackup)
and the required CPAN-modules
should be installed to an self-contained directory tree using only files (no net-access).
E.g. the install.pl should somewhat call internally the cpanm to achieve (as equivalent) for the following cpanm command
cpanm --mirror file://path/to/myapp_repo/distlib --mirror-only My::App
which, should install My::App and all dependencies without network access using only the files from the myapp_repo/distlib
Some questions:
Is possible to use cpanm (called as an locally installed module) without the make?
For creating the myapp_repo/distlib, me thinking about using Pinto. Is it the right tool for achieve the above?
forgot me something? or with other words:
Is the above an viable (read: working) way?
are are any other tools, which i could/should to use for simplifying the creation of such distribution tarball?
#ikegami suggesting some method:
- "install everything" in one fresh-directory on my machine
- transfer this self-contained directory to the target machine
It sound very good, because this directory could contain all the needed app-specific data-files too, unfortunately, I don't understand the details how his solution should be done.
The FatPacked solution looks interesting too - need learn about it.
Don't write your own make or installer. Just copy it make from a different machine (which is basically what apt/yum/etc do anyway, and which you'd have to do even if you wrote your own). You'd be able to use cpan in 5 minutes!
Also, that should allow you to install gcc if you need it (e.g. to install an XS module), although it doesn't sound like you do. If you do install gcc, I'd install my own perl to avoid having to deal with PERL5LIB.
Tools such as minicpan will allow you to install any module from CPAN without internet access. Of course, you can keep using the command you are already using it if mirrors the packages you need.
The above explains how to simply and quickly setup a machine so it can use cpan and thus install any module easily.
If you just want to install a specific module and its dependencies, you can completely avoid using cpan on the target machine. First, you need a fresh install of Perl (preferable of the same version as the one on the target system). Then, simply install the module to a fresh dir on your machine, and transfer that dir to the target machine. That's it; nothing else needs to be done. This even works for XS modules if the two machine are similar enough.
This is what ppm (ActiveState's Perl package manager) does.
Unfortunately, while this solution is almost as simple as the one above, it's not nearly as flexible, it doesn't run the test suite of the modules being installed, etc. It does have the advantage of not requiring the transfer of any binary (if you're not installing any XS modules).
I am relatively new to Perl and I need some help with redistributing the non-core modules. Here's the whole story.
There are two non-core modules that were used in the Perl script: XML::Simple and SOAP::Lite. The version that I'm using (currently on Windows) is Strawberry Perl, so these two modules are already included. However, we don't know if the end users (Unix/Linux system) have these two modules as they might only have the standard version and so have only the core modules. My goal is to make the end users do as little configurations/installs as possible.
First, I tried to see if there's any core modules that's similar to XML::Simple and SOAP::Lite. Unfortunately, I didn't find any (Please correct me if I'm wrong).
So I guess now the only option is to redistribute the two modules. I checked and that these two modules allow redistribution. My problem right now is how to do it. I tried googling with keywords "perl redistribute" but didn't find anything useful. My guess is that we use the exporter tool to achieve this. but these two modules are rather complicated modules and they have several nested folders/pm files (and a whole bunch of other files like MAKE, pod, ini files) so I'm not sure what I should do. The examples I found using exporter are rather simple: They only have 1 pm file and 1 pl file and they are placed into one folder.
Also, I'm open to any other better ways to deal with the problem. The goal is just to make sure all end users can use my script with the least configuration/install efforts as we don't want them to run into a whole bunch of compatibility issues.
Any help would be appreciated. Thanks! =D
I want to elaborate a little bit on #ikegami said.
SOAP::Lite has a large number of CPAN dependencies, so the people installing your module are going to need CPAN access to get it to build, whether you provide it for them, or list it as a dependency. Otherwise, you'll need to provide your entire dependency tree, at which point, you end up using perlbrew, maybe carton, possibly local::lib, and then you might decide you need the next higher level and produce RPMs and DEBs.
Probably better to just provide your script, packaged as a CPAN module, list your dependencies within, and let the chips fall where they may.
Just state a dependency on the modules in your Makefile.PL or Build.PL, then give them the following installation instruction:
cpanm script.tar.gz
One of the best things about Perl is The CPAN, the Comprehensive Perl Archive Network. It's a mirroring service that since about the time Perl 5 originally came out has allowed people to share useful add-on modules, like XML::Simple or SOAP::Lite through a standard, common tool, the cpan client that comes with Perl. Almost all Perl distributions (such as Strawberry Perl and most Perl distributions that come with linux) have a CPAN client configured and included with them. This client lets people download and install modules from CPAN simply by knowing the name of the module.
Almost all module distributions on CPAN follow the exact same layout. They usually have a Makefile.PL file (if it uses ExtUtils::MakeMaker to generate the install script), Build.PL file (if it uses Module::Build to generate the install script), or both. These Perl scripts, once run, create a 'Makefile' or a 'Build' file that can let you install the module and verify that all prerequisites are met.
If you've never made a Perl distribution before, you can download any distribution you want from CPAN and take a look at how things are laid out. The folders and file locations are pretty intuitive once you've seen one. They are usually laid out with the install script and supporting files (like a readme) in the root directory, with the custom modules (modules you make) in the lib directory, and with unit tests in the t directory.
I'd recommend looking at the Build.PL based ones if you're a novice; these are pure Perl based install scripts. If you decide to make a Module.PL based distribution, it's really easy to specify that your module distribution needs XML::Simple and SOAP::Lite. First, create a basic Module::Build based install script. This looks something like:
use Module::Build;
my $build = Module::Build->new(
module_name => 'Foo::Bar',
license => 'perl',
requires => {
'perl' => '5.6.1',
'Some::Module' => '1.23',
'Other::Module' => '>= 1.2, != 1.5, < 2.0',
},
);
$build->create_build_script;
(This is taken right from the Module::Build::Authoring docs).
Then, specify the libraries you need and minimum versions of them. Zero (0) is an acceptable version if you don't care, but that means "anything" is good. I'd recommend specifying at least the version of the libraries installed on the machines you're testing with.
(Neat short cut: you can find out the version of any library that has a $VERSION package variable defined by doing:
perl -MSome::Lib -E "say Some::Lib->VERSION()"
.)
To install the module, the steps look something like this:
cd folder\where\my\lib\is
perl Build.PL
Build
Build test
Build install
This will create the install tool, prepare the folder for testing (usually just copying stuff to a build library area for simple modules), run all .t scripts in the t folder (the "tests", which usually use Test::More for unit testing of the module prior to install), and then finally, install your module to your PC's Perl site libraries.
The Build script, as part of the 'setting things up' phase, will look at your prerequisites, and warn you if you don't have them yet.
Then, as pointed out in ikegami's answer, if you use the cpanm client to install your library, the cpan client will automatically go out, download, test, and install your dependencies for you! Alternatively, Build.PL based installers also have the 'installdeps' option, which will do the same thing. Then any and all dependencies (and potentially recursive dependencies) are automatically downloaded, tested, and installed, even if they change in the future.
I am a very new to porting.
I was trying to port perl to a netbsd system. Since its a custom made build, we wont be able to run configure or make on the target netbsd system. So we are trying to cross-compile it in a host pc and copy the binary over target machine. And in order to do so, we have to make a makefile from scratch, since the format for the makefile in our build is different.
I have some basic doubts regarding this,
Firstly, In order to create a perl makefile for my custom build, what are the basic things will come. Such as ccflags, library paths etc.,?
There are some files like DynaLoader, uudmap.h, myConfig, Config.pm which gets generated while "make". How can i generate them using custum makefile.
How to set various library paths and what are they ?
The #INC, shows the perl search paths, how can i create it ?
Where exactly Perl modules get installed and when it happens?
A perl build normally involves building a stripped down version of perl named miniperl, which is then used extensively in the remainder of the process of building perl and the bundled modules.
There are two basic approaches to cross-compiling: to build miniperl for the target machine and build the modules, etc., there, or to build miniperl for the host and use it to build perl and modules for the target.
The WinCE port uses the latter approach; the rudimentary (last I knew, anyway) support for a -Dusecrosscompile switch to Configure uses the former.
I recommend you ask for advice and help on the perl5-porters mailing list: http://lists.perl.org/list/perl5-porters.html
And be prepared for hard work.
NetBSD's pkgsrc system has perl in it already and has the ability to generate binary packages that you can then install on a target machine.
I'm trying to set up a large-ish project, written in Perl. The IBM MakeMaker tutorial has been very helpful so far, but I don't understand how to link all the modules into the main program. In my project root, I have MANIFEST, Makefile.PL, README, a bin directory, and a lib directory. In my bin directory, I have my main script (Main.pl). In the lib directory, I have each of my modules, divided up into their own respective directories (i.e. Utils::Util1 and Utils::Utils2 in the utils directory, etc). In each module directory, there is also a t directory, containing tests
My MANIFEST file has the following:
bin/Main.pl
lib/Utils/Util1.pm
lib/Utils/Util2.pm
lib/Utils/t/Utils1.t
lib/Utils/t/Utils2.t
Makefile.PL
MANIFEST
README
Makefile.PL is the following:
use ExtUtils::MakeMaker;
WriteMakefile(
'NAME'=>'Foo',
'VERSION_FROM'=>'bin/Main.pl',
'PREREQ_PM'=>{
"XML::Simple"=> 2.18}, #The libraries that we need and their
#minimum version numbers
'EXE_FILES' =>[("bin/Main.pl")]
);
After I make and run, the program crashes, complaining that it cannot find Utils::Util1, and when I run 'make test, it says no tests defined. Can anyone make any suggestions? I have never done a large scale project like this in perl, and I will need to add many more modules
If you are just starting to create Perl modules (which is also Perl's equivalent of a project), don't use Makemaker. Module::Build is the way to go, and it's now part of the standard library. Makemaker is for us old salts who haven't converted to Module::Build yet. :) I'll strike that now that Module::Build is unmaintained and out of favor; I still use MakeMaker.
You should never start off a Perl project by trying to create the structure yourself. It's too much work and you'll always forget something.
There's h2xs, a program that comes with perl and was supposed to be a tool to convert .h files into Perl's glue language XS. It works fine, but its advantage is that it comes with perl:
% h2xs -AXn Module::Name
Something like Module::Starter is a bit more sophisticated, although you have to get it from CPAN. It's the tool we use in Intermediate Perl because it's simple. It fills in some templates with your information:
% module-starter --author=... --email=... --module=...
If you are doing to do this quite a bit, you might then convert that to Distribution::Cooker so you can customize your files and contents. It's a dinky utility I wrote for myself so I could use my own templates.
% dist_cooker Module::Name
If you're really hard core, you might want Dist::Zilla, but that's more for people who already know what they are doing.
Might I also suggest module-starter? It'll automatically create a skeleton project which "Just Works". I learned what little I know about Perl modules organization by reading the generated skeleton files. It's all well-documented, and quite easy to use as a base for growing a larger project in. You can check out the getting-started docs to see what it gives you.
Running module-starter will give you a Perl distribution, consisting of a number of modules (use the command line option --module, such as:
module-starter --distro=Project --module=Project::Module::A,Project::Module::B [...]
to create multiple modules in a single distribution). It's then up to you whether you'd prefer to organize your project as a single distribution consisting of a number of modules working together, or as a number of distributions which can be released separately but which depend on each other (as configured in your Build or Makefile.PL file) to provide a complete system.
Try this structure:
bin/Main.pl
lib/Utils/Util1.pm
lib/Utils/Util2.pm
Makefile.PL
MANIFEST
README
t/Utils1.t
t/Utils2.t
As ysth said, make does not install your modules, it just builds them in a blib directory. (In your case it just copies them there, but if you had XS code, it would be compiled with a C compiler.) Use make install to install your modules for regular scripts to use.
If you want to run your script between make and make install, you can do:
perl -Mblib bin/Main.pl
The -Mblib instructs perl to temporarily add the appropriate directories to the search path, so you can try out an uninstalled module. (make test does that automatically.)
By default, tests are looked for in a top-level t directory (or a test.pl file, but that has some limitations, so should be avoided).
You say "After I make and run"...make puts things into a blib directory structure ready to be installed, but doesn't do anything special to make running a script access them. (make test is special; it does add appropriate paths from blib to perl's #INC to be able to run the tests.) You will need to do a "make install" to install the modules where your script will find them (or use a tool like PAR to package them together with your script).