I'm using DBI and DBD::SQLite, and now I'd like to use the R*Tree feature of SQLite. Since this feature is not compiled by DBD::SQLite by default, I have to add a -DSQLITE_ENABLE_RTREE=1 to the #CC_DEFINE variable in DBD::SQLite's Makefile.PL. If I do a 'perl Makefile.PL && make && make install', everything works fine locally on my machine, but this ultimately needs to be deployable/distributable to end users.
What should I do in a case like this? Should I copy the source, grep the source, and create a DBD::SQLite::WithRTree? Create a private version of DBD::SQLite 1.31.1 (Where 1.31 is the current version of DBD::SQLite)? Perhaps a better way altogether?
All other distributions in the project are deployed/distributed via a non-public CPAN::Mini mirror + CPAN::Mini::Inject.
I have to add a '-DSQLITE_ENABLE_RTREE=1' to the #CC_DEFINE variable in DBD::SQLite's Makefile.PL
You're doing this wrong, perl Makefile.PL DEFINE='-DSQLITE_ENABLE_RTREE=1' works. This is documented in ExtUtils::MakeMaker. Now that you know that, a simple solution involving Distroprefs will likely fall in place.
For similar problems, I have installed the modified distribution in a separate directory (without changing any module names), and using use lib qw(the/special/directory) or setting $PERL5LIB for scripts that need to use the enhanced module.
Tweaking the name of the module would also do the job, but that sound like a lot more work to make and test.
You can do this:
cpan
o conf makepl_arg "DEFINE='-DSQLITE_ENABLE_RTREE=1'"
o conf commit
CPAN will then permanently add that DEFINE to the front of all your Makefile.PL calls.
So, it should just be
cpan DBD::SQLite
And your makefile options should get stuffed onto your compile lines
Related
I love the idea of local::lib. At least, I find it preferable to the default system-wide installs that cpan/perl defualts to in most every way. However, I'm having difficulties with modules that install XS and distribution upgrades to newer Perls. The instructions for local::lib call for adding this line to your .bashrc (or like file)
eval "$(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)"
That executes arbitrary shell code, which is fine. You can see the code it executes by running
perl -I$HOME/perl5/lib/perl5 -Mlocal::lib
In my case it returns this code,
PATH="/home/ecarroll/perl5/bin${PATH:+:${PATH}}"; export PATH;
PERL5LIB="/home/ecarroll/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}"; export PERL5LIB;
PERL_LOCAL_LIB_ROOT="/home/ecarroll/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"; export PERL_LOCAL_LIB_ROOT;
PERL_MB_OPT="--install_base \"/home/ecarroll/perl5\""; export PERL_MB_OPT;
PERL_MM_OPT="INSTALL_BASE=/home/ecarroll/perl5"; export PERL_MM_OPT;
In the above, we're setting the default options for the two most widely used module build systems,
MB is for Module::Build
MM is for ExtUtils::MakeMaker
We're telling those system to install to /home/ecarroll/perl5. The rest of the command sets
PERL5LIB which is used in setting #INC. The #INC variable in Perl is used to resolve calls with use.
and, PATH which prepends the bin/ directory that local::lib installs to.
So basically, install perl modules in your home directory, and look first for files installed in your home directory.
My question is what happens when you install something that has XS? In my case, I have an XS file,
/home/ecarroll/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/auto/Class/Load/XS/XS.so
This was installed, with local::lib using an earlier version of Perl. I've since ran a Debian dist-upgrade and got a newer version of Perl (v5.36). Now a lot of utilities produce an obtuse error about this, including ones installed through the distro itself like in my case Dist::Zilla dzil which I installed with Debian's libdist-zilla-perl package.
$ dzil
XS.c: loadable library and perl binaries are mismatched (got first handshake key 0xeb00080, needed 0xeb80080)
Which is all a result of this module's XS,
perl -MClass::Load::XS -e1
XS.c: loadable library and perl binaries are mismatched (got first handshake key 0xeb00080, needed 0xeb80080)
This seems like very routine use of local::lib:
Am I using local::lib in the way intended? What else should I be doing?
How is local::lib supposed to handle XS?
Is there a way to make local::lib support multiple versions of Perl? My distro reads and writes the XS it manages to /usr/share/perl/$PERL_VERSION. This means a dist-upgrade leaves all the XS from the old version behind. It would be nice to have local::lib install not to /home/ecarroll/perl5/lib/perl5/ but instead to /home/ecarroll/perl5/lib/perl5.30/? Note the added minor version. Then at least a dist-upgrade wouldn't break the distribution.
Perl doesn't maintain ABI compatibility between versions. (5.34.x -> 5.36.y, for example.) After installing a new version of Perl, you will need to re-install modules that install themselves in arch dirs (XS modules and a few others). Might be easiest and safest to remove (rename) the local lib directory (/home/ecarroll/perl5) and start from scratch.
That's why I avoid using anything but distro-provided modules with distro-provided perl. The package manager ensures you always have compatible builds of the modules.
If that's not good enough, I recommend installing your own builds of perl instead of using local::lib. Tools like perlbrew makes this easy. You'll still have to re-install all the modules when you install a new perl, but you'll get the cleaner "Can't locate" error message if you forget one.
I rarely use local::lib, but I'm also a minimalist when it comes to administration. The more layers you have, the worse things get. As you see, you're own answer (now deleted) proposes even more complexity to deal with complexity that isn't likely appropriate for your situation.
But, local::lib has a way to set the locations of everything. You are trying to use the same location for everything instead of changing the locations based on what you are trying to do. However, even though local::lib lets you do that, you're likely to forget to switch things up.
My solution is simpler. I merely install a perl for whatever I'm doing (or reuse a suitable perl). I don't change the default locations at all. I never have to manage that. I simply call the perl I want:
$ ls -d /usr/local/perls/*
/usr/local/perls/perl-5.10.1 /usr/local/perls/perl-5.24.3 /usr/local/perls/perl-5.32.0
/usr/local/perls/perl-5.12.5 /usr/local/perls/perl-5.26.1 /usr/local/perls/perl-5.32.1
/usr/local/perls/perl-5.14.4 /usr/local/perls/perl-5.26.2 /usr/local/perls/perl-5.34.0
/usr/local/perls/perl-5.16.3 /usr/local/perls/perl-5.28.0 /usr/local/perls/perl-5.34.1
/usr/local/perls/perl-5.18.4 /usr/local/perls/perl-5.30.0 /usr/local/perls/perl-5.35.11
/usr/local/perls/perl-5.20.3 /usr/local/perls/perl-5.30.1 /usr/local/perls/perl-5.36.0
/usr/local/perls/perl-5.22.4 /usr/local/perls/perl-5.30.2 /usr/local/perls/perl-5.8.9
I have various symlinks and stable path adjustments to I use the tool name and the version tag I gave it:
% cpan5.36.0 ...
% cpan5.32.1 ...
People use all sorts of tools such as perlbrew, plenv, and now some other tool that you think might solve it. It's a big red flag if you have to constantly create new tools to deal with the tools that you are using. I'd rather spend my time doing something else.
I stand by my other answer—this is too much complexity and you should reduce that. But, lets work with that constraint. You want to use local::lib.
The problem is that you are trying to force the same set of modules on any particular perl, no matter how it was compiled. As ikegami explained, you can't assume that two different perls, or the same library compiled with different perls, are compatible. The version of the perls does not matter. Two different perl5.36, or any version, can be incompatible with each other.
But local::lib gives you what you need to set the appropriate values. The module itself can supply some starter values:
$ perl -I$HOME/perl5/lib/perl5 -Mlocal::lib
Attempting to create directory /Users/brian/perl5
PATH="/Users/brian/perl5/bin${PATH:+:${PATH}}"; export PATH;
PERL5LIB="/Users/brian/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}"; export PERL5LIB;
PERL_LOCAL_LIB_ROOT="/Users/brian/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"; export PERL_LOCAL_LIB_ROOT;
PERL_MB_OPT="--install_base \"/Users/brian/perl5\""; export PERL_MB_OPT;
PERL_MM_OPT="INSTALL_BASE=/Users/brian/perl5"; export PERL_MM_OPT;
But, you don't need to use those values everywhere. You can have as many sets of those variables as you like. You can have a different set for every project, every perl, or however you want to organize it. The only trick is how you select the right set.
The synopsis of local::lib indicates that it expects you to set the appropriate value for your project:
use FindBin;
use local::lib "$FindBin::Bin/../support"; # app-local support library
Knowing that, you can have your per-version libraries by adding $^V in the path you give to local::lib (still with the binary incompatibility problems):
use FindBin;
use local::lib "/my/modules/$^V";
I'm writing a CI script to build a Perl library located at GitHub.
Now I do it this way:
cpanp i Module::Install # and other configure dependencies
perl Makefile.PL
make dist
cpanp i Foo-Bar-6.66.tar.gz
I want to create a PAR packages for all the dependencies I build. That's why I use cpanp.
Unlike cpanm ., cpanp i . is buggy and uses the name of the directory to name the distribution, which is incorrect. That's why I use this make dist step.
I can use something else as long as my PARs are built, of course. There is https://metacpan.org/pod/PAR::Dist#blib_to_par function which can in principle be called during any other build process.
The problem with my current approach is that:
the list of configure dependencies is maintained manually. I'd like to use some tool which does it automatically
on Windows make may be called gmake or dmake depending on perl version. I want to use an existing autodetection logic instead of rolling my own
Finally, I'd like autodetection of Makefile.PL vs Build.PL so I can copy-paste the code for different distributions
My question is: Is there any way to build local sources of a distribution either by using cpanp (this way would solve my problem) or in any other way which works for different versions of Windows Perl and builds PAR distributions of dependencies?
I managed to find a solution myself.
CPANPLUS comes with another tool besides cpanp named cpan2dist. cpan2dist --archive Foo-Bar-6.66.tar.gz works as intended - it builds distributions of dependencies and the current tarball, and autodetects the necessary build tool the way cpanp does.
To determine the real make you can use CPANPLUS::Config->new->get_prog('make'). It again uses the same autodetection logic as CPANPLUS so it's pretty robust.
I have a new FreeBSD 9.0 installation that is having trouble installing modules that require C via CPAN. It seems to be down to cc being called without -I/usr/local/include. I've tried overriding that by starting CPAN with INC="-I/usr/local/include" - while that fixes the problem for some modules, it breaks for others as it seems to be overriding any defaults set in makefiles.
I've got another box with pretty much the same setup on it (slightly older installation, but still FreeBSD 9.0), perl -V on the old box shows that ccflags contains -I/usr/local/include, whereas it doesn't on the new box.
I'm guessing this is why CPAN is calling cc without it.
Question is, how can I configure it?
The settings used by CPAN modules come from the Config module which is part of core Perl. It records the compilation options and settings used when Perl was compiled.
For whatever reason, the person compiling Perl on your new FreeBSD 9.0 installation decided not to include -I/usr/local/include.
Fighting the settings in the Config module is hard work, as you've already discovered. On the whole, it is simplest not to fight it. If you want Perl to use /usr/local/include, then compile Perl so as to do so. You'll probably want to specify -l/usr/local/lib too.
You should check whether the versions of Perl are the same or not on your two similar systems.
If it works, your best bet is probably to add it to the ccflags entry in the file returned by
perl -E'require "Config_heavy.pl"; say $INC{"Config_heavy.pl"};'
Otherwise, there is surely a means of specifying this to ExtUtis::MakeMaker and to Module::Build installers. If so, you can set this cpan using o conf makepl_arg and o conf mbuildpl_arg. (Don't forget to commit!)
I'm trying to set up a large-ish project, written in Perl. The IBM MakeMaker tutorial has been very helpful so far, but I don't understand how to link all the modules into the main program. In my project root, I have MANIFEST, Makefile.PL, README, a bin directory, and a lib directory. In my bin directory, I have my main script (Main.pl). In the lib directory, I have each of my modules, divided up into their own respective directories (i.e. Utils::Util1 and Utils::Utils2 in the utils directory, etc). In each module directory, there is also a t directory, containing tests
My MANIFEST file has the following:
bin/Main.pl
lib/Utils/Util1.pm
lib/Utils/Util2.pm
lib/Utils/t/Utils1.t
lib/Utils/t/Utils2.t
Makefile.PL
MANIFEST
README
Makefile.PL is the following:
use ExtUtils::MakeMaker;
WriteMakefile(
'NAME'=>'Foo',
'VERSION_FROM'=>'bin/Main.pl',
'PREREQ_PM'=>{
"XML::Simple"=> 2.18}, #The libraries that we need and their
#minimum version numbers
'EXE_FILES' =>[("bin/Main.pl")]
);
After I make and run, the program crashes, complaining that it cannot find Utils::Util1, and when I run 'make test, it says no tests defined. Can anyone make any suggestions? I have never done a large scale project like this in perl, and I will need to add many more modules
If you are just starting to create Perl modules (which is also Perl's equivalent of a project), don't use Makemaker. Module::Build is the way to go, and it's now part of the standard library. Makemaker is for us old salts who haven't converted to Module::Build yet. :) I'll strike that now that Module::Build is unmaintained and out of favor; I still use MakeMaker.
You should never start off a Perl project by trying to create the structure yourself. It's too much work and you'll always forget something.
There's h2xs, a program that comes with perl and was supposed to be a tool to convert .h files into Perl's glue language XS. It works fine, but its advantage is that it comes with perl:
% h2xs -AXn Module::Name
Something like Module::Starter is a bit more sophisticated, although you have to get it from CPAN. It's the tool we use in Intermediate Perl because it's simple. It fills in some templates with your information:
% module-starter --author=... --email=... --module=...
If you are doing to do this quite a bit, you might then convert that to Distribution::Cooker so you can customize your files and contents. It's a dinky utility I wrote for myself so I could use my own templates.
% dist_cooker Module::Name
If you're really hard core, you might want Dist::Zilla, but that's more for people who already know what they are doing.
Might I also suggest module-starter? It'll automatically create a skeleton project which "Just Works". I learned what little I know about Perl modules organization by reading the generated skeleton files. It's all well-documented, and quite easy to use as a base for growing a larger project in. You can check out the getting-started docs to see what it gives you.
Running module-starter will give you a Perl distribution, consisting of a number of modules (use the command line option --module, such as:
module-starter --distro=Project --module=Project::Module::A,Project::Module::B [...]
to create multiple modules in a single distribution). It's then up to you whether you'd prefer to organize your project as a single distribution consisting of a number of modules working together, or as a number of distributions which can be released separately but which depend on each other (as configured in your Build or Makefile.PL file) to provide a complete system.
Try this structure:
bin/Main.pl
lib/Utils/Util1.pm
lib/Utils/Util2.pm
Makefile.PL
MANIFEST
README
t/Utils1.t
t/Utils2.t
As ysth said, make does not install your modules, it just builds them in a blib directory. (In your case it just copies them there, but if you had XS code, it would be compiled with a C compiler.) Use make install to install your modules for regular scripts to use.
If you want to run your script between make and make install, you can do:
perl -Mblib bin/Main.pl
The -Mblib instructs perl to temporarily add the appropriate directories to the search path, so you can try out an uninstalled module. (make test does that automatically.)
By default, tests are looked for in a top-level t directory (or a test.pl file, but that has some limitations, so should be avoided).
You say "After I make and run"...make puts things into a blib directory structure ready to be installed, but doesn't do anything special to make running a script access them. (make test is special; it does add appropriate paths from blib to perl's #INC to be able to run the tests.) You will need to do a "make install" to install the modules where your script will find them (or use a tool like PAR to package them together with your script).
I want another developer to run a Perl script I have written. The script uses many CPAN modules that have to be installed before the script can be run. Is it possible to make the script (or the perl binary) to dump a list of all the missing modules? Perl prints out the missing modules’ names when I attempt to run the script, but this is verbose and does not list all the missing modules at once. I’d like to do something like:
$ cpan -i `said-script --list-deps`
Or even:
$ list-deps said-script > required-modules # on my machine
$ cpan -i `cat required-modules` # on his machine
Is there a simple way to do it? This is not a show stopper, but I would like to make the other developer’s life easier. (The required modules are sprinkled across several files, so that it’s not easy for me to make the list by hand without missing anything. I know about PAR, but it seems a bit too complicated for what I want.)
Update: Thanks, Manni, that will do. I did not know about %INC, I only knew about #INC. I settled with something like this:
print join("\n", map { s|/|::|g; s|\.pm$||; $_ } keys %INC);
Which prints out:
Moose::Meta::TypeConstraint::Registry
Moose::Meta::Role::Application::ToClass
Class::C3
List::Util
Imager::Color
…
Looks like this will work.
Check out Module::ScanDeps and the "scandeps.pl" utility that comes with it. It can do a static (and recursive) analysis of your code for dependencies as well as the %INC dump either after compiling or running the program.
Please note that the static source scanning always errs on the side of including too many dependencies. (It is the dependency scanner used by PAR and aims at being easiest on the end-user.)
Finally, you could choose to distribute your script as a CPAN distribution. That sounds much more complicated than it really is. You can use something like Module::Starter to set up a basic skeleton of a tentative App::YourScript distribution. Put your script in the bin/ subdirectory and edit the Makefile.PL to reference all of your direct dependencies. Then, for distribution you do:
perl Makefile.PL
make
make dist
The last step generates a nice App-YourScript-VERSION.tar.gz
Now, when the client wants to install all dependencies, he does the following:
Set up the CPAN client correctly. Simply run it and answer the questions. But you're requiring that already anyway.
"tar -xz App-YourScript-VERSION.tar.gz && cd App-YourScript-VERSION"
Run "cpan ."
The CPAN client will now install all direct dependencies and the dependencies of those distributions automatically. Depending on how you set it up, it will either follow the prerequisites recursively automatically or prompt with a y/n each time.
As an example of this, you might check out a few of the App::* distributions on CPAN. I would think App::Ack is a good example. Maybe one of the App::* distributions from my CPAN directory (SMUELLER).
You could dump %INC at the end of your script. It will contain all used and required modules. But of course, this will only be helpful if you don't require modules conditionally (require Foo if $bar).
For quick-and-dirty, infrequent use, the %INC is the best way to go. If you have to do this with continuous integration testing or something more robust, there are some other tools to help.
Steffen already mentioned the Module::ScanDeps.
The code in Test::Prereq does this, but it has an additional layer that ensures that your Makefile.PL or Build.PL lists them as a dependency. If you make your scripts look like a normal Perl distribution, that makes it fairly easy to check for new dependencies; just run the test suite again.
Aside from that, you might use a tool such as Module::Extract::Use, which parses the static code looking for use and require statements (although it won't find them in string evals). That gets you just the modules you told your script to load.
Also, once you know which modules you loaded, you can combine that with David Cantrell's CPANdeps tool that has already created the dependency tree for most CPAN modules.
Note that you also have to think about optional features too. Your code in this case my not have them, but sometimes you don't load a module until you need it:
sub foo
{
require Bar; # don't load until we need to use it
....
}
If you don't exercise that feature in your trial run or test, you won't see that you need Bar for that feature. A similar problem comes up when a module loads a different set of dependency modules in a different environment (say, mod_perl or Windows, and so on).
There's not a good, automated way of testing optional features like that so you can get their dependencies. However, I think that should be on my To Do list since it sounds like an interesting problem.
Another tool in this area, which is used by Dist::Zilla and its AutoPrereqs plugin, is Perl::PrereqScanner. It installs a scan-perl-prereqs program that will use PPI and a few plugins to search for most kinds of prereq declaration, using the minimum versions you define. In general, I suggest this over scanning %INC, which can bring in bogus requirements and ignores versions.
Today I develop my Perl apps as CPAN-like distributions using Dist::Zilla that can take care of the dependencies through the AutoPrereq plugin. Another interesting piece of code in this area is carton.