We use MyCPAN::App::DPAN (Version 1.28) just like in Question: How can I manage Perl module dependencies?
This works quite well in general. But we ran into troubles with some specific packages:
As expample DBD::Oracle does not have their main *.pm file in DBD-Oracle-1.27/lib/DBD/Oracle/Oracle.pm but in DBD-Oracle-1.27/Oracle.pm.
It seems that dpan does not find this *.pm file, since it is not listed in modules/02packages.details.txt.gz. Whereas DBD::Oracle::Object (other Perl-Module of this distribution, but located DBD-Oracle-1.27/lib/DBD/Oracle/) is listed there.
In dpan.conf we only have set the recommended values:
dpan_dir /some/dir
indexer_id Me
pause_full_name The Admin Team
pause_id TAT
system_id my system id
Other Packages with the same problem:
Apache::DB
Packages that work:
Crypt::Blowfish (no extra modules)
Math::Round (no extra modules)
XML::Twig (no "lib" dir, but extra modules)
Does someone have any tips/hints?
DPAN works by indexing the distributions and storing the results in a file in the indexer_reports directory. As a workaround until I can investigate your problem, you can just edit the report it created for the problem distributions. As long as you haven't set fresh_start in your configuration, it will use the cached report which you have edited to add the missing module.
Related
I maintain multiple Perl written (unix-ish) applications whose current installation process consists of a manually written Makefile and installs configuration files into /etc/.
I'd really like to switch their development to use Dist::Zilla, but so far I haven't found any Dist::Zilla plugin or feature which allows me to put given files into /etc/ when the make install (or ./Build install in case of using Module::Build instead of ExtUtils::MakeMaker) is run by the local administrator who's installing my application.
With pure ExtUtils::MakeMaker, I could define additional make targets in MY::postamble and the let the install target depend on one of them via the depend { install => … } attribute. Doing something similar, but via dzil build, would probably suffice, but I'd appreciate a more obvious way.
One orthogonal approach would be to make the application not to require the files under /etc/ to exist, but for just switching to Dist::Zilla that seems to much change in the actual code despite I only want to change the build system for now.
For the curious: the two applications I currently have in mind for switching to Dist::Zilla are xen-tools and unburden-home-dir.
The best thing to do is to avoid installing files into /etc from any Perl distribution. You cannot ensure that the cpan client (or the installing user) has permissions to install there, and there can be multiple Perls installed on a system, so each one of them would clobber the /etc files of another install. You can't really prevent the file from being overwritten by a subsequent install, so you shouldn't put config data there that you don't want to lose.
You could put the config file in /etc/, if the application knows to look for it there, but you should allow for that path to be customized (say on a test system, look for the file in the local directory, or in a user's home directory).
For installing read-only module-specific data, the best practice in Perl is to install into a Perl-install-specific location, and the module to do that is File::ShareDir::Install. You can use it from Dist::Zilla using the [ShareDir] plugin, Dist::Zilla::Plugin::ShareDir. It is even included in the [#Basic] plugin bundle, so if you use [#Basic] in your dist.ini, you don't need to do anything at all, other than drop your data files into the share/ directory in your distribution repository.
To access the contents of the sharedir from code, use File::ShareDir.
For porting a complex module installer to Dist::Zilla, I recommend my plugins MakeMaker::Custom or ModuleBuild::Custom, depending on which installer you prefer. These allow you to keep your existing Makefile.PL or Build.PL and just have Dist::Zilla plug in necessary bits like the dependencies.
I would like to include a few additional .pl files in my CPAN module. These files are not essential to use the module, but are provide useful functionality/glue when the module is used in some common frameworks and applications.
Currently, I just include the .pl files in a "extras" directory of the distribution. This has the drawback that the files are not installed on make install. Is there a way to include them in the installation and where should they go? (They aren't executables and don't belong in "bin".) Would "share" make sense? Or are these kinds of files usually just not installed and it is left to the user to get them out of the .tgz archive and use as needed?
I use Dist::Zilla to manage my distribution.
I would suggest the following:
If they're actual complete programs or are almost complete, polish then up and make them standalone items that could go into /bin with POD of their own.
If they're utility glue, make a ::Utils module for them to live in and document their usage.
If these are useful code snippets but not something you can install somewhere or are sample usages or handy idioms, create a ::Cookbook all-POD module and include them there with the appropriate illuminating explanation for each one.
I don't know exactly how Dist::Zilla works, but the resulting archive has to be compatible with what ExtUtils::MakeMaker creates.
When you create a module with module-starter, it creates a module template using ExtUtils::MakeMaker. It creates several files and directories like the lib directory where your module lives and the t directories where your tests live.
One thing it doesn't create is a bin directory. However, if you create a bin directory, and you put files under this directory (such as Perl scripts), these files will be installed under the bin directory in your Perl's distribution and linked to /usr/local/bin or /usr/bin. Would this be a good place for your scripts?
I liked #Joe's answer, except that in my case the files were WebWork macros -- individual .pl files that make my module callable from WebWork end-user's code. So they don't fit under any of the categories discussed here, and as .pl files can't be made a module.
This is what I ended up doing:
put all .pl macro files into 'extras/WebWork' in the distribution.
add to "dist.ini" file a [ShareDir] stanza with dir = extras property.
now the WebWork admin can install my distribution from CPAN and then use perl -MFile::ShareDir -e 'print File::ShareDir::dist_dir("Statistics-R-IO")' to find the macros and make them available in WebWork.
If I create a Perl module and install it as
perl Makefile.PL prefix=/path/to/perllib
Then I would expect to be able to set PERL5LIB to /path/to/perllib and the module be found. This doesn't seem to be the case. The module gets installed to /path/to/perllib/site_perl/5.8.5/, so that's what PERL5LIB has to be set to (or passed to "use lib").
Am I doing something wrong in how I install the modules or how I set PERL5LIB?
You're conflating PREFIX= and LIB=.
A distribution consists of more than just modules. Documentation, for one, isn't installed in the same directory as the modules themselves. PREFIX= tells Perl the base path for all components.
LIB= tells Perl where to install modules.
Use (for example)
perl Makefile.PL PREFIX=~ LIB=~/lib/perl5
Then you'd use
export PERL5LIB=~/lib/perl5
No, that's how it works. The prefix indicates the root of the installation, but the library directories are usually somewhere deeper within that directory structure.
If you install multiple modules with the same prefix, they should all be able to use the same PERL5LIB value, though.
Long story short, use INSTALL_BASE instead of PREFIX. It works more like the --prefix of other installers and creates a predictable install pattern. Then you can set PERL5LIB to <INSTALL_BASE>/lib/perl5 and go.
ExtUtils::MakeMaker::FAQ explains:
The behavior of PREFIX is complicated and depends closely on how your Perl is
configured. The resulting installation locations will vary from machine to machine
and even different installations of Perl on the same machine. Because of this, its
difficult to document where prefix will place your modules.
In contrast, INSTALL_BASE has predictable, easy to explain installation locations.
Now that Module::Build and MakeMaker both have INSTALL_BASE there is little reason
to use PREFIX other than to preserve your existing installation locations. If you
are starting a fresh Perl installation we encourage you to use INSTALL_BASE. If you
have an existing installation installed via PREFIX, consider moving it to an
installation structure matching INSTALL_BASE and using that instead.
What version of perl? You should only have to specify the path up to 'site_perl'. For libraries not under site_perl, you should also specify the path up to 'perllib'. More recent versions of perl (5.8 or better?) are better at letting you specify only these two directories, without having to specify version and architecture directories.
I want to setup in-house CPAN for distributing our internal code.
So I was looking at CPAN::Mini as recommended here. But it looks there are other options as CPAN::Site, CPAN::Dark, Dist::Zilla ...
I'm little bit overwhelmed with all these options. What do people mostly use/ recommend?
What I need is a way to push internal modules to repository which can be accesses from several machines.
The quick answer is that you want to use CPAN::Mini to create a local mirror of all that is current on the CPAN, and then CPAN::Mini::Inject to add your own distributions to it.
The long answer is that it helps to understand how a CPAN mirror is constructed. Broadly speaking, it is simply a directory that contains two sub-directories.
The 'modules' directory contains in turn two files, 03modlist.data.gz, whose contents is ignored by modern CPAN clients but there's legacy code that assumes this file exists, so just copy it from an existing mirror. The other is 02packages.details.txt.gz, which I shall describe later.
The 'authors' directory contains a file '01mailrc.txt.gz' which is another relic of the past whose contents can be ignored, so just copy it from another mirror, and it contains the 'id' directory. This in turn contains sub-directories and distributions, whose names follow a pattern. For example, my PAUSE id is DCANTRELL, and one of my distributions is XML-Tiny-2.06.tar.gz, so that file lives at .../authors/id/D/DC/DCANTRELL/XML-Tiny-2.06.tar.gz.
The 02packages.details.txt.gz file is the index that maps module names to distributions, and this must be up to date for your mirror to work properly. It consists of a few header lines, which must be present and correct, followed by a blank line, followed by one line for each module. Those lines are three fields separated by spaces:
module name
module version
distribution filename
eg
XML::Tiny 2.06 D/DC/DCANTRELL/XML-Tiny-2.06.tar.gz
(you may also see .tgz, .zip, and a coupla others)
A distribution may appear in several lines, once for each module it contains. eg
XML::Tiny::DOM 1.1 D/DC/DCANTRELL/XML-Tiny-DOM-1.1.tar.gz
XML::Tiny::DOM::Element 1.1 D/DC/DCANTRELL/XML-Tiny-DOM-1.1.tar.gz
In a normal CPAN mirror, there may be several versions of a distribution, and several versions of a module - for example, the current version and a few older ones, or the current stable one and a dev one. The index file contains the most recent stable version. You can tell dev versions of distributions because they have an underscore in their version, or contain the string '-TRIAL'.
So, knowing all that, you can construct a CPAN-a-like that contains only your code. But using CPAN::Mini and CPAN::Mini::Inject to add your stuff to a "real" CPAN is less work.
Once you've created your CPAN-a-like, you can either expose it on HTTP and access it using any client as normal, or you could just have it in the filesystem and configure the CPAN client to access it using a file:/// URL.
You might also consider Pinto. Pinto allows you to curate your own stable CPAN repository, which can contain any number of both public and private distributions. Pinto also helps you to manage change as your dependencies evolve over time.
DrHyde gave a very nice answer to the question. But if you don't want to maintain a CPAN mirror, you can use MyCPAN::App::DPAN together with MyCPAN::Indexer.
Cave: Both distributions are under development. Not all combinations will work. What I use is the latest version of MyCPAN::App::DPAN on github (1.28_11) and MyCPAN::Indexer version 1.28_10 (later versions don't work with MyCPAN::App::DPAN).
MyCPAN::App::DPAN will create a CPAN-like directory structure on your local disk from the distributions you feed it. You will need to create a config file for it (say, .dpanrc):
# contents of .dpanrc
indexer_id Edward Baudrez <my.email.address#example.org>
dpan_dir /home/ebaudrez/rsync.net/dpan
merge_dirs /home/ebaudrez/rsync.net/dpan/dists
report_dir /home/ebaudrez/rsync.net/dpan/indexer_reports
Put your distribution tarballs in the directory merge_dirs (I think there's no reason that directory should reside under dpan_dir, but I'm too lazy to figure it out right now). Then call dpan:
dpan -f $HOME/.dpanrc
dpan will create a CPAN-like structure in dpan_dir (containing, in particular, authors and modules). This directory can then be used with cpanm (for instance):
cpanm --mirror $HOME/rsync.net/dpan --mirror http://search.cpan.org/CPAN
Note that I use real CPAN as fallback, because the DarkPAN is by definition incomplete. If you also happen to have a mini CPAN mirror, you can also use it here:
cpanm --mirror $HOME/rsync.net/dpan --mirror $HOME/mirrors/minicpan --mirror-only
Do note that, for this scheme to work, you will need to create distribution tarballs from your source code. I like and use Dist::Zilla, but note that you can also generate tarballs from Makefile.PL, so you definitely don't need to use Dist::Zilla. But it takes care of a lot of the details.
Creating a real distribution from your source code may seem like a lot of work, but Dist::Zilla helps lift the burden, and the transition to a real CPAN module, some day in the future ;-), is also simplified a lot when you already have a distribution.
I'm trying to set up a large-ish project, written in Perl. The IBM MakeMaker tutorial has been very helpful so far, but I don't understand how to link all the modules into the main program. In my project root, I have MANIFEST, Makefile.PL, README, a bin directory, and a lib directory. In my bin directory, I have my main script (Main.pl). In the lib directory, I have each of my modules, divided up into their own respective directories (i.e. Utils::Util1 and Utils::Utils2 in the utils directory, etc). In each module directory, there is also a t directory, containing tests
My MANIFEST file has the following:
bin/Main.pl
lib/Utils/Util1.pm
lib/Utils/Util2.pm
lib/Utils/t/Utils1.t
lib/Utils/t/Utils2.t
Makefile.PL
MANIFEST
README
Makefile.PL is the following:
use ExtUtils::MakeMaker;
WriteMakefile(
'NAME'=>'Foo',
'VERSION_FROM'=>'bin/Main.pl',
'PREREQ_PM'=>{
"XML::Simple"=> 2.18}, #The libraries that we need and their
#minimum version numbers
'EXE_FILES' =>[("bin/Main.pl")]
);
After I make and run, the program crashes, complaining that it cannot find Utils::Util1, and when I run 'make test, it says no tests defined. Can anyone make any suggestions? I have never done a large scale project like this in perl, and I will need to add many more modules
If you are just starting to create Perl modules (which is also Perl's equivalent of a project), don't use Makemaker. Module::Build is the way to go, and it's now part of the standard library. Makemaker is for us old salts who haven't converted to Module::Build yet. :) I'll strike that now that Module::Build is unmaintained and out of favor; I still use MakeMaker.
You should never start off a Perl project by trying to create the structure yourself. It's too much work and you'll always forget something.
There's h2xs, a program that comes with perl and was supposed to be a tool to convert .h files into Perl's glue language XS. It works fine, but its advantage is that it comes with perl:
% h2xs -AXn Module::Name
Something like Module::Starter is a bit more sophisticated, although you have to get it from CPAN. It's the tool we use in Intermediate Perl because it's simple. It fills in some templates with your information:
% module-starter --author=... --email=... --module=...
If you are doing to do this quite a bit, you might then convert that to Distribution::Cooker so you can customize your files and contents. It's a dinky utility I wrote for myself so I could use my own templates.
% dist_cooker Module::Name
If you're really hard core, you might want Dist::Zilla, but that's more for people who already know what they are doing.
Might I also suggest module-starter? It'll automatically create a skeleton project which "Just Works". I learned what little I know about Perl modules organization by reading the generated skeleton files. It's all well-documented, and quite easy to use as a base for growing a larger project in. You can check out the getting-started docs to see what it gives you.
Running module-starter will give you a Perl distribution, consisting of a number of modules (use the command line option --module, such as:
module-starter --distro=Project --module=Project::Module::A,Project::Module::B [...]
to create multiple modules in a single distribution). It's then up to you whether you'd prefer to organize your project as a single distribution consisting of a number of modules working together, or as a number of distributions which can be released separately but which depend on each other (as configured in your Build or Makefile.PL file) to provide a complete system.
Try this structure:
bin/Main.pl
lib/Utils/Util1.pm
lib/Utils/Util2.pm
Makefile.PL
MANIFEST
README
t/Utils1.t
t/Utils2.t
As ysth said, make does not install your modules, it just builds them in a blib directory. (In your case it just copies them there, but if you had XS code, it would be compiled with a C compiler.) Use make install to install your modules for regular scripts to use.
If you want to run your script between make and make install, you can do:
perl -Mblib bin/Main.pl
The -Mblib instructs perl to temporarily add the appropriate directories to the search path, so you can try out an uninstalled module. (make test does that automatically.)
By default, tests are looked for in a top-level t directory (or a test.pl file, but that has some limitations, so should be avoided).
You say "After I make and run"...make puts things into a blib directory structure ready to be installed, but doesn't do anything special to make running a script access them. (make test is special; it does add appropriate paths from blib to perl's #INC to be able to run the tests.) You will need to do a "make install" to install the modules where your script will find them (or use a tool like PAR to package them together with your script).