I'm working on a personal Perl module to build a basic script framework and to help me learn more about the language. I've created a new module called "AWSTools::Framework" with ExtUtils::ModuleMaker via the command line tool modulemaker. I'm trying to figure out the appropriate way to test it during development.
The directory structure that was created includes the following:
./AWSTOOLS/Framework/lib/AWSTools/Framework.pm
./AWSTOOLS/Framework/t/001_load.t
The autogenerated 001_load.t file looks like this:
# -*- perl -*-
# t/001_load.t - check module loading and create testing directory
use Test::More tests => 2;
BEGIN { use_ok( 'AWSTools::Framework' ); }
my $object = AWSTools::Framework->new ();
isa_ok ($object, 'AWSTools::Framework');
If I try to run the script directly (either from the command line or inside my TextMate editor), it fails with:
Can't locate AWSTools/Framework.pm in #INC....
If I try to run prove in the ./AWSTOOLS/Framework directory, it fails as well.
The question is: What is the proper way to run the tests on Perl modules while developing them?
If you want to run a single test file, you need to tell perl where to find your modules just like you would for any other program. I use the blib to automatically add the right paths:
$ perl Makefile.PL; make; perl -Mblib t/some_test.t
You can also use prove to do the same thing. I don't use prove, but you can read its documentation to figure it out. The -b switch should do that, but I've had problems with it not doing the right thing (could just be my own idiocy).
If you're using the typical toolchain (ExtUtils::MakeMaker) it will be perl Makefile.PL to generate a makefile, then make test every time afterward. Those commands should be run from the root directory of the module. See http://search.cpan.org/perldoc?ExtUtils::MakeMaker#make_test
Edit: and don't do it all manually, or you will come to hate testing. (Well, more than usual.) You will also want to look at least briefly at Test::Tutorial and https://www.socialtext.net/perl5/testing
You may also want to ask the friendly* people in #perl or related channels on your preferred IRC networks.
*Not actually friendly
I actually think that Dist::Zilla is sufficiently flexible enough to allow you to use it for all development. If you aren't uploading to CPAN, just make sure you don't have [UploadToCPAN] in your dist.ini. Also make sure to [#Filter] it out of any plugin bundles which provide it.
Dist::Zilla may be too much to install for only one quick module that you aren't going to touch very often. If you have more than one dist in development then it is definitely worth a look.
You can easily interface it with your VCS using plugins. (Including Git)
You can create a plugin to deploy onto your server. Which would allow you to make sure that all your test files pass before allowing you to deploy ([TestRelease]).
If you don't like tabs in your source files, you can test for that without writing the test yourself ([NoTabsTests]).
Minimal dist.ini for non-CPAN dist
name = Your-Library
author = E. Xavier Ample <example#example.org>
license = Perl_5
copyright_holder = E. Xavier Ample <example#example.org>
copyright_year = 2012
version = 0.001
[GatherDir]
[PruneCruft]
[PruneFiles]
filename = dist.ini
filename = TODO.txt
match = ^.*[.]te?mp$
[NoTabsTests]
[TestRelease]
[CheckExtraTests]
[ModuleBuild]
[FakeRelease]
Test the dist:
dzil test
dzil xtest
If at a later date, you decide to upload it to CPAN:
Replace [FakeRelease] with [UploadToCPAN].
Get a PAUSE id, and set ~/.pause.
user YOUR-PAUSE-ID
password YOUR-PAUSE-PASSWORD
Run dzil release
DONE
In a quick attempt to help you, I would recommend looking at Testing Files and Test Modules.
Continuing to dig around and experiment, I've found the following two things which work for me:
Use prove -l in the './AWSTOOLS/Framework' directory. According to the prove perldoc page, it adds the "lib" directory to the path when Perl runs all the tests in the "t" directory.
To run the script individually/directly, I'm adding the following to the start of the script above the use Test::More line:
use FindBin qw($Bin);
use lib "$Bin/../lib";
This let's me run the script directly via the commad line and in my editor (TextMate). This is based off this page from the Programming Perl book.
Using the -l flag for prove seems very much like the correct thing to do.
As for the "use lib" solution, I doubt that's actually a best practice. If it was, I would expect that modulemaker would have created the 001_load.t test file with that to begin with.
Related
I'm using DBI and DBD::SQLite, and now I'd like to use the R*Tree feature of SQLite. Since this feature is not compiled by DBD::SQLite by default, I have to add a -DSQLITE_ENABLE_RTREE=1 to the #CC_DEFINE variable in DBD::SQLite's Makefile.PL. If I do a 'perl Makefile.PL && make && make install', everything works fine locally on my machine, but this ultimately needs to be deployable/distributable to end users.
What should I do in a case like this? Should I copy the source, grep the source, and create a DBD::SQLite::WithRTree? Create a private version of DBD::SQLite 1.31.1 (Where 1.31 is the current version of DBD::SQLite)? Perhaps a better way altogether?
All other distributions in the project are deployed/distributed via a non-public CPAN::Mini mirror + CPAN::Mini::Inject.
I have to add a '-DSQLITE_ENABLE_RTREE=1' to the #CC_DEFINE variable in DBD::SQLite's Makefile.PL
You're doing this wrong, perl Makefile.PL DEFINE='-DSQLITE_ENABLE_RTREE=1' works. This is documented in ExtUtils::MakeMaker. Now that you know that, a simple solution involving Distroprefs will likely fall in place.
For similar problems, I have installed the modified distribution in a separate directory (without changing any module names), and using use lib qw(the/special/directory) or setting $PERL5LIB for scripts that need to use the enhanced module.
Tweaking the name of the module would also do the job, but that sound like a lot more work to make and test.
You can do this:
cpan
o conf makepl_arg "DEFINE='-DSQLITE_ENABLE_RTREE=1'"
o conf commit
CPAN will then permanently add that DEFINE to the front of all your Makefile.PL calls.
So, it should just be
cpan DBD::SQLite
And your makefile options should get stuffed onto your compile lines
Do I have to run make and make install each time I change a .pm file for Perl? I'm doing a ton of testing and this is becoming cumbersome.
You don't have to install the module to test it.
If I'm testing inside my distribution directory, I just use the test target:
% make test
Or, if I'm using Module::Build:
% ./Build test
Since make is a dependency management tool, it also takes care of any other steps it needs to perform so it can run the test target. You don't need to run each target separately. Module::Build does the same thing.
If I want to test a single file, I combine the make command with a call to perl that also uses the blib module to set the right #INC:
% make; perl -Mblib t/single_test.t
Some people like using prove for the same thing. No matter which method I use, I'm probably using the arrow keys to move back to a previous command line to re-run it. I do very little typing in any of this.
It depends on module setup, but under the standard MakeMaker I use, "make test" runs a "make" if any files have been modified, so when doing intra-module development "make test" is the only command you need until you've finished.
Evan Carroll got it basically right. To expand on his answer: use the testing tools that come with Perl to tighten the workflow.
Let's say you are in your project directory and you hack on the files in its lib/ subdirectory. Execute prove -l to run all tests. That's easier than messing with absolute paths in the PERL5LIB environment variable.
Presumably you're editing a lib module in a non-lib location, rather than clobbering a global library for each modification - do the sensible thing and change the library path perl uses with PERL5LIB, which will append internally to #INC (the use search path):
PERL5LIB=/home/user/code/perl/project/lib perl myapp.pl
If your program isn't pure-perl and requires a make system, there is no way to do this short of rebuilding, but pure-perl (PP) doesn't really require make under normal circumstances. If you do it this way, running perl under a normal environment will yield the predictable and tested results, running it with your PERL5LIB will allow you to test the program.
I'm trying to set up a large-ish project, written in Perl. The IBM MakeMaker tutorial has been very helpful so far, but I don't understand how to link all the modules into the main program. In my project root, I have MANIFEST, Makefile.PL, README, a bin directory, and a lib directory. In my bin directory, I have my main script (Main.pl). In the lib directory, I have each of my modules, divided up into their own respective directories (i.e. Utils::Util1 and Utils::Utils2 in the utils directory, etc). In each module directory, there is also a t directory, containing tests
My MANIFEST file has the following:
bin/Main.pl
lib/Utils/Util1.pm
lib/Utils/Util2.pm
lib/Utils/t/Utils1.t
lib/Utils/t/Utils2.t
Makefile.PL
MANIFEST
README
Makefile.PL is the following:
use ExtUtils::MakeMaker;
WriteMakefile(
'NAME'=>'Foo',
'VERSION_FROM'=>'bin/Main.pl',
'PREREQ_PM'=>{
"XML::Simple"=> 2.18}, #The libraries that we need and their
#minimum version numbers
'EXE_FILES' =>[("bin/Main.pl")]
);
After I make and run, the program crashes, complaining that it cannot find Utils::Util1, and when I run 'make test, it says no tests defined. Can anyone make any suggestions? I have never done a large scale project like this in perl, and I will need to add many more modules
If you are just starting to create Perl modules (which is also Perl's equivalent of a project), don't use Makemaker. Module::Build is the way to go, and it's now part of the standard library. Makemaker is for us old salts who haven't converted to Module::Build yet. :) I'll strike that now that Module::Build is unmaintained and out of favor; I still use MakeMaker.
You should never start off a Perl project by trying to create the structure yourself. It's too much work and you'll always forget something.
There's h2xs, a program that comes with perl and was supposed to be a tool to convert .h files into Perl's glue language XS. It works fine, but its advantage is that it comes with perl:
% h2xs -AXn Module::Name
Something like Module::Starter is a bit more sophisticated, although you have to get it from CPAN. It's the tool we use in Intermediate Perl because it's simple. It fills in some templates with your information:
% module-starter --author=... --email=... --module=...
If you are doing to do this quite a bit, you might then convert that to Distribution::Cooker so you can customize your files and contents. It's a dinky utility I wrote for myself so I could use my own templates.
% dist_cooker Module::Name
If you're really hard core, you might want Dist::Zilla, but that's more for people who already know what they are doing.
Might I also suggest module-starter? It'll automatically create a skeleton project which "Just Works". I learned what little I know about Perl modules organization by reading the generated skeleton files. It's all well-documented, and quite easy to use as a base for growing a larger project in. You can check out the getting-started docs to see what it gives you.
Running module-starter will give you a Perl distribution, consisting of a number of modules (use the command line option --module, such as:
module-starter --distro=Project --module=Project::Module::A,Project::Module::B [...]
to create multiple modules in a single distribution). It's then up to you whether you'd prefer to organize your project as a single distribution consisting of a number of modules working together, or as a number of distributions which can be released separately but which depend on each other (as configured in your Build or Makefile.PL file) to provide a complete system.
Try this structure:
bin/Main.pl
lib/Utils/Util1.pm
lib/Utils/Util2.pm
Makefile.PL
MANIFEST
README
t/Utils1.t
t/Utils2.t
As ysth said, make does not install your modules, it just builds them in a blib directory. (In your case it just copies them there, but if you had XS code, it would be compiled with a C compiler.) Use make install to install your modules for regular scripts to use.
If you want to run your script between make and make install, you can do:
perl -Mblib bin/Main.pl
The -Mblib instructs perl to temporarily add the appropriate directories to the search path, so you can try out an uninstalled module. (make test does that automatically.)
By default, tests are looked for in a top-level t directory (or a test.pl file, but that has some limitations, so should be avoided).
You say "After I make and run"...make puts things into a blib directory structure ready to be installed, but doesn't do anything special to make running a script access them. (make test is special; it does add appropriate paths from blib to perl's #INC to be able to run the tests.) You will need to do a "make install" to install the modules where your script will find them (or use a tool like PAR to package them together with your script).
I have perl module file and test script pl which I use to run some function, and after browsing through the use lib and this previous question here...
I still don't know how to do this.
Basically I have a production directory which contains the module file, and I have a test directory file which contains the same module and the test script file. each time I try to run the script file, it will automatically calls the module in the directory. by printing out the #INC, it seems that the production directory is hosted in there. thus I try to remove it by using
no lib qw(prod_dir);
and while printing out the #INC shows that the directory is no longer there, somehow the script is still calling that other module...
I know this probably sounds really dumb, but hope someone can help me see the light here :)
thanks.
After you have required or used the module, check %INC to see where it came from.
For example:
use Data::Dumper;
print $INC{'Data/Dumper.pm'}."\n";
Note that "::" becomes "/" and you append ".pm". That might give you a clue.
Remember that the current directory (".") is usually an entry in #INC. But the first step is finding out what directory the module was loaded from.
Another thing to remember is that the directories in #INC are searched in order. use lib prepends to that list (making it the first-searched directory), so you may just need to add the appropriate directory.
Can you say more about what you are trying to do and how you are trying to do it? Is this stuff in a standard Perl distribution structure? If you aren't using the standard distribution structure, can you show us a directory listing so we know where things are? Can you also include the code you use to try to load the module? Just update your original question when you pull together the details.
Typically, I run tests through the build runner, which automatically sets up the right #INC.
If I want to run one test in my distribution, I use the blib module to find the build library which has the development versions of my modules:
% perl -Mblib t/test.t
Some people do the same thing with prove.
If you aren't using the basic distribution set-up, consider using it. The tools and best techniques rely on it.
If you just have your module and test file in the same directory, have you tried adding the current directory to #INC with PERL5LIB?
I want another developer to run a Perl script I have written. The script uses many CPAN modules that have to be installed before the script can be run. Is it possible to make the script (or the perl binary) to dump a list of all the missing modules? Perl prints out the missing modules’ names when I attempt to run the script, but this is verbose and does not list all the missing modules at once. I’d like to do something like:
$ cpan -i `said-script --list-deps`
Or even:
$ list-deps said-script > required-modules # on my machine
$ cpan -i `cat required-modules` # on his machine
Is there a simple way to do it? This is not a show stopper, but I would like to make the other developer’s life easier. (The required modules are sprinkled across several files, so that it’s not easy for me to make the list by hand without missing anything. I know about PAR, but it seems a bit too complicated for what I want.)
Update: Thanks, Manni, that will do. I did not know about %INC, I only knew about #INC. I settled with something like this:
print join("\n", map { s|/|::|g; s|\.pm$||; $_ } keys %INC);
Which prints out:
Moose::Meta::TypeConstraint::Registry
Moose::Meta::Role::Application::ToClass
Class::C3
List::Util
Imager::Color
…
Looks like this will work.
Check out Module::ScanDeps and the "scandeps.pl" utility that comes with it. It can do a static (and recursive) analysis of your code for dependencies as well as the %INC dump either after compiling or running the program.
Please note that the static source scanning always errs on the side of including too many dependencies. (It is the dependency scanner used by PAR and aims at being easiest on the end-user.)
Finally, you could choose to distribute your script as a CPAN distribution. That sounds much more complicated than it really is. You can use something like Module::Starter to set up a basic skeleton of a tentative App::YourScript distribution. Put your script in the bin/ subdirectory and edit the Makefile.PL to reference all of your direct dependencies. Then, for distribution you do:
perl Makefile.PL
make
make dist
The last step generates a nice App-YourScript-VERSION.tar.gz
Now, when the client wants to install all dependencies, he does the following:
Set up the CPAN client correctly. Simply run it and answer the questions. But you're requiring that already anyway.
"tar -xz App-YourScript-VERSION.tar.gz && cd App-YourScript-VERSION"
Run "cpan ."
The CPAN client will now install all direct dependencies and the dependencies of those distributions automatically. Depending on how you set it up, it will either follow the prerequisites recursively automatically or prompt with a y/n each time.
As an example of this, you might check out a few of the App::* distributions on CPAN. I would think App::Ack is a good example. Maybe one of the App::* distributions from my CPAN directory (SMUELLER).
You could dump %INC at the end of your script. It will contain all used and required modules. But of course, this will only be helpful if you don't require modules conditionally (require Foo if $bar).
For quick-and-dirty, infrequent use, the %INC is the best way to go. If you have to do this with continuous integration testing or something more robust, there are some other tools to help.
Steffen already mentioned the Module::ScanDeps.
The code in Test::Prereq does this, but it has an additional layer that ensures that your Makefile.PL or Build.PL lists them as a dependency. If you make your scripts look like a normal Perl distribution, that makes it fairly easy to check for new dependencies; just run the test suite again.
Aside from that, you might use a tool such as Module::Extract::Use, which parses the static code looking for use and require statements (although it won't find them in string evals). That gets you just the modules you told your script to load.
Also, once you know which modules you loaded, you can combine that with David Cantrell's CPANdeps tool that has already created the dependency tree for most CPAN modules.
Note that you also have to think about optional features too. Your code in this case my not have them, but sometimes you don't load a module until you need it:
sub foo
{
require Bar; # don't load until we need to use it
....
}
If you don't exercise that feature in your trial run or test, you won't see that you need Bar for that feature. A similar problem comes up when a module loads a different set of dependency modules in a different environment (say, mod_perl or Windows, and so on).
There's not a good, automated way of testing optional features like that so you can get their dependencies. However, I think that should be on my To Do list since it sounds like an interesting problem.
Another tool in this area, which is used by Dist::Zilla and its AutoPrereqs plugin, is Perl::PrereqScanner. It installs a scan-perl-prereqs program that will use PPI and a few plugins to search for most kinds of prereq declaration, using the minimum versions you define. In general, I suggest this over scanning %INC, which can bring in bogus requirements and ignores versions.
Today I develop my Perl apps as CPAN-like distributions using Dist::Zilla that can take care of the dependencies through the AutoPrereq plugin. Another interesting piece of code in this area is carton.