How can I check whether all Perl modules are imported? - perl

Say I have a Perl module:
use Foo::Bar;
sub add {
return Foo::Bar::func() + Foo::Buzz::func();
}
There is an error in this module, because it forgets to use Foo::Buzz. This error will not always be catched by my unit tests, for example if the test for Foo::Buzz runs earlier and imports Foo::Buzz before add() is run. If I use this module in production code, it will fail with the error that Foo::Buzz is not imported.
How can I check whether all modules that I use in the code are also imported?
Edit: I want to check the code before deploying it in production, to avoid that any errors occur. The example will fail in production, and I want to catch the error before that, for example when I run my unit tests. I want a tool or some code that I can run before deployment that catches this error, like flake8 for python.

The short answer is you can't. Since Perl is a dynamic language, you can't check whether you load all modules before runtime as you can't check whether there are some other bugs in your code.
You still can use some static code analysis, trying to find This::Pattern in files where use This::Pattern; is not presented, but it doesn't guarantee anything.

If Perl was strictly a dynamic language, you could easily check whether or not a module is installed in the program. The problem is that Perl isn't 100% dynamic. It does some compilation and part of that compilation work is done after modules are checked.
Bulrush is on the right track. Unfortunately, you can use use clause in order to do this. use is checked pre-compile, so you'll get an error before your eval executes.
However, there's a clue in the use perldoc page:
use Module LIST
use Module
use VERSION
Imports some semantics into the current package from the named module, generally by aliasing certain subroutine or variable names into your package. It is exactly equivalent to
BEGIN { require Module; Module->import( LIST ); }
There you go! You can use require inside a BEGIN clause which is executed even before the rest of the file is parsed. You can use your eval there to see if this works or not too. You need to use a package variable as a flag to see whether or not this worked because of the scope issues. A regular lexically scoped variable will disappear when you leave the BEGIN clause.
BEGIN {
our $FOO_BAR_available = 0; # Must be a package variable
eval {
require Foo::Bar;
Module->import( qw(...) ); # No need if you don't import any subroutines
};
if (not $# ) {
$FOO_BAR_AVAILABLE = 0;
}
}
Then in your program, you'd have:
our $FOO_BAR_available;
if ( not $FOO_BAR_available ) {
# Here be dragons...
}
else {
# Back to your normal code...
}
The our $FOO_BAR_available is a bit confusing. You're not declaring this variable again, you're merely stating that you want to use this variable without prefixing it with the full package name. The variable was set in the BEGIN clause, and this won't affect the value.
You can skip the use of a package variable entirely, if this module was written correctly. Modules are suppose to set a package variable called $VERSION. You can use this variable as your flag:
BEGIN {
eval {
require Foo::Bar;
Module->import( qw(...) ); # No need if you don't import any subroutines
};
}
Note I not only don't have to declare a package variable, I don't even have to verify if the eval statement worked or not.
Then in your program...
if ( not $FOO::BAR::VERSION ) {
# Here be dragons...
}
else {
# Back to your normal code...
}
If the module set the $VERSION variable, you know it loaded. Otherwise, you know the module was not loaded.
Addendum
I want to check the code before deploying it in production, to avoid that any errors occur. The example will fail in production, and I want to catch the error before that, for example when I run my unit tests.
Here's my recommendations. It isn't as simple as running a script, but it's much better:
First, define your production environment: What version of Perl does it have? What modules are used? This will help developers know what to expect. I know Foo::Bar is good, but I shouldn't use Far::Bu because production doesn't have that. It's the first step. I'm surprised at the number of places that have no idea what's on their production environment.
Use Vagrant. This defines a virtual machine that matches your production environment. Developers can download it to their system, and have on their desktop a copy of the production environment.
Use Jenkins. Jenkins is a continuous build engine. Yes, you don't compile Perl, but you can still benefit from Jenkins:
Jenkins can run your unit tests for you. Automatic testing with each and every change in the code. You catch your errors early on.
Your Jenkins system can match your production machines - Same Perl version, same Perl modules. If it doesn't run on your Jenkins build machine because something's not installed, there's a good chance it won't run on Production.
You install via Jenkins. Jenkins can package your release, and you can use that to install known releases. No pulling code from a developer's system and finding out that there's something on the system that's not in your version control system. I don't know how many times I've seen a developer spool something up from their machine for production (thoroughly tested! Trust me!), and then we discover that we don't have that code in our version control system because the developer forgot to check something in.
You don't normally run flake8 in a production environment. By then, it's a wee bit late.
Perl has a lot of nice tools that perform a similar function:
Perlbrew: This allows your developers to install a separate Perl program with its own CPAN module library into their development system. They can use this to match the Perl version and the modules required to the production environment. This way, they're playing with the same set of rules.
Perlcritic: This checks your module against coding standard set forth by Damian Conway in his Perl Best Practices.
B::Lint: This is like the old lint program in C and can catch coding issues.
However, this is stuff to do before you're all set to run in Production. Use Vagrant to help developers setup their own private production environment for testing. Use Jenkins to make sure you test in a very production like environment and catch errors as soon as they happen rather than after UAT testing.

Related

Checking if imported modules are ever used

For whatever reason my coworkers (and probably myself in the past) have imported modules that we don't use to our projects. It's so bad that modules that don't even exist are being imported.
In Perl, you can say something like:
use Socket;
gethostname("DERP");
It's very hard to tell from that function that it was tied to the socket module.
Is there any program that exists or some functionality of Perl that I'm not aware of that can tell you if you've got bloat imported into your script?
That's why I always explicitly lists imports. e.g. use Socket qw( gethostname ); or use Foo qw( ); (The latter imports nothing rather than defaults.)
One approach would be to go file by file and add these import lists, then stop loading module with an empty import list for which no mention of its package name exists in the file.
Use a profiler to see what modules ARE in use. From that find out which are not. Works for me:
Install the Devel::DProf module if you havent (cpanm Devel::DProf or if ubuntu: apt install libdevel-dprof-perl). Newer profilers also exists, i.e. Devel::NYTProf
Change #!/use/bin/perl to #!/use/bin/perl -d:DProf in your program and then run it. This will make a tmon.out file in your current dir
Run dprofpp -aO9999 to analyze tmon.out. This outputs all executed subs including the module name on each line.
Note: if a sub is called conditionally (inside an if-block for instance) the profiler will not list it if the condition is always false. It helps to have a test suit with full code coverage and run that in profiling mode.

Force locally installed module to use locally installed dependencies

I'm working on a project that requires all third-party (read: CPAN) perl modules to be installed in a perforce repository, so that any code that depends on them can be successfully run without anyone else needing to manually install them. I use cpanminus to install my CPAN modules, so I ran cpanm -L . Moose in the desired directory, and everything installed successfully. However, when I try to compile a module I made with Moose, I sometimes get this error:
Undefined subroutine &Carp::longmess_heavy called at /usr/lib/perl5/5.8.8/Carp.pm line 235.
It looks like, even though Carp was installed into my local directory with Moose, it is using the (outdated) version in /user/lib/perl5/5.8.8. I could upgrade Carp on my machine, but as soon as I check my code into the repository everyone else with their outdated Carps will run into the same issue. So how do I force Moose to use the locally installed Carp, rather than the one in /usr/lib/perl5/5.8.8?
You don't force Moose, you force perl. You've installed the module in a location perl doesn't know anything about, so you need to tell perl about it.
Since you want to affect all scripts, you'd want to place in your login script.
export PERL5LIB=/home/.../lib
If you wanted to only affect that one program, you'd launch the program using the following:
PERL5LIB=/home/.../lib script
or you'd add the following to your script:
use lib '/home/.../lib';
I managed to find a solution. It's messy, but that's the price I have to pay for joining a project that already has a messy system in place.
Before including Moose, I have to explicitly required the new Carp:
require "[path to Carp]/Carp.pm";
This generated a lot of warnings about redefining subroutines, so I had to (temporarily) suppress them:
my $restore_warn = $SIG{'__WARN__'};
$SIG{'__WARN__'} = sub {};
require "[path to Carp]/Carp.pm";
$SIG{'__WARN__'} = $restore_warn;

What is the proper way to test perl modules during development?

I'm working on a personal Perl module to build a basic script framework and to help me learn more about the language. I've created a new module called "AWSTools::Framework" with ExtUtils::ModuleMaker via the command line tool modulemaker. I'm trying to figure out the appropriate way to test it during development.
The directory structure that was created includes the following:
./AWSTOOLS/Framework/lib/AWSTools/Framework.pm
./AWSTOOLS/Framework/t/001_load.t
The autogenerated 001_load.t file looks like this:
# -*- perl -*-
# t/001_load.t - check module loading and create testing directory
use Test::More tests => 2;
BEGIN { use_ok( 'AWSTools::Framework' ); }
my $object = AWSTools::Framework->new ();
isa_ok ($object, 'AWSTools::Framework');
If I try to run the script directly (either from the command line or inside my TextMate editor), it fails with:
Can't locate AWSTools/Framework.pm in #INC....
If I try to run prove in the ./AWSTOOLS/Framework directory, it fails as well.
The question is: What is the proper way to run the tests on Perl modules while developing them?
If you want to run a single test file, you need to tell perl where to find your modules just like you would for any other program. I use the blib to automatically add the right paths:
$ perl Makefile.PL; make; perl -Mblib t/some_test.t
You can also use prove to do the same thing. I don't use prove, but you can read its documentation to figure it out. The -b switch should do that, but I've had problems with it not doing the right thing (could just be my own idiocy).
If you're using the typical toolchain (ExtUtils::MakeMaker) it will be perl Makefile.PL to generate a makefile, then make test every time afterward. Those commands should be run from the root directory of the module. See http://search.cpan.org/perldoc?ExtUtils::MakeMaker#make_test
Edit: and don't do it all manually, or you will come to hate testing. (Well, more than usual.) You will also want to look at least briefly at Test::Tutorial and https://www.socialtext.net/perl5/testing
You may also want to ask the friendly* people in #perl or related channels on your preferred IRC networks.
*Not actually friendly
I actually think that Dist::Zilla is sufficiently flexible enough to allow you to use it for all development. If you aren't uploading to CPAN, just make sure you don't have [UploadToCPAN] in your dist.ini. Also make sure to [#Filter] it out of any plugin bundles which provide it.
Dist::Zilla may be too much to install for only one quick module that you aren't going to touch very often. If you have more than one dist in development then it is definitely worth a look.
You can easily interface it with your VCS using plugins. (Including Git)
You can create a plugin to deploy onto your server. Which would allow you to make sure that all your test files pass before allowing you to deploy ([TestRelease]).
If you don't like tabs in your source files, you can test for that without writing the test yourself ([NoTabsTests]).
Minimal dist.ini for non-CPAN dist
name = Your-Library
author = E. Xavier Ample <example#example.org>
license = Perl_5
copyright_holder = E. Xavier Ample <example#example.org>
copyright_year = 2012
version = 0.001
[GatherDir]
[PruneCruft]
[PruneFiles]
filename = dist.ini
filename = TODO.txt
match = ^.*[.]te?mp$
[NoTabsTests]
[TestRelease]
[CheckExtraTests]
[ModuleBuild]
[FakeRelease]
Test the dist:
dzil test
dzil xtest
If at a later date, you decide to upload it to CPAN:
Replace [FakeRelease] with [UploadToCPAN].
Get a PAUSE id, and set ~/.pause.
user YOUR-PAUSE-ID
password YOUR-PAUSE-PASSWORD
Run dzil release
DONE
In a quick attempt to help you, I would recommend looking at Testing Files and Test Modules.
Continuing to dig around and experiment, I've found the following two things which work for me:
Use prove -l in the './AWSTOOLS/Framework' directory. According to the prove perldoc page, it adds the "lib" directory to the path when Perl runs all the tests in the "t" directory.
To run the script individually/directly, I'm adding the following to the start of the script above the use Test::More line:
use FindBin qw($Bin);
use lib "$Bin/../lib";
This let's me run the script directly via the commad line and in my editor (TextMate). This is based off this page from the Programming Perl book.
Using the -l flag for prove seems very much like the correct thing to do.
As for the "use lib" solution, I doubt that's actually a best practice. If it was, I would expect that modulemaker would have created the 001_load.t test file with that to begin with.

What is the intended development process in Module::Starter's manifest.t?

When Module::Starter initializes a project, it creates a test called manifest.t.
#!perl -T
use strict;
use warnings;
use Test::More;
unless ( $ENV{RELEASE_TESTING} ) {
plan( skip_all => "Author tests not required for installation" );
}
eval "use Test::CheckManifest 0.9";
plan skip_all => "Test::CheckManifest 0.9 required" if $#;
ok_manifest();
When you run tests with Build test, here's part of the output:
t\00-load.t ....... ok
t\boilerplate.t ... ok
t\manifest.t ...... skipped: Author tests not required for installation
I understand the outcome in a narrow sense ($ENV{RELEASE_TESTING} is not set, so the tests are skipped), but I don't fully grasp the big picture. What's the intended development process? I assume it's a good idea to run tests to confirm that my module's manifest is accurate. Should I be setting that environment variable? If so, at what point during the development process?
Many module distributions have tests that check not whether the code works, but whether the distribution is in a suitable state for releasing. Things like the MANIFEST being up to date, whether all functions have been documented in POD, etc.
In order to save time, these tests may be written to skip themselves unless the RELEASE_TESTING environment variable is set. This is an informal standard. That way, these tests don't get run when people install the module, nor do they run when the author is just checking to see if a code change broke anything.
You should run RELEASE_TESTING=1 make test (or the Build equivalent) before releasing your dist. If you use Dist::Zilla (which I highly recommend), you can run release tests with dzil test --release. That flag is also set automatically by the TestRelease plugin, which you should definitely use if you use dzil.
Other environment variables commonly used to control testing are AUTOMATED_TESTING and AUTHOR_TESTING. AUTOMATED_TESTING is set by CPAN testers running automated smoke tests.

How do I find the module dependencies of my Perl script?

I want another developer to run a Perl script I have written. The script uses many CPAN modules that have to be installed before the script can be run. Is it possible to make the script (or the perl binary) to dump a list of all the missing modules? Perl prints out the missing modules’ names when I attempt to run the script, but this is verbose and does not list all the missing modules at once. I’d like to do something like:
$ cpan -i `said-script --list-deps`
Or even:
$ list-deps said-script > required-modules # on my machine
$ cpan -i `cat required-modules` # on his machine
Is there a simple way to do it? This is not a show stopper, but I would like to make the other developer’s life easier. (The required modules are sprinkled across several files, so that it’s not easy for me to make the list by hand without missing anything. I know about PAR, but it seems a bit too complicated for what I want.)
Update: Thanks, Manni, that will do. I did not know about %INC, I only knew about #INC. I settled with something like this:
print join("\n", map { s|/|::|g; s|\.pm$||; $_ } keys %INC);
Which prints out:
Moose::Meta::TypeConstraint::Registry
Moose::Meta::Role::Application::ToClass
Class::C3
List::Util
Imager::Color
…
Looks like this will work.
Check out Module::ScanDeps and the "scandeps.pl" utility that comes with it. It can do a static (and recursive) analysis of your code for dependencies as well as the %INC dump either after compiling or running the program.
Please note that the static source scanning always errs on the side of including too many dependencies. (It is the dependency scanner used by PAR and aims at being easiest on the end-user.)
Finally, you could choose to distribute your script as a CPAN distribution. That sounds much more complicated than it really is. You can use something like Module::Starter to set up a basic skeleton of a tentative App::YourScript distribution. Put your script in the bin/ subdirectory and edit the Makefile.PL to reference all of your direct dependencies. Then, for distribution you do:
perl Makefile.PL
make
make dist
The last step generates a nice App-YourScript-VERSION.tar.gz
Now, when the client wants to install all dependencies, he does the following:
Set up the CPAN client correctly. Simply run it and answer the questions. But you're requiring that already anyway.
"tar -xz App-YourScript-VERSION.tar.gz && cd App-YourScript-VERSION"
Run "cpan ."
The CPAN client will now install all direct dependencies and the dependencies of those distributions automatically. Depending on how you set it up, it will either follow the prerequisites recursively automatically or prompt with a y/n each time.
As an example of this, you might check out a few of the App::* distributions on CPAN. I would think App::Ack is a good example. Maybe one of the App::* distributions from my CPAN directory (SMUELLER).
You could dump %INC at the end of your script. It will contain all used and required modules. But of course, this will only be helpful if you don't require modules conditionally (require Foo if $bar).
For quick-and-dirty, infrequent use, the %INC is the best way to go. If you have to do this with continuous integration testing or something more robust, there are some other tools to help.
Steffen already mentioned the Module::ScanDeps.
The code in Test::Prereq does this, but it has an additional layer that ensures that your Makefile.PL or Build.PL lists them as a dependency. If you make your scripts look like a normal Perl distribution, that makes it fairly easy to check for new dependencies; just run the test suite again.
Aside from that, you might use a tool such as Module::Extract::Use, which parses the static code looking for use and require statements (although it won't find them in string evals). That gets you just the modules you told your script to load.
Also, once you know which modules you loaded, you can combine that with David Cantrell's CPANdeps tool that has already created the dependency tree for most CPAN modules.
Note that you also have to think about optional features too. Your code in this case my not have them, but sometimes you don't load a module until you need it:
sub foo
{
require Bar; # don't load until we need to use it
....
}
If you don't exercise that feature in your trial run or test, you won't see that you need Bar for that feature. A similar problem comes up when a module loads a different set of dependency modules in a different environment (say, mod_perl or Windows, and so on).
There's not a good, automated way of testing optional features like that so you can get their dependencies. However, I think that should be on my To Do list since it sounds like an interesting problem.
Another tool in this area, which is used by Dist::Zilla and its AutoPrereqs plugin, is Perl::PrereqScanner. It installs a scan-perl-prereqs program that will use PPI and a few plugins to search for most kinds of prereq declaration, using the minimum versions you define. In general, I suggest this over scanning %INC, which can bring in bogus requirements and ignores versions.
Today I develop my Perl apps as CPAN-like distributions using Dist::Zilla that can take care of the dependencies through the AutoPrereq plugin. Another interesting piece of code in this area is carton.