From my understanding Perl traditionally has only included core functionality, and people install additional libraries to do all sorts of useful (and sometimes very basic) things. But at some point there came to be "core libraries" which are shipped with Perl by default – so you can use these libraries without installing them.
Coming from Python I'm curious how this is managed. Specifically:
How are libraries chosen?
Do the libraries still have their own version numbers and release schedules?
What kind of backward-compatibility guarantees do you have when using these libraries?
Is it common to upgrade or downgrade these libraries in a system? Is this done system-wide or more specifically?
If there's a bug fix that requires an API change, how does that happen?
How is functionality added to these core libraries (if it is at all)?
Currently, only libraries that are necessary to bootstrap/install other libraries go into the core list.
Some are only in the Perl git repository. Some are dual-life on CPAN and in the repo. Sometimes bugs get fixed in the repo and the changes are backported to the CPAN version. Sometimes there's a new release on CPAN and a Perl maintainer checks in the module into the repo.
You can rely on a core module. There's an very lengthy deprecation timespan before one gets removed, recent prominent example was Switch.
Packagers (e.g. the people who build RPMs for a Linux distribution) never could get this right; the wrong order of include paths (#INC) not their fault, and finally fixed with 5.12. This is the reason where the recommendation comes from to compile your own perl and not mess with the system installation. With 5.12, you are supposed to just use CPAN to install an upgraded version of a core module, and it gets installed addtionally to the one shipped with the system, but since the new one comes before the old one in the include path, the new one gets loaded when you use/require it.
Laid out in perlpolicy.
Program the functionality and tests for it, document the thing, then release on CPAN or respectively have a maintainer apply the changeset. This is accompanied with a discussion on p5p.
Related
I'm working on a Perl app that's intended to be deployed using Module::Build. I've need to install a number of modules through CPAN because they weren't available through Ubuntu's package manager—or, more correctly, the internal apt-get mirror all our servers use. While this is all well and good on our development server, IT is (understandably) reluctant to run code on production machines that isn't cached or otherwise controlled in-house.
As we don't currently have a CPAN mirror, this basically means that I need to get all of these non-Ubuntu modules into one place so they can be archived and/or committed to version control. The ideal solution would be to check the utility out from source control, change a couple config variables for databases and such, maybe run a build/install command, and be done. Fortunately, the development server is a clone of the production server, so modules using XS or other architecture-specific features shouldn't cause an issue.
I think the cleanest way to handle this would be checking in source tarballs for the modules I need and setting Module::Build to use those to resolve its dependencies instead of looking to CPAN, but I don't see an option for that. Is this something that's doable, or is there another way to round up all the modules I need for an essentially offline deployment?
As mentioned in the comments above, Pinto may suit your needs as it creates your own CPAN repo.
Pinto has two primary goals. First, Pinto seeks to address the problem
of instability in the CPAN mirrors. Distribution archives are
constantly added and removed from the CPAN, so if you use it to build
a system or application, you may not get the same result twice.
Second, Pinto seeks to encourage developers to use the CPAN toolchain
for building, testing, and dependency management of their own local
software, even if they never plan to release it to the CPAN.
Pinto accomplishes these goals by providing tools for creating and
managing your own custom repositories of distribution archives. These
repositories can contain any distribution archives you like, and can
be used with the standard CPAN toolchain. The tools also support
various operations that enable you to deal with common problems that
arise during the development process.
Second Answer
Alternatively, if you are only going to deploy to Ubuntu, you can turn CPAN modules - and your own into Debian packages with dh-make-perl. You can then host them in your own repo with reprepro. The beauty of this is that you can update the packages and do a
apt-get update
apt-get upgrade
on the client machines, so long as they have your own repo as a source
Check out Stratopan and Carton.
Also see:
Deploying Perl Application
How can I manage Perl module dependencies?
I'm not sure how common it is, but I've been using perlbrew and Pinto together to solve some of the issues you are talking about.
With perlbrew, I'm not interacting with the "system" perl. There is an application perl and a system perl, and there is no risk of me installing a later version of a module that somehow interferes with something the system perl was doing.
With Pinto, I have archived versions of the CPAN modules that I know will work.
When I deploy, I build a perlbrew perl (with an alias like "prod" or something) and then I install all the necessary modules into that perlbrew perl using the Pinto repository. I currently facilitate this with a cpan bundle module (that module also goes into the Pinto repo) so you can just install the bundle from the repository and it automatically puts all your dependencies in.
Should I put only non-core modules in the PREREQ_PM section of a Makefile.PL or should I put there the core modules too?
Yes, you should specify all dependencies: The Perl Core is not fixed in all eternity. Core modules get added or removed (after a deprecation process) all the time. Specifying all your dependencies…
… will make your program work in future perls that have removed the module from Core. It will still be available from CPAN. For example Term::UI is a Core module since v5.9.5 but was removed in v5.19.0.
… will assert that a sufficiently high version of the core module in question is installed. Some modules have evolved significantly through time, and it is easy to forget that not everything was available five years ago.
… will make your program work on older perls that didn't include the module into Core, but are nevertheless able to use it.
On the other hand these can be very small gains. Nothing will break should you forget to specify such a central module like Carp as a dependency.
Remember: There are three causes for a module to be included in Core:
stuff central to Perl like strict which is not going to be removed.
stuff needed for downloading and installing CPAN modules. This includes filesystem handling. Here changes do happen occasionally.
Historic cruft. Pleeease throw out CGI.pm ;-)
Tip: use the corelist tool from Module::Corelist to see what modules versions are available in which perl release.
There was an interesting discussion on this on PerlMonks not so very long ago.
My personal policy is not to do so.
Although modules are dropped from core occasionally, there is a long (about 2 year) deprecation cycle, giving you plenty of time to re-package your distribution with updated dependencies.
And if the worst comes to the worst, and you don't update your distribution, when they try to install it, they'll get an error message about a missing module, so it should be fairly obvious for them how to proceed. (That is, they should install that module, then try installing yours again.)
I'm have Perl 5.8 installed on all of our servers and wanted to use the DBI and DBD::Oracle modules to access our databases. My main concern is with the newer versions of perl the DBI and DBD modules will stop working with 5.8. Then I'd have to upgrade every server to the newest perl version.
My question is as perl itself becomes later versions and modules are developed for them will they still be backwards compatible? "CPAN does not carry all ancient releases and patchlevels of Perl", if I create documentation saying run "cpan -i DBI" if the newest version of DBI will not function with 5.8?
There is no guarantee.
In general, you want to use the same versions of the modules on all your systems. If you use different versions then you will have different bugs and features available on different servers.
I'd suggest creating Debs / RPMS / etc for the ones you are going to use and then run a package repository that all your servers share.
Not absolutely, but in general perl is pretty gentle about breaking code, with not many breaking changes, and long deprecation cycles on the ones that do happen. A pretty hefty portion of the code that was uploaded to CPAN in 1999 will run without modification in perl 5.14.
Since perl 5.12, the perl release cycle has become shorter, and the deprecation periods have also become shorter, which is cause for concern, but at the same time the concept of feature versioning has gained currency. The idea is that code may declare the version of perl that it's targeting with use VERSION (e.g. use 5.16.0), and any code that doesn't declare a version is assumed to be targeting approximately 5.10. When code that targets an older perl version is run on a newer perl version, newer features that might cause compatibility issues (e.g. new keywords) are disabled, and old misfeatures may be re-enabled in the name of compatibility. This isn't an absolute guarantee, but it will be adhered to as much as practical.
More information about back-compatibility and deprecation is in perlpolicy.
In general, no. There are a lot of great new features in recent releases of Perl (smart match operator, the // operator, for two one example) that are not backwards compatible. Many authors will decide to take advantage of these features rather than keep their modules compatible with older versions of Perl.
Check the CPAN Tester's Matrix of a module, including the link about the max version that passes all of the distribution's tests, to get an idea of what versions of Perl are compatible with each version of a module.
cpan -i Some::Module will indeed attempt to install the latest version of the module Some::Module, but with a little research, it can be used to install older versions too. You need to find or guess the author of an older version and provide the path to the distribution on the CPAN mirror servers. For example,
cpan -i J/JF/JFRIEDL/Yahoo-Search-1.9.12.tar.gz
cpan -i A/AS/ASG/List-Gen-0.80.tar.gz
CPAN authors may delete their older distributions from CPAN. But even then, the distribution is available at the BackPAN if you're willing to download, unpack, and build the distribution yourself.
I'm tasked with replicating a production environment to create many test/sit environments.
One of the things I need to do is build up Perl, with all the modules which have been installed (including internal and external modules) over the years. I could just use CPAN.pm autobundle, but this will result in the test environment having much newer versions of the external modules that production has.
What is the easiest/best way to get and install (a lot of) version specific Perl modules.
bdfoy has the best large scale solution, but if you just want to install a few modules you can ask the CPAN shell to install a specific distribution by referencing a path to a tarball (relative to the top of the CPAN tree).
cpan> install MSCHWERN/Test-Simple-0.62.tar.gz
Throw a URL to BackPAN into your URL list and you can install any older version.
cpan> o conf urllist push http://backpan.perl.org/
This is in the CPAN.pm FAQ under "how do I install a 'DEVELOPER RELEASE' of a module?"
cpan install App::cpanminus
cpanm Your::Module#1.23
(Carton, as referenced in other answers, uses cpanm underneath to resolve explicit version requirements.)
Make your own CPAN mirror with exactly what you want. Stratopan.com, a service, and Pinto, tools that's built on top of, can help you do that.
The CPAN tools only install the latest version of any distribution because PAUSE only indexes the latest version. However, you can create your own, private CPAN that has exactly the distributions that you want. Once you have your own CPAN mirror with only what you want, you point your CPAN tools at only that mirror so it only installs those versions. More on that in a minute.
Now, you want to have several versions of that. You can create as many mirrors as you like, and you can also put the mirrors in source control so you can check out any version of the mirror that you like.
Tools such as CPAN::Mini::Inject can help you set up your own CPAN. Check out my talks on Slideshare for the basic examples, and some of my videos on Vimeo for some of the demonstrations. Look at anything that has "CPAN" or "BackPAN" in the title. I think I might have some stuff about it in The Perl Review too, or should by the next issue. :)
Lately, I've been working on a program called dpan (for DarkPAN) that can look at random directories, find Perl distributions in them, and create the structure and index files that you need. You run dpan, you get a URL to point your CPAN client toward, and off you go. It's part of my MyCPAN-Indexer project, which is in Github. It's not quite ready for unsupervised public use because I mostly work with corporate clients to customize their setup. If you're interested in that, feel free to ask me questions though.
Also, I recently released CPAN::PackageDetails that can help you build the right index file. It's still a bit young too, but again, if you need something special, just ask.
[It's almost five years on and this is a well-asked and well-answered question that has had a lot of views. Since this page must still come up in Google searches, an update can't hurt.]
Carton is worth mentioning here. Carton is a relatively recent tool in the same style as App::cpanminus, App::cpanoutdated, perlbrew, et. al. The author (Miyagawa) calls it "alpha" quality, but even in its current state carton helps simplify the maintenance of multiple environments of version tuned modules across machines.
Pinto too is another recent tool relevant to some of the responses (in fact one of the respondents is a contributor).
Stratopan.com is another alternative. Stratopan provides private CPANs in the cloud. You can fill your Stratopan repository with specific versions of modules (and their dependencies) and then install them using the standard Perl tool chain. The repository changes only when you decide to change it, so you'll get always get the versions of the modules that you want.
Disclaimer: I operate Stratopan.
It seems that creating a cpanfile listing all your modules and desired versions (using the == <version> syntax to lock it to a specific release) could serve well here, too. That would mean using Carton or cpanm for installing the modules.
Doing this would have the benefit of being able to quickly/easily tweak the file to test upgrading specific modules in a dev or staging environment - something that a private CPAN mirror wouldn't let you do (without creating multiple mirrors).
It seems that most of the installers for Perl are centered around installing Perl modules, not applications. Things like ExtUtils::MakeMaker and Module::Build are very well suited for modules, but require some additional work for Web Apps.
Ideally it would be nice to be able to do the following after checking out the source from the repository:
Have missing dependencies detected
Download and install dependencies from CPAN
Run a command to "Build" the source into a final state (perform any source parsing or configuration necessary for the local environment).
Run a command to install the built files into the appropriate locations. Not only the perl modules, but also things like template (.tt) files, and CGI scripts, JS and image files that should be web-accessible.
Make sure proper permissions are set on installed files (and SELinux context if necessary).
Right now we have a system based on Module::Build that does most of this. The work was done by done by my co-worker who was learning to use Module::Build at the time, and we'd like some advice on generalizing our solution, since it's fairly app-specific right now. In particular, our system requires us to install dependencies by hand (although it does detect them).
Is there any particular system you've used that's been particularly successful? Do you have to write an installer based on Module::Build or ExtUtils::MakeMaker that's particular to your application, or is something more general available?
EDIT: To answer brian's questions below:
We can log into the machines
We do not have root access to the machines
The machines are all (ostensibly) identical builds of RHEL5 with SELinux enabled
Currently, the people installing the machines are only programmers from our group, and our source is not available to the general public. However, it's conceivable our source could eventually be installed on someone else's machines in our organization, to be installed by their programmers or systems people.
We install by checking out from the repository, though we'd like to have the option of using a distributed archive (see above).
The answer suggesting RPM is definitely a good one. Using your system's package manager can definitely make your life easier. However, it might mean you also need to package up a bunch of other Perl modules.
You might also take a look at Shipwright. This is a Perl-based tool for packaging up an app and all its Perl module dependencies. It's early days yet, but it looks promising.
As far as installing dependencies, it wouldn't be hard to simply package up a bunch of tarballs and then have you Module::Build-based solution install them. You should take a look at pip, which makes installing a module from a tarball quite trivial. You could package this with your code base and simply call it from your own installer to handle the deps.
I question whether relying on CPAN is a good idea. The CPAN shell always fetches the latest version of a distro, rather than a specific version. If you're interested in ensuring repeatable installs, it's not the right tool.
What are your limitations for installing web apps? Can you log into the machine? Are all of the machines running the same thing? Are the people installing the web apps co-workers or random people from the general public? Are the people installing this sysadmins, programmers, web managers, or something else? Do you install by distributed an archive or checking out from source control?
For most of my stuff, which involves sysadmins familiar with Perl installing in control environments, I just use MakeMaker. It's easy to get it to do all the things you listed if you know a little about MakeMaker. If you want to know more about that, ask a another question. ;) Module::Build is just as easy, though, and the way to go if you don't already like using MakeMaker.
Module::Build would be a good way to go to handle lots of different situations if the people are moderately clueful about the command line and installing software. You'll have a lot of flexibility with Module::Build, but also a bit more work. And, the cpan tool (which comes with Perl), can install from the current directory and handle dependencies for you. Just tell it to install the current directory:
$ cpan .
If you only have to install on a single platorm, you'll probably have an easier time making a package in the native format. You could even have Module::Build make that package for you so the developers have the flexibility of Module::Build, but the installers have the ease of the native process. Sticking with Module::Build also means that you could create different packages for different platforms from a single build tool.
If the people installing the web application really have no idea about command lines, CPAN, and other things, you'll probably want to use a packager and installer that doesn't scare them or make them think about what is going on, and can accurately report problems to you automatically.
As Dave points out, using a real CPAN mirror always gets you the latest version of a module, but you can also make your own "fake" CPAN mirror with exactly the distributions you want and have the normal CPAN tools install from that. For our customers, we make "CPAN on a CD" (although thumb drives are good now too). With a simple "run me" script everything gets installed in exactly the versions they need. See, for instance, my Making my own CPAN talk if you're interested in that. Again, consider the audience when you think about that. It's not something you'd hand to the general public.
Good luck, :)
I'd recommend seriously considering a package system such as RPM to do this. Even if you're running on Windows I'd consider RPM and cygwin to do the installation. You could even set up a yum or apt repository to deliver the packages to remote systems.
If you're looking for a general installer for customers running any number of OSes and distros, then the problem becomes much harder.
Take a look at PAR.
Jonathan Rockway as a small section on using this with Catalyst in his book.