I'm looking for an advice about the software that did not catch my eye during a few days of searches.
Is there any software solution for deployment of multiple projects at once that operates at the level of source-building package manager (like ports, portages, or nix), but could be resided locally?
As for details, we have few loosely connected software projects with the following traits:
Projects are written mostly on C/C ++ and Python, but more languages are represented as a minor mixture (Haskell, Rust, Perl)
Projects are grouped together within git superprojects with certain build/deployment presets carefully tuned for particular environments (and adapted for particular purposes by subset of boolean-like options)
We have already done with well-elaborated CMake build scripts for C++ projects that do support options, build configurations, exporting targets, and so on. It would be expensive to switch from it now.
We are forced to deal with various Linux distributions (from Gentoo and Ubuntu to Debian and CentOS).
We need an unifying build&deployment tool for various environments. The CMake does not integrate well with non-compiling languages (e.g. does not natively support local Python installations via virtualenv).
Instead of changing the things we already developed, I would like to use them in the manner the OS package manager does. For my vision, it should be something pretty similar to the so-called meta-build tool. In fact, the Gentoo Portages are pretty close:
Easy customization with simple boolean options (useflags, profiles)
Delegation of the building procedures to the reliable and customizable tools designed and well-tested especially for this purpose (CMake, autotools, Bazel, etc.)
Offers an ability to change the target compiler installation and specify the building process in a clear declarative way with a standardized instructions set.
Portages can not be ran locally, though (and have other unrelated flaws).
I have to become very confident before switching the whole build system to something like Meson or Bazel or whatever else I could find up to the moment.
Update
To be more specific, I could refer to what we have done up to the moment. The one of the superprojects we're maintaining deals with particular scientific experiment:
All the sub-projects are listed as git submodules
A simple BASH-script maintaining the entire build-install-deploy lifecycle
The presets referring particular settings for a concrete environment are represented as BASH sources and are included by this maintaining shell script.
As few days more had come with no result, I coming up with intention to write my own solution based on experience we already gained due to these shell scripting activity.
For a similar task Bob was developed some time ago. If you still need a build system it might be worth a look. There are some small tutorials available as well as a complex basement example build a linux from scratch.
Basically it is just a environment for a controlled execution of bash scripts with dependency and variant handling, CI integration, IDE support,..
Take a look at Sparrow - Linux scripting platform, it supports projects hierarchy and tasks - configurable scripts, where configuration is made in declarative way in Yaml, Json formats. This hopefully might address many if not all the needs you've mentioned here.
Disclosure - I am the tool author.
I'm using a couple of macros from the autoconf archive in my configure.ac. When aclocal is run, the macros are placed in aclocal.m4. Since this file is automatically generated, I typically wouldn't put it in source control. However, the autogeneration won't work unless the user has the macros installed on their computer in the first place (on Ubuntu I had to do apt-get install autoconf-archive). Is it typical practice to include aclocal.m4 in source control?
Edit: summary: Do not include aclocal.m4 in source control. It is acceptable to include acinclude.m4.
No, it is not best practice to do so. However, it is probably typical practice. Best practice and common practice often diverge widely when dealing with the autotools. In my opinion, the expectation is that a developer who is running the autotools is capable of satisfying the dependencies and making the macros available (eg, by installing autoconf-archive), so the file should not be included in the version control system. It is, however, perfectly acceptable to put the macros in acinclude.m4 and put that file in source control. When invoked, aclocal will look for definitions in acinclude.m4 so that the developer running aclocal (and this is the point that seems to throw a lot of projects; there are really only a small handful of people who should ever be invoking the autotools on a project, and everyone else should be building from a release distribution. If developers working on a project are not modifying the autotool meta-files, there is no reason for them to be running the autotools) does not need to install the autoconf archive.
Whenever I work on a system of any flavor that has a particular way of handling package management, I try to stick with that standard for managing my Perl modules. "When in Rome, etc."
For example, on a Win32 system using ActivePerl, I use PPM for everything and use the great PPM::Make. On a RedHat system I prefer to use RPMs.
Now I am working on a Debian system, and find myself in need of a way to turn an arbitrary CPAN or CPAN style distribution into a deb.
Google shows options like dh-make-perl, CPANPLUS::Dist::Deb and CPAN::Packager::Builder::Deb.
Does anyone with experience with these different tools have any recommendations as to what to use or avoid?
What's the best way to handle building deb files from standard CPAN modules?
Update:
I found an article by Hans Dieter Piercy on this subject - he suggests, for his own needs, CPANPLUS tools. Under some circumstances he recommends dh-make-perl. Jeremiah Foster (who wrote the article brian d foy points to) responds to HDP and makes a case for dh-make-perl.
There's also a post on idimmu.net that describes using dh-make-perl.
ATM, I'm leaning toward dh-make-perl, since that has been thrice recommended (brian d foy as proxy for Jeremy Foster, the idimmu.net author and hillu) vs once for CPANPLUS
dh-make-perl does a good job in taking care of the repetitive and heavy lifting and guessing information from the sources. It has worked correctly for almost all of the CPAN modules that I have packaged up as Debian packages (official or for in-house use only).
That said, the resulting package should only be considered as a starting point for a proper Debian packages. dh-make-perl puts warning notes into the automatically generated such as debian/control (i.e. description of the package and dependencies) and debian/copyright (licensing information).
In response to Manni, I believe that it is a great idea to work with the tools that the OS or distribution provides for package management, not against them. In the case of Debian, this means putting stuff into .deb packages and installing those. Perl's build tools and CPAN do a great job of providing a cross-platform build environment and for distribution of the source code, but compared to package management tools in modern Linux distributions, they perform suboptimally, simply because extra manual intervention is often required that is less easily automated across multiple machines than rolling up a package.
(For one-off and test installations, installing into /usr/local/ and using stow(8) as a poor man's package manager may be okay.)
Even if you are just building the packages for your own use, consider contacting the Debian Perl Group and have someone sponsor an upload to Debian if you feel that the module in question is of use for other people.
I suggest you ask the Debian Perl Maintainers group, rather than here on SO. Just mail the address shown as maintainer on any odd package:
Debian Perl Group <pkg-perl-maintainers#lists.alioth.debian.org>
Back in the day, I added a few modules to Debian, and just 'did it by hand'. I still maintain some. That isn't hard either. but the group now maintains way more package, and has tools.
Jeremiah Foster published an article about turning Perl distros into Debian packages in the Spring 2009 issue of The Perl Review.
There is a very good step by step here as well. (also with links to other good resources and some decent comments. [it is dated 2005, but still mostly relavent and many comments much more recent])
http://www.debian-administration.org/articles/78
here is the debian perl policy (also linked to in article)
http://www.debian.org/doc/packaging-manuals/perl-policy/
You won't like this, but I really think that you should not do this at all. The various Perl Debian packages aren't for developers that need certain Perl modules on their machines. They were built because other applications need them and users want or might want those applications.
Please take a look at the answers to this question before doing something that you probably should not be doing.
I have several applications that I wish to deploy using rpm. Some of the files in my application deployments override files from other deployed packages. Simply including the new files in the deployment package will cause rpm conflicts.
I am looking for the proper way to use rpm to update/replace already installed files.
I have already come up with a few solutions but nothing seems quite right.
Maintain custom versions of the rpms containing the original files.
This seems like a large amount of work for a relatively small reward even though it feels less like a hack than some of the other possible solutions.
Include the files in the rpm with another name and copy them over in the post section.
This would work but will mean littering the system with multiple copies of the files. Also it means additional maintenance in the rpm build spec for each file.
Use wget in the post section to replace the original files from some known server.
This is similar to the copy technique but the files wouldn't even live in the rpm. This might act like a nice central configuration authority though.
Deploy the files as new files, then use symlinks to override the originals.
This is also similar to the copy technique but with less clutter. The problem here is that some files don't behave well as symlinks.
To the best of my knowledge, RPM is not designed to permit updating / replacing existing files, so anything that you do is going to be a hack.
Of the options you list, I'd choose #1 as the least bad hack if the target systems are systems that I admin (as you say, it's more work but is the cleanest solution) and a combination of #2 and #4 (symlinks where possible, copies where not) if I'm creating the RPMs for others' systems (to avoid having to distribute a bunch of RPMs, but I'd make it very clear in the docs what I'm doing).
You haven't described which files need to be updated or replaced and how they need to be updated. Depending on the answers to those questions, you may have a couple of other options:
Many programs are designed to use a single default configuration file and also to grab configuration files from a .d subdirectory. For example, Apache uses /etc/httpd/conf/httpd.conf and /etc/httpd/conf.d/*.conf, so your RPMs could drop files under /etc/httpd/conf.d instead of modifying /etc/httpd/conf/httpd.conf. And if the files that you need to modify are config files that don't follow this pattern but could be made to, you can suggest to the package maintainers that they add this capability; this wouldn't help you immediately but would make future releases easier.
For command-line utilities like sendmail and lpr that can be provided by multiple packages, the alternatives system (see man alternatives) permits more than 1 RPM that provides these utilities to be installed side by side. Again, if the files that you need to modify are command-line utilities that don't follow this pattern but could be made to, you can suggest to the package maintainers that they add this capability.
Config file changes on systems that you administer are better managed through a tool like Cfengine or Puppet rather than through custom RPMs. I think that Red Hat favors Puppet.
If I were creating the RPMs for systems I don't administer, I'd consider using a third-party tool like Bitrock and dumping all of my stuff under /opt just so I wouldn't have to stomp on files installed by other admins' RPMs.
Edit (2019): Nowadays, Software Collections offers a useful alternative. You can create packages that install somewhere under /opt, and the Software Collections tools offer a standardized way for users to opt in to using those instead of whatever's normally installed under /usr. Red Hat uses this to distribute newer versions of tools for their otherwise stable and long-lived (i.e., older) Red Hat Enterprise Linux distributions.
You can also execute rpm -U --replacefiles --replacepkgs ..., which will give you what you want.
See here for more info on RPM %files directives:
http://www.rpm.org/max-rpm/s1-rpm-inside-files-list-directives.html
You can use the arguments from the %post and %pre sections in the RPM scriptlets to determine if you are installing, upgrading or removing packages.
If $1 is 0 - then we're removing old stuff. Targeting 0 packages installed.
If $1 is 1 - then we're installing new stuff. Targeting a total of 1 package to be installed.
If $1 is 2 or more - then we're upgrading this package and $1 represents the number of packages already installed.
These sections help with managing files among the versions.
Keep track of what you're doing between versions and consider what one might do if they were to skip a version or two.
Have consideration for these things and you should be good to go!
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
Although several thousand Emacs Lisp libraries exist, GNU Emacs, until version 24.1 did not have an (internal) package manager.
I guess that most users would agree that it is currently rather inconvenient to find, install and especially keep up-to-date Emacs Lisp libraries.
Pages that make life a bit easier
For versions of Emacs older than 24.1:
Emacs Lisp List - Problem: I see dead people (links).
Emacswiki - Problem: May contain traces of nuts (malicious code).
Emacsmirror - The package repository I am working on. Problem: No package manager supports it natively yet.
Some package managers
It's not that nobody has tried yet. (Some of these did not exist when this question was asked.)
auto-install
borg.el - Assimilate Emacs packages using Git submodules.
el-get.el - Supports many sources.
elinstall.el
epackage aka DELPS - Debian packaging concepts applied to Emacs Lisp packages.
epkg.el - This is now just a tool for browsing the Emacsmirror.
install.el
install-elisp.el
jem-pkg.el
package.el - ELPA. Seems like it will be included in Emacs 24.
UPDATE -- package.el is included in GNU Emacs, starting with version 24.1
pases.el
pelm - Command line installer; using php.
plugin.el
straight.el - Recent and experimental, has not reached 1.0 release yet.
use-package.el
XEmacs package manager
package has been included in the Emacs trunk. epkg is not ready yet and also currently not available. At least install-elisp, plugin and use-package do not seem to be actively maintained anymore.
I have created a git repository containing all these package managers as submodules.
Some utilities that might be useful
Package managers could use these utilities and/or they could be used to maintain a mirror of packages.
date-calc.el - Date calculation and parsing routines.
ell.el - Browse the Emacs Lisp List.
elm.el, elx.el, xpkg.el - Used to maintain the Emacsmirror.
genauto.el - Helps generate autoloads for your elisp packages.
inversion.el - Require specific package versions.
loadhist.el, lib-requires.el, elisp-depend.el - Commands to list Emacs Lisp library dependencies.
project-root.el - Define a project root and take actions based upon it.
strptime.el - Partial implementation of POSIX date and time parsing.
wikirel.el - Visit relevant pages on the Emacs Wiki.
Discussions about the subject at hand
emacs-devel 20080801
comp.emacs 20021121
RationalElispPackaging
The question (finally)
So - I would like to know from you what you consider important/unimportant/supplementary etc. in a package manager for Emacs.
Some ideas
Many packages (the Emacsmirror provides that largest available collection of packages, but there is no explicit support in any package manager yet).
Only packages that have been tested.
Support for more than one package archive (so people can choose between many/tested packages).
Dependency calculated based on required features only.
Dependencies take particular versions into account.
Only use versions that have been released upstream.
Use versions from version control systems if available.
Packages are categorized.
Packages can be uninstalled and updated not only installed.
Support creating fork of upstream version of packages.
Support publishing these forks.
Support choosing a fork.
After installation packages are activated.
Generate autoload files.
Integration with Emacswiki (see wikirel.el).
Users can tag, comment etc. packages and share that information.
Only FSF-assigned/GPL/FOSS software or don't care about license.
Package manager should be integrated be distributed with Emacs.
Support for easily contacting author.
Lots of metadata.
Suggest alternatives before installing a particular package.
I am hoping for these kinds of answers
Pointers to more implementations, discussions etc.
Lengthy descriptions of a set of features that make up your ideal package manager.
Descriptions of one particular desired/undesired feature. Feel free to elaborate on my ideas from above.
Surprise me.
I'm still learning Emacs, so I haven't had a chance to look into package managers, but a great feature would be to inform the user that the package is available if they try to use it but it's not on their system. For example, I wanted to edit a PHP file on a server once, and I tried
M-x php-mode
and Emacs was all like
M-x php-mode [no match]
when it should have been like
php-mode available from ftp.gnu.org. install? (y/n)
and then it would have installed and loaded up php-mode for me. That would have made my day right there.
Automatic publishing from version control
I'd love to see a standard, central, and single Emacs package manager. Right now, I'd put my money on ELPA, but there is still a long way to go.
The biggest thing that would help an Emacs package manager would be to make it super trivial to publish packages. In my opinion, I'd like to see this happen in combination with a version control system like git on a central hosted platform like GitHub -- something that would make it easy for authors to publish their packages and would make it easy for others to contribute back.
Similar to how GitHub (used to) make it easy to publish RubyGems, I'd like to see something similar in an Emacs package manager. For example, tag your repository with "vX.Y.Z" and have your elisp goodness automatically available to all.
The added benefit of using a popular backend like GitHub is that you'd immediately get a lot of exposure which should help drive its success.
What I expect most is that everything useful is on it, and works well. This requires you (or a team of maintainers) to aggressively pursue packaging everything for it, and doing whatever that involves — emailing every author of a useful package, and so on.
For instance, the reason Debian (and its derivatives: Ubuntu etc.) is so good is that you can happily use your system without ever having to install something outside the repositories, and that everything on it is thoroughly tested. The actual features of the package manager are important, but secondary to the managed packages themselves.
Easy configuration synchronization: I, like many people, use Emacs on many different computers and servers, some of them my own and some not. It would be amazing if the package manager had some sort of file which I could transfer from one computer to another; then, on the latter computer, the package manager would bring my Emacs into the state I like it in -- all the packages installed and configurations set. Combined with the ability to be able to easily install either site-wide (if one has root permissions) or as a single user, I could synchronize all of Emacsen everywhere.
I'm nearly positive that the best solution involves submitting more packages to ELPA and adding multi-source support to package.el. The Emacs maintainers have said that they would consider including package.el in version 24 as long as it pointed to an FSF repository by default.
Of course, submission also needs to be an automated process too; the current method of mailing the ELPA maintainer only works on a small scale.
No matter how this is done, the most important thing in my opinion is that it should be trivial to submit packages to the repository. At the same time, we do not want those packages to be instantly available, to guard against malicious code(and due to licensing issues). Unless there is a "trust" system in place, based on crypto signatures.
Also useful:
"metapackages", to install several packages at once.
In the same way, we should be able to install a set of elisp files, for maintainability
"Broken" packages should not be allowed to disrupt Emacs startup. This is easy and I have it implemented in my own .emacs
Ability to install files other than scripts. This is often overlooked, but very useful. You'd be able to, for instance, ship images, for icons, toolbars, etc.
Versioning:package X requires package Y > 1.0
Testing: perform basic sanity checks, testing for conflicts (keybindings, function redefinitions, functions that are expected to be present but aren't, etc).
BUG TRACKING: I can't stress the importance of this enough. Having a centralized place to report package bugs (and being able to track them) is extremely important to assure the quality of the packages.
Some sort of compressed archive seems to be best to do some of the above.
So far, a much improved ELPA seems the way to go.
I once spent some time writing a small package manager for Emacs.
http://gmarceau.qc.ca/plugin.el
I wrote:
Plugin is my attempt at creating a
package manager for Emacs. Plugin
will automatically downloads Emacs
extensions, unpacks them in a
directory, adds that directory to the
load-path, generates auto-load
annotations, and modify your dot-emacs
file. The auto-load annotations are a
little-known feature of Emacs. Once
they are generated, Emacs extensions
load quickly and incrementally, which
is really nice if you have as many
extensions installed as I do.
You will need two library files to get it to run, loop-constructs.el and record.el
I think the hackers for the iPhone got quite close to what I want, as does Ubuntu's "apt".
I like to be able to:
add
remove (package only)
remove user settings
view documentation
upgrade ( after reading the change log)
add new archive ( aka add repository )
see dependencies
see version
search for name, keyword
browse by (date added, date modified, name)
save all installed packages & settings
load set of packages & settings
I would like a main set of things that all work nicely and are the recommended way of doing whatever. Then a global set where everything working gets in. Then the ability for anyone to host their own archive.
It would be nice if this was all tied into git/svn/whatever so that you could install old versions. Make your own patches by forking off etc etc etc....
Besides the mentioned above, i expect something like debian, and other repositories - set of the stable, experemental, untested packages. Ability to add my own repositories - i use lot of the packages directly from VCS, so it could be useful to create my own packages
I think that the package manager should take a lot of inspiration from Rubygems. I also think that it should have a site like Gemcutter.
A central repository could also be nice (like Emacsmirror). This however may not be necessary if a site like Gemcutter exists that collects all packages.
I think these things are important for this to work.
Central location of some kind that collects all packages
Easy to add packages
Easy to maintain packages
Easy to contribute to other packages
Easy to install, uninstall and update packages
Possibility to add package dependencies
Common structure for all packages
So a package manager like Rubygems with a site like Gemcutter and a central repository like Emacsmirror (preferably on Github because of it's social coding) would do Emacs really good.
All in all I think that much inspiration should be taken from Rails and how Rails handles Gems.
I don't know how fresh this question is...
but the model I would like to see is CPAN. I also don't know Rubygems but it sounds similar to CPAN.
CPAN is a perl archive + library management system. When I need to write a perl program that requires... FTP or SOAP or JSON or XML or ZIP, or...etc, I can run the CPAN package manager, select the requisite package for download, view and verify the dependencies, then install everything. CPAN is mirrored .."everywhere".
CPAN works wonderfully for my purposes, and something similar for emacs would be nice to have. It also supports building C/C++ code on demand.
That's what I would like to see in emacs.
Some further comment on requirements.
explicit download of packages. No auto install. No invisible downloads. I want to ask for new libraries or new function.
I should be able to list the name/version/timestamp of installed packages.
If my friend gives me his list, I should be able to diff his emacs state against mine.
check-for-updates function. What updates are available? What do they fix?
depedency checking, verification, and download. If I install csharp-mode and it requires v5.0.28 of cc-mode, then it should confirm with me, that I must also download cc-mode.
there should be some sort of community ranking of these packages, like ranking torrents on isohunt. I want to see if a package has 3 upvotes or 3000.
"transactional" behavior. If an install goes boom, it must unwind to a last-known-good state.
failsafes. If I've put custom mods in linum.el, it should refuse to install a new version over my changes, unless I explicitly allow it. It should warn me before even starting. Do this with checksums/md5's over the existing install.
have the option of running some packages from compressed archives, like zip files. So I never have any doubt that I have not updated any of the embbedded elisp.
ability to use mirrored hosts for package distribution.
all this function should be accessible through M-x library-manageemnt or something.
Finally, it would be nice to have a way to segregate or organize libraries of functions. Hierarchical namespaces. Emacs' flat namespace is very dated. This is sort of independent but complementary to the core function of package management. I'm not a lisp guru so I don't know how hard this would be; maybe there is already a way to do it.
Package managers don't offer anything I value w.r.t. single-file elisp packages with simple dependencies: adding and deleting from site-lisp has never caused problems. It's packages that depend on external programs (e.g., ispell), multi-file packages (e.g., auctex, org-mode) that can be tricky. Can't think of any single-file elisp package with nontrivial dependencies, offhand.
For these, short of a package manager, I'd like emacs' elisp-packages to acquire test suites which can be run en masse, and which provide useful information in the event of dependency failures.