After ~8 years of using Scala, I took a detour to a small language you might have heard of called Go. It's not without its flaws (grow a real type system, boi!), but it does some things much better than Scala could ever hope to.
Go manages dependencies in source form, which any sensible engineer would consider terrifying until she discovers that storing one's dependencies in a vendor/ directory under source control is a "get out of jail free" card for cases when dependency resolution either becomes too complicated for its own good, or depends on flaky 3rd party resources, such as the network.
The latest version of Go's CLI tooling comes with a command called go mod vendor, which does the legwork of downloading the current module's dependencies into a vendor/ directory inside the project, which can subsequently be checked into source control. Setting aside discussions regarding the merits of aggressively and preemptively caching dependencies in this fashion, I would like to state for the record that this command is very convenient.
SBT is notorious for downloading dependencies into ~/.ivy2, which is more of a free-for-all cache shared by all of a user's projects rather than just one. There's a smaller cache in ~/.sbt, which is used by SBT itself as a Humpty Dumpty / Mr Potato Head scratch space. Both directories will be created & populated automatically if they don't exist, but neither is intended to be explicitly managed by the user. Both are internal implementation details of SBT and/or Ivy, and should not be messed with "unless you know what you're doing".
What I want (and now I'll be asking for things) is a sbt vendor command that would do the legwork of populating the unmanaged classpath with all of my project's dependencies. If it can also download all that's needed to run SBT itself into the same directory, that would be just peachy.
Is there a SBT plugin or some sequence of arcane incantations that can be used to accomplish that which I seek?
Before this question gets closed forever by an overzealous posse of moderators, I'm going to post here the hack which got me over this particular hump. As usual, the solution ended up being a shell script:
#!/bin/bash
root="$(readlink -f "$(dirname "$0")")"
sbt="$root/.sbt-launcher"
if [ ! -x "$sbt" ]; then
echo "$sbt does not exist or is not executable"
exit 1
fi
exec "$sbt" \
-ivy "$root/.ivy2" \
-sbt-dir "$root/.sbt" \
-sbt-boot "$root/.sbt/boot" \
-sbt-launch-dir "$root/.sbt/launchers" \
$#
Let's unpack this really quickly. First, the shell script is a wrapper for the real SBT launcher, which is located in the same directory and named .sbt-launcher. Any recent version should work; you too can download one from http://git.io/sbt.
My wrapper ensures that four flags are always passed to the real SBT launcher:
-ivy specifies a custom location for the Ivy cache.
-sbt-dir, -sbt-boot, and -sbt-launch-dir together force SBT to stop using the account-wide ~/.sbt directory as a dumping ground for SBT JARs and other things.
I saved this shell script as sbt inside my project, placed .sbt-launcher from http://git.io/sbt right next to it, and began using the wrapper instead of the real SBT. I then checked into source control the directories .ivy2 and .sbt which were created inside my project.
That's all it took. It's not an elegant solution, but it does well to isolate my CI/CD process from volatility in Internet artifact repositories.
Related
I have a simple CMake find module I've written, for a library of mine used by other projects. It's pretty simplistic, with its full text available here. Mainly there's one find_path() and one find_library(), and then some variables are set.
Now, I want CMake, when trying to find my package, to fall back on:
git-cloning or downloading the package/library from its GitHub repository,
Unpacking the archive, if it was a download
Building the package, either be using the running CMake itself somehow (the package has its own CMakeLists.txt), or by running an arbitrary shell command in the directory into which the packages was downloaded/cloned
The specifics of what happens post-download are less important to me than actually having a download fall-back.
How can I / how should I make this happen?
Notes:
Of course if the download/git clone fails, than finding the package has failed.
No need to worry about specific versions at the repo, although you can if you want to.
I'm using a couple of macros from the autoconf archive in my configure.ac. When aclocal is run, the macros are placed in aclocal.m4. Since this file is automatically generated, I typically wouldn't put it in source control. However, the autogeneration won't work unless the user has the macros installed on their computer in the first place (on Ubuntu I had to do apt-get install autoconf-archive). Is it typical practice to include aclocal.m4 in source control?
Edit: summary: Do not include aclocal.m4 in source control. It is acceptable to include acinclude.m4.
No, it is not best practice to do so. However, it is probably typical practice. Best practice and common practice often diverge widely when dealing with the autotools. In my opinion, the expectation is that a developer who is running the autotools is capable of satisfying the dependencies and making the macros available (eg, by installing autoconf-archive), so the file should not be included in the version control system. It is, however, perfectly acceptable to put the macros in acinclude.m4 and put that file in source control. When invoked, aclocal will look for definitions in acinclude.m4 so that the developer running aclocal (and this is the point that seems to throw a lot of projects; there are really only a small handful of people who should ever be invoking the autotools on a project, and everyone else should be building from a release distribution. If developers working on a project are not modifying the autotool meta-files, there is no reason for them to be running the autotools) does not need to install the autoconf archive.
I am working on perl module that I would like to submit in CPAN.
But I have a small query in regards to the directory structure of module.
As per the perlmonk article the module code directory structure should be as below:
Foo-Bar-0.01/Bar.pm
Foo-Bar-0.01/Makefile.PL
Foo-Bar-0.01/MANIFEST
Foo-Bar-0.01/Changes
Foo-Bar-0.01/test.pl
Foo-Bar-0.01/README
But when I am using the command, the structure is generated as below
h2xs -AX Foo::Bar
Writing Foo-Bar/lib/Foo/Bar.pm
Writing Foo-Bar/Makefile.PL
Writing Foo-Bar/README0
Writing Foo-Bar/t/Foo-Bar.t
Writing Foo-Bar/Changes
Writing Foo-Bar/MANIFEST
The article in question is advocating a considerably-older module structure. It certainly could be used, but it loses a lot of the advancements that have been put into place as far as good testing, building, and distribution practices.
To break down the differences:
modules have moved from the top level to the lib/ directory. This unifies the location where your module "lives" (i.e., the place where you work on the code and create the baseline modules to be tested and eventually distributed). It also makes it easier to set up any hierarchy that you need (e.g. subclasses, or helper modules); the newer setup will just pick these up. The older one may but I'm not familiar enough with it to say yes or no.
Makefile.PL in the newer setup will, when "make" is run. create a library called "blib", the *b*uild *lib*rary - this is where the code is built for actual testing. It will pretty much be a copy of lib/ unless you have XS code, in which case this is where the compiled XS code ends up. This makes the process of building and testing the code simpler; if you update a file in lib/, the Makefile will rebuild the file into blib before trying to test it.
the t/ directory replaces test.pl; "make test" will execute all the *.t files in t/, as opposed to you having to put all your tests in test.pl. This makes it far easier to write tests, as you can be sure you have a consistent state at the beginning of each test.
MANIFEST and Changes are the same in both: MANIFEST (built by "make manifest") is used to determine which files in the build library should be redistributed when the module is packaged for upload, and used to verify that a package is complete when it's downloaded and unpacked for building. Changes is simply a changelog, which you edit by hand to record the changes made in each distributed version.
As recommended in the comments on your question, using Module::Starter or Dist::Zilla (be warned that Dist::Zilla is Moose-based and will install a lot of prereqs) is a better approach to building modules in a more modern way. Of the two, the h2xs version is closer to modern packaging standards, but you're really better off using one of the recommended package starter options (and probably Module::Build, which uses a Build Perl script instead of a Makefile to build the code).
I have downloaded a project which uses sbt as its build system and I want to build it. You'd think it would be as simple as typing "sbt" or something, but no.
I thought I'd add a question for this because it can take literally hours to figure this out on your own. I'm not joking.
tl;dr:
sbt compile
If you want to run it:
sbt run
To see what other targets are available:
sbt tasks
To get some (other) help, but mostly targeted at commands typed from the sbt console (ie, running sbt without parameters):
sbt help
This all assumes sbt version >= 0.10.0. To see what version of sbt is in use, do:
grep sbt.version project/build.properties
If there's no such file, and there's a file with extension ".sbt" on the base directory (not the project directory), then it's >= 0.10.0. Of course, if the grep works, it should tell you the version.
First, you'll want to use sbt-extras, because that automatically downloads and uses the right version of sbt. Trying to use the wrong version of sbt (newer or older than what the project you're trying to build says it requires) won't necessarily work, and may cause strange errors.
Run it:
~/path/to/sbt-extras/sbt
Wait for it to start up and download everything. If you need to use an authenticated proxy, you'll need to edit the script to specify the username and password for the proxy.
Check the version of Scala that sbt thinks it needs to build against (at the end of the output, if everything worked). If this is OK, fine, you don't need to do anything. If it isn't, you can temporarily specify a version explicitly with ++, e.g.:
++2.8.1
(If you want to make this permanent, you can edit the build definition files, but as that involves making a change to files under version control, that might not be what you want to do.)
Now, if you are using an older version of sbt, don't skip the next step! You could get strange errors if you do.
update
Now you can build and test what you've built:
test
If you get an error "Filename too long", this is not an sbt-specific problem, it's a scala problem, which most frequently affects Ubuntu users (technically, for Unbuntu users it's generally related to home directories encrypted with encfs). If you are using Scala >= 2.9, edit the build to use the scalac command-line option that allows you to specify a maximum filename length. Otherwise, if you are on Linux, you can redirect the build to /dev/shm or /tmp, by running these commands in a shell prompt (don't background sbt with CTRL+Z on Unix, because it may appear to stop working properly):
rm -rf target
ln -s /dev/shm target
(you may have to execute these commands in project/build instead or as well.)
Actually, it's probably better, and may even be more secure, to create a subdirectory of /dev/shm or /tmp and use that instead.
The compilation result should appear in target. You might then want to run it, if it's something you can run:
run
If everything looks OK, you can optionally publish the result locally so that the result can then be picked up automatically by other sbt builds:
publish-local
I don't think I could explain it better that the Getting Started Guide could. Please read the first 6 parts of it, which shouldn't too long time, to get it up and running.
I posted this question looking for something similar to Buildout for Perl. I think Shipwright is what I'm looking for but I'm not really sure. I've played around with it and I created a project, imported all of my source and dependencies and I've exported everything to a vessel then the documentation sort of just stopped. What do I do with a shipyard vessel? Do I do my actual development work in the vessel, or do I do my development in the Shipyard? I'm assuming that the vessel is only for deployment, but how do I actually deploy a vessel to a web server (say I'm using linux, apache and just running straight cgi).
Is Shipwright the right thing for what I'm trying to accomplish or is there something else that would be more appropriate? Ideally I could use Shipwright similar to how I use Buildout. I use Buildout to create a nice isolated environment for my development, and also I use Buildout when deploying to live servers to manage all of my application's dependencies.
EDIT: Here are the highlights of what I can do with Buildout that I would like to be able to do in Perl.
With Buildout, I have a file in my codebase that lists dependencies (which for Perl would either be CPAN modules or other source repositories). I can run a bootstrap script that will fetch all of those dependencies and drop them into a directory within my project and NOT install them at a system level. Buildout also creates utility scripts which can do anything you want (run tests, other command line tools, anything really) and those scripts explicitly add the dependencies to the path so that as my scripts are running all of my dependencies are available to be imported.
What this really does very well is that it allows me to manage my dependencies without having to ever install anything at a system level. Which makes changing from one version to another very easy. Also, it allows me to have multiple Buildout projects running on the same system using different versions of the same module. Finally, one huge benefit is that with Buildout's directory structure, I can just commit the dependencies to source control and to deploy to a new machine I just need to do a checkout and all of my dependencies are already satisfied without having to touch anything installed at a system level.
I don't think you'll find anything exactly like Buildout in Perl, but you could put together a couple of things that would do the trick.
You could use a standard Build.PL script for Module::Build for managing your dependencies and having commands to run tests, etc.
Then you could use cpanminus to do the installation of those dependencies into a local (non-system) directory.
Then you might be able to use Shipwright to do the bundling and deployment of the project with these now-local dependencies.