I have a fairly normal Scala project currently being built using Maven. I would like to support both Scala 2.9.x and the forthcoming 2.10, which is not binary or source compatible. I am willing to entertain converting to SBT if necessary, but I have run into some challenges.
My requirements for this project are:
Single source tree (no branching). I believe that trying to support multiple concurrent "master" branches for each Scala version will be the quickest way to miss bugfixes between the branches.
Version specific source directories. Since the Scala versions are not source compatibile, I need to be able to specify an auxiliary source directory for version specific sources.
Version specific source jars. End users should be able to download the correct source jar, with the correct version specific sources, for their version of Scala for IDE integration.
Integrated deployment. I currently use the Maven release plugin to deploy new versions to the Sonatype OSS repository, and would like to have a similarly simple workflow for releases.
End-user Maven support. My end users are often Maven users, and so a functional POM that accurately reflects dependencies is critical.
Shaded jar support. I need to be able to produce a JAR that includes a subset of my dependenices and removes the shaded dependencies from the published POM.
Things I have tried:
Maven profiles. I created a set of Maven profiles to control what version of Scala is used to build, using the Maven build-helper plugin to select the version specific source tree. This was working well until it came time to publish;
Using classifiers to qualify versions doesn't work well, because the source jars would also need custom classifiers ('source-2.9.2', etc.), and most IDE tools wouldn't know how to locate them.
I tried using a Maven property to add the SBT-style _${scala.version} suffix to the artifact name, but Maven does not like properties in the artifact name.
SBT. This works well once you can grok it (no small task despite extensive documentation). The downside is that there does not seem to be an equivalent to the Maven shade plugin. I've looked at:
Proguard. The plugin is not updated for SBT 0.12.x, and won't build from source because it depends on another SBT plugin that has changed groupIds, and doesn't have a 0.12.x version under the old name. I have not yet been able to work out how to instruct SBT to ignore/replace the plugin dependency.
OneJar. This uses custom class loading to run Main classes out of embedded jars, which is not the desired result; I want the class files of my project to be in the jar along with (possibly renamed) class files from my shaded dependencies.
SBT Assembly plugin. This can work to a degree, but the POM file appears to include the dependencies that I'm trying to shade, which doesn't help my end users.
I accept that there may not be a solution that does what I want for Scala, and/or I may need to write my own Maven or Scala plugins to accomplish the goal. But if I can I'd like to find an existing solution.
Update
I am close to accepting #Jon-Ander's excellent answer, but there is still one outstanding piece for me, which is a unified release process. The current state of my build.sbt is on GitHub. (I'll reproduce it here in an answer later for posterity).
The sbt-release plugin does not support multi-version builds (i.e., + release does not behave as one might like), which makes a sort of sense as the process of release tagging doesn't really need to happen across versions. But I would like two parts of the process to be multi-version: testing and publishing.
What I'd like to have happen is something akin to two-stage maven-release-plugin process. The first stage would do the administrative work of updating Git and running the tests, which in this case would mean running + test so that all versions are tested, tagging, updating to snapshot, and then pushing the result to upstream.
The second stage would checkout the tagged version and + publish, which will rerun the tests and push the tagged versions up to the Sonatype repository.
I suspect that I could write releaseProcess values that do each of these, but I'm not sure if I can support multiple releaseProcess values in my build.sbt. It probably can work with some additional scopes, but that part of SBT is still strange majick to me.
What I currently have done is changed the releaseProcess to not publish. I then have to checkout the tagged version by hand and run + publish after the fact, which is close to what I want but does compromise, especially since the tests are only run on the current scala version in the release process. I could live with a process that isn't two-stage like the maven plugin, but does implement multi-version test and publish.
Any additional feedback that can get me across the last mile would be appreciated.
Most of this is well supported in sbt within a single source tree
Version specific source directories are usually not need. Scala programs tends to be source compatible - so often in fact that
crossbuilding (http://www.scala-sbt.org/release/docs/Detailed-Topics/Cross-Build) has first class support in sbt.
If you really need version specific code, you can add extra source folders.
Putting this in your build.sbt file will add "src/main/scala-[scalaVersion]" as a source directory for each version as you crossbuild in addition to the regular "src/main/scala".
(there is also a plugin available for generating shims between version, but I haven't tried it - https://github.com/sbt/sbt-scalashim)
unmanagedSourceDirectories in Compile <+= (sourceDirectory in Compile, scalaVersion){ (s,v) => s / ("scala-"+v) }
version specific source jars - see crossbuilding, works out of the box
integrated deployment - https://github.com/sbt/sbt-release (has awesome git integration too)
Maven end-users - http://www.scala-sbt.org/release/docs/Detailed-Topics/Publishing.html
Shaded - I have used this one https://github.com/sbt/sbt-assembly which have worked fine for my needs.
Your problem with the assembly plugin can be solved by rewriting the generated pom.
Here is an example ripping out joda-time.
pomPostProcess := {
import xml.transform._
new RuleTransformer(new RewriteRule{
override def transform(node:xml.Node) = {
if((node \ "groupId").text == "joda-time") xml.NodeSeq.Empty else node
}
})
}
Complete build.sbt for for reference
scalaVersion := "2.9.2"
crossScalaVersions := Seq("2.9.2", "2.10.0-RC5")
unmanagedSourceDirectories in Compile <+= (sourceDirectory in Compile, scalaVersion){ (s,v) => s / ("scala-"+v) }
libraryDependencies += "joda-time" % "joda-time" % "1.6.2"
libraryDependencies += "org.mindrot" % "jbcrypt" % "0.3m"
pomPostProcess := {
import xml.transform._
new RuleTransformer(new RewriteRule{
override def transform(node:xml.Node) = {
if((node \ "groupId").text == "joda-time") xml.NodeSeq.Empty else node
}
})
}
I've done something similar to this with SBT as an example:
https://github.com/seanparsons/scalaz/commit/21298eb4af80f107181bfd09eaaa51c9b56bdc28
It's made possible by SBT allowing all the settings to be determined based on other settings, which means that most other things should "just work".
As far as the pom.xml aspect I can only recommend asking the question in the SBT mailing list, I would be surprised if you couldn't do that however.
My blog post http://www.day-to-day-stuff.blogspot.nl/2013/04/fixing-code-and-binary.html contains an example of a slightly more finegrained solution for attaching different source directories; one per major S. Also it explains how to create scala-version-specific code that can be used by not-specific code.
Update 2016-11-08: Sbt now supports this out of the box: http://www.scala-sbt.org/0.13/docs/sbt-0.13-Tech-Previews.html#Cross-version+support+for+Scala+sources
Related
New to Scala and sbt; coming from a python world, and rather confused by library dependencies, versioning, and what sbt can and cannot automatically download (i.e. when a .jar needs to be manually placed in /lib).
I want to use classes from the Scala geoscript project (http://geoscript.org/scala/quickstart.html) in my Scala app.
I'm using IntelliJ with Scala 2.11.8.
Can I ultimately do something like:
libraryDependencies += "org.geoscript" % "some-artifact" % some=version
Or is this going to be an "unmanaged dependency"? And if so, what's the cleanest way to do that?
If a dependency jar is published to Maven Central and some other repositories, sbt will automatically be able to resolve and download it when declaring it as libraryDependency. This is a "managed dependency". Usually a library's documentation will tell you the "coordinates" (group id, artifact id, version) you need to install it.
From the page you linked and the Maven search, it looks like Geoscript is not published, so you will have to add it to your /lib folder in the project. That should be enough to put it in the classpath. You may need to refresh the project for IntelliJ to pick it up and offer completions.
With the recent issue reported against sbt-cross-building (that errors when .value macro is used for attributes), I wonder what are the use cases for the plugin for sbt 0.13.2 and on? Isn't the built-in crossScalaVersions facility enough (as described in Cross-building)?
What are the use cases that necessitate the existence of the sbt-cross-building plugin in sbt 0.13 and on?
sbt-cross-building is like a musician's music — a plugin author's plugin. Since sbt now retains binary compatibility in between minor releases, there's now less demand for sbt-cross-building, but it used to be a regular ritual among the sbt plugin authors to republish every time sbt release comes out. So when this plugin came out, it used to be my favorite plugin of all time.
There was a talk even about merging the plugin into the sbt mothership, but instead previous owners of sbt came up with a mechanism to cross publish sbt 0.13 plugins from sbt 0.12.4:
sbtVersion in Global := "0.13.0-RC1"
scalaVersion in Global := "2.10.2"
This worked well enough that sbt-cross-publishing kind of missed the wave during 0.12 to 0.13 jump. One of the neat features on sbt-cross-publishing is that you can create a custom Scala source directories that are used for specific sbt versions. This allowed a single code base of plugins to be used even when the source-level compatibility was broken from the sbt side.
The concept of cross building across sbt versions is useful one just like cross building across Scala versions. At the same time, partially due to the stasis during the minor releases, the major release jump is big enough that many of the plugins I maintain required cutting a new version to take advantage of new Scala version, library, DSL syntax etc.
Disclaimer: I'm scala beginner. All defaults works nice for me, but whenever I want to have custom layout/build I run into a problem.
So, as part of my build I need to pull .war(web app) from project A and .jar(jetty launcher) from project B into some directory of project C(tanuki service wrapper).
Can anybody please provide an an example how to do this in the most effective way.
Not sure if it works with war files, but for making jars locally available you could use sbt's publish-local command. Say you have an sbt project "mylibrary" and another sbt project "mymain". If you locally publish "mylibrary.jar", you can add it as a dependency to "mymain" just like you add any other sbt-managed dependency, i.e., by adding a line such as
libraryDependencies += "foo.bar.com" %% "mylibrary" % "0.1-SNAPSHOT"
to the build.sbt of "mymain".
If that is not possible you might want to write an sbt plugin/command that copies the files into a given directory. I don't have experience with extending sbt, so I can't help with that, but other stackoverflowers surely can :-)
EDIT: (addressing a comment by the OP)
No, I don't have a particular Sbt tutorial. If I need help I turn to the usual suspects, the wiki, the mailing list, Stackoverflow, Sbt's source code. Sbt has an IO package which offers a copyFile method, which, according to this thread, comes in handy. Searching for 'copying files' on the mailing list also yields other results that might help you.
I would like to give IntelliJ IDEA a try, but I have no idea how to get going.
I am simply trying to create a new project that uses Finagle Echo Server, hosted on github, as starting point.
Assuming I'm starting with a clean install on Mac. I installed IDEA and added the Scala and SBT plugins. What steps should I take to create a project, that uses Finagle and run the code in the http server example?
PLEASE help. I realize my question sounds like a stupid question, but there are so many different approaches to working with Scala projects from SBT command line, Scala-IDE, Idea, etc, that I simply don't know how to get a comfortable development environment going.
A manual solution that doesn't require you to use SBT for your project might be more straightforward, given the SBT versioning issues. You'll still use SBT to build finagle, though.
Install the SBT runner script per step 1 above. (It can handle SBT 0.7 projects too).
Manually git clone git://github.com/twitter/finagle.git.
cd to the finagle directory and type "sbt package". Its dependencies should end up under lib_managed, and it should produce the finagle jars themselves under target/ or some such (just note the locations in the command output).
Create an IDEA project from scratch, and manually add dependencies to the finagle jars and their dependencies (under Project Structure->Dependencies).
This answer assumes that you want to use SBT. Also, I should qualify that this is my usual procedure, but I haven't confirmed that it works with finagle in particular.
0. Install IDEA, with Scala and SBT plugins. (Done by the OP; here for others)
1. Install SBT (automatic method). Copy this handy sbt runner script to a convenient location (or, if you want to keep it up to date, git clone https://github.com/paulp/sbt-extras.git and symlink the script into ~/bin), and make sure it's executable. It will automatically download whatever it needs based on the sbt.version specified in your build.properties.
2. Install sbt-idea. sbt-idea is an SBT plugin (not an IDEA plugin) that generates IDEA module files from an SBT project. It's convenient to install this globally since it's not project-specific. You don't have to download anything manually; just add this to ~/.sbt/plugins/build.sbt:
resolvers += "sbt-idea-repo" at "http://mpeltonen.github.com/maven/"
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "0.11.0")
3. Create SBT project. Create a directory for your project, and a "project" directory within it. Create project/Build.scala as follows:
import sbt._
object MyBuild extends Build {
lazy val root = Project("root", file(".")) dependsOn finagle
lazy val finagle = RootProject(uri("git://github.com/twitter/finagle.git"))
}
See the SBT documentation for lots more options re configuring your project. Note we have to use the Full Configuration here (not just build.sbt) in order to express the github dependency.
It's also a good idea to create project/build.properties:
sbt.version=0.11.2
project.version=0.1
build.scala.versions=2.9.1
4. Generate IDEA project. cd to the directory containing the sbt-based project. type "sbt gen-idea". If all goes well, the directory will now have ".idea" and ".idea_modules" subdirectories.
5. Open the project in IDEA. It may be necessary to fix the target JDK version in the project settings. Aside from that, the project should be ready to go, with all source paths, library dependencies, etc. properly configured.
I have a Scala project that depends on a number of libraries built
against 2.9.0-1. I'm interested in applying a new small patch from
upstream that fixes the Scala REPL to no longer execute each line in a
new thread. I should be able to download the Scala sources, apply the
patch, and ant-build everything myself, but is there any way to make
sbt (or at least the console command) use this build?
It's also important to be able to reproduce this environment on all
our dev boxes, so we're interested in minimizing out-of-band hacks. We
have a corp maven repository, but not sure how to "override" things
such that Scala 2.9.0-1 is fetched from here (if that's even the best
approach for this problem).
Publish the custom Scala jars to the corporate repository with a unique version and use that unique version as the value for your scalaVersion setting. Reusing the same version for different artifacts is problematic for caches as well as keeping track of exactly which jar is being used.
To set the Scala version used to locate cross-built managed dependencies, such as
"net.databinder" %% "dispatch-http" % "0.8.5"
use the following setting:
scalaVersion in update := "2.9.0-1"
First thing I would do is to ask Paul if that feature could be included in the 2.9.1.
Next try would be just using a working nightly.
Last chance will be applying patches.