Suggestions needed to improve packaging all sources and javadoc of sbt projects - scala

Question

To avoid version-related problems with scala (2.9, 2.10, 2.11, …), we want to include all necessary jar files to use scala in a java application. To facilitate debugging & development, we want to include the sources & javadocs of all such libraries too.
I know this topic has been asked many times before; however, I haven't found a solution that could work for us (scala 2.11 & sbt 0.13.5).
I managed to prototype an approximate solution with an sbt project configured as follows:
./build.sbt:
val packAllCommand = Command.command("packAll") {
state =>
"clean" :: "update" :: "updateClassifiers" ::
"pack" :: "dependencyGraph" :: "dependencyDot" ::
state
}
commands += packAllCommand
./project/plugins.sbt:
resolvers +=
"sonatype-releases" at "https://oss.sonatype.org/content/repositories/releases/"
addSbtPlugin("org.xerial.sbt" % "sbt-pack" % "0.6.1")
addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.7.4")
./project/Build.scala
import sbt._
import Keys._
import net.virtualvoid.sbt.graph.Plugin.graphSettings
import xerial.sbt.Pack._
/**
* Goal:
*
* use sbt to package all the jars/sources/javadoc for scala & related libraries needed to use scala in a java application
* without requiring scala to be installed on the system.
*
* #author Nicolas.F.Rouquette#jpl.nasa.gov
*/
object BuildWithSourcesAndJavadocs extends Build {
object Versions {
val scala = "2.11.2"
val config = "1.2.1"
val scalaCheck = "1.11.5"
val scalaTest = "2.2.1"
val specs2 = "2.4"
val parboiled = "2.0.0"
}
lazy val scalaLibs: Project = Project(
"scalaLibs",
file( "scalaLibs" ),
settings = Defaults.coreDefaultSettings ++ Defaults.runnerSettings ++ Defaults.baseTasks ++ graphSettings ++ packSettings ++ Seq(
scalaVersion := Versions.scala,
packExpandedClasspath := true,
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-library" % scalaVersion.value % "compile" withSources () withJavadoc (),
"org.scala-lang" % "scala-compiler" % scalaVersion.value % "compile" withSources () withJavadoc (),
"org.scala-lang" % "scala-reflect" % scalaVersion.value % "compile" withJavadoc () withJavadoc () ),
( mappings in pack ) := { extraPackFun.value } ) )
lazy val otherLibs: Project = Project(
"otherLibs",
file( "otherLibs" ),
settings = Defaults.coreDefaultSettings ++ Defaults.runnerSettings ++ Defaults.baseTasks ++ graphSettings ++ packSettings ++ Seq(
scalaVersion := Versions.scala,
packExpandedClasspath := true,
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-library" % Versions.scala % "provided",
"org.scala-lang" % "scala-compiler" % Versions.scala % "provided",
"org.scala-lang" % "scala-reflect" % Versions.scala % "provided",
"com.typesafe" % "config" % Versions.config % "compile" withSources () withJavadoc (),
"org.scalacheck" %% "scalacheck" % Versions.scalaCheck % "compile" withSources () withJavadoc (),
"org.scalatest" %% "scalatest" % Versions.scalaTest % "compile" withSources () withJavadoc (),
"org.specs2" %% "specs2" % Versions.specs2 % "compile" withSources () withJavadoc (),
"org.parboiled" %% "parboiled" % Versions.parboiled % "compile" withSources () withJavadoc () ),
( mappings in pack ) := { extraPackFun.value } ) ).dependsOn( scalaLibs )
lazy val root: Project = Project( "root", file( "." ) ) aggregate ( scalaLibs, otherLibs )
val extraPackFun: Def.Initialize[Task[Seq[( File, String )]]] = Def.task[Seq[( File, String )]] {
def getFileIfExists( f: File, where: String ): Option[( File, String )] = if ( f.exists() ) Some( ( f, s"${where}/${f.getName()}" ) ) else None
val ivyHome: File = Classpaths.bootIvyHome( appConfiguration.value ) getOrElse sys.error( "Launcher did not provide the Ivy home directory." )
// this is a workaround; how should it be done properly in sbt?
// goal: process the list of library dependencies of the project.
// that is, we should be able to tell the classification of each library dependency module as shown in sbt:
//
// > show libraryDependencies
// [info] List(
// org.scala-lang:scala-library:2.11.2,
// org.scala-lang:scala-library:2.11.2:provided,
// org.scala-lang:scala-compiler:2.11.2:provided,
// org.scala-lang:scala-reflect:2.11.2:provided,
// com.typesafe:config:1.2.1:compile,
// org.scalacheck:scalacheck:1.11.5:compile,
// org.scalatest:scalatest:2.2.1:compile,
// org.specs2:specs2:2.4:compile,
// org.parboiled:parboiled:2.0.0:compile)
// but... libraryDependencies is a SettingKey (see ld below)
// I haven't figured out how to get the sequence of modules from it.
val ld: SettingKey[Seq[ModuleID]] = libraryDependencies
// workaround... I found this API that I managed to call...
// this overrides the classification of all jars -- i.e., it is as if all library dependencies had been classified as "compile".
// for now... it's a reasonable approaximation of the goal...
val managed: Classpath = Classpaths.managedJars( Compile, classpathTypes.value, update.value )
val result: Seq[( File, String )] = managed flatMap { af: Attributed[File] =>
af.metadata.entries.toList flatMap { e: AttributeEntry[_] =>
e.value match {
case null => Seq()
case m: ModuleID => Seq() ++
getFileIfExists( new File( ivyHome, s"cache/${m.organization}/${m.name}/srcs/${m.name}-${m.revision}-sources.jar" ), "lib.srcs" ) ++
getFileIfExists( new File( ivyHome, s"cache/${m.organization}/${m.name}/docs/${m.name}-${m.revision}-javadoc.jar" ), "lib.javadoc" )
case _ => Seq()
}
}
}
result
}
}
Thanks to the sbt-pack and sbt-dependency-graph plugins, the above produces what I need:
scalaLibs/target/dependencies-compile.dot
scalaLibs/target/pack/lib
scalaLibs/target/pack/lib.srcs
scalaLibs/target/pack/lib.javadoc
otherLibs/target/dependencies-compile.dot
otherLibs/target/pack/lib
otherLibs/target/pack/lib.srcs
otherLibs/target/pack/lib.javadoc
The dot files can be visualized with GraphViz; it helps explain why a particular library is included…
I would like to improve this approach in terms of the following:
some libraries in scalaLibs are duplicated in otherLibs,
this approach ignores library dependency classification & overrides (not used here)
Suggestions?
Nicolas.