Cross-compiling aggregate projects with different crossScalaVersions for subprojects - scala

We have a project with several subprojects which can compile under both Scala 2.10 and 2.11, one subproject which only compiles under 2.10 (actually, Scala-Virtualized 2.10.2) and one subproject which only compiles under 2.11. Is there a simple way to create an aggregate project which cross-builds all possible subprojects for both 2.10 and 2.11? Or, alternately, to have different default projects for 2.10 and 2.11?
In particular, here is current Build.scala. If I add lmsBackend to root, I get
> show scalaVersion
[info] common/*:scalaVersion
[info] 2.10.4
[info] lms-backend/*:scalaVersion
[info] 2.10.2
[info] meta/*:scalaVersion
[info] 2.10.4
[info] community-edition/*:scalaVersion
[info] 2.10.4
[info] core/*:scalaVersion
[info] 2.10.4
[info] scalan/*:scalaVersion
[info] 2.10.4
> show crossScalaVersions
[info] common/*:crossScalaVersions
[info] List(2.10.4, 2.11.5)
[info] lms-backend/*:crossScalaVersions
[info] List(2.10.2)
[info] meta/*:crossScalaVersions
[info] List(2.10.4, 2.11.5)
[info] community-edition/*:crossScalaVersions
[info] List(2.10.4, 2.11.5)
[info] core/*:crossScalaVersions
[info] List(2.10.4, 2.11.5)
[info] scalan/*:crossScalaVersions
[info] List(2.10.4, 2.11.5)
SBT is able to run update, compile, etc. fine on this aggregate project. However, once I try any cross-building, things break:
> +update
[info] Setting version to 2.10.4
[info] Reapplying settings...
[info] Set current project to scalan (in build file:/home/aromanov/IdeaProjects/scalan-lite/)
...
[info] Updating {file:/home/aromanov/IdeaProjects/scalan-lite/}lms-backend...
[info] Resolving org.scala-lang.virtualized#scala-library;2.10.4 ...
[warn] module not found: org.scala-lang.virtualized#scala-library;2.10.4
...
[info] Resolving org.scala-lang.virtualized#scala-compiler;2.10.4 ...
[warn] module not found: org.scala-lang.virtualized#scala-compiler;2.10.4
...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.scala-lang.virtualized#scala-library;2.10.4: not found
[warn] :: org.scala-lang.virtualized#scala-compiler;2.10.4: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] org.scala-lang.virtualized:scala-library:2.10.4 ((sbt.Classpaths) Defaults.scala#L1169)
[warn] +- com.huawei.scalan:lms-backend_2.10:0.2.6-SNAPSHOT
[warn] org.scala-lang.virtualized:scala-compiler:2.10.4
[warn] +- com.huawei.scalan:lms-backend_2.10:0.2.6-SNAPSHOT
[trace] Stack trace suppressed: run last lms-backend/*:update for the full output.
[error] (lms-backend/*:update) sbt.ResolveException: unresolved dependency: org.scala-lang.virtualized#scala-library;2.10.4: not found
[error] unresolved dependency: org.scala-lang.virtualized#scala-compiler;2.10.4: not found
[error] Total time: 2 s, completed Jan 28, 2015 1:33:49 PM
show scalaVersion now shows 2.10.4 for all subprojects. Is there any way to include lms-backend in the aggregate project and still avoid this problem?

I had a similar question. My question had an additional complication of classpath dependencies (dependsOn), rather than just aggregation, but one solution for my problem fixes this one quite well.
sbt-doge, a deceptively cutesy GitHub project, replaces the implementation of + with one of its interchangeable prefixes: much, so, such, very.
Current implementation of + cross building operator does not take in account for the crossScalaVersions of the sub projects. Until that's fixed, here's an alternative implementation of it.
The creator of the project is one of the main contributors to SBT.
Add
addSbtPlugin("com.eed3si9n" % "sbt-doge" % "0.1.3")
to project/plugins.sbt.
Then
> very compile

I believe sbt-doge might not be enough for your requirements as mine were not met by this plugin as well.
I would recommend sbt-cross
This plugin allows you for much more flexibility among your modules, as well as, it provides a way to aggregate modules of different Scala versions, without the need to have any version in common!
Example:
lazy val common = (project in file("common")).cross
lazy val common_2_11 = common("2.11.8")
lazy val common_2_10 = common("2.10.6")
lazy val A = (project in file("A"))
.settings(scalaVersion := "2.10.6")
.dependsOn(common_2_10)
lazy val B = (project in file("B"))
.settings(scalaVersion := "2.11.8")
.dependsOn(common_2_11)
lazy val root = (project in file("."))
.aggregate(common, A, B)
I have already posted a similar answer.
Enjoy!

Related

Add docker publish step to sbt-release process with new tag

I'm integrating the sbt-release plugin to our projects, in order to delegate all the build+publish tasks to it.
It basically does all we need, but I'm adding an additional ReleaseStep to it: publishing a Docker image. This is my current releaseProcess (as per this):
releaseProcess := Seq[ReleaseStep](
checkSnapshotDependencies,
inquireVersions,
runTest,
setReleaseVersion,
commitReleaseVersion,
tagRelease,
//publishArtifacts,
releaseStepCommand("docker"),
setNextVersion,
commitNextVersion
//pushChanges,
)
(commented push-related stuff not to break anything upstream)
It's mostly fine and dandy, except the tag that the resulting Docker image gets.
For example, if the project was version 0.17.0-SNAPSHOT, and I wanted to release version 1.0.0, something like this would happen:
[develop] ✓ [17:54:22] lithium : ~/devel/some-company/some-app
➤ sbt clean "release skip-tests"
[info] Loading project definition from /opt/devel/some-company/some-app/project
[info] Updating {file:/opt/devel/some-company/some-app/project/}some-app-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Set current project to someapp (in build file:/opt/devel/some-company/some-app/)
[warn] Multiple resolvers having different access mechanism configured with same name 'local'. To avoid conflict, Remove duplicate project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
[warn] Credentials file /var/jenkins_home/credentials/nexus_creds does not exist
[success] Total time: 0 s, completed May 18, 2017 5:54:56 PM
[info] Starting release process off commit: a7acde4d0e943be1e4befa6cc70dc873e164044a
[warn] Multiple resolvers having different access mechanism configured with same name 'local'. To avoid conflict, Remove duplicate project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
[warn] Credentials file /var/jenkins_home/credentials/nexus_creds does not exist
[info] Updating {file:/opt/devel/some-company/some-app/}root...
[info] Done updating.
[warn] Found intransitive dependency (net.logstash.logback:logstash-logback-encoder:4.7) while publishMavenStyle is true, but Maven repositories
[warn] do not support intransitive dependencies. Use exclusions instead so transitive dependencies
[warn] will be correctly excluded in dependent projects.
[warn]
Release version [0.17.0] : 1.0.0
Next version [1.0.1-SNAPSHOT] :
[info] Setting version to '1.0.0'.
[info] Reapplying settings...
[info] Set current project to someapp (in build file:/opt/devel/some-company/some-app/)
[info] [develop db69b0a] Setting version to 1.0.0
[info] 1 file changed, 1 insertion(+), 1 deletion(-)
[info] Reapplying settings...
[info] Set current project to someapp (in build file:/opt/devel/some-company/some-app/)
[info] Packaging /opt/devel/some-company/some-app/target/scala-2.11/someapp_2.11-0.17.0-SNAPSHOT-sources.jar ...
[warn] Multiple resolvers having different access mechanism configured with same name 'local'. To avoid conflict, Remove duplicate project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
[warn] Credentials file /var/jenkins_home/credentials/nexus_creds does not exist
[info] Done packaging.
[info] Wrote /opt/devel/some-company/some-app/target/scala-2.11/someapp_2.11-0.17.0-SNAPSHOT.pom
Warning: node.js detection failed, sbt will use the Rhino based Trireme JavaScript engine instead to run JavaScript assets compilation, which in some cases may be orders of magnitude slower than using node.js.
[info] Packaging /opt/devel/some-company/some-app/target/scala-2.11/someapp_2.11-0.17.0-SNAPSHOT-web-assets.jar ...
[info] Done packaging.
[info] Compiling 107 Scala sources and 36 Java sources to /opt/devel/some-company/some-app/target/scala-2.11/classes...
[info] Main Scala API documentation to /opt/devel/some-company/some-app/target/scala-2.11/api...
[warn] Class javax.annotation.CheckReturnValue not found - continuing with a stub.
[warn] /opt/devel/some-company/some-app/app/services/DOPXHandlerActor.scala:44: non-variable type argument services.DOPX in type pattern List[services.DOPX] (the underlying of List[services.DOPX]) is unchecked since it is eliminated by erasure
[warn] case reqs:List[DOPX] => withMDC { // PASAR A CASE CLASS
[warn] ^
[warn] /opt/devel/some-company/some-app/app/services/DOPXHandlerActor.scala:44: non-variable type argument services.DOPX in type pattern List[services.DOPX] (the underlying of List[services.DOPX]) is unchecked since it is eliminated by erasure
[warn] case reqs:List[DOPX] => withMDC { // PASAR A CASE CLASS
[warn] ^
[warn] /opt/devel/some-company/some-app/app/controllers/TransactionController.scala:79: match may not be exhaustive.
[warn] It would fail on the following input: (None, Success(_))
[warn] paymentConfirmationsService.confirm(siteId, chargeId, operationResource, request.headers.get("user")) map { _ match {
[warn] ^
[warn] there were 15 feature warnings; re-run with -feature for details
[warn] /opt/devel/some-company/some-app/app/services/DistributedTransactionProcessor.scala:139: match may not be exhaustive.
[warn] It would fail on the following input: Some(_)
[warn] failureResponse match {
[warn] ^
[warn] /opt/devel/some-company/some-app/app/services/DistributedTransactionProcessor.scala:312: match may not be exhaustive.
[warn] It would fail on the following inputs: (_, Failure(_)), (_, Success(_))
[warn] distributedOperationProcessor.processDistributedOPx(opDataFatherFixed.chargeId, Rechazada(), opDataFatherFixed, None, meanPayment, Some(refundSubPaymentOperations)).map( pr => pr match {
[warn] ^
model contains 337 documentable templates
[warn] /opt/devel/some-company/some-app/app/services/refunds/RefundService.scala:346: match may not be exhaustive.
[warn] It would fail on the following input: None
[warn] operation.sub_transactions.find { subTx => subTx.subpayment_id.get == subpaymentId } match {
[warn] ^
[warn] /opt/devel/some-company/some-app/app/services/ApplicationTimer.scala:9: Could not find any member to link for "ApplicationLifecycle".
[warn] /**
[warn] ^
[warn] three warnings found
[info] Main Scala API documentation successful.
[info] Packaging /opt/devel/some-company/some-app/target/scala-2.11/someapp_2.11-0.17.0-SNAPSHOT-javadoc.jar ...
[info] Done packaging.
[warn] there were 15 feature warnings; re-run with -feature for details
[warn] 30 warnings found
[warn] bootstrap class path not set in conjunction with -source 1.6
[info] /opt/devel/some-company/some-app/app/legacy/some-company/sps/domain/DBParametros.java: Some input files use unchecked or unsafe operations.
[info] /opt/devel/some-company/some-app/app/legacy/some-company/sps/domain/DBParametros.java: Recompile with -Xlint:unchecked for details.
[info] Packaging /opt/devel/some-company/some-app/target/scala-2.11/someapp_2.11-0.17.0-SNAPSHOT.jar ...
[info] Done packaging.
[info] Packaging /opt/devel/some-company/some-app/target/scala-2.11/someapp_2.11-0.17.0-SNAPSHOT-sans-externalized.jar ...
[info] Done packaging.
[info] Sending build context to Docker daemon 108.9MB
[info] Step 1/8 : FROM lapp-dvde004:5000/java-alpine:latest
[info] ---> fd94b5262b7b
[info] Step 2/8 : MAINTAINER Redbee
[info] ---> Using cache
[info] ---> 2905e1a8a792
[info] Step 3/8 : WORKDIR /opt/docker
[info] ---> Using cache
[info] ---> 0b2f169737f6
[info] Step 4/8 : ADD opt/ /opt
[info] ---> a973fb66a793
[info] Removing intermediate container f6da68f8a1c3
[info] Step 5/8 : ADD opt/docker/conf/jce_policy-8.tar.gz /usr/lib/jvm/default-jvm/jre/lib/security/
[info] ---> 3fda5972ddd5
[info] Removing intermediate container cb10f1fcf2dd
[info] Step 6/8 : RUN chown -R daemon:daemon .
[info] ---> Running in 867093c046e0
[info] ---> 60a969ff427c
[info] Removing intermediate container 867093c046e0
[info] Step 7/8 : USER daemon
[info] ---> Running in a7fa422e326c
[info] ---> 8857a5a5392b
[info] Removing intermediate container a7fa422e326c
[info] Step 8/8 : ENTRYPOINT /opt/docker/conf/wrapper.sh
[info] ---> Running in 31887383f984
[info] ---> 2418dba2d69e
[info] Removing intermediate container 31887383f984
[info] Successfully built 2418dba2d69e
[info] Successfully tagged lapp-dvde004:5000/someapp:0.17.0-SNAPSHOT
[info] Built image lapp-dvde004:5000/someapp:0.17.0-SNAPSHOT
[info] Update Latest from image lapp-dvde004:5000/someapp:0.17.0-SNAPSHOT
[success] Total time: 58 s, completed May 18, 2017 5:56:07 PM
[info] Setting version to '1.0.1-SNAPSHOT'.
[info] Reapplying settings...
[info] Set current project to someapp (in build file:/opt/devel/some-company/some-app/)
[info] [develop a1dd63d] Setting version to 1.0.1-SNAPSHOT
[info] 1 file changed, 1 insertion(+), 1 deletion(-)
Sadly I cannot highlight code, but it says:
[info] Setting version to '1.0.0'.
[info] Reapplying settings...
And then the built image is:
[info] Built image lapp-dvde004:5000/someapp:0.17.0-SNAPSHOT
In my limited understanding, it seems like sbt-release and sbt-docker (or sbt-native-packager, I tried with both) do not share the same build context? Something along those lines?
sbt-release successfully changes the version.sbt file before sbt-docker kicks in, but still the latter pays no mind to it.
Any ideas?
PS: Funfact: This question apparently had the answer (or at least his requirements matched mine), so, pandaforme, if you read this, why did you delete that gist!? :(
Thanks to the amazing pandaforme who re-uploaded the aforementioned gist, it's now working.
So, I implemented two things: A 'release' process which also builds a Docker image, and also a custom task 'publishSnapshot' which is sort of an extension to 'publish' (publishes the snapshot and it's docker image; also known as "overengineering when I could just do sbt publish dockerBuildAndPush").
Implementation, if anyone needs it:
val publishDocker = ReleaseStep(action = st => {
val extr: Extracted = Project.extract(st)
val ref: ProjectRef = extr.get(thisProjectRef)
extr.runAggregated(
sbtdocker.DockerKeys.dockerBuildAndPush in sbtdocker.DockerPlugin.autoImport.docker in ref,
st
)
st
})
lazy val publishSnapshot = taskKey[Unit]("Publish a Snapshot and it's Docker image")
publishSnapshot in Compile := Def.sequential(
publish in Compile,
sbtdocker.DockerKeys.dockerBuildAndPush in sbtdocker.DockerPlugin.autoImport.docker
).value
releaseProcess := Seq[ReleaseStep](
checkSnapshotDependencies,
inquireVersions,
runTest,
setReleaseVersion,
commitReleaseVersion,
tagRelease,
publishArtifacts,
publishDocker,
setNextVersion,
commitNextVersion,
pushChanges
)

Why does sbt use a different version of Scala to the one I asked for?

It even though I have scalaVersion set to 2.11.7, my project wants to build with Scala 2.10.x.
Every time I recompile a new directory src/main/scala2.10 appears in my project. I'm pretty sure that this must be the result of an SBT reconfiguration, but what exactly is causing this, and how do I fix it?
I have a very simple [project_root]/build.sbt file:
name := "coolproduct"
version := "0.0.1"
lazy val scalaVersion = "2.11.7"
lazy val akkaVersion = "2.4.1"
libraryDependencies += "com.typesafe.akka" %% "akka-actor" % akkaVersion
When I run this script it seems to be trying to use Scala 2.10 - a totally different version to what I actually asked for:
> sbt compile
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
[info] Loading project definition from C:\Users\salim\workspace\funproxy\project
[info] Set current project to funproxy (in build file:/C:/Users/salim/workspace/funproxy/)
[info] Updating {file:/C:/Users/salim/workspace/funproxy/}funproxy...
[info] Resolving com.typesafe.akka#akka-actor_2.10;2.4.1 ...
[warn] module not found: com.typesafe.akka#akka-actor_2.10;2.4.1
[warn] ==== local: tried
[warn] C:\Users\salim\.ivy2\local\com.typesafe.akka\akka-actor_2.10\2.4.1\ivys\ivy.xml
[warn] ==== jcenter: tried
[warn] https://jcenter.bintray.com/com/typesafe/akka/akka-actor_2.10/2.4.1/akka-actor_2.10-2.4.1.pom
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/com/typesafe/akka/akka-actor_2.10/2.4.1/akka-actor_2.10-2.4.1.pom
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.typesafe.akka#akka-actor_2.10;2.4.1: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] com.typesafe.akka:akka-actor_2.10:2.4.1 (C:\Users\salim\workspace\funproxy\build.sbt#L22-23)
[warn] +- default:funproxy_2.10:0.0.1
I'm running an up to date version of SBT:
> sbt about
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
[info] Loading project definition from C:\Users\salim\workspace\funproxy\project
[info] Set current project to funproxy (in build file:/C:/Users/salim/workspace/funproxy/)
[info] This is sbt 0.13.8
[info] The current project is {file:/C:/Users/salim/workspace/funproxy/}funproxy 0.0.1
[info] The current project is built against Scala 2.10.4
[info] Available Plugins: sbt.plugins.IvyPlugin, sbt.plugins.JvmPlugin, sbt.plugins.CorePlugin, sbt.plugins.JUnitXmlReportPlugin
[info] sbt, sbt plugins, and build definitions are using Scala 2.10.4
The short answer: change lazy val scalaVersion = "2.11.7" to:
scalaVersion := "2.11.7"
scalaVersion is not just a local scala var (or val) you can override; it's defined internally as a Key (see Keys). With the := operation you're changing the internal state of that key.

sbt 0.12.4 - there were x feature warning(s); re-run with -feature for details

I get an error there were 15 feature warning(s); re-run with -feature for details :
$ /usr/local/sbt/bin/sbt
[info] Loading project definition from /home/alex/Documents/projects/my_app123/project
[info] Set current project to sbt-android (in build file:/home/alex/Documents/projects/my_app123/)
> compile -feature
[error] Expected end of input.
[error] compile -feature
[error] ^
> sbt-version
[info] 0.12.4
> compile
[warn] Credentials file /home/alex/.ivy2/.credentials does not exist
[info] Compiling 20 Scala sources to /home/alex/Documents/projects/my_app123/target/scala-2.10/sbt-0.12/classes...
[error] there were 15 feature warning(s); re-run with -feature for details
[error] one error found
[error] (compile:compile) Compilation failed
[error] Total time: 27 s, completed 01 12, 15 1:20:12 PM
In build.sbt
scalaVersion := "2.10.4"
I can't find out what that error is because it doesn't say that, even when I run sbt as $ /usr/local/sbt/bin/sbt -feature.
$ /usr/local/sbt/bin/sbt -feature
[info] Loading project definition from /home/alex/Documents/projects/my_app123/project
[info] Set current project to sbt-android (in build file:/home/alex/Documents/projects/my_app123/)
$
It's JDK 1.7
What's the error is and how do I fix it?
UPDATE:
build.sbt
scalacOptions in ThisBuild ++= Seq("-unchecked", "-deprecation")
or in sbt itself:
> set scalacOptions in ThisBuild ++= Seq("-unchecked", "-deprecation")
The error remains the same.
I don't get a chance to use sbt on a daily basis, so this is just to confirm the suggestion in the linked answer:
$ sbt
[info] Set current project to Compiler options (in build file:/home/apm/tmp/sbtwarn/)
> compile
[info] Compiling 1 Scala source to /home/apm/tmp/sbtwarn/target/scala-2.11/classes...
[warn] there was one feature warning; re-run with -feature for details
[warn] one warning found
[success] Total time: 4 s, completed Jan 12, 2015 9:45:17 AM
> set scalacOptions in ThisBuild ++= Seq("-feature")
[info] Defining {.}/*:scalacOptions
[info] The new value will be used by compile:scalacOptions
[info] Reapplying settings...
[info] Set current project to Compiler options (in build file:/home/apm/tmp/sbtwarn/)
> compile
[info] Compiling 1 Scala source to /home/apm/tmp/sbtwarn/target/scala-2.11/classes...
[warn] /home/apm/tmp/sbtwarn/src/main/scala/Sample.scala:6: postfix operator head should be enabled
[warn] by making the implicit value scala.language.postfixOps visible.
[warn] This can be achieved by adding the import clause 'import scala.language.postfixOps'
[warn] or by setting the compiler option -language:postfixOps.
[warn] See the Scala docs for value scala.language.postfixOps for a discussion
[warn] why the feature should be explicitly enabled.
[warn] Console println (List(1,2,3) head)
[warn] ^
[warn] one warning found
[success] Total time: 1 s, completed Jan 12, 2015 9:45:46 AM
Hey, it worked!
You should add
scalacOptions += "-feature"
to your build.sbt and execute reload if your sbt console is running (or restart it).
Alternatively if you want to set it only for a single session, while in sbt console, you can write set scalacOptions += "-feature", this setting is applied immediately, no need to reload or restart sbt console.
Re-run with -feature for details

How to declare dependency on Scalding in sbt project?

I am trying to figure out how to create an build.sbt file for my own Scalding-based project.
Scalding source structure has no build.sbt file. Instead it has project/Build.scala build definition.
What would be the right way to integrate my own sbt project with Scalding, so I could also import it later in Eclipse with sbt-eclipse plugin?
Update:
For the following code:
import cascading.tuple.Fields
import com.twitter.scalding._
class Scan(args: Args) extends Job(args) {
val output = TextLine("tmp/out.txt")
val wordsList = List(
("john"),
("liza"),
("nina"),
("x"))
val orderedPipe =
IterableSource[(String)](wordsList, ('word))
.debug
.write(output)
}
With this build.sbt:
name := "Scan"
version := "1.0"
libraryDependencies := Seq("com.twitter" %% "scalding" % "0.11.1")
I get errors:
$ sbt
[info] Loading global plugins from /home/test/.sbt/0.13/plugins
[info] Set current project to Scan (in build file:/home/test/Cascading/Scala/Scan/)
> compile
[info] Updating {file:/home/test/Cascading/Scala/Scan/}scan...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] downloading http://repo1.maven.org/maven2/com/twitter/scalding_2.10/0.11.1/scalding_2.10-0.11.1.jar ...
[info] [SUCCESSFUL ] com.twitter#scalding_2.10;0.11.1!scalding_2.10.jar (641ms)
[info] Done updating.
[info] Compiling 1 Scala source to /home/test/Cascading/Scala/Scan/target/scala-2.10/classes...
[error] /home/test/Cascading/Scala/Scan/src/main/scala/Scan.scala:1: not found: object cascading
[error] import cascading.tuple.Fields
[error] ^
[error] /home/test/Cascading/Scala/Scan/src/main/scala/Scan.scala:2: object twitter is not a member of package com
[error] import com.twitter.scalding._
[error] ^
[error] /home/test/Cascading/Scala/Scan/src/main/scala/Scan.scala:5: not found: type Job
[error] class Scan(args: Args) extends Job(args) {
[error] ^
[error] /home/test/Cascading/Scala/Scan/src/main/scala/Scan.scala:5: not found: type Args
[error] class Scan(args: Args) extends Job(args) {
[error] ^
[error] /home/test/Cascading/Scala/Scan/src/main/scala/Scan.scala:5: too many arguments for constructor Object: ()Object
[error] class Scan(args: Args) extends Job(args) {
[error] ^
[error] /home/test/Cascading/Scala/Scan/src/main/scala/Scan.scala:6: not found: value TextLine
[error] val output = TextLine("tmp/out.txt")
[error] ^
[error] /home/test/Cascading/Scala/Scan/src/main/scala/Scan.scala:15: not found: value IterableSource
[error] IterableSource[(String)](wordsList, ('word))
[error] ^
[error] 7 errors found
[error] (compile:compile) Compilation failed
Update 2
After doing git clone git#github.com:twitter/scalding.git their repository and sbt publishLocal I still have the same compilation errors.
BUT adding two lines that you suggested to build.sbt allowed me to compile my code. So the following build.sbt really works, thanks!
name := "BlockScan"
version := "1.0"
libraryDependencies := Seq("com.twitter" %% "scalding" % "0.11.1")
lazy val scaldingCore = ProjectRef(uri("https://github.com/twitter/scalding.git"), "scalding-core")
lazy val myProject = project in file(".") dependsOn scaldingCore
'sbt eclipse' creates Eclipse project wich does not compile under Eclipse and reports these errors:
Project 'Scan' is missing required Java project: 'scalding-core'
More than one scala library found in the build path (/home/test/usr/eclipse-scala-3.0.3/configuration/org.eclipse.osgi/bundles/290/1/.cp/lib/scala-library.jar, /home/test/wks/Cascading/Scala/scalding/target/scala-2.9.3/scalding-assembly-0.10.0.jar).At least one has an incompatible version. Please update the project build path so it contains only compatible scala libraries.
scalacheck_2.9.3-1.10.0.jar is cross-compiled with an incompatible version of Scala (2.9.3).
specs_2.9.3-1.6.9.jar is cross-compiled with an incompatible version of Scala (2.9.3).
Since they don't seem to publish their libraries to remote repositories where you could pull down the necessary dependencies, you'll have to declare the source dependency on the GitHub repository for the project.
lazy val scaldingCore = ProjectRef(uri("https://github.com/twitter/scalding.git"), "scalding-core")
lazy val myProject = project in file(".") dependsOn scaldingCore
With the above build definition, sbt will git clone the RootProject and load the build.
➜ scalding xsbt
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
Cloning into '/Users/jacek/.sbt/0.13/staging/e1da2accb95841ffb1df/scalding'...
[info] Loading project definition from /Users/jacek/.sbt/0.13/staging/e1da2accb95841ffb1df/scalding/project
[info] Updating {file:/Users/jacek/.sbt/0.13/staging/e1da2accb95841ffb1df/scalding/project/}scalding-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] downloading http://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.eed3si9n/sbt-assembly/scala_2.10/sbt_0.13/0.10.2/jars/sbt-assembly.jar ...
[info] [SUCCESSFUL ] com.eed3si9n#sbt-assembly;0.10.2!sbt-assembly.jar (3600ms)
[info] Done updating.
[info] Compiling 3 Scala sources to /Users/jacek/.sbt/0.13/staging/e1da2accb95841ffb1df/scalding/project/target/scala-2.10/sbt-0.13/classes...
[warn] there were 8 deprecation warning(s); re-run with -deprecation for details
[warn] there were 2 feature warning(s); re-run with -feature for details
[warn] two warnings found
[info] Set current project to myProject (in build file:/Users/jacek/sandbox/scalding/)
> projects
[info] In file:/Users/jacek/sandbox/scalding/
[info] * myProject
[info] In https://github.com/twitter/scalding.git
[info] maple
[info] scalding
[info] scalding-args
[info] scalding-avro
[info] scalding-commons
[info] scalding-core
[info] scalding-date
[info] scalding-hadoop-test
[info] scalding-jdbc
[info] scalding-json
[info] scalding-parquet
[info] scalding-repl
The build setup should give you access to scalding classes.
> console
[info] Starting scala interpreter...
[info]
Welcome to Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_60).
Type in expressions to have them evaluated.
Type :help for more information.
scala> import com.twitter.scalding._
import com.twitter.scalding._
And the Scan class compiles fine - it's in src/main/scala directory.
> show sources
[info] ArrayBuffer(/Users/jacek/sandbox/scalding/src/main/scala/Scan.scala)
[success] Total time: 0 s, completed Jul 15, 2014 12:21:14 AM
> compile
[info] Updating {file:/Users/jacek/sandbox/scalding/}myProject...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 1 Scala source to /Users/jacek/sandbox/scalding/target/scala-2.10/classes...
[success] Total time: 4 s, completed Jul 15, 2014 12:21:20 AM
You could also git clone git#github.com:twitter/scalding.git their repository and sbt publishLocal to be able to declare binary dependency in build.sbt as follows:
libraryDependencies := Seq("com.twitter" %% "scalding" % "0.11.1")
With the dependency in (either way), execute sbt eclipse and be done with it!

SBT dependsOn RootProject: doesn't compile the dependency

I have a pretty simple configuration:
//lazy val bananaRdfProject = RootProject( uri("git://github.com:stample/banana-rdf.git#"+bananaGitBranch) )
// lazy val bananaRdfProject = RootProject( uri("https://github.com/stample/banana-rdf.git#"+bananaGitBranch) )
// lazy val bananaRdfProject = ProjectRef( uri("https://github.com/stample/banana-rdf.git#"+bananaGitBranch) ,"banana-rdf")
lazy val bananaRdfProject = RootProject( file("../banana-rdf") )
lazy val main = play.Project(appName, appVersion, appDependencies).settings(...)
.dependsOn( bananaRdfProject )
I tried using the 4 different project declarations above for bananaRdfProject.
As I may edit this banana-rdf locally, I want it to be recompiled each time I build my play project, so that I do not have to publish the banana-rdf...
But when I try to compile my main play project, that uses banana-rdf, it doesn't compile banana-rdf, but tries to compile the main project: the compilation fails because banana-rdf classes are missing in the classpath.
sebastien#clemence-XPS-L412Z:rww-play (master *%)$ ./play.old/play
[info] Loading project definition from /home/sebastien/Bureau/rww-play/project
[warn] there were 1 deprecation warning(s); re-run with -deprecation for details
[warn] one warning found
[info] Loading project definition from /home/sebastien/Bureau/banana-rdf/project
[info] Updating {file:/home/sebastien/Bureau/banana-rdf/project/}banana-rdf-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 2 Scala sources to /home/sebastien/Bureau/banana-rdf/project/target/scala-2.10/sbt-0.13/classes...
[warn] there were 11 deprecation warning(s); re-run with -deprecation for details
[warn] there were 2 feature warning(s); re-run with -feature for details
[warn] two warnings found
[info] Set current project to RWWeb (in build file:/home/sebastien/Bureau/rww-play/)
_
_ __ | | __ _ _ _
| '_ \| |/ _' | || |
| __/|_|\____|\__ /
|_| |__/
play 2.2-TLS built with Scala 2.10.3-RC3 (running Java 1.7.0_45), http://www.playframework.com
> Type "help play" or "license" for more information.
> Type "exit" or use Ctrl+D to leave this console.
[RWWeb] $ compile
[info] Updating {file:/home/sebastien/Bureau/banana-rdf/}banana...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/home/sebastien/Bureau/rww-play/}RWWeb...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 53 Scala sources and 1 Java source to /home/sebastien/Bureau/rww-play/target/scala-2.10/classes...
[error] /home/sebastien/Bureau/rww-play/app/controllers/CORSProxy.scala:4: object banana is not a member of package org.w3
[error] import org.w3.banana.plantain.Plantain
[error] ^
[error] /home/sebastien/Bureau/rww-play/app/controllers/CORSProxy.scala:7: not found: type Plantain
[error] object CORSProxy extends org.www.readwriteweb.play.CORSProxy[Plantain](webClient)
.................
Isn't it supposed to compile banana-rdf before trying to compile my main project? If not, what is the point of depending on an external RootProject?
The RootProject(file("../banana-rdf")) project reference just references the root project, and what you really need is a reference to the banana-rdf subproject (in the rdf subdirectory).
The available subprojects are defined in https://github.com/w3c/banana-rdf/blob/master/project/build.scala. There are a few:
[main]> projects
[info] In file:/Users/jacek/sandbox/stackoverflow/19832655/
[info] * main
[info] In https://github.com/w3c/banana-rdf.git
[info] banana
[info] banana-jena
[info] banana-rdf
[info] banana-rdf-test-suite
[info] banana-sesame
[info] examples
[info] experimental
[info] ldp
[info] patch
To reference banana-rdf you should then use the following ProjectRef pointing at a right module (subproject) in a build configuration. Note ProjectRef as well as the name of the subproject - banana-rdf.
lazy val bananaRdfProject =
ProjectRef(uri("https://github.com/w3c/banana-rdf.git"), "banana-rdf")
With the ProjectRef you should be able to resolve any types defined in the banana-rdf subproject.
There is a difference between aggregate projects and dependsOn. I think you need to aggregate to get all projects to build together, dependsOn only makes sure the classes from that project is on the classpath, but it could of course be an old artifact if you haven't built it recently. Check out the sbt docs for more info: http://www.scala-sbt.org/0.12.3/docs/Getting-Started/Multi-Project.html