New to Scala and sbt; coming from a python world, and rather confused by library dependencies, versioning, and what sbt can and cannot automatically download (i.e. when a .jar needs to be manually placed in /lib).
I want to use classes from the Scala geoscript project (http://geoscript.org/scala/quickstart.html) in my Scala app.
I'm using IntelliJ with Scala 2.11.8.
Can I ultimately do something like:
libraryDependencies += "org.geoscript" % "some-artifact" % some=version
Or is this going to be an "unmanaged dependency"? And if so, what's the cleanest way to do that?
If a dependency jar is published to Maven Central and some other repositories, sbt will automatically be able to resolve and download it when declaring it as libraryDependency. This is a "managed dependency". Usually a library's documentation will tell you the "coordinates" (group id, artifact id, version) you need to install it.
From the page you linked and the Maven search, it looks like Geoscript is not published, so you will have to add it to your /lib folder in the project. That should be enough to put it in the classpath. You may need to refresh the project for IntelliJ to pick it up and offer completions.
Related
I am new to scala, when I create a scala project in intellij it took really so long to download all the jar files. I have installed the scala plugin for idea IDE. Could anyone please tell me what should I do to get on the right track? Thanks very much.
Since the scala language is really a library on top of the JVM, creating a scala project for the first time requires the download of the specified scala version from a maven repository (usually this is at least the compiler and library, which for 2.10 comes to ~20MB). This will happen the first time, even if you've installed that same version of scala on your machine outside of sbt.
Once you go through this the first time, though, the next project you create--whether via sbt on the command line or via an IntelliJ sbt project--will pull these dependencies from your local ivy cache instead.
However, should you change the language version in your project, it will once again have to download the full language dependencies for that version.
In the screenshot it try to download the source files of one of of the dependencies. That mean it will try to download not only the compiled jars, but also it's source and docs.
only after downloading all of them, the project will be ready.
Disable to downloading of the sources and docs, and it will be much faster. How to disable it depends on how you create the project. (eg if you create sbt project, make sure to uncheck the "download sources and docs" in the creation/import wizard)
I'm trying to add the apache commons email library to my Play project and I'm having trouble.
Firstly I have both build.sbt and plugins.sbt in my project and I'm not sure which one I should be putting the import into, does anyone know?
Also, I'm not sure why there even is the separate project module in my project, intelliJ created it as part of the project. Could anyone explain the purpose of the two separate modules and why they are there?
Thanks!
So, in sbt, you have your project. This is specified in build.sbt (or more correctly, any *.sbt file in your projects base directory). Any libraries that your applications code needs, for example, if your application needs to send emails using the commons email library, go in to the librarDependencies seeing in here.
But build.sbt itself is Scala code that needs to be compiled, but it's not part of your applications runtime. So in sbt, your projects build is a project itself, one that has to be compiled. It has its own classpath, which consists of the sbt plugins you're using, so for example, if you need a less compiler to compile your less files, that's not something that gets done at runtime, so you don't want your application code depending on that, it goes into your project builds libraryDependencies, which gets specified in project/plugins.sbt (or in fact any *.sbt in the project directory). So, once you add it there, you can use the Scala code it provides from build.sbt. IntelliJ imports this project for you so that you can have syntax highlighting and other IDE features in build.sbt.
But it doesn't stop there. How does project/plugins.sbt get compiled, where is its classpath? Well, your projects builds projects builds project is also an sbt project itself too... It keeps going down. IntelliJ stops at that point though, it doesn't keep importing these meta sbt projects because it's actually very rare to need additional sbt plugins for your projects builds projects builds project, so it just uses the same classpath as your projects build project for syntax highlighting in project/plugins.sbt.
I have a fairly normal Scala project currently being built using Maven. I would like to support both Scala 2.9.x and the forthcoming 2.10, which is not binary or source compatible. I am willing to entertain converting to SBT if necessary, but I have run into some challenges.
My requirements for this project are:
Single source tree (no branching). I believe that trying to support multiple concurrent "master" branches for each Scala version will be the quickest way to miss bugfixes between the branches.
Version specific source directories. Since the Scala versions are not source compatibile, I need to be able to specify an auxiliary source directory for version specific sources.
Version specific source jars. End users should be able to download the correct source jar, with the correct version specific sources, for their version of Scala for IDE integration.
Integrated deployment. I currently use the Maven release plugin to deploy new versions to the Sonatype OSS repository, and would like to have a similarly simple workflow for releases.
End-user Maven support. My end users are often Maven users, and so a functional POM that accurately reflects dependencies is critical.
Shaded jar support. I need to be able to produce a JAR that includes a subset of my dependenices and removes the shaded dependencies from the published POM.
Things I have tried:
Maven profiles. I created a set of Maven profiles to control what version of Scala is used to build, using the Maven build-helper plugin to select the version specific source tree. This was working well until it came time to publish;
Using classifiers to qualify versions doesn't work well, because the source jars would also need custom classifiers ('source-2.9.2', etc.), and most IDE tools wouldn't know how to locate them.
I tried using a Maven property to add the SBT-style _${scala.version} suffix to the artifact name, but Maven does not like properties in the artifact name.
SBT. This works well once you can grok it (no small task despite extensive documentation). The downside is that there does not seem to be an equivalent to the Maven shade plugin. I've looked at:
Proguard. The plugin is not updated for SBT 0.12.x, and won't build from source because it depends on another SBT plugin that has changed groupIds, and doesn't have a 0.12.x version under the old name. I have not yet been able to work out how to instruct SBT to ignore/replace the plugin dependency.
OneJar. This uses custom class loading to run Main classes out of embedded jars, which is not the desired result; I want the class files of my project to be in the jar along with (possibly renamed) class files from my shaded dependencies.
SBT Assembly plugin. This can work to a degree, but the POM file appears to include the dependencies that I'm trying to shade, which doesn't help my end users.
I accept that there may not be a solution that does what I want for Scala, and/or I may need to write my own Maven or Scala plugins to accomplish the goal. But if I can I'd like to find an existing solution.
Update
I am close to accepting #Jon-Ander's excellent answer, but there is still one outstanding piece for me, which is a unified release process. The current state of my build.sbt is on GitHub. (I'll reproduce it here in an answer later for posterity).
The sbt-release plugin does not support multi-version builds (i.e., + release does not behave as one might like), which makes a sort of sense as the process of release tagging doesn't really need to happen across versions. But I would like two parts of the process to be multi-version: testing and publishing.
What I'd like to have happen is something akin to two-stage maven-release-plugin process. The first stage would do the administrative work of updating Git and running the tests, which in this case would mean running + test so that all versions are tested, tagging, updating to snapshot, and then pushing the result to upstream.
The second stage would checkout the tagged version and + publish, which will rerun the tests and push the tagged versions up to the Sonatype repository.
I suspect that I could write releaseProcess values that do each of these, but I'm not sure if I can support multiple releaseProcess values in my build.sbt. It probably can work with some additional scopes, but that part of SBT is still strange majick to me.
What I currently have done is changed the releaseProcess to not publish. I then have to checkout the tagged version by hand and run + publish after the fact, which is close to what I want but does compromise, especially since the tests are only run on the current scala version in the release process. I could live with a process that isn't two-stage like the maven plugin, but does implement multi-version test and publish.
Any additional feedback that can get me across the last mile would be appreciated.
Most of this is well supported in sbt within a single source tree
Version specific source directories are usually not need. Scala programs tends to be source compatible - so often in fact that
crossbuilding (http://www.scala-sbt.org/release/docs/Detailed-Topics/Cross-Build) has first class support in sbt.
If you really need version specific code, you can add extra source folders.
Putting this in your build.sbt file will add "src/main/scala-[scalaVersion]" as a source directory for each version as you crossbuild in addition to the regular "src/main/scala".
(there is also a plugin available for generating shims between version, but I haven't tried it - https://github.com/sbt/sbt-scalashim)
unmanagedSourceDirectories in Compile <+= (sourceDirectory in Compile, scalaVersion){ (s,v) => s / ("scala-"+v) }
version specific source jars - see crossbuilding, works out of the box
integrated deployment - https://github.com/sbt/sbt-release (has awesome git integration too)
Maven end-users - http://www.scala-sbt.org/release/docs/Detailed-Topics/Publishing.html
Shaded - I have used this one https://github.com/sbt/sbt-assembly which have worked fine for my needs.
Your problem with the assembly plugin can be solved by rewriting the generated pom.
Here is an example ripping out joda-time.
pomPostProcess := {
import xml.transform._
new RuleTransformer(new RewriteRule{
override def transform(node:xml.Node) = {
if((node \ "groupId").text == "joda-time") xml.NodeSeq.Empty else node
}
})
}
Complete build.sbt for for reference
scalaVersion := "2.9.2"
crossScalaVersions := Seq("2.9.2", "2.10.0-RC5")
unmanagedSourceDirectories in Compile <+= (sourceDirectory in Compile, scalaVersion){ (s,v) => s / ("scala-"+v) }
libraryDependencies += "joda-time" % "joda-time" % "1.6.2"
libraryDependencies += "org.mindrot" % "jbcrypt" % "0.3m"
pomPostProcess := {
import xml.transform._
new RuleTransformer(new RewriteRule{
override def transform(node:xml.Node) = {
if((node \ "groupId").text == "joda-time") xml.NodeSeq.Empty else node
}
})
}
I've done something similar to this with SBT as an example:
https://github.com/seanparsons/scalaz/commit/21298eb4af80f107181bfd09eaaa51c9b56bdc28
It's made possible by SBT allowing all the settings to be determined based on other settings, which means that most other things should "just work".
As far as the pom.xml aspect I can only recommend asking the question in the SBT mailing list, I would be surprised if you couldn't do that however.
My blog post http://www.day-to-day-stuff.blogspot.nl/2013/04/fixing-code-and-binary.html contains an example of a slightly more finegrained solution for attaching different source directories; one per major S. Also it explains how to create scala-version-specific code that can be used by not-specific code.
Update 2016-11-08: Sbt now supports this out of the box: http://www.scala-sbt.org/0.13/docs/sbt-0.13-Tech-Previews.html#Cross-version+support+for+Scala+sources
I have inherited a big project with several subprojects.
all of them use several jar files, all of them located under each project's lib directory. I want to take all the projects and migrate them to maven, but dependencies are a problem (too many of them), some of them are commonly used libraries (apache projects, xerces, jms, etc) and others are not.
is there a way to autogenerate maven dependencies for those jars that can be found on public maven repositories. for example, see that my project use the spice-jndikit-1.2.jar file and automatically get the appropiate depedency with group, artifact and (if possible) version?
thank you
I wrote a groovy script to generate a starting set of Apache ivy files.
https://github.com/myspotontheweb/ant2ivy
In my case, I wanted to "Maven-ize" my ANT builds without switching completely away from ANT.
It is feasible to extend this code to generate a Maven POM, if people were interested in this feature.
You can convert a project to Maven using the m2e plugin, but this erases your jar references, and should not be used.
I doubt that such a thing exists since typical jars (unless themselves built with Maven) don't have the necessary information to correlate the groupId, artifactId and version back to a repository to get the proper path.
You might be able to write something that parses the file name for the name and version, but you still have the package-based path to figure out.
If you're building using Ant, you might also consider using Apache Ivy, and its file-system based resolution (very fast and easy to configure), to get you started, and slowly role over to the Maven repos for the artifacts, this way you're not spending a lot of time up-front finding Maven dependencies.
I would like to give IntelliJ IDEA a try, but I have no idea how to get going.
I am simply trying to create a new project that uses Finagle Echo Server, hosted on github, as starting point.
Assuming I'm starting with a clean install on Mac. I installed IDEA and added the Scala and SBT plugins. What steps should I take to create a project, that uses Finagle and run the code in the http server example?
PLEASE help. I realize my question sounds like a stupid question, but there are so many different approaches to working with Scala projects from SBT command line, Scala-IDE, Idea, etc, that I simply don't know how to get a comfortable development environment going.
A manual solution that doesn't require you to use SBT for your project might be more straightforward, given the SBT versioning issues. You'll still use SBT to build finagle, though.
Install the SBT runner script per step 1 above. (It can handle SBT 0.7 projects too).
Manually git clone git://github.com/twitter/finagle.git.
cd to the finagle directory and type "sbt package". Its dependencies should end up under lib_managed, and it should produce the finagle jars themselves under target/ or some such (just note the locations in the command output).
Create an IDEA project from scratch, and manually add dependencies to the finagle jars and their dependencies (under Project Structure->Dependencies).
This answer assumes that you want to use SBT. Also, I should qualify that this is my usual procedure, but I haven't confirmed that it works with finagle in particular.
0. Install IDEA, with Scala and SBT plugins. (Done by the OP; here for others)
1. Install SBT (automatic method). Copy this handy sbt runner script to a convenient location (or, if you want to keep it up to date, git clone https://github.com/paulp/sbt-extras.git and symlink the script into ~/bin), and make sure it's executable. It will automatically download whatever it needs based on the sbt.version specified in your build.properties.
2. Install sbt-idea. sbt-idea is an SBT plugin (not an IDEA plugin) that generates IDEA module files from an SBT project. It's convenient to install this globally since it's not project-specific. You don't have to download anything manually; just add this to ~/.sbt/plugins/build.sbt:
resolvers += "sbt-idea-repo" at "http://mpeltonen.github.com/maven/"
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "0.11.0")
3. Create SBT project. Create a directory for your project, and a "project" directory within it. Create project/Build.scala as follows:
import sbt._
object MyBuild extends Build {
lazy val root = Project("root", file(".")) dependsOn finagle
lazy val finagle = RootProject(uri("git://github.com/twitter/finagle.git"))
}
See the SBT documentation for lots more options re configuring your project. Note we have to use the Full Configuration here (not just build.sbt) in order to express the github dependency.
It's also a good idea to create project/build.properties:
sbt.version=0.11.2
project.version=0.1
build.scala.versions=2.9.1
4. Generate IDEA project. cd to the directory containing the sbt-based project. type "sbt gen-idea". If all goes well, the directory will now have ".idea" and ".idea_modules" subdirectories.
5. Open the project in IDEA. It may be necessary to fix the target JDK version in the project settings. Aside from that, the project should be ready to go, with all source paths, library dependencies, etc. properly configured.