I want to use http://dispatch.databinder.net/Dispatch.html .
The site indicates I must add this to project/plugins.sbt:
libraryDependencies += "net.databinder.dispatch" %% "core" % "0.9.1"
which I did. I then restarted the play console and compiled.
Importing doesnt work:
import dispatch._
Guess I have been silly, but then I never used a build system when using Java.
How must I trigger the process that downloads/builds the package? Where are the jars (or equivalent) stored; can I reuse them? When is the package available for use by the Play application?
It doesn't say you should add it to project/plugins.sbt. That is the wrong place. It says to add to the build.sbt file, on the root of your project. Being a Play project, project/Build.scala might be more appropriate -- I don't know if it will pick up settings from build.sbt or not.
To add the dependency in your Build.scala:
val appDependencies = Seq(
"net.databinder.dispatch" %% "core" % "0.9.1"
)
You probably need to run sbt update.
From the sbt Command Line Reference:
update Resolves and retrieves external dependencies as described in library dependencies.
Related
I'm a bit of newbie to the Scala and the Play framework (2.6.x). See git push heroku master failure screenshot below.
I'm requiring the jsoup dependency in build.sbt (the first one):
libraryDependencies += "org.jsoup" % "jsoup" % "1.11.3"
libraryDependencies += guice
libraryDependencies += "org.scalatestplus.play" %% "scalatestplus-play" % "3.1.2" % Test
And using it in my controller:
import org.jsoup.Jsoup
import org.jsoup.nodes.Document
...
val res = scala.io.Source.fromURL(data.url)("ISO-8859-1").mkString
val s = Jsoup.parse(res).title
In addition I attempted to use it as an unmanaged dependency by adding it to the lib/ folder, though I still get the same Heroku error.
Interestingly, the app works OK and without errors locally. Is there something I'm missing? Thanks.
Edit:
Beginning of build log:
Failure:
My guess is, that you created Play project from a template. Initial template contains both build.sbt but also gradle build (build.gradle, gradlew, gradlew.bat. Locally, you use sbt for compilation. However, Heroku picks up Gradle build.
Problem: You added the dependency only to build.sbt, but not into gradle.build file.
If you do not really need Gradle, than I suggest to delete Gradle build files from your repository and try to push again.
If you want to keep Heroku using Gradle, than you have to maintain both types of build files.
There also probably is a way how to instruct Heroku to use sbt as preferred choice.
I want to build a 'fat' jar of my code. I understand how to do this mostly but all the examples I have use the idea that the jar is not local and I am not sure how to include into my assembled jar another JAR that I built that the scala code uses. Like what folder does this JAR I have to include reside in?
Normally when I run my current code as a test using spark-shell it looks like this:
spark-shell --jars magellan_2.11-1.0.6-SNAPSHOT.jar -i st_magellan_abby2.scala
(the jar file is right in the same path as the .scala file)
So now I want to build a build.sbt file that does the same and includes that SNAPSHOT.jar file?
name := "PSGApp"
version := "1.0"
scalaVersion := "2.11.8"
resolvers += "Spark Packages Repo" at "http://dl.bintray.com/spark-packages/maven"
//provided means don't included it is there. already on cluster?
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.2.0" % "provided",
"org.apache.spark" %% "spark-sql" % "2.2.0" % "provided",
"org.apache.spark" %% "spark-streaming" % "2.2.0" % "provided",
//add magellan here somehow?
)
So where would I put the jar in the SBT project folder structure so it gets picked up when I run sbt assembly? Is that in the main/resources folder? Which the reference manual says is where 'files to include in the main jar' go?
What would I put in the libraryDependencies here so it knows to add that specific jar and not go out into the internet to get it?
One last thing, I was also doing some imports in my test code that doesn't seem to fly now that I put this code in an object with a def main attached to it.
I had things like:
import sqlContext.implicits._ which was right in the code above where it was about to be used like so:
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._
import org.apache.spark.sql.functions.udf
val distance =udf {(a: Point, b: Point) =>
a.withinCircle(b, .001f); //current radius set to .0001
}
I am not sure can I just keep these imports inside the def main? or do I have to move them elsewhere somehow? (Still learning scala and wrangling the scoping I guess).
One way is to build your fat jar using the assembly plugin (https://github.com/sbt/sbt-assembly) locally and publishLocal to store the resulting jar into your local ivy2 cache
This will make it available for inclusion in your other project based on build.sbt settings in this project, eg:
name := "My Project"
organization := "org.me"
version := "0.1-SNAPSHOT"
Will be locally available as "org.me" %% "my-project" % "0.1-SNAPSHOT"
SBT will search local cache before trying to download from external repo.
However, this is considered bad practise, because only final project should ever be a fat-jar. You should never include one as dependency (many headaches).
There is no reason to make project magellan a fat-jar if library is included in PGapp. Just publishLocal without assembly
Another way is to make projects dependant on each other as code, not library.
lazy val projMagellan = RootProject("../magellan")
lazy val projPSGApp = project.in(file(".")).dependsOn(projMagellan)
This makes compilation in projPSGApp tigger compilation in projMagellan.
It depends on your use case though.
Just don't get in a situation where you have to manage your .jar manually
The other question:
import sqlContext.implicits._ should always be included in the scope where dataframe actions are required, so you shouldn't put that import near the other ones in the header
Update
Based on discussion in comments, my advise would be:
Get the magellan repo
git clone git#github.com:harsha2010/magellan.git
Create a branch to work on, eg.
git checkout -b new-stuff
Change the code you want
Then update the versioning number, eg.
version := "1.0.7-SNAPSHOT"
Publish locally
sbt publishLocal
You'll see something like (after a while):
[info] published ivy to /Users/tomlous/.ivy2/local/harsha2010/magellan_2.11/1.0.7-SNAPSHOT/ivys/ivy.xml
Go to your other project
Change build.sbt to include
"harsha2010" %% "magellan" % "1.0.7-SNAPSHOT" in your libraryDependencies
Now you have a good (temp) reference to your library.
Your PSGApp should be build as an fat jar assembly to pass to Spark
sbt clean assembly
This will pull in the custom build jar
If the change in the magellan project is usefull for the rest of the world, you should push your changes and create a pull request, so that in the future you can just include the latest build of this library
I am very new to SBT, Breeze and IntelliJ, though I have a decent grasp of Scala and I am trying to install the Breeze library, which I think is managed.
What I've done:
I followed the instructions on this page and added this script to the build.sbt file in my project:
libraryDependencies ++= Seq(
// other dependencies here
"org.scalanlp" %% "breeze" % "0.10",
// native libraries are not included by default. add this if you want them (as of 0.7)
// native libraries greatly improve performance, but increase jar sizes.
"org.scalanlp" %% "breeze-natives" % "0.10"
)
resolvers ++= Seq(
// other resolvers here
"Sonatype Releases" at "https://oss.sonatype.org/content/repositories/releases/"
)
// Scala 2.9.2 is still supported for 0.2.1, but is dropped afterwards.
scalaVersion := "2.11.1" // or 2.10.3 or later
I then ran sbt update in the project directory (via the terminal), and saw that all the pieces of Breeze downloaded.
I then tried re-running sbt update, but this did not trigger another download.
Issue:
The problem is that I cannot access the library via IntelliJ. import breeze._ gives the standard Cannot resolve symbol breeze and I couldn't find any mention of Breeze in "Project Structure." It isn't in the lib directory of the project either.
Am I missing a step?
Sounds like a bug in the IntelliJ project, try removing the .idea directory from the project directory and then re-import the project into IntelliJ using the wizard.
I've downloaded the ScalaTest jar and have used it as in the example, but now I would like to start using it with sbt. Where do I place the downloaded jar so I can use it with sbt across multiple projects?
You don't download dependencies like ScalaTest manually. The point of using sbt is to declare your project's dependencies and let sbt download them for you automatically.
Add this line in your build.sbt file:
libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.0" % "test"
For more details see official doc on setting this up.
Even after reading: Scala, problem with a jar file, I'm still a bit confused. I am trying to import some packages into my Scala file, and the interpreter is not recognizing them even after adding to classpath.
One example:
I have the import statement:
import org.json4s._
I downloaded the .jar from here: http://mvnrepository.com/artifact/org.json4s/json4s-native_2.10/3.2.4
and added to the interpreter classpath using:
scala> :cp /Users/aspangher13/Downloads/json4s-native_2.10-3.2.4.jar
Scala acknowledges the classpath:
Your new classpath is: ".:/Users/aspangher13/Downloads/json4s-native_2.10-3.2.4.jar:/Users/aspangher13/Downloads/jna-3.5.2.jar"
But still throws this error:
<console>:7: error: object scalatra is not a member of package org
import org.json4s._
Can anyone see what I'm doing wrong? Thanks!!
And as a followup, does anyone know where to find the package: JsonAST._?
Go the simple and create a little sbt project.
First step - create a project
For your purposes you don't need a complex build. So just create two files:
./build.sbt
name := "name your project"
version := "0.1"
scalaVersion := "2.10.2" // or whatever you prefer
./project/build.properties
sbt.version=0.12.4
The just go to the project root folder and call sbt
Second step - add dependencies
Open your ./build.sbt file and add:
libraryDependency ++= Seq(
"org.scalatra" %% "scalatra" % "2.2.1",
"org.scalatra" %% "scalatra-scalate" % "2.2.1",
"org.scalatra" %% "scalatra-specs2" % "2.2.1" % "test",
"org.json4s" %% "json4s-native % "3.2.4",
"net.java.dev.jna" & "jna" & "3.5.2"
)
Step three - run the console
Don't forget to reload sbt with reload task, and then call console or console-quick task. This should work.
But there are easier ways to do this:
1) Use gitter8 - Scalatra gitter8 project
2) Read little into about Scalatra sbt dependencies
Still not sure how :cp works but if you execute
scala -classpath "list of jars colon separated"
then inside the REPL do your imports it should work
import org.json4s._
import org.xyz
However, when you try to use the classes you are likely to be missing transitive dependencies required by json4s and so we come back to the sbt example # 4lex1v describes, which will handle this. Creating a little project and running sbt console will indeed greatly simplify this.
Seems like the -classpath and :cp are primarily meant to make your code available in the shell and then only if you understand all of the transitive dependencies, or have none.