I'm following this tutorial to create a scala jar to upload to streamsets to use in a spark evaluator.
I'm using Intellij 2017.3.4.
If I use Intellij to build the artifact into the out folder, it's over 100mb in size.
If i use the 'sbt clean package' option from the command line, the jar is 3mb.
They both work fine.
Could anyone tell me why there's such a difference, and how I'd setup my Intellij so I create the smaller version?
Thanks
Matt
Without further knowledge of your build.sbt and your IntelliJ settings I can only guess that your build.sbt contains a line like this:
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0" % Provided
Notice the Provided. It means that spark is not bundled in the jar that sbt prepares, but is expected to already be present at the server in which you run your program (on your class path).
IntelliJ uses all the jars (including those marked provided) so that you can debug/run your program via IntelliJ. Since you did not add your build settings in IntelliJ this is the best I can do for now.
If you use sbt assembly to package your artifact, and you want to do this with a run configuration in IntelliJ you can go to Run/Debug Configurations, click the + button, select sbt task and fill in assembly as the task.
Read more about packaging your spark app here.
I hope that helps :)
how to integrate SBT scala in eclipse? i am following this post as well as
i want to add existing project in scala. how to compile it and how to use build.sbt in eclipse?
SBT integration test setup
and http://grosdim.blogspot.in/2013/01/quick-sbt-tutorial.html
but not finding any useful please help me
how to compile and execute a existing project in eclipse using build.sbt??
You can't use build.sbt in Eclipse directly. Instead you need to generate an Eclipse project from your SBT project using sbteclipse plugin. Quoting the README:
Add sbteclipse to your plugin definition file. You can use either:
the global file (for version 0.13 and up) at ~/.sbt/0.13/plugins/plugins.sbt
the project-specific file at PROJECT_DIR/project/plugins.sbt
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "2.4.0")
In sbt use the command eclipse to create Eclipse project files
> eclipse
In Eclipse use the Import Wizard to import Existing Projects into Workspace
See the github wiki for more usage details. Don't forget to rerun eclipse command after changes in your SBT project.
is any IDE supporting SBT in a proper way (like Maven for example)? Because I've found a lot of tools that generate IDE-related configuration files but I haven't found any plugins that give any support of SBT interaction form within IDE.
I want to make an IDE-agnostic project based on SBT, but also I want to be able to use full spectrum of features that IDE provide and not just use it as an editor and do all the other stuff from console.
Does Intellij fit the bill ? It has an SBT plugin (and a Scala plugin, obviously!)
I know this isn't exactly what you're looking for, but putting it here as a work-around for working with SBT in eclipse for whoever is interested.
SBT generates eclipse config files, but after you import it, it works fine from within eclipse. You just need to set up the project for the first time outside of Eclipse, run SBT to resolve dependencies, generate eclipse structure using the eclipse sbt plugin and import into Eclipse. After that, you can run the code directly from Eclipse and it works fine.
Here're the steps in detail:
Create the folder structure as follows:
Create a file called plugins.sbt in the project folder and add the following line to it:
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "2.1.0")
Create build.sbt in the root directory i.e. ScalaSBTProject with content similar to the following. I'm using akka here, but add and remove libraries as you require:
name := "ScalaSBTProject"
version := "1.0"
scalaVersion := "2.10.0-RC2"
resolvers += "Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/"
libraryDependencies += "com.typesafe.akka" % "akka-cluster-experimental_2.10.0-RC2" % "2.1.0-RC2"
Open command prompt and run sbt in the directory ScalaSBTProject. SBT will download and resolve whatever dependencies are required
Run the command eclipse at the SBT command line. This will generate all the eclipse related project files
Import ScalaSBTProject into Eclipse using File->Import->Existing Project to workspace, and make sure you check Import into workspace
EDIT: Just as a Post-Script, you can quite easily create a batch file to take the name of the project and generate the eclipse compatible project, just a way to speed up the process.
I manage my project using Maven and SBT at same time. The reasons for this are:
Intellij IDEA cannot import SBT
project.(idea-sbt plugin doesn't
work very well)
I don't know how can get sources and
javadocs from SBT.(I'd like to see any answers about this)
The problem is I don't know how to let Maven download SBT dependency. I search through maven repository and couldn't find anything about sbt. I wanna use Maven or SBT to manage all the jars in my project.
If you put a pom.xml to the root of your project, it will be recognized by SBT. When you specify no managed dependencies in the project definition, SBT relies on Maven dependencies.
As it said in SBT doumentation,
sbt performs this dependency handling
when the update action is executed. By
default, sbt does not update your
dependencies before every compilation,
but only does so when you execute
update. sbt supports three ways of
specifying these dependencies:
* Declarations in your project definition
* Maven POM files
* Ivy configuration and settings files
Maven knows nothing about SBT as of now (at least, I've not heard about any plugins so far), so, the best you can do to manage your project both in Maven and SBT, is to generate POMs by SBT. See SBT to Maven Converter for more details.
idea-sbt plugin works great for me with IDEA 10 - all it's really intended to do is open an SBT shell within the IDE and it does that well enough.
A plugin you should look into if you're interested in getting the Maven out of your build is sbt-idea plugin ( https://github.com/mpeltonen/sbt-idea ). This is a great plugin that generates IDEA files from an SBT project. It couldn't be easier to use. At an SBT prompt, run the following commands:
*sbtIdeaRepo at http://mpeltonen.github.com/maven/
*idea is com.github.mpeltonen sbt-idea-processor 0.3.0
update
idea
Note the asterisks - they should be included.
At this point, you can open your project in IDEA. It won't complain about the SBT dependencies. Any time you add new dependencies to your project file, simply run the 'idea' command again to tell IDEA about it. I do that in the SBT window provided by idea-sbt.
As far as getting sources and docs with dependencies, you can do something like this (from the SBT docs):
val sc = "org.scalacheck" % "scalacheck" % "1.5" withSources()
There is a corresponding withJavadoc() method. Hope that helps.
I just got started with Scala/LiftWeb/Sbt developing, and I'd like to import a Sbt project in IntelliJ Idea.
Actually, I managed to import my project in two different ways:
1) with Maven. I created a Maven project, and of top of that I created a Sbt project, which I then imported in IntelliJ. I could then easily start, stop the jetty server, and do other stuff.
But that's not what I want. I want to do the same stuff, just Maven-free.
That lead me to
2) with Eclipse. So, I created a new Sbt project (with a little script I wrote, configuring the Sbt project to be a WebProject). I used then the sbt-eclipsify plugin to 'convert' the project for Eclipse, which I then imported in IntelliJ (existing source -> eclipse).
But the problems started here: I cannot get the IntelliJ Sbt plugin to work.
Can anyone help me with this?
There are three basic ways how to create a project - modern versions of IntelliJ can import sbt project out of the box, otherwise you can either use sbt plugin to generate IntelliJ project, or use IntelliJ Scala plugin to create sbt project. Basic features work out of the box using both solutions, some complex builds can have problems, so try other tools to see if it works there.
IntelliJ
IntelliJ IDEA has become so much better these days. The current version (14.0.2) supports sbt projects out of the box with the Scala plugin. Just install the plugin and you should be able to open up Scala/sbt projects without any troubles.
With the plugin, just point at a sbt project and IDEA is going to offer you a wizard to open that kind of project.
IntelliJ Scala Plugin
IntelliJ plugin can be found here
http://confluence.jetbrains.com/display/SCA/Scala+Plugin+for+IntelliJ+IDEA or can be installed directoly from within the IDE using Settings -> Plugins dialog. Afterwards one can just do File -> New Project -> Scala -> SBT based. IntelliJ will generate basic build.sbt, download necessary dependencies and open project.
SBT Plugin
Sbt plugin that generate an idea project based on the sbt files can be found here: https://github.com/mpeltonen/sbt-idea
SBT 12.0+ & 13.0+
Simply add addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.5.2") to your build.sbt; no additional resolvers are needed.
Older Versions:
SBT 0.11+
Create and add the following lines to ~/.sbt/plugins/build.sbt OR PROJECT_DIR/project/plugins.sbt
resolvers += "sbt-idea-repo" at "http://mpeltonen.github.com/maven/"
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.6.0")
Use gen-idea in sbt to create IDEA project files.
By default, classifiers (i.e. sources and javadocs) of sbt and library dependencies are loaded if found and references added to IDEA project files. If you don't want to download/reference them, use command gen-idea no-classifiers no-sbt-classifiers.
SBT 0.10.1
(according to the plugin author, 0.10.0 won't work!)
Create and add the following lines to ~/.sbt/plugins/build.sbt:
resolvers += "sbt-idea-repo" at "http://mpeltonen.github.com/maven/"
libraryDependencies += "com.github.mpeltonen" %% "sbt-idea" % "0.10.0"
Use gen-idea sbt task to create IDEA project files.
By default, classifiers (i.e. sources and javadocs) of sbt and library dependencies are loaded if found and references added to IDEA project files. If you don't want to download/reference them, use command gen-idea no-classifiers no-sbt-classifiers.
SBT 0.7
To use it, simply run this from your sbt shell, it will use the plugin as an external program:
> *sbtIdeaRepo at http://mpeltonen.github.com/maven/
> *idea is com.github.mpeltonen sbt-idea-processor 0.4.0
...
> update
...
> idea
...
You can also add trait in your project definition, as you want:
import sbt._
class MyProject(info: ProjectInfo) extends ParentProject(info) with IdeaProject {
lazy val mySubProject = project("my-subproject", "my-subproject", new DefaultProject(_) with IdeaProject)
// ...
}
For now I do this by hand. It is quite simple.
Create the project with SBT
Create a new IDEA Project with the same root path
Create a module with the same root path
Set src/main/scala as a src path on the module
Set src/test/scala as a test path on the module
Add scala-library.jar as a library
Add lib (if it is present) as a jar directory within a module library
Add lib_managed/compile (if it is present) as a jar directory within a module library
Add lib_managed/test (if it is present) as a jar directory within a module library
That's it from memory. It would be better if it were automated, but it's no big deal as it is now.
One note of caution: The above approach doesn't work well with new-school sbt, i.e. versions 0.10 and newer, because it doesn't copy dependencies into lib_managed by default. You can add
retrieveManaged := true
to your build.sbt to make it copy the dependencies into lib_managed.
Tempus fugit and IntelliJ IDEA has become so much better these days. It's 2015 after all, isn't it?
Having said that, the latest version of IntelliJ IDEA 14.0.2 supports sbt projects out of the box with the Scala plugin. Just install the plugin and you should be able to open up Scala/sbt projects without much troubles.
I'm using the Early Access version of the plugin which is 1.2.67.6.EAP as of the time of the writing.
With the plugin just point at a sbt project and IDEA is going to offer you a wizard to open that kind of project.
About sbt-idea in sbt 0.12.4
For sbt 0.12.4 the system-wide plugin configuration file - ~/.sbt/plugins/build.sbt or PROJECT_DIR/project/plugins.sbt - should have the following lines:
resolvers += "Sonatype snapshots" at "http://oss.sonatype.org/content/repositories/snapshots/"
addSbtPlugin(dependency="com.github.mpeltonen" % "sbt-idea" % "1.5.0-SNAPSHOT")
Run sbt gen-idea to generate IDEA project files.
Read the sbt-idea plugin website for more up-to-date information. You may also find my blog entry Importing sbt-based project to IntelliJ IDEA 13 (with sbt-idea, Scala 2.11 and sbt 0.12) useful.
For sbt 0.7
See the answer elsewhere on this page.
For sbt 0.10
Clone and build Ismael's sbt-idea:
git clone https://github.com/ijuma/sbt-idea.git
cd sbt-idea
git checkout sbt-0.10
./sbt package
Create an sbt plugin lib directory if you don't have one already
mkdir -p ~/.sbt/plugins/lib
Copy the jar built in step one into here
cp sbt-idea/target/scala-2.8.1.final/*.jar ~/.sbt/plugins/lib
Restart or reload sbt, then you can run gen-idea (or gen-idea with-classifiers if you want sources and javadoc in intelliJ too)
Source: Tackers' suggestion on the message group.
In IntelliJ IDEA 13.x itself
You can open an SBT-based project in IDEA nowadays. It will create the necessary project and modules, and keep your dependencies up-to-date whenever you make changes to the build scripts.
I just went through all this pain. I spend days trying to get an acceptable environment up and have come to the conclusion that ENSIME, SBT and JRebel are going to be my development environment for some time. Yes, it is going back to Emacs, but ENSIME turns it into a bit or an idea with refactoring, debugging support, navigation, etc. It's not nowhere near as good as Eclipse (Java), but unless the scala plugins work better it's the best we have.
Until the Scala development environments get up to snuff (Eclipse or IntelliJ) I'm not going to bother. They're just way too buggy.
See the discussion on the lift site.
http://groups.google.com/group/liftweb/browse_thread/thread/6e38ae7396575052#
Within that thread, there is a link to a HOWTO for IntelliJ, but although it kinda works, there are many issues that render it a little less that useful.
http://blog.morroni.com/2010/07/14/setup-intellij-9-for-lift-framework-development/comment-page-1/
The answers are old for 2014.
In IntelliJ 13.x, the plugin Scala is ver 0.41.2 ( SBT is included).
My SBT version is 0.13.5 (terminal : sbt sbtVersion )
Go to the project's root folder and enter in the terminal
sbt idea
You will see two new hidden folders .idea and .idea_modules.
Then in IntelliJ, File > Open > select the project.
It should open the project without any problem.
Before you start creating your SBT project, make sure that the Scala plugin is downloaded and enabled in IntelliJ IDEA.
below link explains everything you need to know.
https://www.jetbrains.com/help/idea/2016.1/getting-started-with-sbt.html