I'm following this tutorial to create a scala jar to upload to streamsets to use in a spark evaluator.
I'm using Intellij 2017.3.4.
If I use Intellij to build the artifact into the out folder, it's over 100mb in size.
If i use the 'sbt clean package' option from the command line, the jar is 3mb.
They both work fine.
Could anyone tell me why there's such a difference, and how I'd setup my Intellij so I create the smaller version?
Thanks
Matt
Without further knowledge of your build.sbt and your IntelliJ settings I can only guess that your build.sbt contains a line like this:
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0" % Provided
Notice the Provided. It means that spark is not bundled in the jar that sbt prepares, but is expected to already be present at the server in which you run your program (on your class path).
IntelliJ uses all the jars (including those marked provided) so that you can debug/run your program via IntelliJ. Since you did not add your build settings in IntelliJ this is the best I can do for now.
If you use sbt assembly to package your artifact, and you want to do this with a run configuration in IntelliJ you can go to Run/Debug Configurations, click the + button, select sbt task and fill in assembly as the task.
Read more about packaging your spark app here.
I hope that helps :)
Related
I have created basic Scala Play application with https://www.playframework.com/getting-started play-scala-seed. This project compiles and runs with sbt run. But I have another Scala project that compiles and runs and which I have submitted to my local Ivy repository with command sbt publishLocal. This other project was saved at C:\Users\tomr\.ivy2\local\com.agiintelligence\scala-isabelle_2.13\master-SNAPSHOT as a result of this command.
Then I imported (exactly so - imported, no just opened) my Play project in IntelliJ and I used Project - Open Module Settings - Project Settings - Libraries to add com.agiintelligence jar from my ivy2 location. After such operations IntelliJ editor recognizes com.agiintelligence classes. That is fine.
But when I am trying to run my Play application with sbt run, I experience the error message not found: object com that is exactly when compiling import com.agiintelligence line in my Scala controller file of Play application.
Of course - such error has been reported and resolved with, e.g. object play not found in scala application
But that solution suggests to append build.sbt file. My build.sbt file is pretty bare:
name := """agiintelligence"""
organization := "com.agiintelligence"
version := "1.0-SNAPSHOT"
lazy val root = (project in file(".")).enablePlugins(PlayScala)
scalaVersion := "2.13.5"
libraryDependencies += guice
libraryDependencies += "org.scalatestplus.play" %% "scalatestplus-play" % "5.0.0" % Test
// Adds additional packages into Twirl
//TwirlKeys.templateImports += "com.skaraintelligence.controllers._"
// Adds additional packages into conf/routes
// play.sbt.routes.RoutesKeys.routesImport += "com.skaraintelligence.binders._"
My Play application contains (as can bee seen from the IntelliJ project pane) some tens of 'external libraries' (it shows my com.agiintelligence jar as well), but why should I add my own ivy2 library in build.sbt file if no other libraries are listed here? What is different with my library? It is on my computer, in the repository as expected already?
Of course, I can try to add it build.sbt and issue sbt update and see what happens, but I can not understand this logic? Can someone explain it and provide some clue to intelligible solution of my error message?
My Play application contains (as can bee seen from the IntelliJ project pane) some tens of 'external libraries'
Those are probably just transitive dependencies of your Play dependency, that is why sbt downloaded all of them and put them in your classpath so you could use them without you needing to tell it about them; because the pom of Play already did.
It is not that the build tool or the IDE magically added all those dependencies for you because they read your mind and magically understood you wanted them. And that for some reason the magic stopped working for your own library.
Why it is not sufficient to list it Project-Setting--External Libraries in IntelliJ only?
That is sufficient for the IDE to work, but not for the build tool. The build tool is independent of the IDE; it doesn't know about it. sbt just knows about the dependencies you configured in your definition file.
Even more, you should always configure your dependencies on your build tool and then import that in the IDE; rather than the opposite. IDEs are graphical tools, so their state can not be committed, can not be shared, can not keep track of changes, can not be used in CI / CD environments; additionally, different teammates may want to use different IDEs.
I resolved the error message by adding line in build.sbt file
libraryDependencies += "de.unruh" %% "scala-isabelle" % "master-SNAPSHOT"
and by subsequent run of sbt update.
Error is solved, but the main question still stand - why I had to do this? Why there are tens of dependencies that are not listed in build.sbt and why should I list my dependency in build.sbt and why it is not sufficient to list it Project-Setting--External Libraries in IntelliJ only?
OK, comment by #Luis_Miguel_Mejía_Suárez gave the explanation, that comment is the actual and expected answer to my question.
I'm having trouble getting IntelliJ to recognize Ammonite imports in my Scala script. This is a new feature, reported here, where there are also instructions. I followed the instructions and I started a new sbt project in IntelliJ to try it out, but it's not working.
I see the Ammonite dependencies in the project's "External Libraries", which I specified in the build.sbt:
libraryDependencies += "com.lihaoyi" %% "ammonite-ops" % "1.0.3"
libraryDependencies += "com.lihaoyi" % "ammonite" % "1.0.3" cross CrossVersion.full
The (quite simple) project seems to successfully build. I also specified that Scala worksheet files should be treated as "Always Ammonite".
All this, yet the IDE shows the script like:
What more do I need to do to get this to work?
Versions:
Java 1.8.0
Scala 2.12.4
sbt 1.0.3
IntelliJ CE 2017.3.2
IntelliJ scala plugin 2017.3.11
I had similar issues with IntelliJ, so let me share how I made it work.
My versions:
IntelliJ Ultimate 2017.3.2 (so basically the same version as yours)
IntelliJ Scala plugin 2017.3.11
Kubuntu, so keyboard shortcuts below may have to be adjusted
Now, I started from the same code as you have, with updated version of dependency 0.6.7.
Already now pressing the green arrow to next to 1st line runs Ammonite. The dependencies are downloaded and link is displayed. That is mandatory step for IntelliJ to recognize that new jar is available.
Also at this stage Intellij shows me a small popup with information that I can import the jar. I did not use it.
To make IntelliJ recognize the imports, I press Alt+Enter Show Intention Actions on the broken com.lihaoyi::scalatags:0.6.7 dependency and use Create library from jar...
This makes Intellij recognize the library, imports and it can start giving you hints.
The library is added to the Project's dependencies (Ctrl+Alt+Shift+s). You can see also dependencies coming from other files below.
I had similar issues with IntelliJ and seem to be figthing bugs in IntelliJ.
My versions:
IntelliJ Ultimate 2018.2.4
IntelliJ Scala plugin 2018.2.11
OS: Windows
Scala: 2.12.6 (at the moment Ammonite current release 1.2.1 is not released for Scala 2.12.7, see #879 and below)
Without IntelliJ Project
If I just open the .sc file without open project then I get prompted with
Project JDK is not defined
No Scala SDK in module
the notification: 'Add all Ammonite standard dependencies to the project?'
The notification quickly dissapears. If I click 'Add' button before fixing the JDK, the imports does not work, and the add button is gone. fix the jdk and scala, and then find the 'Add all Ammonite...' in the eventlog and click 'add' there, then it works.
However all changes in settings are lost next time I reopen IntelliJ.
With IntelliJ Project
Create new project Scala sbt project, default directory based project.
I place my sc-files in there somewhere
Now IntelliJ remembers settings for sc-files. I can even open sc files directly from explorer/total commander in IntelliJ after configuring 'open with...' to intellij, though I prefer.
Do NOT allow IntelliJ to auto-import sbt project settings via notification, it seems to ruin the library imports, and I need to recreate project to recover.
Scala Version issues
The library auto-import 'add' button will only work for Scala versions for which Ammonite is released. See Maven Central Ammonite releases. At the time of writing Ammonite 1.2.1 is not released for scala-2.12.7, though the current unstable version of ammonite is. See Ammonite bug-report #879. The add button just fails silently, which sucks, esp. for a newbie like me.
The auto-generated sbt build file looks like this
name := "Ammonite"
version := "0.1"
scalaVersion := "2.12.6"
This is how it looks in project settings after libraries have successfully loaded:
and here are the dependencies that make it compile. Note that the 'SBT : ' dependencies are those auto-generated via SBT-file (it seems), and the others are libraries added by scala plugin ammonite notifications feature
The above library list is empty when IntelliJ fails to load the ammonite libraries.
IntelliJ with SBT (because IntelliJ loses ammonite imports, and for $ivy)
IntelliJ seems to frequently for lose its ammonite imports. If I have SBT setup to give the imports, then it does not matter so much. Also IntelliJ $ivy support for ammonite only works when files are already cached locally in ivy-cache, and running Ammonite script on my windows caches files in different location, which means IntelliJ $ivy imports effectively does not work. Hence I use SBT to setup the sum of the imports I use in my scripts.
My build.sbt
scalaVersion := "2.12.7"
// https://mvnrepository.com/artifact/com.lihaoyi/ammonite
libraryDependencies += "com.lihaoyi" % ("ammonite_" + scalaVersion) % "1.2.1-16-48ce533"
// Extra example dependency
libraryDependencies += "org.jsoup" % "jsoup" % "1.11.3"
I think IntelliJ IDEA won't find a dependency in a local cache if you're working mostly with Scala 2.13 and SBT 1.3+ with the Coursier library management. We need to help IntelliJ IDEA by downloading dependencies to the Ivy cache.
I solved this issue by:
Creating the ivy.xml somewhere with required dependencies, e.g:
<ivy-module version="1.0">
<info organisation="test" module="download-deps"/>
<dependencies>
<dependency org="com.softwaremill.sttp.client" name="core_2.12" rev="2.1.1"/>
<dependency org="com.softwaremill.sttp.client" name="okhttp-backend_2.12" rev="2.1.1"/>
<dependency org="com.github.tototoshi" name="scala-csv_2.12" rev="1.3.6"/>
</dependencies>
</ivy-module>
Note, I specified Scala 2.12 here.
Running ivy -refresh from the directory with the created ivy.xml.
Then we need is to "Create library from jar..." from the red import's context actions in IntelliJ IDEA.
That's all!
you need to associate .sc files with ammonite.
in you settings, it should look like this
(solution taken from here)
I would also recommend reading the following blog post (scroll down to Ammonite scripts support) to make sure you have all needed requirements
is any IDE supporting SBT in a proper way (like Maven for example)? Because I've found a lot of tools that generate IDE-related configuration files but I haven't found any plugins that give any support of SBT interaction form within IDE.
I want to make an IDE-agnostic project based on SBT, but also I want to be able to use full spectrum of features that IDE provide and not just use it as an editor and do all the other stuff from console.
Does Intellij fit the bill ? It has an SBT plugin (and a Scala plugin, obviously!)
I know this isn't exactly what you're looking for, but putting it here as a work-around for working with SBT in eclipse for whoever is interested.
SBT generates eclipse config files, but after you import it, it works fine from within eclipse. You just need to set up the project for the first time outside of Eclipse, run SBT to resolve dependencies, generate eclipse structure using the eclipse sbt plugin and import into Eclipse. After that, you can run the code directly from Eclipse and it works fine.
Here're the steps in detail:
Create the folder structure as follows:
Create a file called plugins.sbt in the project folder and add the following line to it:
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "2.1.0")
Create build.sbt in the root directory i.e. ScalaSBTProject with content similar to the following. I'm using akka here, but add and remove libraries as you require:
name := "ScalaSBTProject"
version := "1.0"
scalaVersion := "2.10.0-RC2"
resolvers += "Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/"
libraryDependencies += "com.typesafe.akka" % "akka-cluster-experimental_2.10.0-RC2" % "2.1.0-RC2"
Open command prompt and run sbt in the directory ScalaSBTProject. SBT will download and resolve whatever dependencies are required
Run the command eclipse at the SBT command line. This will generate all the eclipse related project files
Import ScalaSBTProject into Eclipse using File->Import->Existing Project to workspace, and make sure you check Import into workspace
EDIT: Just as a Post-Script, you can quite easily create a batch file to take the name of the project and generate the eclipse compatible project, just a way to speed up the process.
I manage my project using Maven and SBT at same time. The reasons for this are:
Intellij IDEA cannot import SBT
project.(idea-sbt plugin doesn't
work very well)
I don't know how can get sources and
javadocs from SBT.(I'd like to see any answers about this)
The problem is I don't know how to let Maven download SBT dependency. I search through maven repository and couldn't find anything about sbt. I wanna use Maven or SBT to manage all the jars in my project.
If you put a pom.xml to the root of your project, it will be recognized by SBT. When you specify no managed dependencies in the project definition, SBT relies on Maven dependencies.
As it said in SBT doumentation,
sbt performs this dependency handling
when the update action is executed. By
default, sbt does not update your
dependencies before every compilation,
but only does so when you execute
update. sbt supports three ways of
specifying these dependencies:
* Declarations in your project definition
* Maven POM files
* Ivy configuration and settings files
Maven knows nothing about SBT as of now (at least, I've not heard about any plugins so far), so, the best you can do to manage your project both in Maven and SBT, is to generate POMs by SBT. See SBT to Maven Converter for more details.
idea-sbt plugin works great for me with IDEA 10 - all it's really intended to do is open an SBT shell within the IDE and it does that well enough.
A plugin you should look into if you're interested in getting the Maven out of your build is sbt-idea plugin ( https://github.com/mpeltonen/sbt-idea ). This is a great plugin that generates IDEA files from an SBT project. It couldn't be easier to use. At an SBT prompt, run the following commands:
*sbtIdeaRepo at http://mpeltonen.github.com/maven/
*idea is com.github.mpeltonen sbt-idea-processor 0.3.0
update
idea
Note the asterisks - they should be included.
At this point, you can open your project in IDEA. It won't complain about the SBT dependencies. Any time you add new dependencies to your project file, simply run the 'idea' command again to tell IDEA about it. I do that in the SBT window provided by idea-sbt.
As far as getting sources and docs with dependencies, you can do something like this (from the SBT docs):
val sc = "org.scalacheck" % "scalacheck" % "1.5" withSources()
There is a corresponding withJavadoc() method. Hope that helps.
I just got started with Scala/LiftWeb/Sbt developing, and I'd like to import a Sbt project in IntelliJ Idea.
Actually, I managed to import my project in two different ways:
1) with Maven. I created a Maven project, and of top of that I created a Sbt project, which I then imported in IntelliJ. I could then easily start, stop the jetty server, and do other stuff.
But that's not what I want. I want to do the same stuff, just Maven-free.
That lead me to
2) with Eclipse. So, I created a new Sbt project (with a little script I wrote, configuring the Sbt project to be a WebProject). I used then the sbt-eclipsify plugin to 'convert' the project for Eclipse, which I then imported in IntelliJ (existing source -> eclipse).
But the problems started here: I cannot get the IntelliJ Sbt plugin to work.
Can anyone help me with this?
There are three basic ways how to create a project - modern versions of IntelliJ can import sbt project out of the box, otherwise you can either use sbt plugin to generate IntelliJ project, or use IntelliJ Scala plugin to create sbt project. Basic features work out of the box using both solutions, some complex builds can have problems, so try other tools to see if it works there.
IntelliJ
IntelliJ IDEA has become so much better these days. The current version (14.0.2) supports sbt projects out of the box with the Scala plugin. Just install the plugin and you should be able to open up Scala/sbt projects without any troubles.
With the plugin, just point at a sbt project and IDEA is going to offer you a wizard to open that kind of project.
IntelliJ Scala Plugin
IntelliJ plugin can be found here
http://confluence.jetbrains.com/display/SCA/Scala+Plugin+for+IntelliJ+IDEA or can be installed directoly from within the IDE using Settings -> Plugins dialog. Afterwards one can just do File -> New Project -> Scala -> SBT based. IntelliJ will generate basic build.sbt, download necessary dependencies and open project.
SBT Plugin
Sbt plugin that generate an idea project based on the sbt files can be found here: https://github.com/mpeltonen/sbt-idea
SBT 12.0+ & 13.0+
Simply add addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.5.2") to your build.sbt; no additional resolvers are needed.
Older Versions:
SBT 0.11+
Create and add the following lines to ~/.sbt/plugins/build.sbt OR PROJECT_DIR/project/plugins.sbt
resolvers += "sbt-idea-repo" at "http://mpeltonen.github.com/maven/"
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.6.0")
Use gen-idea in sbt to create IDEA project files.
By default, classifiers (i.e. sources and javadocs) of sbt and library dependencies are loaded if found and references added to IDEA project files. If you don't want to download/reference them, use command gen-idea no-classifiers no-sbt-classifiers.
SBT 0.10.1
(according to the plugin author, 0.10.0 won't work!)
Create and add the following lines to ~/.sbt/plugins/build.sbt:
resolvers += "sbt-idea-repo" at "http://mpeltonen.github.com/maven/"
libraryDependencies += "com.github.mpeltonen" %% "sbt-idea" % "0.10.0"
Use gen-idea sbt task to create IDEA project files.
By default, classifiers (i.e. sources and javadocs) of sbt and library dependencies are loaded if found and references added to IDEA project files. If you don't want to download/reference them, use command gen-idea no-classifiers no-sbt-classifiers.
SBT 0.7
To use it, simply run this from your sbt shell, it will use the plugin as an external program:
> *sbtIdeaRepo at http://mpeltonen.github.com/maven/
> *idea is com.github.mpeltonen sbt-idea-processor 0.4.0
...
> update
...
> idea
...
You can also add trait in your project definition, as you want:
import sbt._
class MyProject(info: ProjectInfo) extends ParentProject(info) with IdeaProject {
lazy val mySubProject = project("my-subproject", "my-subproject", new DefaultProject(_) with IdeaProject)
// ...
}
For now I do this by hand. It is quite simple.
Create the project with SBT
Create a new IDEA Project with the same root path
Create a module with the same root path
Set src/main/scala as a src path on the module
Set src/test/scala as a test path on the module
Add scala-library.jar as a library
Add lib (if it is present) as a jar directory within a module library
Add lib_managed/compile (if it is present) as a jar directory within a module library
Add lib_managed/test (if it is present) as a jar directory within a module library
That's it from memory. It would be better if it were automated, but it's no big deal as it is now.
One note of caution: The above approach doesn't work well with new-school sbt, i.e. versions 0.10 and newer, because it doesn't copy dependencies into lib_managed by default. You can add
retrieveManaged := true
to your build.sbt to make it copy the dependencies into lib_managed.
Tempus fugit and IntelliJ IDEA has become so much better these days. It's 2015 after all, isn't it?
Having said that, the latest version of IntelliJ IDEA 14.0.2 supports sbt projects out of the box with the Scala plugin. Just install the plugin and you should be able to open up Scala/sbt projects without much troubles.
I'm using the Early Access version of the plugin which is 1.2.67.6.EAP as of the time of the writing.
With the plugin just point at a sbt project and IDEA is going to offer you a wizard to open that kind of project.
About sbt-idea in sbt 0.12.4
For sbt 0.12.4 the system-wide plugin configuration file - ~/.sbt/plugins/build.sbt or PROJECT_DIR/project/plugins.sbt - should have the following lines:
resolvers += "Sonatype snapshots" at "http://oss.sonatype.org/content/repositories/snapshots/"
addSbtPlugin(dependency="com.github.mpeltonen" % "sbt-idea" % "1.5.0-SNAPSHOT")
Run sbt gen-idea to generate IDEA project files.
Read the sbt-idea plugin website for more up-to-date information. You may also find my blog entry Importing sbt-based project to IntelliJ IDEA 13 (with sbt-idea, Scala 2.11 and sbt 0.12) useful.
For sbt 0.7
See the answer elsewhere on this page.
For sbt 0.10
Clone and build Ismael's sbt-idea:
git clone https://github.com/ijuma/sbt-idea.git
cd sbt-idea
git checkout sbt-0.10
./sbt package
Create an sbt plugin lib directory if you don't have one already
mkdir -p ~/.sbt/plugins/lib
Copy the jar built in step one into here
cp sbt-idea/target/scala-2.8.1.final/*.jar ~/.sbt/plugins/lib
Restart or reload sbt, then you can run gen-idea (or gen-idea with-classifiers if you want sources and javadoc in intelliJ too)
Source: Tackers' suggestion on the message group.
In IntelliJ IDEA 13.x itself
You can open an SBT-based project in IDEA nowadays. It will create the necessary project and modules, and keep your dependencies up-to-date whenever you make changes to the build scripts.
I just went through all this pain. I spend days trying to get an acceptable environment up and have come to the conclusion that ENSIME, SBT and JRebel are going to be my development environment for some time. Yes, it is going back to Emacs, but ENSIME turns it into a bit or an idea with refactoring, debugging support, navigation, etc. It's not nowhere near as good as Eclipse (Java), but unless the scala plugins work better it's the best we have.
Until the Scala development environments get up to snuff (Eclipse or IntelliJ) I'm not going to bother. They're just way too buggy.
See the discussion on the lift site.
http://groups.google.com/group/liftweb/browse_thread/thread/6e38ae7396575052#
Within that thread, there is a link to a HOWTO for IntelliJ, but although it kinda works, there are many issues that render it a little less that useful.
http://blog.morroni.com/2010/07/14/setup-intellij-9-for-lift-framework-development/comment-page-1/
The answers are old for 2014.
In IntelliJ 13.x, the plugin Scala is ver 0.41.2 ( SBT is included).
My SBT version is 0.13.5 (terminal : sbt sbtVersion )
Go to the project's root folder and enter in the terminal
sbt idea
You will see two new hidden folders .idea and .idea_modules.
Then in IntelliJ, File > Open > select the project.
It should open the project without any problem.
Before you start creating your SBT project, make sure that the Scala plugin is downloaded and enabled in IntelliJ IDEA.
below link explains everything you need to know.
https://www.jetbrains.com/help/idea/2016.1/getting-started-with-sbt.html