I need the ability to parse XML files in Scala for a regression modelling project of mine. It seems that there is no longer a scala.xml package ready form the get-go so we need it externally. Solution seems to be the project scala.xml from GitHub: https://github.com/scala/scala-xml
Thing is, in order for me to get the JAR file to Eclipse it seems I need to use sbt. I have sbt installed but the regression modelling project was originally made in Eclipse by File -> New -> Scala Project.
Last time I used sbt was when I tried to get ScalaFX to work in Eclipse. I then understood why they changed the name from Simple Build Tools to Scala Build Tools. It was pure hell to get the JAR file (which I did not get by using sbt).
The only way to get the library scala.xml in Scala version 2.12 is via sbt. So now the situation is that I need to make modifications to sbt.build file which doesn't (?) exist in the Eclipse project as the project wasn't made using sbt. How do I do this?
Answering to my own question for anyone having this same problem:
You can make a Git project into a JAR file really easily. What you need is git commands and sbt installed. Here's what you do
i) Open up any directory. Preferably make a new directory with descriptive name.
ii) Go to the Git project you want, click the green box "Clone or download" and copy the url.
iii) Open console, go to the directory you want the project and type git clone _ where _ is the url of the project.
iv) Once the project is cloned open it with console.
v) Type sbt and wait until sbt sets everything up.
vi) Depending on the Scala version you want to use you can do it now after compiling.
vii) Type compile and wait for the sbt to compile.
viii) Once compiling is done type package and you have the JAR file in the projects target directory. After compiling path is shown in the console.
Related
I have installed spark-hadoop env in my Red Hat 64. And I also want to read and write code in spark source code project in intelliJ idea. I have downloaded spark source code and make everything ready. But I had some errors when compiling spark project in IntelliJ idea.
Here are errors:
/home/xuch/IdeaProjects/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystQI.scala
Error:(809, 34) not found: value SparkSqlParser
case ast if ast.tokenType == SparlSqlParser.TinyintLiteral =>
Error:(812, 34) not found: value SparkSqlParser
case ast if ast.tokenType == SparlSqlParser.SmallintLiteral =>
... ...
But actually I did not find a file named SparkSqlParser.scala in the whole project neither a scala class named SparkSqlParser.
However, I had searched the web for some files named SparkSqlParser.scala, but they don't have attribute like "TinyintLiteral", "SmallintLiteral", etc.
Here are the files link:
https://github.com/yjshen/zzzzobspk/blob/master/sql/core/src/main/scala/org/apache/spark/sql/SparkSQLParser.scala
https://apache.googlesource.com/spark/+/c152dde78f73d5ce3a483fd60a47e7de1f1916da/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SparkSQLParser.scala
I meet the same problem. Here is my solution:
Download the antlr4 (i.e. antlr v4) plugin of IntelliJ. Then you can see the file "spark-2.0.1\sql\catalyst\src\main\antlr4\org\apache\spark\sql\catalyst\parser\SqlBase.g4" can be recognized by IntelliJ IDEA.
Navigate to View->Tool Windows->Maven Projects tab. select the project "Spark Project Catalyst". Right click on it. Then select "Generate sources and update folders"
After that you can see some files added into the "spark-2.0.1\sql\catalyst\target\generated-sources\antlr4"
Then you can build success of the project.
Hope it can help you.
None of the advice here worked for me. I noticed, however, that the generated code depends on Antlr 3.x while Antlr 4.x is what is in the dependencies (mvn dependecy:tree). I don't know why this was the case. Maybe because I had earlier built it from the command line (?).
Anyway, try cleaning your Catalyst sub-project then rebuild the autogenerated sources. To do this in IntelliJ, go to View -> Tools Window -> Maven Projects.
Then navigate to the "Spark Project Catalyst" in the "Maven Project" tab.
Navigate to clean -> clean:clean and double click it. Navigate to Plugins -> antlr4 -> antlr4:antlr4 and double click it.
Now, you'll see the autogenerated sources of the Antlr classes are different and they should compile. YMMV.
1) First build your Spark from command line using build instructions given in http://spark.apache.org/docs/latest/building-spark.html#building-with-buildmvn
2) Then check
$SPARK_HOME/sql/catalyst/target/generated-sources/antlr3/org/apache/spark/sql/catalyst/parser folder.
Some of the generated classes like SparkSqlLexer.java is there.
List of classes it generates are
SparkSqlLexer.java[enter link description here][1]
SparkSqlParser.java
SparkSqlParser_ExpressionParser.java
SparkSqlParser_FromClauseParser.java
SparkSqlParser_IdentifiersParser.java
SparkSqlParser_KeywordParser.java
SparkSqlParser_SelectClauseParser.java
3) Open Module Settings. Click on spark-catalyst module. Go to Source tab in the right. Make target/generated-source as a source folder.
I also faced similar problem when I updated my fork to latest master. Unfortunately, could not find a way to make it work from IDEA. What I did is compiled the project from command line. It generated the antlr classes which is required. I then added the generated-source target/generated-source/antlr as source directory. After that I could run tests from Idea. Ideally Idea generate source should have generated the code. Need to check more why it did not. May be because I have maven3.3.3 configured.
I have did as the intruction from Rishitesh Mishra and get stuck in the first step. I have always errors when executing "build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package".
I have both tried on source code from https://spark.apache.org and fork on github.
I have attached the log screenshot in a new answer as following.
error log image
I followed the getting start video on Jetbrains website to setup IntelliJ IDEA 13.1 Community Edition to work with Scala. Scala plugin v0.36.431 had been installed. While I created a new Scala SBT project with wizard, there was no src/ directory structure generated in the project. Only two sbt files were generated:
scala-course/
├── build.sbt
└── project
└── plugins.sbt
From the video and other document I know that there should be a src/ directory structure, including src/main/scala, src/test/scala, etc. sbt uses the same directory structure as Maven for source files by default.
I can create those folders manually and mark it as source root. However it is trivial. So my question is: Why IntelliJ IDEA new project wizard doesn't generate the directory structure as said in document? Was I doing something wrong? I checked the preferences and couldn't find anything that seems related.
Normally it should create these folders automatically. It may take a while though - it takes couple of seconds in my case.
When creating project make sure you have selected Scala -> SBT, then proceed with the wizard.
Once the Finish is clicked, the project will be loaded. This part takes couple of seconds, and I can see no src/main/scala nor src/test/scala generated until it's done. Observe the bottom of the screen to see when it's done.
Once the process is finished, you'll see the folders.
If that's not the case, check the settings. You should have the Create directories for empty content roots automatically checked. You may want to check Use auto-import to automatically propagate changes in the build.sbt.
After changing the settings (if the change is required) you may need to refresh the project, as seen in picture below.
This can also happen if you do not have a JDK selected. For some reason you no longer get the option to select an SDK so you must make sure you have configured this before hand. To fix this do the following:
From the welcome screen, go to
Configure -> Project defaults -> Project structure and add the jdk.
Source:
What's the reason for "Error:Cannot determine Java VM executable in selected JDK"?
Thanks to lpiepiora, with his hint I find out the reason.
Because my sbt is newly installed, there is nothing in ~/.ivy2/cache/ and ~/.sbt/boot/. sbt needed to download required dependencies from repositories on network. It happened that my proxy to internet had something wrong, download stuck.
And also need to notice that, if quit IntelliJ IDEA when sbt is running in background, the next time you'll get error of waiting for some lock file. Have to remove the lock file on filesystem and restart IntelliJ IDEA again.
After fixed the network problem, everything work as promised. It requires several minutes, depends on network speed, to download required jar files. After finished, the src/ directory structure is created.
I followed the instructions in this thread but I had a java crash in the final phase in sbt with the configuration bellow and I think this info maybe useful:
The problem happened with IDEA 2016.2, sbt 0.13.8 (I tried later to import using 0.13.12 but the crash was the same), scala 2.11.8 and ubuntu 16.04.
The only way I could make it work was to use java 8 instead of 9.
error: error while loading package, Missing dependency 'object java.lang.Object in compiler mirror', required by /home/jbamaral/.sbt/boot at xsbt.boot.Boot.main(Boot.scala)
...
stack log here
...
[error] scala.reflect.internal.MissingRequirementError: object java.lang.Object in compiler mirror not found.
I have developed a scala application for the first time, but I have to deploy it with a "one-click" type script that can run and build the scala application from source WITHOUT ECLIPSE.
Since I'm completely new to scala I don't know how to tell it where all my source files are etc... to get it to build my app from the command line. I also have 2 3rd party .jar libraries that I need to tell the scala compiler to link to...
Any documentation on this? Or example command lines? My project hierarchy is:
src/packagename: contains all .scala
bin/packagename: contains all.class files
libs/ -> contains 2 .jar files I will need to import somehow
I'm working on debian linux
EDIT: I found this ability to export in eclipse so I created a .java file and called my main scala object from it. Then I exported as a runnable jar. However, when I go to run the new runnable jar "sudo java runnable.jar" it says "class not found exception: runnable.jar"
You should take a look at https://github.com/harrah/xsbt/wiki which is the common way to build a Scala project. Run through the tutorial in the wiki to learn how you should organise your directory structure, so that everything may run fine.
If you want to combine it with eclipse, checkout this plugin: https://github.com/typesafehub/sbteclipse
I'm added a dependence to my build.sbt (casbah). I did a sbt update, I did check my ~/.ivy2/cache directory and all jars are there. Do I have to add this ~/.ivy/cache directory to my Build Path and add the casbah as external Jar to my project? If no, probably no because I did try it, what should I do to be able to use this jar in my scala project?
EDIT
I found this instructions that helped me, but still a hack
Establish a simple project (general/project) named "IvyCache"
located at your ".ivy2/cache" folder just for library reference
purposes.
Establish a Scala project located at your "project" folder.
Add the following libraries to the Scala project by means of "Add
JARs" to the "Java Build Path":
3.a) All jars from "/IvyCashe/org.scala-tools.sbt" filterred by
"*2.9.1-0.11.2" or any other Scala/SBT version numbers.
3.b) A single sbinary_*.jar from "/IvyCache/org.scala-tools.sbinary".
3.c) A single test-interface*.jar from "/IvyCache/org.scala-
tools.testing".
Now your build files should compile within Eclipse.
The easiest way to manage this is to use the eclipse plugin for sbt. Then you can just say sbt eclipse on the command line any time you change the dependencies in build.sbt, and the Eclipse files will be automatically updated for you.
Doing it this way means that you will never have to manually configure your Eclipse build path. After all, sbt already knows how to construct the build path, so there's no reason you would have to do it manually.
I have a project that is build with cmake. In my cmakelists I have a
add_subdirectory(externals/foo)
to build the dependency "foo" which has it's own cmakelists.
Now it is so that also the whole foo sources and headers are included in the generated Project file (I'm using Eclipse). But all I want is to only have my project available in Eclipse (Eclipse has problems with subprojects in the same folder structure).
So that the cmakelists from "foo" is only used to build "foo" automatically and link it to my project. I don't want to see it in my IDE however.
Is this possible? If yes: How?
When I use Eclipse with CMake, I create the Eclipse project manually (with the New Project wizard) and for CMake I use the standard makefile generator. It requires a little extra setup: you need to set the build directory in the project properties if you're doing an out-of-source build, and I usually set the build command to make VERBOSE=1.
I'm not sure since I haven't used the Eclipse generator(s), but for the lack of a better solution, perhaps this method would solve your issue, since it gives you more control over the Eclipse project.
No, how should the ide know what to compile if you don't tell it what to compile? If you don't want to have the project in you project file, just don't add it.
Just compile the external lib by itself (use "cmake externals/foo") and then add the libraries in your project's CMakeLists.txt with
target_link_libraries(your_project externals/foo/bin/foo.lib)