I have installed spark-hadoop env in my Red Hat 64. And I also want to read and write code in spark source code project in intelliJ idea. I have downloaded spark source code and make everything ready. But I had some errors when compiling spark project in IntelliJ idea.
Here are errors:
/home/xuch/IdeaProjects/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystQI.scala
Error:(809, 34) not found: value SparkSqlParser
case ast if ast.tokenType == SparlSqlParser.TinyintLiteral =>
Error:(812, 34) not found: value SparkSqlParser
case ast if ast.tokenType == SparlSqlParser.SmallintLiteral =>
... ...
But actually I did not find a file named SparkSqlParser.scala in the whole project neither a scala class named SparkSqlParser.
However, I had searched the web for some files named SparkSqlParser.scala, but they don't have attribute like "TinyintLiteral", "SmallintLiteral", etc.
Here are the files link:
https://github.com/yjshen/zzzzobspk/blob/master/sql/core/src/main/scala/org/apache/spark/sql/SparkSQLParser.scala
https://apache.googlesource.com/spark/+/c152dde78f73d5ce3a483fd60a47e7de1f1916da/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SparkSQLParser.scala
I meet the same problem. Here is my solution:
Download the antlr4 (i.e. antlr v4) plugin of IntelliJ. Then you can see the file "spark-2.0.1\sql\catalyst\src\main\antlr4\org\apache\spark\sql\catalyst\parser\SqlBase.g4" can be recognized by IntelliJ IDEA.
Navigate to View->Tool Windows->Maven Projects tab. select the project "Spark Project Catalyst". Right click on it. Then select "Generate sources and update folders"
After that you can see some files added into the "spark-2.0.1\sql\catalyst\target\generated-sources\antlr4"
Then you can build success of the project.
Hope it can help you.
None of the advice here worked for me. I noticed, however, that the generated code depends on Antlr 3.x while Antlr 4.x is what is in the dependencies (mvn dependecy:tree). I don't know why this was the case. Maybe because I had earlier built it from the command line (?).
Anyway, try cleaning your Catalyst sub-project then rebuild the autogenerated sources. To do this in IntelliJ, go to View -> Tools Window -> Maven Projects.
Then navigate to the "Spark Project Catalyst" in the "Maven Project" tab.
Navigate to clean -> clean:clean and double click it. Navigate to Plugins -> antlr4 -> antlr4:antlr4 and double click it.
Now, you'll see the autogenerated sources of the Antlr classes are different and they should compile. YMMV.
1) First build your Spark from command line using build instructions given in http://spark.apache.org/docs/latest/building-spark.html#building-with-buildmvn
2) Then check
$SPARK_HOME/sql/catalyst/target/generated-sources/antlr3/org/apache/spark/sql/catalyst/parser folder.
Some of the generated classes like SparkSqlLexer.java is there.
List of classes it generates are
SparkSqlLexer.java[enter link description here][1]
SparkSqlParser.java
SparkSqlParser_ExpressionParser.java
SparkSqlParser_FromClauseParser.java
SparkSqlParser_IdentifiersParser.java
SparkSqlParser_KeywordParser.java
SparkSqlParser_SelectClauseParser.java
3) Open Module Settings. Click on spark-catalyst module. Go to Source tab in the right. Make target/generated-source as a source folder.
I also faced similar problem when I updated my fork to latest master. Unfortunately, could not find a way to make it work from IDEA. What I did is compiled the project from command line. It generated the antlr classes which is required. I then added the generated-source target/generated-source/antlr as source directory. After that I could run tests from Idea. Ideally Idea generate source should have generated the code. Need to check more why it did not. May be because I have maven3.3.3 configured.
I have did as the intruction from Rishitesh Mishra and get stuck in the first step. I have always errors when executing "build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package".
I have both tried on source code from https://spark.apache.org and fork on github.
I have attached the log screenshot in a new answer as following.
error log image
Related
I've created a Play Framework program via Typesafe Activator (so it follows the template exactly).
I used sbteclipse-plugin version 3.0.0 to create an Eclipse project and imported that into Scala IDE 4.0.0. These are all the latest versions at the time of writing.
The Scala IDE definitely seems to support the Play Framework. It has syntax highlighting for the custom formats, including the routing file and templates. Yet, it doesn't seem to be able to find the views from the controllers. In particular, the call to views.html.index triggers an error: "object index is not a member of package views.html".
I tried enabling refreshing using native hooks or pooling as detailed here, but it had no affect.
I should note that while the code has been compiled in the command line (with activator ~run), it hasn't been compiled in Scala IDE, since I don't know how to (it doesn't seem to be documented anywhere).
What can I do to get rid of these false errors?
EDIT: After running activator clean ~run, I have another error: The project cannot be built until build path errors are resolved. There's no further details on what these build path errors are.
Update: Just upgrade to sbteclipse version 5.1.0 and everything should work out of the box. Also make sure you follow the Play documentation on how to set up Eclipse/ScalaIDE.
This is a known bug in sbteclipse, which probably will be fixed soon.
For now, you can add the following line to your build.sbt:
EclipseKeys.createSrc := EclipseCreateSrc.All
Kill the SBT console and run sbt eclipse again. That should add the following line to the .classpath file within your project folder as a workaround:
<classpathentry kind="src" path="target/scala-2.11/twirl/main"/>
Refresh your Eclipse project to pick up the change.
I had the same issue, also with Scala IDE 4.0.0 . I followed mkurz instuctions and they worked like a charm. But instead of changing the .classpath file in the project folder manually I used Eclipse interface:
In the top menu of the main window, click on Project and then on Properties.
In the Properties window, click on Java Build Path option (options list is on the left)
In the Source tab, click on Add Folder... button.
In the Source Folder Selection window, choose the target/scala-2.11/twirl/main folder, so it is included in the compilation path. Click Ok button.
Click Ok in the Properties window.
Now the project should compile just fine :) . With that I was able to finish the play setup example in Scala IDE website
I tried #mkurz solution first, but also ran into the same error as #matt. I became frustrated that I could not generate the eclipse project without having to go to the Eclipse project properties to manually fix the build errors. After some investigation, I discovered the solution that removed all errors entirely. Add this your build.sbt:
unmanagedSourceDirectories in Compile <+= twirlCompileTemplates.target
Or if that does not work for you, you could also use:
unmanagedSourceDirectories in Compile <+= target.zipWith(scalaBinaryVersion) { (b,v) => b / s"scala-$v/twirl/main" }
Good bye, build errors!
I got the same error message.
Are you using java8 as jre in eclipse?
After switching back from java8 to java7, everything worked fine again.
If, after following Mkurz' instructions (adding EclipseKeys.CreateSrc... ), your problems are not solved, click on Project -> Properties -> Java Build Path. Look at the source folders tab.
You may find a duplicate file folder named .../src_managed/main (Thanks Matt). If so, close the project. Remove ONE of the two ../src_managed/main entries from the .classpath file (located in the base of the activator/SBT project directory). Reopen and clean the project and you should be good to go.
For me, it turned out that installed JRE in the Scala IDE was openjdk, changed it to Oracle Java 8 and it worked.
I've got a scala project that compiles, runs and tests fine when using SBT from the command line. However, when building the project in intellij, it seems every class in the project has this error in the event log, causing the build to fail:
SendCommandToService is already defined as case class SendCommandToService
case class SendCommandToService(service: String, commandName: String, keys: Array[String], values: Array[String])
^
For me, the reason is that both myproject/src and myproject/src/main/scala are marked as Source. So IntelliJ failed to build myproject/src/main/scala due to above errors. Unmark Source from myproject/src (in IntelliJ, File->Project structure, select myproject Module, select src folder in Sources Tab, remove it from Source in the "Add Content Root" pane) solved the problem.
It means there are two compiled classes with identical package and class name found in your classpath. One compiled by sbt, one compiled by IntelliJ.
One of the following should be able to solve the issue:
try to generate IntelliJ .iml file with sbt-idea rather than import directly.
sbt clean before click Build -> Rebuild in IntelliJ
when rebuilding with IntelliJ, make sure sbt is not running
I ran into this issue today on IntelliJ 2021.2.1 and according to this page it's some issue with IntelliJ's incremental compiler for Scala, so the solution is to change the "Incrementality Type" from "IDEA" to "Zinc" in Preferences -> Build, Execution, Deployment -> Compiler -> Scala Compiler
For me, the solution was to double check the source folders in each of my modules in IntelliJ.
File > Project Structure > Modules and for each module, double check that the Source Folders only contain your intended folders, e.g. src/main/scala, and do not contain any generated sources (e.g. target/scala-2.12/src_managed/main.
I had the same problem and #Max is right, there is a conflict with the compiled classes, but the solution provided didn't work for me. It turns out that I was using sbt-idea to generate the IDEA project structure as a workaround of an Intellij IDEA 14 + scala plugin bug on the SBT import, that is not fixed yet at the time I write this.
In order to fix it, I had to remove src_managed/main/controller that was in conflict with src_managed/main in the Module settings because of an sbt-idea bug. So double-check your module source folders and make sure you don't have subfolders in conflict with a parent folder already declared as source.
You need to change "Settings -> Build,Execution,Deployment -> Scala Compiler -> Compile order" from "Mixed" to "Java then Scala".
If you have compile the project previous, you should first run "sbt clean".
I had a similar issue repeatedly both within Idea and without: plain SBT.
It turned out that CVS stores copies of some *.scala files in subdirectory CVS/Base, which SBT apparently tries to compile. The problem went away when I deleted the CVS subdirectories.
Problem is caused by duplicated line in .idea/modules/<your_project_name>.iml file. Check if you do not have duplicated <source_folder> tag.
In my case I had the same problem with all classes in src/test/scala path, and after removal duplicated tag for this path, project build fine.
Do you have any other files in your project with an SendCommandToService in them?
You could try renaming it to something else, see if that works
If you want to keep the same names, you can put them into separate packages.
Or have them in different encapsulating objects
object traitdemo{
object Ex1{
...
}
}
object otherdemo{
object Ex1 {
...
}
}
that will work even in the same file
In my case problem solved by change ScalaTest template configuration in Idea. I select use sbt, disable print info, remove build before launch.
I like to use SBT for clean/package/test on specific module. I also use mixed Java/Scala classes in test (but I replace compile order to Java than Scala).
At least now I can test from IDE withot this error.
PS: Now I disable use sbt. My tests work fine (but I'm not sure, that they will work).
PPS: New tests not runs before compilation. It is disadvantage of removing build (and, maybe, of disabling use sbt). But this extra build cause problem with dublication, as I think
File -> Invalid Caches/Restart worked for me. All other answers here did not.
After the sbt compile I had to mark the folder as Generated Sources Root because I needed those files for compilation.
I'll just add mine to the list in case anyone else made this beginner mistake: I temporarily "saved my progress" by doing cp Foo.scala Foo-save.scala, forgetting that sbt would try to compile all the .scala files in the directory.
(I don't know, I guess I was thinking of programming languages where any file not explicitly included was ignored ...)
Of course, since both the main file and the "temporary backup" file defined the same classes ... yeah.
I had the same error message and it turned out that IntelliJ for some reason created duplicate copies of some existing source files. For example I had a file Attribute.scala that was tracked with git and then there was an untracked file Atrribute 2.scala with the same contents in the same directory (which I never created). This was of course a problem, because the compiler considers them part of the project, hence the duplicate object definition error.
I am not 100% sure when this happened (I suspect it was during git rebase). So, if you run into this problem again, it's also worth checking with git status if you have some untracked files which duplicate contents of tracked files.
Remove the untracked files and the problem is solved.
Kudos to this question thread, it helped me to solve this issue.
My case is a project with the mix of Scala, Java and Avro schemas.
IDE: IntelliJ IDEA 2022.1.3
How I solved it step by step (in IntelliJ):
File -> Project Structure
Project Settings -> Modules
Now we need to fix the "source" files. I searched for target/scala-2.12/src_managed in each module and marked it like “Source directory” (blue colour).
Unmarked all other paths like target/scala-2.12/* in all modules (in my case it was target/scala-2.12/src_managed/main/compiled_avro). As an example from my project, I left only these sources in one of the modules: target/scala-2.12/src_managed , src/main/scala.
Save the changes and rebuild the project.
ADDITION: Oh, and looks like sometimes this error occurs when you compile your scala project (with avro files) outside of the IntelliJ. For example when you build a project using sbt externally and then run some tests using the IntelliJ -> class already defined error occurs (or is not a member of package error). In such case: you need to run sbt clean compile externally first and go to step #1.
this happen when you incorrectly src/main/any other folder as Sources Root. Please check if you have any such cases. If yes, then Unmark those by righclick on it. then clic one level above of your groupId starting. Lets say your package is com.company.test and com may comes under java or scala, then right click on that(java/scala) then Mark as Sources Root.
In my case, the problem was the protobuf Idea plugin:
Remove the idea protbuf plugin.
Close Idea
Remove all folders related with idea (.idea and .idea_modules)
Open Idea and Import the project again.
Problem: I cannot setup Run Configurations to run scalatest for the Scalatests in my project.
Steps to reproduce:
Right click on Scala Suite and click on Run as -> Run configurations..
On the left, I see a configuration template for ScalaTest. I click on New and fill the Name but it cannot find the suite-class.
Note: It is mentioned here that I should see Run as -> ScalaTest - Suite but I do not see that option. I tried using context menu in the editor, and in the package explorer
Steps taken:
Using: Scala IDE for Eclipse version: 2.1.0.m3-2_09
Using SBT, assemble project, run eclipse command and then import project and dependencies into Eclipse
Project compiles. ScalaTest code compiles(scalatest_2.9.2-1.8.jar is in the 'Referenced Libraries' configuration)
I've been fighting a similar problem for the past few days; Lily / Jimbo's answer didn't quite match my situation, but helped me find the right direction.
In my case, I was using a third-party library that I'd copied in. The package names of the classes and tests matched, but the folder structure did not -- all of my tests were directly in Play's "/test" folder, rather than in folders that matched the package names. This didn't show any errors, but was broken: packages ought to match folders. When I built the right folder structure underneath test, and recompiled, the expected "Run As -> ScalaTest - Suite" options showed up.
Don't know if your problem is the same, but you might check this if you haven't already found the issue...
This could be caused by a misalignment between the scalatest and the scala eclipse IDE version. Try scalatest_2.9.0-2.0.M5b.jar or scalatest_2.10-2.0.M5b.jar. The former jar definitely works with ide 2.0.9.x so maybe the new version needs the 2.10 jar. Pick your version carefully from here
Willem's answer is what worked for me. Getting both plugins from the same update site (from the list on Scalatest's github site), seemed to work for me using Kepler.
for my case, one click on 'Reimport All Maven Projects' icon, like 'Refresh' icon, solved the problem.
Not sure if you fixed the error, but I had similar a error yesterday and was pulling my hair trying to fix it (none of the suggestions I found by googling seemed to help me). So for me, it turns out that it's as easy as package hierarchy in my test suite.
I am using the play framework, so naturally my folders look like this controllers.package1.package2.... and this applies to my test folders also.
Now my test classes however, have the package definition package1.package2..... (no "controllers" as prefix).
If I run the tests on sbt/play command prompt, it's not a problem. But running them through Eclipse would give me the problem you described.
So anyways... thought I'd share this, in case this could help.
it's a bit annoying combined with the view template compile issue in play framework. but my approach is to regenerate the eclipse project file and add view template path into the class path
In Eclipse with groovy plugin, I try to launch a test #Test public void testToLaunch() {...} but I have the error:
The input type of the launch configuration does not exist
What input type is in the context of launch configuration? (can't find such an entry in launch configuration window)
Note: I try sts 2.8.1 and e3.7
This happens normally when the folder in which test case is present is not a source folder, please check this post as well.
Hope that helps!
This can also happen if there is a problem with the groovy class. A few things to check:
1) Ensure that the class name exactly matches the filename (filename = MyTest.groovy)
package com.mypackage;
import groovy.util.GroovyTestCase;
class MyTest extends GroovyTestCase {}
2) Ensure that the package defined in the file matches the package the file is actually in.
In Eclipse you can do
Right click -> properties -> Java build path
Notice test folder is not available in sources. Add it.
Add folder -> Select test -> OK
Now rerun you unit test cases.
This happened to me, and I just restarted Eclipse (GGTS) and everything was fine again.
I had a spelling mistake which lead to that error message. My test class file name was named JUnit5Test.java (with upper U) and the class itself was named Junit5Test (with lower u).
I was using Spring Tool Suite 4 (4.8.0.RELEASE).
This also happened to me. But these tests are written in Groovy. The problem I encountered has to do with how the IDE (Eclipse Kepler, Java EE) first opens a Groovy project after executing "mvn eclipse:eclipse".
The Build Paths do not reference the Groovy source files correctly.
To resolve, I:
Right-click on the project, select "Build Path" > "Configure Build Path..."
Select "Source" tab
For test and src folders (.../src/main/groovy, and .../src/test/groovy)
make sure "**/*.groovy" is set as "Inclusion patterns", not "**/*.java"
Hope this saves time for someone.
Cheers!
I had the same error message when I head the test-class duplicated both in the main Java source folder and the testsrc folder. Removing the incorrectly placed one in the main Java source folder solved the problem for me.
2019 Update: This drove me crazy for days even with latest Eclipse and fresh installs (Mac, Grails 4, Gradle 5.1.1, Java 8). Some above examples led me to the solution.
My problem was more that the code I was testing included a mix of groovy and java src/main code. It gave me NoClassDefFound on the .groovy classes when I ran my Spec as JUnit.
Solution: I had to modify my Run/Debug Configuration to include build/classes/groovy/main. Then it worked. It's a little bit of a pain to remember to that for every new Configuration, but, it keeps me going. I hope it helps you.
Whenever you create a Junit test in eclipse, make sure your Junit test file is inside src/test/java folder.
I had a similar problem. Like others have already pointed out, it was about source folders. I had to change my source folder setup. There was an empty src-folder that disappeared after I right-clicked on it and selected 'remove from build path' from Build path menu. After that I right-clicked both java/src and java/test folders and chose Build path > Use as a source folder. And suddenly my tests were JUnited!
In similar situations I'd advice to remove all source folders from build path and add them again when you're sure you've got the right ones. Your source folders should be those with Java package structure under them. In case of proj/java/test/com/stackoverflow/main it's the 'test' folder.
This is what resolved for me (Eclipse Oxygen). I had already done what Robert suggested in the earlier post. I was still getting the error. When I went to edit the configuration for junit launch, I saw that the Test Class field just had the class name. I had to hit the Search button to the right. The Test Class field now had the complete name for the class
com.mycompany.mypackage.MyClass
With this I am able to run the JUnit. But I have to keep fixing this for every run.
Found another way to cause this message. The cause turned out to be an empty copy of MyTest.java under src/main/java, as well as the real one under src/test/java.
Think the empty file was a hangover from some refactoring and was oddly causing no compile errors either. Deleting it enabled the test to run again.
I am new to intellij idea and I decided to shift because I found the scala plugin on eclipse to be annoyingly buggy.
But, intellij idea, which I have heard to be very good, looks difficult to use.
I looked through the getting started with scala plugin page but couldn't find documentation specific to the latest version of lightweight ide of intellij idea 11 for scala.
I have created a project and made a hello world object in scala but I am not able to run it.
On trying to run it, it shows the scala interpreter (type in expressions to get them evaluated).
I don't exactly understand what changes to make in the project structure to be able to run scala code. As of now, i added the jsdk to point to my $JAVA_HOME and the scala library is also added. The scala plugin is also working in that syntax highlighting etc is fine.
The intellij idea user interface is also not very friendly and I don't get how to run the code. What modules am I supposed to add? Also, does compiled scala code run with jvm?
I'll be grateful to anyone who helps out.. I have been putting off writing scala code because I am not able to find THE IDE.
EDIT: Thanks for all the help! I am now able to run a basic scala program. I am trying to add external jars. What is the equivalent of doing Project right click-> Configure Build Path -> adding external jar files in eclipse? I tried adding "dependencies " under modules tab in project structure. It's adding the jar files but somehow the classes in the jar files are not still recognised.
With the Scala plugin installed either
A. Create a new project and select the Scala facet in the creation phase. It will create the Scala library lib and Scala compiler lib and setup the facet for you
B. If you already have a project. Go to Project Structure -> Modules and right click the module and go Add facet and add a Scala facet. Now you need to add scala-library.jar as a library of the module and go into the Scala facet and point it to a library containing scala-compiler.jar
Some more information
this is what your module should look like under project settings
select the Scala facet and this is what you should see (Library name for the compiler is unimportant as long as it says (version xxx) next to the library name
these are the jar files in my scala-compiler lib
and these are the jar files in my scala-library lib
With everything setup like that you should be able to right click -> new Scala class (Select object from the dropdown) in a source directory and add
def main(args:Array[String]) {
println("Hello world")
}
then right click on the class and select Run Classnam.main() and IntelliJ will setup a run configuration for you.
Edit: I can see someone posted somewhat the same I want to say here. I hope this can give a few more in-depths hints, so I will post it anyway :-)
If you would like to run Scala native in IntelliJ (I shifted to IntelliJ and still can't get my hands down) you need to make sure a few things are in order. Most of this happens under File -> Project Structure which requires you to create a project (when you get a hang of it, it is much more logical than Eclipse -- and Netbeans for that matter). I am also assuming you have selected a jdk, but this can also be done in the Project Structure.
Now, to Scala:
Make sure you have the Scala plugin. Visit the Settings (File -> Settings; can also be found at the start-up screen) and locate 'Plugins' (for me it's nr. 8 from the bottom). Under "Browse Repositories" you should be able to search for "scala", find and install a plugin.
1 A. If no plugins pops up you are probably using a version of intellij where scala is not (yet) supported. If that happens you can go to the homepage for the plugin and download the nightly build.
Import Scala as a library in File -> Project Structure -> Global Libraries. Click the plus sign, select Java (Scala is a Java-lib), browse to your scala-dir (which can be downloaded here: http://plugins.intellij.net/plugin/?id=1347) and select the lib directory. Press ok and you should see the library popping up on the list.
Make sure the facet (Scala framework) is in place. Goto File -> Project Structure -> Facets. Click on the plus-sign and find Scala. A sublink should appear (if not double-click the Scala link). Clicking the sublink make sure you have selected a compiler before continuing. This should be provided by the modules selected before. I normally use FSC (Fast Scala Compiler) but it's probably a good idea to test the basic settings before experimenting.
Make sure the Scala Module is in place. In Project Structure -> Modules click on the module you want to compile in scala and click the plus icon above and select Scala. Again be sure to select a compiler library.
If you want to, setup configurations for the compiler under the facet
Use sbt with sbt-idea plugin to generate idea project files for complex projects.
Or
For simple projects.
Add scala nature to the project, and you can right click and run any file extending App.
A good walk-through on how to set up a Scala project in Intellij is located here:
http://sonyarouje.com/2011/03/18/running-scala-in-intellij-idea-10/