I am facing issue in compiling 'TokenizerExample' that comes along with Spark-Scala package.
I have setup my environment in IntelliJ and I am able to successfully compile other Spark-Scala examples such as NaiveBayes, CosineSimilarity etc.
But when I load the 'TokenizerExample' into IntelliJ IDE, system displays an error message on the below line stating 'Cannot resolve reference transform with such signature':
val tokenized = tokenizer.transform(sentenceDataFrame)
val regexTokenized = regexTokenizer.transform(sentenceDataFrame)
I have not done any edit and I could observe that the issue is with the transform method. Could you please help me address this issue? Appreciate your support.
Thanks!
SBT/Maven, make sure you have all the dependencies listed in the .sbt or pom.xml file for the mllib module and rebuild. I personally didn't work on mllib, but this kind of errors will show up if we don't resolve the dependencies well enough. Thanks.
Related
I'm learning Scala and following this video https://www.youtube.com/watch?v=yU85EowqhY4
But when I tried to run my AkkaHttpJson class I end up getting this error
Symbol 'type akka.stream.scaladsl.Flow' is missing from the classpath. This symbol is required by 'method akka.http.scaladsl.server.RouteResult.routeToFlow'. Make sure that type Flow is in your classpath and check for conflicting dependencies with `-Ylog-classpath`. A full rebuild may help if 'RouteResult.class' was compiled against an incompatible version of akka.stream.scaladsl. Http().newServerAt("localhost", 8081).bind(route)
Why am I getting this error? Everything is exactly as shown in the video.
Is it the dependencies that are causing these issues?
Also, If I make a project using Scala 2.* and then want to convert to Scala 3, would I always face issues?
I'm really new to Scala, so I appreciate all help!
Posted the question here and got the answer :)
I am trying to use brunel on a spark scala notebook on IBM datascience experience.
%AddJar -magic https://brunelvis.org/jar/spark-kernel-brunel-all-2.2.jar
%%brunel data(leadsDF) map x(state) y(count) color(state)
I always get this error:
Name: Error parsing magics!
Message: Magics [brunel] do not exist!
StackTrace:
Is there a import needed for brunel?
I tested this Scala 2.11 with spark 2.0 and it worked for me.
%AddJar -magic https://brunelvis.org/jar/spark-kernel-brunel-all-2.2.jar
Then i used below to display and it showed me the map.
%%brunel data('co2agg') map(low) x(CO2_per_capita) color(Mean_Co2) tooltip(#all):: width=800, height=500
Example reference is from
https://github.com/Brunel-Visualization/Brunel/tree/master/spark-kernel/examples
I have fully functioning example here for scala 2.11:-
https://apsportal.ibm.com/analytics/notebooks/97e83c35-06a2-476a-aa57-078a20f04356/view?access_token=8d2f4f749aab9abbf45a08351f7b50f7fb09f06ad9aa7bab5ebc9a2cc98e902f
I tested this with scala 2.10 and i get some dependecy error, i am guessing it is because of brunel 2.2 may be dependent scala 2.11.
I hope that helps.
Thanks,
Charles.
I had the same problem where magics where not working correctly. For me this action from the "Known Issues" fixed it:
Run the following code in a Python notebook to remove the existing Scala libraries: !rm -rvf ~/data/libs/*
When I develop an IDEA plugin, I want to use akka, but have some problems.
I created a demo project here: https://github.com/freewind/idea-plugin-akka-demo
You can just clone it and reproduce the problem on your computer. (Notice the Setup section)
And I copy the problem here:
Problems
1. Can't use default akka configuration
If I removed:
src/main/resources/application.conf
src/main/scala/freewind/MyAkkaConfig
and run this plugin, it will report this error when starting:
com.intellij.ide.plugins.PluginManager$StartupAbortedException:
com.intellij.diagnostic.PluginException: No configuration setting found for key 'akka'
[Plugin: com.yourcompany.unique.plugin.id]
2. Can't load the configuration from file
Then I copied the reference.conf from akka jar, to src/main/resources/application.conf, but it still report the same error. Seems akka in IDEA plugin can't find this file automatically.
3. ClassNotFoundException: akka.actor.LightArrayRevolverScheduler
So I have to use MyAkkaConfig.scala to hardcode the configuration in scala code, but this time, it reports another error:
com.intellij.ide.plugins.PluginManager$StartupAbortedException:
com.intellij.diagnostic.PluginException: ClassNotFoundException: akka.actor.LightArrayRevolverScheduler
[Plugin: com.yourcompany.unique.plugin.id]
The akka.actor.LightArrayRevolverScheduler is used in MyAkkaConfig.scala, and is included in akka-actor_2.11:2.3.12:jar. But why IDEA can't load it?
For the 3rd problem, it can be fixed by passing the classloader:
val system = ActorSystem("my-actor", MyAkkaConfig.config, this.getClass.getClassLoader)
But we also can remove the MyAkkaConfig.config, to use the file application.conf under resources
I am very interested at using the Scala script engine that we have now in scala 2.11
I saw that within the REPL everything works but if I do it from a test (I tried from sbt and maven) then the ScriptEngine for scala is not found.
I tried that to see which ScriptEngine are present
println("manager:" + manager.getEngineFactories.size())
manager.getEngineFactories.toList.foreach(factory => println(factory.getClass.getSimpleName))
But I only get NashornScriptEngine (I have JDK8)
Does anybody have an idea why I can't find it? I had previously issues with NashornScript in SBT that could be resolved by added fork in Test but this is not the issue here.
It is as if the ScriptEngine is not present in the classpath when you are outside the REPL
Did you add the REPL (and, maybe, the compiler?) to the classpath? I imagine that if you don't specify the dependency and bring only the library, it would definitely not work.
If you did do that, perhaps if you could be more forthcoming with details on how you are doing this test?
I have Intellij-IDEA 13.1.2. [edited, previously 13.0.2]
I use the scala-plugin.
I'm trying to use worksheets to evaluate code.
But all I got are two errors :
bad macro impl binding: versionFormat is supposed to be there
Unable to read an event from: rO0ABXNyADVvcmcuamV0YnJhaW5zLmpwcy5pbmNyZW1lbnRhbC...
I can run the scala console normally and execute my code in it,
but the worksheet doesn't function.
If my code is incorrect, it outputs an error indicating the interpreter failed to parse my code. I got the "bad macro impl binding" error only if my code is correct.
I tried creating a new project, but it didn't work.
I followed tutorial to configure scala in intellij but it didn't help either.
Is there an important configuration step I may have missed ? What does that error mean ?
EDIT : I tried the simplest thing in my worksheet like 1 or var x = 1 or println("Hello World!")
EDIT2: I'm not sure what I changed but now I have another error :
Error:error while loading MacroPrinter, class file needed by MacroPrinter is missing.
reference value macros of package reflect refers to nonexisting symbol.
UPDATE: Now it works fine under Intellij 13.1.5, Scala plugin 0.41.2 with both scala-2.10 and 2.11
I had the "Unable to read an event from" issue and switching from Scala 2.11.0 to Scala 2.10.4 fixed it for me.
I've shared a workaround in a similar question and I think it can work for this question as well:
I'm having the same issue with latest Idea and Scala plugin.
It seems that the worksheet has a problem executing any line which evaluates to Unit. Assigning is Unit, that's why your tableTest(0) = "zero" fails.
I've temporarily solved it with the following workaround:
this line will fail with error Error:Unable to read an event from:...
println("Will fail")
You can fix it by defining this helper method and using it for any Unit expression:
def unit(f: => Unit): String = {f; ""}
unit(println("Will work"))
You just have to ignore the line it generates in the output panel with res0: String =
You also can put this method in some object and import in any WS you need.
I am using IntelliJIDEA 13.1.2 and Scala Plugin 0.36.431. I tried to create a Scala Non-SBT project and created a worksheet file, then I met the "bad macro impl binding" problem. But if I created a Scala SBT project with a worksheet file, it worked well.
After updating to IntelliJ 13.1.5 build 135.1289 with JRE 1.7.0_60 and Scala 2.11.0 I had the same issue.
I fixed the problem by disabling the checkbox in settings -> IDE Settings -> Scala -> "Run compile server".