Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 2 years ago.
Improve this question
What am i doing wrong here? I am trying to explode the json column, it looks ok to me, but keep getting this error? Can someone help me please?
import org.apache.spark.sql.functions._
import spark.implicits._
val yearname = baby_names.withColumn("data".explode($"data"))
.withColumn("year",$"data"(8))
.withColumn("name",$"data"(9))
.select("year","name")
command-3936897808825418:4: error: not enough arguments for method withColumn: (colName: String, col: org.apache.spark.sql.Column)org.apache.spark.sql.DataFrame.
Unspecified value parameter col.
val yearname = baby_names.withColumn("data".explode($"data"))
I think you mean withColumn("data",explode($"data")) with a comma , separating the 2 arguments. That way it matches the profile:
def withColumn(colName: String, col: Column): DataFrame
The problem is due to the third line , I suggest to change it to:
import org.apache.spark.sql.{functions => F}
val yearname = baby_names.withColumn("data", F.explode(F.col("data")))
Related
Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 9 months ago.
Improve this question
This is some of the strange issue, the program is only printing hello scala, but it seems that I am getting syntax weird error. Please correct me if I miss anything
Program
object hello{
def main(args:Array[String]){
println("Hello Scala")
}
}
Error:- '=' expected, but '{' found
As the error message suggests, you need a = before the {:
def main(args: Array[String]) = {
println("Hello Scala")
}
The syntax you used was originally supported so you might see it in some example code. But it was deprecated in later versions and is now no longer allowed.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
Actually, I have several Qs regarding to this code snippet:
Does '???' cause this exception?
What could be assigned instead of '???' ?
What does '???' stand for?
object InfixTypesHard2 extends App {
val x0: Int --- String --- Boolean = ???
}
class ---[T, V](t: T, v: V)
Stacktrace:
Exception in thread "main" scala.NotImplementedError: an implementation is missing
at scala.Predef$.$qmark$qmark$qmark(Predef.scala:344)
at section3_OOP_operatorOverloading.InfixTypesHard2$.delayedEndpoint$section3_OOP_operatorOverloading$InfixTypesHard2$1(InfixTypesHard2.scala:5)
at section3_OOP_operatorOverloading.InfixTypesHard2$delayedInit$body.apply(InfixTypesHard2.scala:3)
at scala.Function0.apply$mcV$sp(Function0.scala:39)
at scala.Function0.apply$mcV$sp$(Function0.scala:39)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
at scala.App.$anonfun$main$1(App.scala:76)
at scala.App.$anonfun$main$1$adapted(App.scala:76)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
at scala.App.main(App.scala:76)
at scala.App.main$(App.scala:74)
at section3_OOP_operatorOverloading.InfixTypesHard2$.main(InfixTypesHard2.scala:3)
at section3_OOP_operatorOverloading.InfixTypesHard2.main(InfixTypesHard2.scala)
Is just a method in Predef nothing special about it from the language point of view.
As you can see the implementation is just throwing a new NotImplementedError.
The idea of that method is that you can use it as a filler to "implement" any method, and the idea of the fancy name was that it could be noticed easily since the idea is that any usage of ??? should be temporary and fixed latter.
And if you wonder why it can be used in any place, it is because ??? itself is of type Nothing; this because the act of throwing an exception has such type. And since Nothing is the subtype of all types (also known as the bottom type) it can always be used due Liskov.
What could be assigned instead of '???' ?
Something of type: Int --- String --- Boolean; which is just a weird tuple re-implementation.
// For example:
new ---(new ---(1, "text"), false)
// Thanks to Jasper Moeys for providing the right syntax.
I guess this is some homework, no idea what you were asked to do here.
Or maybe, the whole idea of the snippet is to show that you can use types as infix when they have two type parameters.
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 5 years ago.
Improve this question
Newbie to Scala.
I encountered this error while compile the code.
Error:(84, 130) type mismatch;
found : String
required: Array[String]
val mappingStr = "Mapping Strings: \n" + stringIndexers.map(r=>Array(r.getInputCol, r.labels.mkString(", "))).reduce(_+"\n"+_.mkString(": \n")) + "\n"
^
the hat char points at "Array" of my code.
I didn`t see any issue, can anyone help to explain why?
You map a list of some items into a list of Array[String], because it's what Array() apparently returns for each element of stringIndexers.
Then you try to reduce this List[Array[String]] by +-ing Strings. This expects _ in reduce to be a String, but it is not, it's an Array[String].
You should provide a way to convert your arrays into strings, or maybe flatten your list of arrays first, it's hard for me to tell which your intention is.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 5 years ago.
Improve this question
I'm putting some test cases and having lots of parameters yet it is cumbersome and not so easy to understand putting them into a liner in scala. I rather specify in a visually acceptable manner but correctly parse them in scala...
I would use String Interpolation to define the arguments such as below
val kafkaServers = "server1:9092,server2:9092"
val pleasentArgs = s"""
--master='spark://someserver'
--metricWindowSize=50
--outputDirectory='/tmp/someGoodDirectory'
--zkQuorum=$zkQuorum
--saveToHDF=false
--userName='jack#somewhereElse.com'
--kafkaServers=$kafkaServers
"""
And have a method to clean up all the newline, tab etc characters and then generate the args as shown below:
def cleanUp(str:String) = str.split('\n').map(_.trim.filter(_ >= ' ')).mkString(" ")
def genArgs(args:String): Array[String] =cleanUp(args).split(" ").filter(!_.trim.isEmpty)
So eventually we can just call genArgs method
val args = genArgs(pleasentArgs)
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
as the questions suggests. Would it ever make sense to even think about generics on non-collections? I have been trying to think this through. A collection is really just a 'wrapper' of other objects, so there has to be a 'wrapper' to direct a generic at?
thanks
Yep. As a minimal example, here is a class that wraps something of type T, where T is generic:
scala> class Wrapper[T](val x: T)
defined class Wrapper
If you give it an Int, then it's a Wrapper[Int]:
scala> new Wrapper(5)
res0: Wrapper[Int] = Wrapper#578ef2b6
If you give it a String, then it's a Wrapper[String]:
scala> new Wrapper("this")
res1: Wrapper[String] = Wrapper#3b16bf07
This is directly analogous to a collection of items of type T:
scala> Vector(1,2,3)
res2: scala.collection.immutable.Vector[Int] = Vector(1, 2, 3)
Yes. It is possible and it makes perfect sense. Pretty much all of the type system of Scala revolves around them. Well, them and the "abstract types".
To learn more you can start from here. Googling for "Scala type parameters" will also give you plenty of useful results.