I have a string e.g. "100,123,124" and wanted to convert into Set[Int] data type. e.g Set[100,123,124]
How to convert this in Scala?
Simply use:
"100,123,124".split(",").map(_.toInt).toSet
Related
My underlying function is defined like this:
def rowToSHA1(s: Seq[Any]): String = {
//return sha1 of sequence
}
}
Here is the definition of my udf:
val toSha = udf[String, Seq[Any]](rowToSHA1)
df.withColumn("shavalue",(toSha(array($"id",$"name",$"description",$"accepted")))
It works when i pass only a list of string as parameter but i get an error when there is a boolean.
org.apache.spark.sql.AnalysisException: cannot resolve 'array(`id`, `name`,
`description`, `accepted`)' due to data type mismatch: input to function
array should all be the same type, but it's [string, string, string,
boolean];;
I'm exploring the use of a generic function, is it a good idea?
FIX: converted my column to string before applying the function
df.withColumn("shavalue",(toSha(array($"id",$"name",$"description",$"accepted".cast("string)))
The best solution I know for this kind of situation is just convert everything to String, When you read/create the DataFrame make sure everything is String or convert it at some point. Later you can convert if back to any other type.
I have a String:
val x = """{foo:"value1", bar:"value2"}"""
and I want to convert it into JsString.
val converted = JsString(x)
Now, if i print converted, following result is printed:
"{foo:\"value1\", bar:\"value2\"}"
However, I don't want the \ added in the string. Is there any other way avoid this auto escaping without using string.replace?
Try
println(JsString("""{foo:"value1", bar:"value2"}""").value)
"{foo:"value1", bar:"value2"}"
is not a valid JSON, that's why quotes are escape by JsString. Indeed, how would a JSON parser interpret this, if the internal quotes are not escaped?
If you want a (JVM) String with the JSON object inside, you already have it. If you want a JSON string, representing a JSON object, you MUST have escaping characters.
If you want the JSON object, you can always use
val obj: JsValue = Json.parse("""{foo:"value1", bar:"value2"}""")
I have an array in Scala e.g. Array("11","112","as2") and I want to parse it into a string:
"['11','112','as2']"
How can I do it in Scala?
mkString is one way to go. The scala REPL is great for this sort of thing:
scala> Array("11","112","as2")
res0: Array[String] = Array(11, 112, as2)
scala> "['"+res0.mkString("','")+"']"
res1: String = ['11','112','as2']
But if you are creating JSON, perhaps something from your JSON library would be more appropriate?
So I'm basically doing this
// data type
type FeatureTuple = ((String, String), Double)
featureTuple.flatMap(_._1._2)
But it's returning a RDD[Char] and not RDD[String], why would this happen?
The flatMap is causing the String to be implicitly converted to a WrappedString which is a TraversableOnce[Char] (because the flatMap method of RDD is expecting an argument of type T => TraversableOnce[U] and a String can be converted into a TraversableOnce[Char]), and so you will end up with a RDD[Char] instead of an RDD[String]. If that implicit conversion did not exist (it is defined in scala.Predef), then you'd just get a compile error.
Anyway, to fix this just use map instead of flatMap.
I am writing scala application, and want to utilize my java library. I have java API function that takes inputs as Object; however, the scala application that calling such function has input type as string. my code is as following:
val data = Array("foo", "bar")
val dataSource = new MyJavaAPIDataProvider(data)
Because I am using MyJavaAPIDataProvider(Object[] data), the scala code keeps telling me type mismatch. How do I change my scala code data array from a string array to Object array? Thanks
You have to cast the array to Array[Object]
val objs = data.asInstanceOf[Array[Object]]
val dataSource = new MyJavaAPIDataProvider(objs)
Edit:
to complete this answer with Seth Tisue's comment, you also can use:
val data = Array[AnyRef]("foo", "bar")
To avoid having to cast. AnyRef is equivalent to Java's Object.
Based on that, another possibility that comes to my mind is to create an array of AnyRef with the elements of your string array, like this:
Array[AnyRef](data: _*)