So I have changed the "t" that is TimeStamp and i have to convert it in Double. I defined this class:
case class RawData(sessionId: String,
t: Double,
channel: Int,
signalName: String,
physicalValue: Double,
messageId: Long,
vehicleId: String)
And I have problem casting "t" into double in this code:
def raw(): Unit = {
import rawData.sqlContext.implicits._
//TODO solve timestamp
val datDMY = rawData
.map(row => {
cal.setTimeInMillis(row.t.)
RawDataExtended(
row.sessionId,
row.t,
row.channel,
row.signalName,
row.physicalValue,
row.messageId,
cal.get(Calendar.YEAR),
cal.get(Calendar.MONTH) + 1,
cal.get(Calendar.DAY_OF_MONTH)
)
})
Maybe this could help you:
case class InputModel(id: Int, time: Timestamp)
case class Foo(id: Int, timeLong: Long, timeDouble: Double)
val xs = Seq((1, Timestamp.from(Instant.now())), (2, Timestamp.from(Instant.now()))).toDF("id", "time")
val ys = xs
.select('id, 'time cast (DataTypes.LongType) as "timeLong", 'time cast (DataTypes.DoubleType) as "timeDouble")
.as[Foo]
val zs = xs
.as[InputModel]
.map(row => {
Foo(row.id, row.time.getTime.toLong, row.time.getTime.toDouble)
})
xs.show(false)
ys.show(false)
zs.show(false)
Although be careful time zones and also with precision when converting to Double - notice how Timestampt it is represented as Long compared to Double.
Related
Let's say I have two case classes:
case class C1(a: Int, b: String)
case class C2(a: Int, b: String, c: Long = 0)
I want to convert instance of C1 to C2 and then set additional field c. I found out the following solution:
C1.unapply(C1(1, "s1")).map(v => C2(v._1, v._2, 7l))
But specifying parameters one by one is not applicable, because real case class will have at least 15 params. Any ideas how to solve it?
This solution can be done by doing something like this thread.
How to append or prepend an element to a tuple in Scala
implicit class TupOps2[A, B](val x: (A, B)) extends AnyVal {
def :+[C](y: C) = (x._1, x._2, y)
def +:[C](y: C) = (y, x._1, x._2)
}
Usage:
val c1 = new C1(1, "s1")
val longVal = 0L
scala> C1.unapply(c1).map(r => (r :+ longVal))
res0: Option[(Int, String, Long)] = Some((1,s1,0))
scala> C1.unapply(c1).map(r => (C2.apply _) tupled (r:+ longVal))
res45: Option[C2] = Some(C2(1,s1,0))
Hope it helps :)
I think what you need is closer to
https://github.com/scalalandio/chimney
case class MakeCoffee(id: Int, kind: String, addict: String)
case class CoffeeMade(id: Int, kind: String, forAddict: String, at: ZonedDateTime)
val command = MakeCoffee(id = Random.nextInt,
kind = "Espresso",
addict = "Piotr")
import io.scalaland.chimney.dsl._
val event = command.into[CoffeeMade]
.withFieldComputed(_.at, _ => ZonedDateTime.now)
.withFieldRenamed(_.addict, _.forAddict)
.transform
I have a case class looks like this
case class EmotionData(
fearful: Double,
angry: Double,
sad: Double,
neutral: Double,
disgusted: Double,
surprised: Double,
happy: Double
)
I receive an Option[EmotionData] and I need each emotion data as an Option[Double].
What I did was:
val (fearful, angry, sad, neutral, disgusted, surprised, happy) = videoResult.emotion match {
case Some(e) => (Some(e.fearful), Some(e.angry), Some(e.sad), Some(e.neutral), Some(e.disgusted), Some(e.surprised), Some(e.happy))
case None => (None, None, None, None, None, None, None)
}
This way I have each field as an Option[Double] value.
But isn't there a way to do this in Scala where I can iterate through all fields of an object and extract them without rewriting each field?
Here's a slightly different approach that might be, perhaps, a little more palatable.
val vidEmo :Option[EmotionData] = videoResult.emotion
val (fearful, angry, sad, neutral, disgusted, surprised, happy) =
(vidEmo.map(_.fearful)
,vidEmo.map(_.angry)
,vidEmo.map(_.sad)
,vidEmo.map(_.neutral)
,vidEmo.map(_.disgusted)
,vidEmo.map(_.surprised)
,vidEmo.map(_.happy))
But really, you should just keep vidEmo around and extract what you need when you need it.
Yes, there is a way to iterate through the fields of an object by using productIterator. It would look something like this:
val List(fearful, angry, sad, neutral, disgusted, surprised, happy) =
videoResult.emotion.map(_.productIterator.map(f => Some(f.asInstanceOf[Double])).toList)
.getOrElse(List.fill(7)(None))
As you can see, this isn't much better than what you already have, and is more prone to error. The problem is that the number and order of fields is explicit in the result you have specified, so there are limits to how much this can be automated. And this only works because the type of all the fields is the same.
Personally I would keep the value as Option[EmotionData] as long as possible, and pick out individual values as needed, like this:
val opt = videoResult.emotion
val fearful = opt.map(_.fearful) // Option[Double]
val angry = opt.map(_.angry) // Option[Double]
val sad = opt.map(_.sad) // Option[Double]
val happy = opt.fold(0)(_.happy) // Double, default is 0 if opt is None
val ok = opt.forall(e => e.happy > e.sad) // True if emotion not set or more happy than sad
val disgusted = opt.exists(_.disgusted > 1.0) // True if emotion is set and disgusted value is large
Maybe this?
case class EmotionData(
fearful: Double,
angry: Double,
sad: Double,
neutral: Double,
disgusted: Double,
surprised: Double,
happy: Double
)
val s = Some(EmotionData(1,2,3,4,5,6,7))
val n:Option[EmotionData] = None
val emotionsOpt = s.map { x =>
x.productIterator.toVector.map(x => Some(x.asInstanceOf[Double]))
}.getOrElse(List.fill(7)(None))
// Or if you want an iterator:
val emotionsOptItr = n.map { x =>
x.productIterator.map(x => Some(x.asInstanceOf[Double]))
}.getOrElse(List.fill(7)(None))
println(emotionsOpt)
println(emotionsOptItr)
Which results in:
Vector(Some(1), Some(2), Some(3), Some(4), Some(5), Some(6), Some(7))
List(None, None, None, None, None, None, None)
You can do Something like that:
val defaultEmotionData=(0.0,0.0,0.0,0.0,0.0,0.0,0.0)
object Solution1 extends App{
case class EmotionData(
fearful: Double,
angry: Double,
sad: Double,
neutral: Double,
disgusted: Double,
surprised: Double,
happy: Double
)
case class EmotionDataOption(
fearfulOpt: Option[Double],
angryOpt: Option[Double],
sadOpt: Option[Double],
neutralOpt: Option[Double],
disgustedOpt: Option[Double],
surprisedOpt: Option[Double],
happyOpt: Option[Double]
)
val emotion = Some(EmotionData(1.2, 3.4, 5, 6, 7.8, 3, 12))
val ans: EmotionDataOption = emotion.getOrElse(defaultEmotionData).toOption
implicit class ToOption(emotionData: EmotionData) {
def toOption = EmotionDataOption(Some(emotionData.fearful), Some(emotionData.angry), Some(emotionData.sad), Some(emotionData
.neutral), Some(emotionData.disgusted), Some(emotionData.surprised), Some(emotionData.happy))
}
}
Now where ever you will have an object of type EmotionData you can use toOption on that and it will convert it's values into EmotionDataOption which will have values Option[Double].
If you will return Tuple7 then it will be tough to access values, that's why I think converting it into another case class EmotionDataOption is a good idea and you will be able to access the values easily with the parameter name.
I want to sum one columns data based on the key specified. Stream is like id(String) Key, value(Long).
val aggtimelogs: KTable[String, java.lang.Long] = stream
.groupByKey()
.aggregate(
() => 0L,
(key: String, value: java.lang.Long, aggregate: java.lang.Long) => value + aggregate)
//Failing here
Getting
Unspecified value parameters: : Materialized[K, VR, KeyValueStore[Bytes, Array[Byte]]
How to do in Scala?
Kafka version is
compile "org.apache.kafka:kafka-clients:2.0.0"
compile (group: "org.apache.kafka", name: "kafka-streams", version: "2.0.0"){
exclude group:"com.fasterxml.jackson.core"
}
Even I tried this
val reducer = new Reducer[java.lang.Long]() {
def apply(value1: java.lang.Long, value2: java.lang.Long): java.lang.Long = value1 + value2
}
val agg = stream
.groupByKey()
.reduce(reducer)
Also this
val reducer : Reducer[Long] = (value1: Long, value2: Long) => value1 + value2
Says
StreamAggregation.scala:39: type mismatch;
found : (Long, Long) => Long
required: org.apache.kafka.streams.kstream.Reducer[Long]
val reducer : Reducer[Long] = (value1: Long, value2: Long) => value1 + value2
I did it like this
val aggVal = streams.groupByKey().reduce(new Reducer[Double]() {
def apply(val1: Double, val2: Double): Double = val1 + val2
})
I have 2 questions
I am trying to learn scalacheck
Question 1)
Here is the test I am writing which is throwing the error. Can you please point to which page from docmentation i should read to understand reason behind this error.
case class Student(name:String, age:Int, mathsScore:Int, scienceScore:Int){
require(name != null ,"Name cannot be blank")
require(age > 3 ,"Age should be more than 3")
require(mathsScore >= 0 , "Score should not be negative")
require(scienceScore >= 0 ,"Score should not be negative")
val totalScore = mathsScore + scienceScore
}
Test is
object CaseStudySpecification extends Properties("Case Study Specification") {
property("nullName") = forAll { (name: String, age: Int, ms: Int, ss: Int) =>
if (name == null)
Prop.throws(classOf[IllegalArgumentException]) {
val x = Student(name, age, ms, ss)
}
}
}
Error is
No implicit view available from AnyVal => org.scalacheck.Prop.
[error] property("nullName") = forAll { (name: String, age: Int, ms: Int, ss: Int) =>
[error] ^
Question 2)
The official documentation gives one example test class as
property("stringLength") = Prop.forAll { s: String =>
val len = s.length
(s+s).length == len+len
}
I also read that it can be written as
val stringLength = Prop.forAll { s: String =>
val len = s.length
(s+s).length == len+len
}
How can i run the second form of test code , as when i run sbt test nothing happens for second version.
Both of the above snippets are in
object Ch3 extends Properties("String") {
}
The signature of Prop.forAll being called requires a function returning Prop (or at least something that can be implicitly converted to Prop) but, as written, the function:
(name: String, age: Int, ms: Int, ss: Int) => {
if (name != null) Prop.throws(...)
}
has an inferred signature of (String, Int, Int, Int) => AnyVal, and there does not exist an implicit conversion into a property. Hence the compilation error. The function could be trivially fixed by making sure it always returns a Boolean as follows:
property("nullName") = forAll { (name: String, age: Int, ms: Int, ss: Int) =>
if (name == null) Prop.throws(classOf[IllegalArgumentException]) {
Student(name, age, ms, ss)
}
else true
}
This results in the implicit Boolean => Prop function being applied, and the code compiles. A more idiomatic fix would be to rewrite the property using the implication operator:
property("nullName") = forAll { (name: String, age: Int, ms: Int, ss: Int) =>
name == null ==>
Prop.throws(classOf[IllegalArgumentException]) {
Student(name, age, ms, ss)
}
}
However it is not a good idea to reject too much of the generated input and since the very first value that scalacheck generates is in fact null, the property ends up as 'undecided' so the test still fails. You could just simplify your property to:
property("nullName") = forAll { (age: Int, ms: Int, ss: Int) =>
Prop.throws(classOf[IllegalArgumentException]) {
Student(null, age, ms, ss)
}
}
As this isn't a scalacheck-specific problem but rather a general Scala one it isn't specifically covered in the scalacheck documentation; you can read up on implicit views for more background.
I'm trying to parse the csv files and I need to determine the type of each field starting from its string value.
for examples:
val row: Array[String] = Array("1/1/06 0:00","3108 OCCIDENTAL DR","3","3C","1115")
this is what I would get:
row(0) --> Date
row(1) --> String
row(2) --> Int
Ecc....
how can I do?
------------------------------------ SOLUTION ------------------------------------
This is the solution I've found to recognize the fields String, Date, Int, Double and Boolean.
I hope that someone can serve in the future.
def typeDetection(x: String): String = {
x match {
// Matches: [12], [-22], [0] Non-Matches: [2.2], [3F]
case int if int.matches("^-?[0-9]+$") => "Int"
// Matches: [2,2], [-2.3], [0.2232323232332] Non-Matches: [.2], [,2], [2.2.2]
case double if double.matches("^-?[0-9]+(,|.)[0-9]+$") => "Double"
// Matches: [29/02/2004 20:15:27], [29/2/04 8:9:5], [31/3/2004 9:20:17] Non-Matches: [29/02/2003 20:15:15], [2/29/04 20:15:15], [31/3/4 9:20:17]
case d1 if d1.matches("^((((31\\/(0?[13578]|1[02]))|((29|30)\\/(0?[1,3-9]|1[0-2])))\\/(1[6-9]|[2-9]\\d)?\\d{2})|(29\\/0?2\\/(((1[6-9]|[2-9]\\d)?(0[48]|[2468][048]|[13579][26])|((16|[2468][048]|[3579][26])00))))|(0?[1-9]|1\\d|2[0-8])\\/((0?[1-9])|(1[0-2]))\\/((1[6-9]|[2-9]\\d)?\\d{2})) *(?:(?:([01]?\\d|2[0-3])(\\-|:|\\.))?([0-5]?\\d)(\\-|:|\\.))?([0-5]?\\d)")
=> "Date"
// Matches: [01.1.02], [11-30-2001], [2/29/2000] Non-Matches: [02/29/01], [13/01/2002], [11/00/02]
case d2 if d2.matches("^(?:(?:(?:0?[13578]|1[02])(\\/|-|\\.)31)\\1|(?:(?:0?[1,3-9]|1[0-2])(\\/|-|\\.)(?:29|30)\\2))(?:(?:1[6-9]|[2-9]\\d)?\\d{2})$|^(?:0?2(\\/|-|\\.)29\\3(?:(?:(?:1[6-9]|[2-9]\\d)?(?:0[48]|[2468][048]|[13579][26])|(?:(?:16|[2468][048]|[3579][26])00))))$|^(?:(?:0?[1-9])|(?:1[0-2]))(\\/|-|\\.)(?:0?[1-9]|1\\d|2[0-8])\\4(?:(?:1[6-9]|[2-9]\\d)?\\d{2})$")
=> "Date"
// Matches: [12/01/2002], [12/01/2002 12:32:10] Non-Matches: [32/12/2002], [12/13/2001], [12/02/06]
case d3 if d3.matches("^(([0-2]\\d|[3][0-1])(\\/|-|\\.)([0]\\d|[1][0-2])(\\/|-|\\.)[2][0]\\d{2})$|^(([0-2]\\d|[3][0-1])(\\/|-|\\.)([0]\\d|[1][0-2])(\\/|-|\\.)[2][0]\\d{2}\\s([0-1]\\d|[2][0-3])\\:[0-5]\\d\\:[0-5]\\d)$")
=> "Date"
case boolean if boolean.equalsIgnoreCase("true") || boolean.equalsIgnoreCase("false") => "Boolean"
case _ => "String"
}
}
val row: Array[String] = Array("1/1/06 0:00","3108 OCCIDENTAL DR","3","3C","1115")
val types: Array[String] = row.map(x => x match {
case string if string.contains("/") => "Date probably"
case string if string.matches("[0-9]+") => "Int probably"
case _ => "String probably"
})
types.foreach( x => println(x))
Outputs:
Date probably
String probably
Int probably
String probably
Int probably
But in all honesty I wouldn't use this approach, this is so error prone and there are so many things that could go wrong that I don't even want to think about it, the simplest example is what if a string contains a /, this small piece of code would match that as a Date.
I don't know your use-case but in my experience it's always a bad idea to create something that tries to guess types form unsecure data, if you have control over it you could introduce some identifier, for example "1/1/06 0:00 %d%" where %d% would indicate a date and so on and then remove it from the string, and even then you'll never be 100% sure that this won't fail.
For each string: try parsing it into the type you want. You'll have to write a function for each type. Keep trying in order until one of them works, order is important. You can use your favorite Date/Time library.
import java.util.Date
def stringdetect (s : String) = {
dateFromString(s) orElse intFromString(s) getOrElse s
}
def arrayDetect(row : Array[String]) = row map stringdetect
def arrayTypes(row : Array[String]) = {
arrayDetect(row) map { _ match {
case x:Int => "Int"
case x:Date => "Date"
case x:String => "String"
case _ => "?"
} }
}
def intFromString(s : String): Option[Int] = {
try {
Some(s.toInt)
} catch {
case _ : Throwable => None
}
}
def dateFromString(s : String): Option[Date] = {
try {
val formatter = new java.text.SimpleDateFormat("d/M/yy h:mm")
formatter.format(new java.util.Date)
Some(formatter.parse(s))
} catch {
case _ : Throwable => None
}
}
From the REPL / worksheet:
val row: Array[String] = Array("1/1/06 0:00","3108 OCCIDENTAL DR","3","3C","1115")
//> row : Array[String] = Array(1/1/06 0:00, 3108 OCCIDENTAL DR, 3, 3C, 1115)
arrayDetect(row)
//> res0: Array[Any] = Array(Sun Jan 01 00:00:00 CST 2006, 3108 OCCIDENTAL DR, 3 , 3C, 1115)
arrayTypeDisplay(row)
//> res1: Array[String] = Array(Date, String, Int, String, Int)