I am trying to create a generic code to read using spark sql from a view. Where T can be any object passed at runtime, I should be able to get DataSet of T.
Method
def queryStream[T](sparkSession: SparkSession, query: String, filterParams: Array[String])(implicit tag: ClassTag[T]): Dataset[T] = sparkSession.sql(query)
.as(Encoders.bean[KafkaMessage](classOf[KafkaMessage]))
.map(f => {
init()
getObjectMapper.readValue(f.message, classTag[T].runtimeClass).asInstanceOf[T]
})(Encoders.bean[T](classOf[T]))
Invocation :
queryStream[Student](sparkSession,"select cast(value as string) as message,"
+ "cast(key as string) as messageKey, " +
"cast(partition as int) partition, " +
"cast(offset as long) offset from events",null)
KafkaMessage:
class KafkaMessage {
#BeanProperty
var messageKey: String = _
#BeanProperty
var message: String = _
#BeanProperty
var partition: Int = _
#BeanProperty
var offset: Long = _
override def toString(): String = s"Message Key: $messageKey, Message: $message, Partition:$partition, Offset:$offset"
}
Related
I am trying to parse a complex json which has different schema for the same field in a same message. So I am trying to parse it via Either, but play json doesn't supports Either natively. So I tried to write custom reads and write function which has some syntax issue and quite tedious to figure out. Can some one help me with this?
Input json
{
"settings":[
{
"active":false,
"parameter":{
"x1":0,
"x2":"Simple"
},
"type":"sometype"
},
{
"active":true,
"parameter":[
{
"y1":100,
"y2":"east",
"y3":{
"blue":true,
"green":false
}
}
],
"type":"someDifferentType"
}
]
}
I have tried below code which has syntax problem
case class Xparameter(x1: Int, x2: String)
object Xparameter {
implicit val format: Format[Xparameter] = Json.format[Xparameter]
}
case class Yparameter(y1: Int, y2: String, y3: Color)
object Yparameter {
implicit val format: Format[Yparameter] = Json.format[Yparameter]
}
case class Color(blue: Boolean, green: Boolean)
object Color {
implicit val format: Format[Color] = Json.format[Color]
}
case class SettingLeft(
active: Boolean,
`type`: String,
parameter: Xparameter
)
object SettingLeft {
implicit val format: Format[SettingLeft] = Json.format[SettingLeft]
}
case class SettingRight(
active: Boolean,
`type`: String,
parameter: List[Yparameter]
)
object SettingRight {
implicit val format: Format[SettingRight] = Json.format[SettingRight]
}
case class Setting(
active: Boolean,
`type`: String,
parameter: Either[Xparameter, List[Yparameter]]
)
object Setting {
implicit def apply(active: Boolean,
`type`: String,
parameter: Xparameter
): Setting = Setting(active,`type`, Left(parameter))
implicit def apply(active: Boolean,
`type`: String,
parameter: List[Yparameter]
): Setting = Setting(active,`type`, Right(parameter))
implicit val format: Format[Setting] = new Format[Setting] {
override def reads(json: JsValue): JsResult[Setting] = json
.validate[SettingLeft]
.map(SettingLeft.apply) // <-- not sure if its possible to do
.orElse(
json
.validate[SettingRight]
.map(SettingRight.apply)
) // <--- syntax error
override def writes(json: Setting): JsValue = json.either match {
case Left(value) => Json.toJson(value)
case Right(value) => Json.toJson(value) // <--- syntax error
}
}
}
I'm using Scala with Play framework (2.8.1) and have a Sort class for capturing sort based query string parameters. Sample url: http://myurl:9000?sortBy=name&sortOrder=asc. Here, the sortOrder field is optional and will have default value of "asc" if nothing is provided. I have implemented own QueryStringBindable class as below:
object Sort {
val asc = "asc"
implicit def queryStringBinder(implicit stringBinder: QueryStringBindable[String]) = new QueryStringBindable[Sort] {
override def bind(key: String, params: Map[String, Seq[String]]): Option[Either[String, Sort]] = {
for {
sortBy <- stringBinder.bind("sortBy", params)
if(params.contains("sortOrder")) {sortOrder <- stringBinder.bind("sortOrder", params)}
} yield {
(sortBy, sortOrder) match {
case (Right(sortBy), Right(sortOrder)) => Right(Sort(sortBy, Some(sortOrder)))
case _ => Left("Unable to bind Sort")
}
}
}
override def unbind(key: String, sort: Sort): String = {
stringBinder.unbind("sortBy", sort.sortBy) + "&" + stringBinder.unbind("sortOrder", sort.sortOrder.getOrElse(asc))
}
}
}
case class Sort(sortBy: String, sortOrder: Option[String] = Some(Sort.asc))
However, I'm unable to capture optional field sortOrder with default value if nothing is provided in the query string of url. I would want http://myurl:9000?sortBy=name to still sort by ascending order (default) even if &sortOrder isn't provided.
It might be easier not to use a for comprehension:
object Sort {
val asc = "asc"
implicit def queryStringBinder(implicit stringBinder: QueryStringBindable[String]) = new QueryStringBindable[Sort] {
override def bind(key: String, params: Map[String, Seq[String]]): Option[Either[String, Sort]] = {
val sortBy = stringBinder.bind("sortBy", params) // Option[Either[String, String]]
val sortOrder = stringBinder.bind("sortOrder", params) // Option[Either[String, String]]
val result = // Either[String, Sort]
(sortBy, sortOrder) match {
case (Some(Right(field)), Some(Right(order))) => Right(Sort(field, Some(order)))
case (Some(Right(field)), None) => Right(Sort(field))
case _ => Left("Unable to bind Sort")
}
Option(result)
}
override def unbind(key: String, sort: Sort): String = {
stringBinder.unbind("sortBy", sort.sortBy) + "&" + stringBinder.unbind("sortOrder", sort.sortOrder.getOrElse(asc))
}
}
}
case class Sort(sortBy: String, sortOrder: Option[String] = Some(Sort.asc))
I am trying to create 2 auxiliary constructors .But not able to do so.
class learnMultipleAuxuliaryConstructor(firstname: String,
lastName: String,
middleName: String) {
println("This is primary constructor")
println("Complete Name is " + firstname + lastName + middleName)
def this(firstname: String) {
this(firstname, "", "")
println("This is Auxiliary constructor with firstname")
println("First Name is " + firstname)
}
def this(lastname: String) {
this("", lastname, "")
println("This is Auxiliary constructor with lastname")
println("lastname is " + lastname)
}
}
Can i do this?
You have to have distinct constructor signatures. So:
class learnMultipleAuxuliaryConstructor(firstname: String,
lastName: String,
middleName: String) {
def this(firstname: String) ...
def this(lastname: String) ...
}
is not possible. For such case it would be better to have either default parameters and named parameters:
class learnMultipleAuxuliaryConstructor(firstname: String = "",
lastName: String = "",
middleName: String = "")
new learnMultipleAuxuliaryConstructor(lastName = "Custom")
or separate methods in companion objects with distinct names:
object learnMultipleAuxuliaryConstructor {
def fromFirstName(firstName: String) = ...
def fromLastName(lastName: String) = ...
}
Using this class I'm attempting to modify the String parameter of the constructor and convert it to date :
class OrderInstance(var size: Double,
var side: String,
var trade_id: Int,
var price: Double,
var time: java.util.Date,
var code: String) {
def getTime(time: String): java.util.Date = new java.util.Date()
def this(size: String,
side: String,
trade_id: Int,
price: String,
time: String, code: String) = this(size.toDouble: Double,
side: String,
trade_id: Int,
price.toDouble: Double,
getTime(time: String) : java.util.Date, code: String)
// Step 3 - proper signature for `equals`
// Steps 4 thru 7 - implement a `match` expression
override def equals(that: Any): Boolean =
that match {
case that: OrderInstance => {
that.canEqual(this) &&
this.trade_id == that.trade_id
}
case _ => false
}
// Step 1 - proper signature for `canEqual`
// Step 2 - compare `a` to the current class
def canEqual(a: Any) = a.isInstanceOf[OrderInstance]
// Step 8 - implement a corresponding hashCode c=method
override def hashCode: Int = {
val prime = 31
var result = 1
result = prime * result + trade_id;
result = prime * result + (if (code == null) 0 else code.hashCode)
result
}
override def toString = {
String.valueOf(trade_id)
}
}
This line :
def getTime(time: String): java.util.Date = new java.util.Date()
should be called in order to return a date instead of String. But I receive error :
Error:(54, 7) not found: value getTime
getTime(time: String) : java.util.Date, code: String)
This error is reported at run time, not compile time. Have I not defined the function getTime correctly ?
Update :
I've attempted to make more clear what I'm trying to achieve :
class OrderInstance(var size: Double,
var side: String,
var trade_id: Int,
var price: Double,
var time: String,
var code: String){
this(time) = this(OrderInstance.getTime(time))
def this(size: String,
side: String,
trade_id: Int,
price: String,
time: String, code: String) = this(size.toDouble: Double,
side: String,
trade_id: Int,
price.toDouble: Double,
time : String, code: String)
// Step 3 - proper signature for `equals`
// Steps 4 thru 7 - implement a `match` expression
override def equals(that: Any): Boolean =
that match {
case that: OrderInstance => {
that.canEqual(this) &&
this.trade_id == that.trade_id
}
case _ => false
}
// Step 1 - proper signature for `canEqual`
// Step 2 - compare `a` to the current class
def canEqual(a: Any) = a.isInstanceOf[OrderInstance]
// Step 8 - implement a corresponding hashCode c=method
override def hashCode: Int = {
val prime = 31
var result = 1
result = prime * result + trade_id;
result = prime * result + (if (code == null) 0 else code.hashCode)
result
}
override def toString = {
String.valueOf(trade_id)
}
}
object OrderInstance {
def getTime(time: String): java.util.Date = new java.util.Date()
}
new OrderInstance(1.0 , "" , 1 , 1.0 , "2019-09-13T16:27:19.881Z" , "")
This returns error :
Error:(60, 26) OrderInstance does not take parameters
this(time) = this(OrderInstance.getTime(time))
for line
this(time) = this(OrderInstance.getTime(time))
How to update time to be converted to Date instead of String ?
getTime is an instance method, which you're trying to invoke from a constructor, i.e. before the instance has been created. Since getTime doesn't actually use any instance variables, you'll want to place it in a companion object (the scala equivalent of static)
class OrderInstance(
//...
) {
this(string) = this(OrderInstance.getTime(string))
}
object OrderInstance {
def getTime(time: String): java.util.Date = new java.util.Date()
}
Thanks to answer from user user1186491 this works as expected :
class OrderInstance(var time: java.util.Date){
def this(time: String) = this(
OrderInstance.getTime(time))
}
object OrderInstance {
def getTime(time: String): java.util.Date = new java.util.Date()
}
val o = new OrderInstance("2019-09-13T16:27:19.881Z" )
println(o.time)
I have big object:
case class BigObject(val str: String, val number: Int) {
val someVal = ...
val someVal2 = ...
}
I'd like to copy this object without re-valuation of values. Is it possible? Right now I'm using this approach:
val newBigObject = oldBigObject.copy(str = newStr)
As I see from the logs/debugger, "someVal" and "someVal2" are re-valuated. Is it possible to avoid it? As my BigObject is really big and value re-valuation takes some time, but performance is really important for me.
Thanks for your answers!
Here's a way:
Make the someVal and someVal2 fields which are also passed to the constructor and pull out the initialization logic for those fields in the companion object.
In your case:
class BigObject private(val str: String,
val number: Int,
val someVal: SomeType,
val someVal2: SomeType) {
def copy(newStr: String = str, newNumber: Int = number) = {
new BigObject(newStr, newNumber, someVal, someVal2)
}
}
object BigObject {
def apply(str: String, number: Int): BigObject = {
val someVal = initialize() //your initialization logic here
val someVal2 = initialize2()
new BigObject(str, number, someVal, someVal2)
}
}
Now, you can copy without re-evaling the inner fields:
val bigObj = BigObject("hello", 42)
val anotherBigObj = bigObj.copy(newStr = "anotherStr")
Alternatively, if you don't like companion objects, you can make two constructors. The primary one includes all the fields (also the non visible ones) and will be private. The public one will have only the two visible parameters:
class BigObject private(val str: String,
val number: Int,
val someVal: Any,
val someVal2: Any) {
def this(str: String, number: Int) = this(str, number, initializeVal, initializeVal2)
def copy(newStr: String = str, newNumber: Int = number) = {
new BigObject(newStr, newNumber, someVal, someVal2)
}
}
Usage:
val bigObj = new BigObject("hello", 42)
val anotherBigObj = bigObj.copy(newStr = "anotherStr")