I have the followeing classes:
case class RolepermissionRow(roleid: Int, permissionid: Int)
and
case class RolePermissionJSON(
var id: Option[Long],
var num: Option[Int],
var name: Option[String],
var perms: Map[String, Boolean])
I create a map:
var s = Map("1" -> true)
I create a RolePermissionJSON:
val f = RolePermissionJSON(Some(0), Some(0), Some('test'), s)
And I would like to convert the perms Map to RolepermissionRow using the following code:
scala> f.perms.map { case (key, value) => if (value) RolepermissionRow(key.toInt, 1) }.toSeq
res7: Seq[Any] = List(RolepermissionRow(1,1))
The result is Seq[Any] but I would like to have Seq[RolepermissionRow]
Simple changes required:
val z = f.perms.map { case (key, value) if (value) => RolepermissionRow(key.toInt, 1) }.toSeq
Now description:
In your code resulted Seq contains of RolepermissionRows if value is true and Units otherwise.
You should filter out "empty" elements of map that gives you Unit.
UPD:
#nafg advice to use collect to prevent match error in runtime.
f.perms.filter(_._2).map{case (key, value) => RolepermissionRow(key.toInt, 1) }.toSeq
I think you should use filter firstly to filter the map avoid empty Map.
Related
I am trying to send a List[KeyValue] to the topic, in a Kafka Streams app. But the stream expects a single KeyValue. How I can send the KeyValues to the stream, instead of the whole list?
class MainTopology {
val createTopology = (sourceTopic: String, sinkTopic: String) => {
val builder = new StreamsBuilder()
builder.stream(sourceTopic)
.map[String, Option[JsonMessage]]((key, value) => toJsonEvent(key, value))
.map[String, String]((key, value) => toFormattedEvents(key, value))
.to(sinkTopic)
builder.build()
}
private val toJsonEvent = (key: String, value: String) => {
println(value)
val jsonEventsAsCaseClasses = jsonToClass(value)
new KeyValue(key, jsonEventsAsCaseClasses)
}
private val toFormattedEvents = (key: String, value: Option[JsonMessage]) => {
val jsonEvents: List[String] = formatEvents(value)
jsonEvents.map(event => new KeyValue(key,event))
}
}
The second map is not compiling due to this.
Expression of type List[KeyValue[String, String]] doesn't conform to expected type KeyValue[_ <: String, _ <: String]
I update my code, but now it is throwing another error:
val builder = new StreamsBuilder()
builder.stream(sourceTopic)
.map[String, Option[JsonMessage]]((key, value) => toJsonEvent(key, value))
.flatMap(
(key, value) => toFormattedEvents(key, value)
)
.to(sinkTopic)
builder.build()
}
private val toJsonEvent = (key: String, value: String) => {
println(value)
val jsonEventsAsCaseClasses = jsonToClass(value)
new KeyValue(key, jsonEventsAsCaseClasses)
}
private val toFormattedEvents = (key: String, value: Option[JsonMessage]) => {
val jsonEvents: List[String] = formatEvents(value)
jsonEvents.map(event => new KeyValue(key,event)).toIterable.asJava
}
Error:(15, 8) inferred type arguments [?1,?0] do not conform to method flatMap's type parameter bounds [KR,VR]
.flatMap(
Error:(16, 20) type mismatch;
found : org.apache.kafka.streams.kstream.KeyValueMapper[String,Option[org.minsait.streams.model.JsonMessage],Iterable[org.apache.kafka.streams.KeyValue[String,String]]]
required: org.apache.kafka.streams.kstream.KeyValueMapper[_ >: String, _ >: Option[org.minsait.streams.model.JsonMessage], _ <: Iterable[_ <: org.apache.kafka.streams.KeyValue[_ <: KR, _ <: VR]]]
(key, value) => toFormattedEvents(key, value)
Take a look at flatMap() and flatMapValues() to replace the second map().
https://kafka.apache.org/22/javadoc/org/apache/kafka/streams/kstream/KStream.html
For example:
.flatMap[String, Long]((key, value) => Seq(("foo", 1L), ("bar", 2L)))
If you do intent to keep using map for some reason, then see below.
The second map is not compiling due to this.
Expression of type List[KeyValue[String, String]] doesn't conform to expected type KeyValue[_ <: String, _ <: String]
The corresponding code, for reference:
.map[String, String]((key, value) => toFormattedEvents(key, value))
What you need is something like:
.map[String, List[KeyValue[KR, VR]]]((key, value) => toFormattedEvents(key, value))
where KR and VR are the key and value types, respectively, as returned by toFormattedEvents() (the actual return types are not clear from your question). For this to work you must also have Serde for the List[KeyValue[KR, VR]] type.
Let me illustrate this with a few different types so it's easier to understand which part of the method call refers to which type. Assume we want to the map output to have a key type of Integer and a value type of List[KeyValue[String, Long]]:
.map[Integer, List[KeyValue[String, Long]]]((key, value) => (5, List(KeyValue.pair("foo", 1L), KeyValue.pair("bar", 2L))))
Note that this example assigns the same values to every mapped record, but it's not the point of the example.
I am trying to parse a csv row here and each field can be a different type. To handle the error accumulation I am using Either[String, B] where the String is an error message and B is the value. The issue here is that B can be different types, Option[Int], String, Array[String], resulting in my Map being type (String, Either[String,java.io.Serializable]) effectively making the Map unreusable. Is there a way (I'm definitely sure there is) to more elegantly accumulate errors while also reusing those values to populate properties on an object?
override def parseCsv(implicit fields: Map[String, String]): Either[String, User] = {
val parsedValues = Map(Headers.Id -> getFieldAsString(Headers.Id),
Headers.FirstName -> getFieldAsString(Headers.FirstName),
Headers.LastName -> getFieldAsString(Headers.LastName),
Headers.Email -> getFieldAsString(Headers.Email),
Headers.TimeZone -> getFieldAsString(Headers.TimeZone),
Headers.Region -> getOption(Headers.Region),
Headers.Phone -> getOption(Headers.Phone),
Headers.ProfileImage -> getFieldAsString(Headers.ProfileImage),
Headers.Roles -> getFieldAsArray(Headers.Roles))
val errors = parsedValues.collect { case (key, Left(errors)) => errors }
if (!errors.isEmpty) Left(errors.mkString(", "))
else {
val user = new User
user.id = getFieldAsString(Headers.Id).right.get
user.firstName = getFieldAsString(Headers.FirstName).right.get
user.lastName = getFieldAsString(Headers.LastName).right.get
user.email = getFieldAsString(Headers.Email).right.get
user.timeZone = getFieldAsString(Headers.TimeZone).right.get
user.phoneNumber = (for {
region <- getOption(Headers.Region).right.get
phone <- getOption(Headers.Phone).right.get
_ = validatePhoneNumber(phone, region)
} yield {
new PhoneNumber(region, phone)
}).orNull
user.profileImageUrl = getFieldAsString(Headers.ProfileImage).right.get
user.roles = getFieldAsArray(Headers.Roles).right.get
Right(user)
}
}
Create case classes for all types of Bs. These case classes must extend some common trait. While populating the user object just pattern match and retrieve values.
sealed trait Result {
val paramName: String
}
case class OptionInt(override val paramName: String, value: Option[Int]) extends Result
case class ArrayString(override val paramName: String, value: Array[String]) extends Result
case class StringValue(override val paramName: String, value: String) extends Result
now the final type would be like Either[String, Result]
after parsing the whole file create a List[Result]
If you are expecting age as Option[Int] and firstName as String then do this
list.foreach { result =>
result match {
case Option("age", value) => userId.age = value.getOrElse(defaultAge)
case StringValue("firstName", value) => userId.firstName = value
case StringValue("email", value) => userId.email = value
case _ => //do nothing
}
}
The following Iterable can be o size one, two, or (up to) three.
org.apache.spark.rdd.RDD[Iterable[(String, String, String, String, Long)]] = MappedRDD[17] at map at <console>:75
The second element of each tuple can have any of the following values: A, B, C. Each of these values can appear (at most) once.
What I would like to do is sort them based on the following order (B , A , C), and then create a string by concatenating the elements of the 3rd place. If the corresponding tag is missing then concatenate with a blank space: ``. For example:
this:
CompactBuffer((blah,A,val1,blah,blah), (blah,B,val2,blah,blah), (blah,C,val3,blah,blah))
should result in:
val2,val1,val3
this:
CompactBuffer((blah,A,val1,blah,blah), (blah,C,val3,blah,blah))
should result in:
,val1,val3
this:
CompactBuffer((blah,A,val1,blah,blah), (blah,B,val2,blah,blah))
should result in:
val2,val1,
this:
CompactBuffer((blah,B,val2,blah,blah))
should result in:
val2,,
and so on so forth.
In your case when A, B and C appear at most once, you could add the corresponding values to a temporary map and retrieve the values from the map in the correct order.
If we use getOrElse to get the values from the map, we can specify the empty string as default value. This way we still get the correct result if our Iterable doesn't contain all the tuples with A, B and C.
type YourTuple = (String, String, String, String, Long)
def orderTuples(order: List[String])(iter: Iterable[YourTuple]) = {
val orderMap = iter.map { case (_, key, value, _, _) => key -> value }.toMap
order.map(s => orderMap.getOrElse(s, "")).mkString(",")
}
We can use this function as follows :
val a = ("blah","A","val1","blah",1L)
val b = ("blah","B","val2","blah",2L)
val c = ("blah","C","val3","blah",3L)
val order = List("B", "A", "C")
val bacOrder = orderTuples(order) _
bacOrder(Iterable(a, b, c)) // String = val2,val1,val3
bacOrder(Iterable(a, c)) // String = ,val1,val3
bacOrder(Iterable(a, b)) // String = val2,val1,
bacOrder(Iterable(b)) // String = val2,,
def orderTuples(xs: Iterable[(String, String, String, String, String)],
order: (String, String, String) = ("B", "A", "C")) = {
type T = Iterable[(String, String, String, String, String)]
type KV = Iterable[(String, String)]
val ord = List(order._1, order._2, order._3)
def loop(xs: T, acc: KV, vs: Iterable[String] = ord): KV = xs match {
case Nil if vs.isEmpty => acc
case Nil => vs.map((_, ",")) ++: acc
case x :: xs => loop(xs, List((x._2, x._3)) ++: acc, vs.filterNot(_ == x._2))
}
def comp(x: String) = ord.zipWithIndex.toMap.apply(x)
loop(xs, Nil).toList.sortBy(x => comp(x._1)).map(_._2).mkString(",")
}
I was trying to recursively create a key, value pair map from a hierarchical case class structure, but no luck.
case class Country(id: Option[Long], name: String, states: Option[List[State]])
case class State(id: Option[Long], name: String, cities: Option[List[City]])
case class City(id: Option[Long], name: String)
So I was trying to extract that in Lists and zip to get the key value pair, then trying to do some recursive computation to obtain the same in the Lists of objects.
val keys = country.getClass.getDeclaredFields.map(f => f.getName)
val values = country.productIterator
val m = (keys zip values.toList).toMap
This gives me 3 keys with their values.
My problem is to solve the recursive way in the List[City] inside State and List[State] inside Country.
Is there any good way to do that? I can't think in any good solution. I was trying to iterate over the map, matching where there is a List and iterate and trying to save this progress in a BufferList.
Has anyone trying to do something like this?
EDIT
I wasn't specific about the desirable output.
I want the child to be other maps, so getting the example from #Tienan Ren I tried to do something like this:
def toMap[T: TypeTag](clazz: scala.Product): Map[String, Any] = {
def recursion(key: String, list: List[T]): Map[String, Any] = {
list.toMap
}
val keys = clazz.getClass.getDeclaredFields.map(_.getName)
val values = (keys zip clazz.productIterator.toList) map {
case (key, value: List[T]) => recursion(key, value)
case (key, value) => (key, value)
}
values.toMap
}
Where the recursion should receive the list and back a Map.
Not sure why would you use Option[List[..]], empty list should be capable of representing the None case. Here is my implementation after removing Option for Lists.
case class Country(id: Option[Long], name: String, states: List[State])
case class State(id: Option[Long], name: String, cities: List[City])
case class City(id: Option[Long], name: String)
The toMap function:
import reflect.runtime.universe._
def toMap[T: TypeTag](c: scala.Product): Map[String, Any] = {
val keys = c.getClass.getDeclaredFields.map(_.getName)
val z = (keys zip c.productIterator.toList) map {
case (key, value: List[T]) if typeOf[T] <:< typeOf[scala.Product] =>
(key, value.map(v => toMap(v.asInstanceOf[scala.Product])))
case (key, value) => (key, value)
}
z.toMap
}
Output:
val country = Country(None, "US", List(State(None, "CA", City(None, "LA") :: Nil)))
println(toMap(country))
This gives you:
Map(id -> None, name -> US, states -> List(Map(id -> None, name -> CA, cities -> List(Map(id -> None, name -> LA)))))
In scala, how would i declare & instantiate a map that returns a function (for the sake of argument? A function that accepts two variables, one a String, one an Int)?
I am envisioning:
val myMap = Map[String, (String,Int)=>Boolean](
WHAT GOES HERE???
)
Let's just map the string "a" to this cool function. I don't care much about what the function does - returns true, perhaps?
Try this:
val myMap = Map[String, (String, Int) => Boolean](
"Test" -> ((s, i) => true)
)
you can do something like this:
val map = Map("key" -> { (str: String, n: Int) =>
str.indexOf(n) == -1
})
result:
> map: scala.collection.immutable.Map[String,(String, Int) => Boolean] = Map(key - <function2>)