In Scala, is there a way to map over a collection while passing a value along fold-style? Something like:
case class TxRecord(name: String, amount: Int)
case class TxSummary(name: String, amount: Int, balance: Int)
val txRecords: Seq[TxRecord] = txRecordService.getSortedTxRecordsOfUser("userId")
val txSummarys: Seq[TxSummary] = txRecords.foldMap(0)((sum, txRecord) =>
(sum + txRecord.amount, TxSummary(txRecord.name, txRecord.amount, sum + txRecord.amount)))
Related
I have a class Dimensions(Int, Int, Int) and a Shape(String name), put into a Tuple(Shape, Dimensions)
My dataset is:
(Cube, Dimensions(5,5,5))
(Sphere, Dimensions(5,10,15))
(Cube, Dimensions(3,3,3))
I need to return this:
(Cube, Dimensions(8,8,8))
(Sphere, Dimensions(5,10,15))
where I group by the name of the shape then sum up all of the dimension values. Currently I am able to map into a (Name, Int, Int, Int) but I am unsure of how to wrap it back to a Dimension object.
data.map(_._2.map(x => (x.length,x.width,x.height)))
Any help would be appreciated
Assuming there are no very specific special cases and you have a RDD. You just need an aggregateByKey.
case class Dimensions(i1: Int, i2: Int, i3: Int)
val initialRdd: RDD[(Shape, Dimensions)] = ???
def combineDimensions(dimensions1: Dimensions, dimensions2: Dimensions): Dimensions =
Dimensions(
dimensions1.i1 + dimensions2.i1,
dimensions1.i2 + dimensions2.i2,
dimensions1.i3 + dimensions2.i3
)
val finalRdd: RDD[(Shape, Dimensions)] =
initialRdd
.aggregateByKey(Dimensions(0, 0, 0))(
{ case (accDimensions, dimensions) =>
combineDimensions(accDimensions, dimensions)
},
{ case (partitionDimensions1, partitionDimensions2) =>
combineDimensions(partitionDimensions1, partitionDimensions1)
}
)
I want to parse province to case class, it throws mismatch
scala.MatchError: Some(USA) (of class scala.Some)
val result = EntityUtils.toString(entity,"UTF-8")
val address = JsonParser.parse(result).extract[Address]
val value.province = Option(address.province)
val value.city = Option(address.city)
case class Access(
device: String,
deviceType: String,
os: String,
event: String,
net: String,
channel: String,
uid: String,
nu: Int,
ip: String,
time: Long,
version: String,
province: Option[String],
city: Option[String],
product: Option[Product]
)
This:
val value.province = Option(address.province)
val value.city = Option(address.city)
doesn't do what you think it does. It tries to treat value.province and value.city as extractors (which don't match the type, thus scala.MatchError exception). It doesn't mutate value as I believe you intended (because value apparently doesn't have such setters).
Since value is (apparently) Access case class, it is immutable and you can only obtain an updated copy:
val value2 = value.copy(
province = Option(address.province),
city = Option(address.city)
)
Assuming the starting point:
val province: Option[String] = ???
You can get the string with simple pattern matching:
province match {
case Some(stringValue) => JsonParser.parse(stringValue).extract[Province] //use parser to go from string to a case class
case None => .. //maybe provide a default value, depends on your context
}
Note: Without knowing what extract[T] returns it's hard to recommend a follow-up
I have a class that has some Option values, and I need to apply all the values that come as Some in the class to a list of objects.
Example:
class Thing(name: String, age: Int)
class Filter(name: Option[String], age: Option[Int], size: Option[Int])
val list: List[Thing] = functionThatReturnsAListOfThings(param)
val filter: Filter = functionThatReturnsAFilter(otherParam)
list.filter{ thing =>
if filter.name.isDefined {
thing.name.equals(filter.name.get)
}
if filter.age.isDefined {
thing.age == filter.age.get
}
}.take{
if filter.size.isDefined filter.size.get
else list.size
}
How can I apply the filter to the list properly with FP?
First off we need to make a small change so that the constructor arguments are public members.
class Thing(val name: String, val age: Int)
class Filter(val name : Option[String]
,val age : Option[Int]
,val size : Option[Int])
Next, it's not clear, from your example code, what should happen when filter.name and filter.age are both None. I'll assume that None means true, i.e. not filtered out.
list.filter { thing =>
filter.name.fold(true)(_ == thing.name) &&
filter.age.fold(true)(_ == thing.age)
}.take(filter.size.getOrElse(Int.MaxValue))
Note that take(Int.MaxValue) is a bit more efficient than take(list.size).
I would like to implement a HbaseConnector.
I'm actually reading the guide but there is a part that I don't understand and I can't find any information about it.
In the part 2 of the guide we can see the following code :
case class HBaseRecord(col0: String, col1: Boolean,col2: Double, col3: Float,col4: Int, col5: Long, col6: Short, col7: String, col8: Byte)
object HBaseRecord {def apply(i: Int, t: String): HBaseRecord = { val s = s”””row${“%03d”.format(i)}””” HBaseRecord(s, i % 2 == 0, i.toDouble, i.toFloat, i, i.toLong, i.toShort, s”String$i: $t”, i.toByte) }}
val data = (0 to 255).map { i => HBaseRecord(i, “extra”)}
I do understand that they store the future column in the HbaseRecord case class but I don't understand the specific use of this line :
val s = s”””row${“%03d”.format(i)}”””
Could someone care to explain ?
It is used to generate row ids like row001, row002 etc. which will populate column0 of your table. Try out simpler way with function
def generate(i: Int): String = { s"""row${"%03d".format(i)}"""}
I have List collection with data below:
case class Expense_detail(po_id: Long, supplier_id: String, price: String)
Expense_detail(1,"S00001","1000.0"),
Expense_detail(2,"S00001","2000.0"),
Expense_detail(3,"S00002","3,000.0"),
Expense_detail(4,"S00003","4,000.0")
Is it possible to map it into below Map collection:
"S00001" -> ((1,2), "3000.0")
"S00002" -> ((3), "3000.0")
"S00003" -> ((4), "4000.0")
Yes with groupBy an mapValues.
case class ExpenseDetail(poId: Long, supplierId: String, price: String)
val details : List[ExpenseDetail] = ...
details.
groupBy( _.supplierId ).
mapValues( details => ( (details.map(_.poId)), details.map(_.price.toInt).sum ))
This should work.
I changed the naming to honor the Scala/Java best practices to use CamelCase instead of snake_case.