Slick Queries With GroupBy - scala

I'm new to Scala and Slick. I'm trying to run a query with a grouping by an object and creating two sequences for each object.
This is my code :
val firstQuery = {
for {
table1Id <- Table1.filter(_.someid === someid).map(_.someid)
table2 <- Table2.filter(_.someid === table1Id)
table3 <- Table3.filter(_.someId === someid)
_ <- {
Table4.filter(table4 => table4.someid === table3.someid &&
table4.name === "$$$")
}
table5 <- Table5.filter(_.someid === table2.someid)
} yield {
(table2, table3, table5)
}
}
val finalQuery = {
for {
(table6, firstQueryTuple) <- {
Table6 joinRight firstQuery on
((firstTable, secondTable) => firstTable.someid === secondTable._1.someid)
}
} yield {
(table6, firstQueryTuple._1, firstQueryTuple._2, firstQueryTuple._3)
}
}
dataBaseConnection.run(finalQuery.result).map { resultSequence =>
val grouping = resultSequence.groupBy { case (table6, table2, _, _) => (table6, table2) }
val groupingFiltered = grouping.map { case (table6table2tuple, sequence) => (table6table2tuple, sequence.map(_._4) (which is Table5), sequence.map(_._3) (which is table3)) }.toSeq
groupingFiltered match {
case tupleQuery if tupleQuery.nonEmpty => Success(tupleQuery.map(x => (x._1._2, x._1._1, x._2, x._3) ))
case _ => Failure(NonEmptyList(DBMissingRecordProblem(s"Error")))
}
}
Now what I need to achieve is this result
Seq[(Table2, Option[Table6], Seq[Table5], Seq[Table3])]
Is there a way to group by and achieve two sequences? I know it can be done for one sequence, I mean group by an object and achieve (object, Seq[anotherObject])
Please help me if you can.

Related

Slick how to sort by a option column got from left join

At first, get all the children comments in a subquery
val getChildCommentStatQuery =
Tbcomments
.filter(row =>
row.itargetid === commentTargetId && row.iparentid =!= 0 && row.istatus === statusNormal)
.groupBy(_.iparentid)
.map { case (parentId, group) =>
// return parent commentId,the number of all the children,and the sum of like of all the children
(parentId, group.size, group.map(_.ilikecount).sum)
}
Secondly, use Table Tbcomments to left join with subquery table got in the first step.
val query =
Tbcomments
.joinLeft(getChildCommentStatQuery)
.on(_.iid == _._1)
.filter { case (comments, _) =>
// got the first level comment whose parentid equals to 0
comments.itargetid === commentTargetId && comments.iparentid === 0 && comments.istatus === statusNormal
}
.map { case (comments, childComments) =>
val replyCount = childComments.map(_._2)
val likeCountSum = comments.ilikecount + childComments.map(_._3.getOrElse(0))
val hot = replyCount + likeCountSum
val support = comments.ilikecount - comments.inotlikecount
(comments.iid, replyCount, likeCountSum, hot, support, comments.dtcreated)
}
.sortBy(row =>
sortOrder match {
case LATEST_COMMENT_ORDER => row._6.desc
case EARLIEST_COMMENT_ORDER => row._6.asc
case SUPPORT_COMMENT_ORDER => row._5.desc
// case HEAT_COMMENT_ORDER => row._4.get
})
The problem is, the row._4 is a field aggregate based on right table, and in the left join case it's an option. It seems can't sortBy the option field directly.
Thanks!

Scala functional programming dry run

Could you please help me in understanding the following method:
def extractGlobalID(custDimIndex :Int)(gaData:DataFrame) : DataFrame = {
val getGlobId = udf[String,Seq[GenericRowWithSchema]](genArr => {
val globId: List[String] =
genArr.toList
.filter(_(0) == custDimIndex)
.map(custDim => custDim(1).toString)
globId match {
case Nil => ""
case x :: _ => x
}
})
gaData.withColumn("globalId", getGlobId('customDimensions))
}
The method applies an UDF to to dataframe. The UDF seems intended to extract a single ID from column of type array<struct>, where the first element of the struct is an index, the second one an ID.
You could rewrite the code to be more readable:
def extractGlobalID(custDimIndex :Int)(gaData:DataFrame) : DataFrame = {
val getGlobId = udf((genArr : Seq[Row]) => {
genArr
.find(_(0) == custDimIndex)
.map(_(1).toString)
.getOrElse("")
})
gaData.withColumn("globalId", getGlobId('customDimensions))
}
or even shorter with collectFirst:
def extractGlobalID(custDimIndex :Int)(gaData:DataFrame) : DataFrame = {
val getGlobId = udf((genArr : Seq[Row]) => {
genArr
.collectFirst{case r if(r.getInt(0)==custDimIndex) => r.getString(1)}
.getOrElse("")
})
gaData.withColumn("globalId", getGlobId('customDimensions))
}

How to update the list only when certain conditions match

I'm doing the following to map and update a list:
if (colors.map(_.id).contains(updated.id)) {
val newColorList = updated :: colors.filterNot(s => s.id == updated.id)
SomeMethod(newColorList)
}
else {
this
}
The above works ok, However, I want to add a condition such as this: If, and only if, the updated quantity is 0 then also update the enddate to todays date.
I've done the following
if (color.map(_.id).contains(updated.id) && updated.quantity == 0) {
val newColorList = updated.copy(enddate = Instant.now().toEpochMilli) :: colors.filterNot(s => s.id == updated.id)
}
else if (color.map(_.id).contains(updated.id)) {
val newColorList = updated :: colors.filterNot(s => s.id == updated.id)
SomeMethod(newColorList)
}
else {
this
}
Is there a better way to do this than multiple if / else statements
something like
val hasColor=color.map(_.id).contains(updated.id)
newColorList = (hasColor,updated.quantity) match {
case (true,0) => updated.copy(enddate = Instant.now().toEpochMilli) :: colors.filterNot(s => s.id == updated.id)
case (true,_) => updated :: colors.filterNot(s => s.id == updated.id)
otherwise => this
}
You can use collect over the colors collection.
I assume there is a simple a case class Color like this
case class Color(id:String, quantity: Int, endDate : LocalDate)
And updated is also of type Color
import java.time.LocalDate
case class Color(id:String, quantity: Int, endDate : LocalDate)
val updated = Color("one", 0, LocalDate.now)
val colors = List(
Color("one", 1, LocalDate.now().minusMonths(2)),
Color("two", 2, LocalDate.now.minusMonths(4)),
Color("three", 3, LocalDate.now.minusMonths(6))
)
colors.collect{
case e if e.id == updated.id && updated.quantity == 0 =>
updated.copy(endDate = LocalDate.now) //add element to new list, call method
case e if e.id == updated.id => updated
case e => e
}
//List(Color(one,0,2018-12-06), Color(two,2,2018-08-06), Color(two,2,2018-06-06))
Hope this helps you.

How do you flatten a Sequence of Sequence in DBIOaction SLICK?

Hey guys i am new in slick, how can I flatten this sequence of sequence? so that can return the commented code
def insertIfNotExists(mapCountryStates: Map[String, Iterable[StateUtil]]): Future[Seq[Seq[StateTable]]] /*: Future[Seq[StateTable]]*/ = {
val interaction = DBIO.sequence(mapCountryStates.toSeq.map { case (alpha2Country, statesUtil) =>
val codes = statesUtil.map(_.alpha3Code)
for {
countryId <- Countries.filter(_.alpha2Code === alpha2Country).map(_.id).result.head
existing <- States.filter(s => (s.alpha3Code inSet codes) && s.countryId === countryId).result
stateTables = statesUtil.map(x => StateTable(0L, x.name, x.alpha3Code, countryId))
statesInserted <- StatesInsertQuery ++= stateTables.filter(s => !existing.exists(x => x.alpha3Code == s.alpha3Code && x.countryId == s.countryId))
} yield existing ++ statesInserted
})
db.run(interaction.transactionally)
}
if I write it here:
val interaction = DBIO.sequence(...).flatten
or here:
db.run(interaction.flatten.transactionally)
[error] Cannot prove that Seq[Seq[StateRepository.this.StateTableMapping#TableElementType]] <:< slick.dbio.DBIOAction[R2,S2,E2].
but when the application runs, because the IDE does not detect it as an error:
I update my definition with DBIO.fold:
It looks like you might be after DBIO.fold. This provides a way to take a number of actions and reduce them down to a single value. In this case, your single value is a Seq[StateTable] from a Seq[Seq[StateTable]].
A sketch of how this could look might be...
def insertIfNotExists(...): DBIO[Seq[StateTable]] = {
val interaction: Seq[DBIO[Seq[StateTable]]] = ...
val startingPoint: Seq[StateTable] = Seq.empty
DBIO.fold(interaction, startingPoint) {
(total, list) => total ++ list
}
}
It looks like the types will line up using fold. Hope it's of some use in your case.
There's some more information about fold in Chapter 4 of Essential Slick.
A viable solution should be to flatten the sequence once the Future has been completed:
def insertIfNotExists(mapCountryStates: Map[String, Iterable[StateUtil]]): Future[Seq[StateTable]] = {
val interaction = DBIO.sequence(mapCountryStates.toSeq.map { case (alpha2Country, statesUtil) =>
val codes = statesUtil.map(_.alpha3Code)
for {
countryId <- Countries.filter(_.alpha2Code === alpha2Country).map(_.id).result.head
existing <- States.filter(s => (s.alpha3Code inSet codes) && s.countryId === countryId).result
stateTables = statesUtil.map(x => StateTable(0L, x.name, x.alpha3Code, countryId))
statesInserted <- StatesInsertQuery ++= stateTables.filter(s => !existing.exists(x => x.alpha3Code == s.alpha3Code && x.countryId == s.countryId))
} yield existing ++ statesInserted
})
db.run(interaction.transactionally).map(_.flatten)
}

Tuple seen as Product, compiler rejects reference to element

Constructing phoneVector:
val phoneVector = (
for (i <- 1 until 20) yield {
val p = killNS(r.get("Phone %d - Value" format(i)))
val t = killNS(r.get("Phone %d - Type" format(i)))
if (p == None) None
else
if (t == None) (p,"Main") else (p,t)
}
).filter(_ != None)
Consider this very simple snippet:
for (pTuple <- phoneVector) {
println(pTuple.getClass.getName)
println(pTuple)
//val pKey = pTuple._1.replaceAll("[^\\d]","")
associate() // stub prints "associate"
}
When I run it, I see output like this:
scala.Tuple2
((609) 954-3815,Mobile)
associate
When I uncomment the line with replaceAll(), compile fails:
....scala:57: value _1 is not a member of Product with Serializable
[error] val pKey = pTuple._1.replaceAll("[^\\d]","")
[error] ^
Why does it not recognize pTuple as a Tuple2 and treat it only as Product
OK, this compiles and produces the desired result. But it's too verbose. Can someone please demonstrate a more concise solution for dealing with this typesafe stuff?
for (pTuple <- phoneVector) {
println(pTuple.getClass.getName)
println(pTuple)
val pPhone = pTuple match {
case t:Tuple2[_,_] => t._1
case _ => None
}
val pKey = pPhone match {
case s:String => s.replaceAll("[^\\d]","")
case _ => None
}
println(pKey)
associate()
}
You can do:
for (pTuple <- phoneVector) {
val pPhone = pTuple match {
case (key, value) => key
case _ => None
}
val pKey = pPhone match {
case s:String => s.replaceAll("[^\\d]","")
case _ => None
}
println(pKey)
associate()
}
Or simply phoneVector.map(_._1.replaceAll("[^\\d]",""))
By changing the construction of phoneVector, as wrick's question implied, I've been able to eliminate the match/case stuff because Tuple is assured. Not thrilled by it, but Change is Hard, and Scala seems cool.
Now, it's still possible to slip a None value into either of the Tuple values. My match/case does not check for that, and I suspect that could lead to a runtime error in the replaceAll call. How is that allowed?
def killNS (s:Option[_]) = {
(s match {
case _:Some[_] => s.get
case _ => None
}) match {
case None => None
case "" => None
case s => s
}
}
val phoneVector = (
for (i <- 1 until 20) yield {
val p = killNS(r.get("Phone %d - Value" format(i)))
val t = killNS(r.get("Phone %d - Type" format(i)))
if (t == None) (p,"Main") else (p,t)
}
).filter(_._1 != None)
println(phoneVector)
println(name)
println
// Create the Neo4j nodes:
for (pTuple <- phoneVector) {
val pPhone = pTuple._1 match { case p:String => p }
val pType = pTuple._2
val pKey = pPhone.replaceAll(",.*","").replaceAll("[^\\d]","")
associate(Map("target"->Map("label"->"Phone","key"->pKey,
"dial"->pPhone),
"relation"->Map("label"->"IS_AT","key"->pType),
"source"->Map("label"->"Person","name"->name)
)
)
}
}