I've got a very simple database table called regions where each region may have a parent region.
mysql> describe region;
+---------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+-------------+------+-----+---------+-------+
| region_code | char(3) | NO | PRI | NULL | |
| region_name | varchar(50) | NO | | NULL | |
| parent_region | char(3) | YES | MUL | NULL | |
+---------------+-------------+------+-----+---------+-------+
Now I'd like to hydrate this data to a Scala object graph of case classes that each have a parent of the same type.
case class Region(code: String, name: String, parent: Option[Region])
I do this with the following code. It works but it creates duplicate objects which I'd like to avoid if possible.
class RegionDB #Inject() (db: Database) {
def getAll(): Seq[Region] = {
Logger.debug("Getting all regions.")
db.withConnection { implicit conn =>
val parser = for {
code <- str("region_code")
name <- str("region_name")
parent <- str("parent_region").?
} yield (code, name, parent)
val results = SQL("SELECT region_code, region_name, parent_region from region").as(parser.*)
// TODO: Change this so it doesn't create duplicate records
def toRegion(record: (String, String, Option[String])): Region = {
val (code, name, parent) = record
val parentRecord = parent.map(p => results.find(_._1 == p)).getOrElse(None)
new Region(code, name, parentRecord.map(toRegion).orElse(None))
}
val regions = results map toRegion
regions.foreach(r => Logger.debug("region: " + r))
regions
}
}
}
I know how to do this in the imperative way but not the functional way. I know there has got to be an expressive way to do this with recursion but I can't seem to figure it out. Do you know how? Thanks!
I was able to solve this issue by restructuring the Region case class so that the parent region is a var and by adding a collection of children. It would be nice to do this without a var but oh well.
case class Region(code: String, name: String, subRegions: Seq[Region]){
var parentRegion: Option[Region] = None
subRegions.foreach(_.parentRegion = Some(this))
}
The recursion is more natural going from the root down.
def getAll(): Seq[Region] = {
Logger.debug("Getting all regions.")
db.withConnection { implicit conn =>
val parser = for {
code <- str("region_code")
name <- str("region_name")
parent <- str("parent_region").?
} yield (code, name, parent)
val results = SQL("SELECT region_code, region_name, parent_region from region").as(parser.*)
def toRegion(record: (String, String, Option[String])): Region = {
val (regionCode, name, parent) = record
val children = results.filter(_._3 == Some(regionCode)).map(toRegion)
Region(regionCode, name, children)
}
val rootRegions = results filter(_._3 == None) map toRegion
rootRegions.foreach(r => Logger.debug("region: " + r))
rootRegions
}
}
Related
I have two tables:
Shop
class ShopTable(tag: Tag) extends GenericTableShop, UUID {
def * = (id.?, name, address) <> (Shop.tupled, Shop.unapply _)
def name = column[String]("name")
def address = column[String]("address")
}
val shopTable = TableQuery[ShopDAO.ShopTable]
Order
class OrderTable(tag: Tag) extends GenericTableOrder, UUID {
def * = (id.?, shopId, amount) <> (Order.tupled, Order.unapply _)
def shopId = column[UUID]("shop_id")
def amount = column[Double]("amount")
}
val orderTable = TableQuery[OrderDAO.OrderTable]
I can get statictic (orders count, orders amount sum) for shops:
def getStatisticForShops(shopIds: List[UUID]): Future[Seq(UUID, Int, Double)] = {
searchStatisticsByShopIds(shopIds).map(orderStatistics =>
shopIds.map(shopId => {
val o = orderStatistics.find(_._1 == shopId)
(
shopId,
o.map(_._2).getOrElse(0),
o.map(_._3).getOrElse(0.0)
)
})
)
}
def searchStatisticsByShopIds(shopIds: List[UUID]): Future[Seq(UUID, Int, Double)] =
db.run(searchStatisticsByShopIdsCompiled(shopIds).result)
private val searchStatisticsByShopIdsCompiled = Compiled((shopIds: Rep[List[(UUID)]]) =>
orderTable.filter(_.shopId === shopIds.any)
.groupBy(_.shopId)
.map { case (shopId, row) =>
(shopId, row.length, row.map(_.amount).sum.get)
}
)
By I need sorting and filtering shopTable by orders count.
How I can join grouped orderTable to shopTable with zero values for missing shops?
I want have such a request:
| id(shopId) | name | address | ordersCount | ordersAmount |
| id1 | name | address | 4 | 200.0 |
| id2 | name | address | 0 | 0.0 |
| id3 | name | address | 2 | 300.0 |
I use scala 2.12.6, slick 2.12:3.0.1, play-slick 3.0.1, slick-pg 0.16.3
P.S. I may have found a solution
val shopsOrdersQuery: Query[(Rep[UUID], Rep[Int], Rep[Double]), (UUID, Int, Double), Seq] = searchShopsOrdersCompiled.extract
// Query shops with orders for sorting and filtering
val allShopsOrdersQueryQuery = shopTable.joinLeft(shopsOrdersQuery).on(_.id === _._1)
.map(s => (s._1, s._2.map(_._2).getOrElse(0), s._2.map(_._3).getOrElse(0.0)))
private val searchShopsOrdersCompiled = Compiled(
orderTable.groupBy(_.shopId)
.map { case (shopId, row) =>
(shopId, row.length, row.map(_.amount).sum.get)
}
)
Yes, this solution is work fine
val shopsOrdersQuery: Query[(Rep[UUID], Rep[Int], Rep[Double]), (UUID, Int, Double), Seq] = searchShopsOrdersCompiled.extract
// Query shops with orders for sorting and filtering
val allShopsOrdersQueryQuery = shopTable.joinLeft(shopsOrdersQuery).on(_.id === _._1)
.map(s => (s._1, s._2.map(_._2).getOrElse(0), s._2.map(_._3).getOrElse(0.0)))
private val searchShopsOrdersCompiled = Compiled(
orderTable.groupBy(_.shopId)
.map { case (shopId, row) =>
(shopId, row.length, row.map(_.amount).sum.get)
}
)
I need to read from a config file and map the config to a case class .It works fine if i have one table as below
CONFIG
mapping {
target {
oracle = {
type = "oracle"
schema = "orcl"
tableName = "my_table"
query = "select key from my_table where dob='2020-01-01'
}
}
SCALA CODE SNIPPET
val targetConfig:Map[String,QueryEngine] = config.getObject("mapping.target")
.entrySet()
.asScala
.foldLeft(Map.empty[String , QueryEngine]) { case ( acc , entry ) =>
val target = entry.getKey
val targetConfig = entry.getValue match {
case validElement if validElement.valueType() == ConfigValueType.OBJECT => validElement.asInstanceOf[ConfigObject].toConfig
case invalidElement => sys.error("illegal syntax at $invalidElement")
}
targetConfig.getString("type") match {
case "oracle" => acc + (target -> new OracleQueryEngine(vars,target,targetConfig.getString("schema"),targetConfig.getString("tableName"),targetConfig.getString("query"),targetConfig.getString("param")))
case x => sys.error(s"unknow target not defined $targetConfig with $targetConfig")
}
}
NOW i updated CONFIG with MULTIPLE tables in the target mapping.
mapping {
target {
oracle =
emp = {
type = "oracle"
schema = "orcl"
tableName = "emp"
query = "select key from emp where dob='2020-01-01'
}
dept = {
type = "oracle"
schema = "orcl"
tableName = "dept"
query = "select key from dept where dob='2020-01-01'
}
}
}
CODE SNIPPET for the multiple table scenario This is giving error Expression of mutable.Set[Map[String,QueryEngine]] doesnt confirm to Map[Query,String]
val targetConfig:Map[String,QueryEngine] = config.getObject("mapping.target")
.entrySet()
.asScala
.foldLeft(Map.empty[String , QueryEngine]) { case ( acc , entry ) =>
val target = entry.getKey
val targetConfig = entry.getValue match {
case validElement if validElement.valueType() == ConfigValueType.OBJECT => validElement.asInstanceOf[ConfigObject].toConfig
case invalidElement => sys.error("illegal syntax at $invalidElement")
}
targetConfig.getObject(s"mapping.target.$target").keySet().asScala.map { key =>
target match {
case "oracle" => acc + (target -> new OracleQueryEngine(vars,target,targetConfig.getString("schema"),targetConfig.getString("tableName"),targetConfig.getString("query"),targetConfig.getString("param")))
case x => sys.error(s"unknow target not defined $targetConfig with $targetConfig")
}
}
}
PURE CONFIG CODE
import pureconfig._
import pureconfig.generic.ProductHint
import pureconfig.generic.auto._
import com.typesafe.config.ConfigFactory
import pureconfig._
import pureconfig.generic.auto._
object PureconfigExample {
case class QueryEngine(`type`: String, schema: String, tableName: String, query: String)
type DatabaseConfig = Map[String, Map[String, QueryEngine]]
case class TargetConfig(target: DatabaseConfig)
case class ApplicationConfig(mapping: TargetConfig)
def main(args: Array[String]): Unit = {
val configString =
"""
|mapping {
| target {
| oracle {
| emp {
| type = "oracle"
| schema = "orcl"
| tableName = "emp"
| query = "select key from emp where dob='2020-01-01'"
| }
| dept {
| type = "oracle"
| schema = "orcl"
| tableName = "dept"
| query = "select key from dept where dob='2020-01-01'"
| }
| }
| }
|}
|""".stripMargin
import pureconfig.generic.auto._
// See for more details - https://pureconfig.github.io/docs/overriding-behavior-for-case-classes.html
implicit def hint[T]: ProductHint[T] = ProductHint[T](ConfigFieldMapping(CamelCase, CamelCase))
val config = ConfigSource.string(configString).load[ApplicationConfig]
println(config)
}
}
POM.XML
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>test2</groupId>
<artifactId>test2</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>com.github.pureconfig</groupId>
<artifactId>pureconfig-generic-base_2.12</artifactId>
<version>0.12.2</version>
</dependency>
<dependency>
<groupId>com.github.pureconfig</groupId>
<artifactId>pureconfig-generic_2.12</artifactId>
<version>0.12.0</version>
</dependency>
</dependencies>
</project>
COMPILER ERROR
PureConfigExample.scala
Error:(43, 56) Cannot find an implicit instance of pureconfig.ConfigReader[PureconfigExample.ApplicationConfig].
If you are trying to read or write a case class or sealed trait consider using PureConfig's auto derivation by adding `import pureconfig.generic.auto._`
val config = ConfigSource.string(configString).load[ApplicationConfig]
Error:(43, 56) not enough arguments for method load: (implicit reader: pureconfig.Derivation[pureconfig.ConfigReader[PureconfigExample.ApplicationConfig]])pureconfig.ConfigReader.Result[PureconfigExample.ApplicationConfig].
Unspecified value parameter reader.
val config = ConfigSource.string(configString).load[ApplicationConfig]
Excuse me, for probably, not the answer you probably expected, but I strongly encourage avoid manual config parsing, because there are tools that doing it for you automatically, without boiler-plate code and type-safe at the same time. The most popular and probably the best library in Scala eco-system is PureConfig.
So in you case the solution would look like something like this:
import pureconfig._
import pureconfig.generic.ProductHint
import pureconfig.generic.auto._
object PureconfigExample {
case class QueryEngine(`type`: String, schema: String, tableName: String, query: String)
type DatabaseConfig = Map[String, Map[String, QueryEngine]]
case class TargetConfig(target: DatabaseConfig)
case class ApplicationConfig(mapping: TargetConfig)
def main(args: Array[String]): Unit = {
val configString =
"""
|mapping {
| target {
| oracle {
| emp {
| type = "oracle"
| schema = "orcl"
| tableName = "emp"
| query = "select key from emp where dob='2020-01-01'"
| }
| dept {
| type = "oracle"
| schema = "orcl"
| tableName = "dept"
| query = "select key from dept where dob='2020-01-01'"
| }
| }
| }
|}
|""".stripMargin
// See for more details - https://pureconfig.github.io/docs/overriding-behavior-for-case-classes.html
implicit def hint[T]: ProductHint[T] = ProductHint[T](ConfigFieldMapping(CamelCase, CamelCase))
val config = ConfigSource.string(configString).load[ApplicationConfig]
println(config)
}
}
which in my case produces next result:
Right(ApplicationConfig(TargetConfig(Map(oracle -> Map(emp -> QueryEngine(oracle,orcl,emp,select key from emp where dob='2020-01-01'), dept -> QueryEngine(oracle,orcl,dept,select key from dept where dob='2020-01-01'))))))
Hope this help!
I use IntelliJ IDEA to execute the code shown below. The content of df is the following:
+------+------+
|nodeId| p_i|
+------+------+
| 26|0.6914|
| 29|0.6914|
| 474| 0.0|
| 65|0.4898|
| 191|0.4445|
| 418|0.4445|
I get Task serialization error at line result.show() when I run this code:
class MyUtils extends Serializable {
def calculate(spark: SparkSession,
df: DataFrame): DataFrame = {
def myFunc(a: Double): String = {
var result: String = "-"
if (a > 1) {
result = "A"
}
return result
}
val myFuncUdf = udf(myFunc _)
val result = df.withColumn("role", myFuncUdf(df("a")))
result.show()
result
}
}
Why do I get this error?
Update:
This is how I run the code:
object Processor extends App {
// ...
val mu = new MyUtils()
var result = mu.calculate(spark, df)
}
I had to add extends Serializable to the specification of a class MyUtils.
I am new to Scala.
Currently trying to write a program which fetches table metadata from a database as a list of lists in the below format, and convert it to Objects of user defined type TableFieldMetadata.
The getAllTableMetaDataAsVO function does this conversion.
Can you tell me if I can write this function much better in functional way.
| **Table Name** | **FieldName** | **Data type** |
| table xxxxxx | field yyyyyy | type zzzz |
| table xxxxxx | field wwwww| type mmm|
| table qqqqqq| field nnnnnn| type zzzz |
Note: Here table name can repeat as it usually has multiple columns.
User defined classes:
1. TableFieldMetadata:
/**
* Class to hold the meta data for a table
*/
class TableFieldMetadata (name: String){
var tableName: String = name
var fieldMetaDataList: List[FieldMetaData] = List()
def add(fieldMetadata: FieldMetaData) {
fieldMetaDataList = fieldMetadata :: fieldMetaDataList
}
}
2. FieldMetaData :
/**
* Class to hold the meta data for a field
*/
class FieldMetaData (fieldName: String, fieldsDataType: String) {
var name:String = fieldName
var dataType:String = fieldsDataType
}
Function:
/**
* Function to convert list of lists to user defined objects
*/
def getAllTableMetaDataAsVO(allTableMetaData: List[List[String]]):List[TableFieldMetadata] = {
var currentTableName:String = null
var currentTable: TableFieldMetadata = null;
var tableFieldMetadataList: List[TableFieldMetadata] = List()
allTableMetaData.foreach { tableFieldMetadataItem =>
var tableName = tableFieldMetadataItem.head
if (currentTableName == null || !currentTableName.equals(tableName)) {
currentTableName = tableName
currentTable = new TableFieldMetadata(tableName)
tableFieldMetadataList = currentTable :: tableFieldMetadataList
}
if (currentTableName.equals(tableName)) {
var tableField = tableFieldMetadataItem.tail
currentTable.add(new FieldMetaData(tableField(0), tableField(1)))
}
}
return tableFieldMetadataList
}
Here is one solution. NOTE: I just used Table and Field for easy of typing into the REPL. You should use case classes.
scala> case class Field( name: String, dType : String)
defined class Field
scala> case class Table(name : String, fields : List[Field])
defined class Table
scala> val rawData = List( List("table1", "field1", "string"), List("table1", "field2", "int"), List("table2", "field1", "string") )
rawData: List[List[String]] = List(List(table1, field1, string), List(table1, field2, int), List(table2, field1, string))
scala> val grouped = rawData.groupBy(_.head)
grouped: scala.collection.immutable.Map[String,List[List[String]]] = Map(table1 -> List(List(table1, field1, string), List(table1, field2, int)), table2 -> List(List(table2, field1, string)))
scala> val result = grouped.map { case(k,v) => { val fields = for { r <- v } yield { Field( r(1), r(2) ) }; Table( k, fields ) } }
result: scala.collection.immutable.Iterable[Table] = List(Table(table1,List(Field(field1,string), Field(field2,int))), Table(table2,List(Field(field1,string))))
I have this method that I need to evaluate an expression based on these Boolean values:
var Ma, Co, Ar, Le, St, Ri: Boolean = false
def Logic: Boolean =
(Ma & (!Ma | Co) & (!Ma | Ar) & (!Co | Ma) &
(!Ar | Ma) & (!Ar | Le | St | Ri))
In main() method I have:
def main(args: Array[String]) {
val ls: List[String] = List("Ma", "Co", "Ar")
//now I need to change the value of Ma, Co and Ar variables to "true"??
}
Is there a general way that may help to change the value of only these Boolean variables to true that are found in this list?
You can use an Enumeration, and use its ValueSet to store your true values instead of individual vars. This lets you refer to them by String name:
object MyBoolValue extends Enumeration {
type MyBoolValue = Value
val Ma, Co, Ar, Le, St, Ri = Value
}
class MyBoolValueContext {
var trueValues = MyBoolValue.ValueSet.empty
def isTrue(v: MyBoolValue) = trueValues contains v
def apply(v: MyBoolValue) = isTrue(v)
}
So then you can do:
import MyBoolValue._
val c = new MyBoolValueContext
c(Ma)
c.trueValues += Le
c.trueValues -= St
def logic: Boolean = (c(Ma) &
(!c(Ma) | c(Co)) &
(!c(Ma) | c(Ar)) &
(!c(Co) | c(Ma)) &
(!c(Ar) | c(Ma)) &
(!c(Ar) | c(Le) | c(St) | c(Ri)))
And you can use withName to handle String input:
c.trueValues ++= List("Ar", "Le", "St").map(MyBoolValue.withName)
You can get a little fancier by making the context implicit:
implicit case class ResolveMyBoolValue(self: MyBoolValue) extends AnyVal {
def get(implicit context: MyBoolValueContext): Boolean = context.isTrue(self)
}
implicit val context = new MyBoolValueContext
val result = Ar.get | !St.get
Or use an implicit conversion, though this could cause some confusion if misused:
implicit def resolveMyBoolValue(v: MyBoolValue)
(implicit context: MyBoolValueContext) = {
context.isTrue(v)
}
val result: Boolean = Le
Not without reflection, I think. But if you keep the name->value mapping in a map, you can do something like this:
val v = scala.collection.mutable.Map[String, Boolean]("Ma" -> false, "Co"-> false, "Ar" -> false, "Le" -> false, "St" -> false, "Ri" -> false)
//> v : scala.collection.mutable.Map[String,Boolean] = Map(Ar -> false, Le -> f
//| alse, Co -> false, Ma -> false, St -> false, Ri -> false)
def Logic:Boolean = (v("Ma") & (!v("Ma") | v("Co")) & (!v("Ma") | v("Ar")) & (!v("Co") | v("Ma")) &
(!v("Ar") | v("Ma")) & (!v("Ar") | v("Le") | v("St") | v("Ri")))
//> Logic: => Boolean
val ls: List[String] = List("Ma", "Co", "Ar") //> ls : List[String] = List(Ma, Co, Ar)
v("Ma") //> res0: Boolean = false
ls.foreach(v(_) = true)
v("Ma") //> res1: Boolean = true
Logic //> res2: Boolean = false
One of the ways to do it:
def Logic(params:List[String]) = {
val p = params.map(par => (par, true)).toMap.withDefaultValue(false)
(
p("Ma") & (
!p("Ma") | p("Co")
) & (
!p("Ma") | p("Ar")
) & (
!p("Co") | p("Ma")
) & (
!p("Ar") | p("Ma")
) & (
!p("Ar") | p("Le") | p("St") | p("Ri")
)
)
}
scala> Logic(List("Ma","Co","Ar"))
res0: Boolean = false
scala> Logic(List("Ma","Co","Ar","Le"))
res1: Boolean = true