Can this be made prettier? Matching string to case class field - scala

Sorry about the title, please edit it to be more descriptive if you can!
Is there a way to generalize this with scala? I have quite a few fields that can be filtered against, and this is just plain ugly! The problem I ran against was matching parameter name against the case class field, can it be done in a more general way, without this much code duplication?
get("/MostClicked") { request =>
val res = MongoDbOps.findMostClicked()
val res1 = request.params.get("source") match {
case None => res
case Some(f) => res.filter(_.source == f)
}
val res2 = request.params.get("category") match {
case None => res1
case Some(f) => res1.filter(_.category == f)
}
// more of the same...
render.plain {
res2.toJson.prettyPrint
}.toFuture
}

You can try either of the two approaches below.
case class MostClicked(
source: String,
category: String,
rating: String)
object MongoDbOps {
def findMostClicked() = List[MostClicked]()
}
class Request {
val params = Map[String, String]()
}
def get(path: String)(f: Request => String) = {
f(new Request)
}
First is to use a List of matchers and then apply them sequentially using foldLeft:
get("/MostClicked") { request =>
val res = MongoDbOps.findMostClicked()
val kfun = List(
"source" -> ((x: MostClicked, y: String) => x.source == y),
"category" -> ((x: MostClicked, y: String) => x.category == y),
"rating" -> ((x: MostClicked, y: String) => x.rating == y))
val r = kfun.foldLeft(res) { (x, y) =>
request.params.get(y._1)
.map(f => res.filter(y._2(_, f)))
.getOrElse(x)
}
r.toString
// more of the same...
render.plain {
r.toJson.prettyPrint
}.toFuture
}
Or simply make it more readable:
get("/MostClicked") { request =>
val res = MongoDbOps.findMostClicked()
val res1 = request.params.get("source")
.map(f => res.filter(_.source == f))
.getOrElse(res)
val res2 = request.params.get("category")
.map(f => res.filter(_.category == f))
.getOrElse(res1)
val res3 = request.params.get("rating")
.map(f => res.filter(_.rating == f))
.getOrElse(res2)
// more of the same...
render.plain {
res3.toJson.prettyPrint
}.toFuture
}

I had to break it down to parts and work out the types, thus the smaller methods.
It works in my simple experiment and gets rid of the duplication where it can, but might not be as readable?
Note that unless you get into reflection, you still need to create the name of the parameter and how it should be filtered.
This looks to be the same as tuxdna's answer except with types to increase readability and maintainability
SETUP
case class Request(params: Map[String, String])
case class Result(category: String, source: String)
type Filterer = (Result, String) => Boolean
case class FilterInfo(paramName: String, filterer: Filterer)
type Analyzer = FilterInfo => List[Result]
val request = Request(Map("source"->"b"))
EXTRACTION METHODS
def reduce(filterInfos: List[FilterInfo], results: List[Result]) = {
filterInfos.foldLeft(results) { (currentResult, filterInfo) =>
request.params.get(filterInfo.paramName)
.map(filterVal => currentResult.filter(filterInfo.filterer(_, filterVal)))
.getOrElse(currentResult)
}
}
USAGE
val filterInfos = List(
FilterInfo("source", (result, filterVal) => result.source == filterVal),
FilterInfo("category", (result, filterVal) => result.category == filterVal))
val res = List(Result("a","a"), Result("b", "b"))
reduce(filterInfos, res)
Used in your example it would be more like this:
get("/MostClicked") { request =>
val res = MongoDbOps.findMostClicked()
val filterInfos = List(
FilterInfo("source", (result, filterVal) => result.source == filterVal),
FilterInfo("category", (result, filterVal) => result.category == filterVal))
val finalResult = reduce(filterInfos, res)
render.plain {
finalResult.toJson.prettyPrint
}.toFuture
}

Related

Convert Map[String, Any] to nested case class with controlled Type casting

I'd like to convert a Map[String, Any] to a given case class, and the map can be a nested map.
The case class can have certain fields as Optional, and if the map doesn't have the field in question , it'll have a default value.
eg.
case class CC(a: Int, b: Option[Int], c: String)
Map(a -> 1, c -> "abc")
result case class : CC(1, None, "abc")
The only caveat is I need to control the typeSignatures of these fields.
ex:
Map(a ->, b -> "10", c ->"abc"). \\Note b is string here
result case class : CC(1, Some(10), "abc")
I got some code that can do this exact behavior(code below), but I cant seem to make it work for nested case class
import scala.reflect.runtime.universe._
object DynamicClassFactory {
val classLoaderMirror = runtimeMirror(getClass.getClassLoader)
}
class DynamicClassFactory[T: TypeTag] {
import DynamicClassFactory.classLoaderMirror
val tpe = typeOf[T]
val classSymbol = tpe.typeSymbol.asClass
val companion = tpe.typeSymbol.companion
val classMirror = classLoaderMirror reflectClass classSymbol
val instanceMirror = classLoaderMirror.reflectModule(companion.asModule)
val objMirror = classLoaderMirror.reflect(instanceMirror.instance)
val constructorSymbol = tpe.decl(termNames.CONSTRUCTOR)
val constructorArgs = constructorSymbol.asMethod.paramLists.flatten.map{
p => (p.name.decodedName.toString, p.typeSignature)
}
val defaultConstructor =
if (constructorSymbol.isMethod) constructorSymbol.asMethod
else {
val ctors = constructorSymbol.asTerm.alternatives
ctors.map { _.asMethod }.find { _.isPrimaryConstructor }.get
}
val constructorMethod = classMirror reflectConstructor defaultConstructor
def buildParameters: List[String] = constructorArgs.map ( x => x._1 )
def getCompanionParam(name: String): Any =
if(classSymbol.isCaseClass) {
objMirror.reflectMethod(instanceMirror.symbol.info.decl(TermName(name)).asMethod)("test")
}
def getDefaults = {
val defaults = companion.typeSignature.members.filter { m => m.isMethod && m.name.toString.contains("apply$default")}.toList.map { x => (x.asMethod.name.toTermName.toString.replaceAll("[^0-9]", "").toInt-1, getCompanionParam(x.asMethod.name.toTermName.toString)) }.toMap
val defaultValues = scala.collection.mutable.Map[String, Any]()
for( (x, i) <- buildParameters.view.zipWithIndex ) {
if(defaults.contains(i)) {
defaultValues(x) = defaults(i)
}
}
defaultValues.toMap
}
def buildWith(args: Seq[_]): T = {
constructorMethod(args: _*).asInstanceOf[T]
}
def buildSafe(configuration: Map[String, Any]): T = {
val config = getDefaults ++ configuration
val safeConfig = safeTypeConvert(config, constructorArgs.toMap)
buildWith(constructorArgs.map { x => safeConfig(x._1) })
}
def safeTypeConvert(v: Map[String, Any], t: Map[String, Type]): Map[String, Any] = {
val o = scala.collection.mutable.Map[String, Any]()
val z = v.filter{
case (k, v) =>
t.contains(k)
}
z.foreach {
case (k, v) =>
val typeS: Type = t(k)
try {
if (typeS <:< typeOf[Option[String]]) {
v match {
case x: Int => o(k) = Some(x.toString)
case y: Option[Nothing] => o(k) = None
case _ => o(k) = Some(v.asInstanceOf[String])
}
}
else if (typeS <:< typeOf[Option[Int]]) {
v match {
case x: String => o(k) = Some(x.toFloat.toInt)
case y: Option[Nothing] => o(k) = None
case _ => o(k) = Some(v.asInstanceOf[Int])
}
}
else if (typeS <:< typeOf[Option[Boolean]]) {
v match {
case x: String => o(k) = Some(x.toLowerCase.toBoolean)
case y: Option[Nothing] => o(k) = None
case _ => o(k) = v
}
}
else if (typeS <:< typeOf[Int]) {
v match {
case x: String => o(k) = x.toFloat.toInt // this is to handle decimals returned from json
case _ => o(k) = v.asInstanceOf[Int]
}
} else if (typeS =:= typeOf[String]) {
v match {
case x: Int => o(k) = x.toString
case _ => o(k) = v.asInstanceOf[String]
}
} else if (typeS =:= typeOf[Boolean]) {
v match {
case x: String => o(k) = x.toLowerCase.toBoolean
case _ => o(k) = v
}
}
else if (typeS <:< typeOf[List[_]]) {
if (v == Nil) {
o(k) = List[String]()
} else {
o(k) = v.asInstanceOf[List[_]]
}
} else if (typeS <:< typeOf[Map[_, _]]) {
if (v == None) {
o(k) = Map()
} else o(k) = v
}
else {
throw new Exception(s"sdf $k must be of type ${typeS.typeSymbol.name.decoded}, got ${v.getClass.getName}: ${v.toString}")
}
} catch {
case e#(_: ClassCastException | _: NumberFormatException | _: IllegalArgumentException) =>
throw new Exception(s"$k must be of type ${typeS.typeSymbol.name.decoded}, got ${v.getClass.getName}: ${v.toString}")
}
}
return o.toMap
}
}
The above works great for a map which is not a nested map, but fails for a nested map
case class Address(addLine1: String, addLine2: Optional[String], zip: Long)
case class Inner(id: Option[String], addr: Option[Address], isPath: Option[Boolean])
case class Outer(name: String, addtlnInfo: Inner)
val map: Map[String, Any] = Map("name" -> "john", "addtlnInfo" -> Map("id" -> 123, "addr" -> Map("addLine1" -> "901 Lincoln St", "zip" -> "80101")))
val ins = new DynamicClassFactory[Outer]
ins.buildSafe(map)

How to define an empty Tuple?

I have the following function definition:
private def extractUrl: String => (String, String)
= url =>
url
.split("/")
.toList
.filter(_.startsWith("localhost"))
.flatMap(e => e.split(":").toList)
.foldLeft[(String, String)](("", "")) { (acc, x) =>
acc match {
case ("", "") => (x, "")
case (a, "") => (a, x)
case z => z
}
}
the question is, is there another way to define an empty Tuple instead of ("", "")?
Empty tuple
("", "") is a tuple of empty strings with type (String, String).
Empty is unclear in that context, could be (None, None) or even (null, null) (bad)
You seem to use "" to represents a value that is not present. Try using None and Some[String], both sub types of Option[String], to indicate that a value is not present.
Analysis and comments
Potentially, your method seems not to do what is intended. (execute code below)
Think about using already present functions / methods / libraries for manipulating URLs (also see example below)
Think about using Option
object Fun {
import java.net.URL
def main(args: Array[String]): Unit = {
val url1 = "http://localhost:4000/a/b/c?x=1&y=2#asdf"
val url2 = "http://example.com:4000/a/localhostb/c?x=1&y=2#asdf"
val urls = List(url1, url2)
// your approach
println("Your approach")
urls.map( urlString => extractUrl(urlString ))
.foreach(println)
println("New approach")
urls.map(x => extractUrl2(x))
.filter( x => x.host.startsWith("localhost") )
.foreach(println)
}
case class HostPort(host: String, port: Option[String])
def extractUrl2: String => HostPort = urlString => {
val url = new URL(urlString)
HostPort(url.getHost,
url.getPort match {
case -1 => None
case i => Some(i.toString)
})
}
def extractUrl: String => (String, String) = url =>
url
.split("/")
.toList
.filter(_.startsWith("localhost"))
.flatMap(e => e.split(":").toList)
.foldLeft[(String, String)](("", "")) { (acc, x) =>
acc match {
case ("", "") => (x, "")
case (a, "") => (a, x)
case z => z
}
}
}
yields
Your approach
(localhost,4000)
(localhostb,)
New approach
HostPort(localhost,Some(4000))
I don't think it is possible to define an empty Tuple.
I tried to use (->) but that resolves to a Tuple2.type and not a Tuple2.
If the values of your Tuple are optional, use the type system to express that:
(Option[String], Option[String])
Better yet, you could define a case class for your data structure:
case class HostAndPort(host: Option[String], port: Option[String])
This would provide default values for each type within the tuple:
let my_tuple: (String, usize) = Default::default();

scoobi concatenate DLists issue

I have a scala scoobi code (that i execute locally for now) of the following structure:
def run() = {
val myvalue1 = processSource.source1("mypath/inputData/", references) // has several subfolders with input files that should be processed and some lines in these files are to be rejected due to some reasons and written to a log
val (partitionedVal1, partitionedVal2) = myvalue1.partition {
case Left(x) => true
case _ => false}
val output = processData.process(partitionedVal1) // some processing
val getRidOfRight = partitionedVal2.map(x => x.right.get)
persist(getRidOfRight.toTextFile("output/log", overwrite = true), output.toTextFile("output/processed_data", overwrite = true))
}
And the processSource is an object as:
object processSource {
def source1(a:DList[Either[Class1, String]] , b:DList[Either[Class2, String]] ) = {
val (a1, a2) = a.partition {
case Left(x) => true
case _ => false}
val (b1, b2) = b.partition {
case Left(x) => true
case _ => false}
val joinedAB1:DList[Either[Class1, String]] = a1.by(_.myKey).joinLeft(b1.by(_.myOtherKey)).map {
case (key, (val1, Some(val2))) => Left(val1.withCommon(val2))
case (key,(val1, None)) => Left(val1)}
val AB2:DList[Either[Class1, String]] = b2.map{ x => Right(x.right.get)}
AB1 ++ AB2 ++ b1
}
}
Now, my problem is that the getRidOfRight returns only part of the output it should return. WHY?
github.com/NICTA/scoobi/issues/328 its a bug in scoobi - solution is to use scoobi 0.9.1 and "collect" instead of "partition" ("partition" is not fixed)

How to check if there's None in List[Option[_]] and return the element's name?

I have multiple Option's. I want to check if they hold a value. If an Option is None, I want to reply to user about this. Else proceed.
This is what I have done:
val name:Option[String]
val email:Option[String]
val pass:Option[String]
val i = List(name,email,pass).find(x => x match{
case None => true
case _ => false
})
i match{
case Some(x) => Ok("Bad Request")
case None => {
//move forward
}
}
Above I can replace find with contains, but this is a very dirty way. How can I make it elegant and monadic?
Edit: I would also like to know what element was None.
Another way is as a for-comprehension:
val outcome = for {
nm <- name
em <- email
pwd <- pass
result = doSomething(nm, em, pwd) // where def doSomething(name: String, email: String, password: String): ResultType = ???
} yield (result)
This will generate outcome as a Some(result), which you can interrogate in various ways (all the methods available to the collections classes: map, filter, foreach, etc.). Eg:
outcome.map(Ok(result)).orElse(Ok("Bad Request"))
val ok = Seq(name, email, pass).forall(_.isDefined)
If you want to reuse the code, you can do
def allFieldValueProvided(fields: Option[_]*): Boolean = fields.forall(_.isDefined)
If you want to know all the missing values then you can find all missing values and if there is none, then you are good to go.
def findMissingValues(v: (String, Option[_])*) = v.collect {
case (name, None) => name
}
val missingValues = findMissingValues(("name1", option1), ("name2", option2), ...)
if(missingValues.isEmpty) {
Ok(...)
} else {
BadRequest("Missing values for " + missingValues.mkString(", ")))
}
val response = for {
n <- name
e <- email
p <- pass
} yield {
/* do something with n, e, p */
}
response getOrElse { /* bad request /* }
Or, with Scalaz:
val response = (name |#| email |#| pass) { (n, e, p) =>
/* do something with n, e, p */
}
response getOrElse { /* bad request /* }
if ((name :: email :: pass :: Nil) forall(!_.isEmpty)) {
} else {
// bad request
}
I think the most straightforward way would be this:
(name,email,pass) match {
case ((Some(name), Some(email), Some(pass)) => // proceed
case _ => // Bad request
}
A version with stone knives and bear skins:
import util._
object Test extends App {
val zero: Either[List[Int], Tuple3[String,String,String]] = Right((null,null,null))
def verify(fields: List[Option[String]]) = {
(zero /: fields.zipWithIndex) { (acc, v) => v match {
case (Some(s), i) => acc match {
case Left(_) => acc
case Right(t) =>
val u = i match {
case 0 => t copy (_1 = s)
case 1 => t copy (_2 = s)
case 2 => t copy (_3 = s)
}
Right(u)
}
case (None, i) =>
val fails = acc match {
case Left(f) => f
case Right(_) => Nil
}
Left(i :: fails)
}
}
}
def consume(name: String, email: String, pass: String) = Console println s"$name/$email/$pass"
def fail(is: List[Int]) = is map List("name","email","pass") foreach (Console println "Missing: " + _)
val name:Option[String] = Some("Bob")
val email:Option[String]= None
val pass:Option[String] = Some("boB")
val res = verify(List(name,email,pass))
res.fold(fail, (consume _).tupled)
val res2 = verify(List(name, Some("bob#bob.org"),pass))
res2.fold(fail, (consume _).tupled)
}
The same thing, using reflection to generalize the tuple copy.
The downside is that you must tell it what tuple to expect back. In this form, reflection is like one of those Stone Age advances that were so magical they trended on twitter for ten thousand years.
def verify[A <: Product](fields: List[Option[String]]) = {
import scala.reflect.runtime._
import universe._
val MaxTupleArity = 22
def tuple = {
require (fields.length <= MaxTupleArity)
val n = fields.length
val tupleN = typeOf[Tuple2[_,_]].typeSymbol.owner.typeSignature member TypeName(s"Tuple$n")
val init = tupleN.typeSignature member nme.CONSTRUCTOR
val ctor = currentMirror reflectClass tupleN.asClass reflectConstructor init.asMethod
val vs = Seq.fill(n)(null.asInstanceOf[String])
ctor(vs: _*).asInstanceOf[Product]
}
def zero: Either[List[Int], Product] = Right(tuple)
def nextProduct(p: Product, i: Int, s: String) = {
val im = currentMirror reflect p
val ts = im.symbol.typeSignature
val copy = (ts member TermName("copy")).asMethod
val args = copy.paramss.flatten map { x =>
val name = TermName(s"_$i")
if (x.name == name) s
else (im reflectMethod (ts member x.name).asMethod)()
}
(im reflectMethod copy)(args: _*).asInstanceOf[Product]
}
(zero /: fields.zipWithIndex) { (acc, v) => v match {
case (Some(s), i) => acc match {
case Left(_) => acc
case Right(t) => Right(nextProduct(t, i + 1, s))
}
case (None, i) =>
val fails = acc match {
case Left(f) => f
case Right(_) => Nil
}
Left(i :: fails)
}
}.asInstanceOf[Either[List[Int], A]]
}
def consume(name: String, email: String, pass: String) = Console println s"$name/$email/$pass"
def fail(is: List[Int]) = is map List("name","email","pass") foreach (Console println "Missing: " + _)
val name:Option[String] = Some("Bob")
val email:Option[String]= None
val pass:Option[String] = Some("boB")
type T3 = Tuple3[String,String,String]
val res = verify[T3](List(name,email,pass))
res.fold(fail, (consume _).tupled)
val res2 = verify[T3](List(name, Some("bob#bob.org"),pass))
res2.fold(fail, (consume _).tupled)
I know this doesn't scale well, but would this suffice?
(name, email, pass) match {
case (None, _, _) => "name"
case (_, None, _) => "email"
case (_, _, None) => "pass"
case _ => "Nothing to see here"
}

Returning multiple collections from flatMap

I am working on a method that has 3 possible outcomes for multiple items: Error, Invalid and Success. For each of these I need to return a json list identifying which items were in error, invalid and successful.
My current attempt follows. I have used Object to represent the class my objects are as fully explaining would take too long. The Object class has a method process which returns a boolean to indicate success or error and throws an exception when the object is invalid:
def process(list: List[Objects]) = {
val successIds = new ListBuffer[Int]();
val errorIds = new ListBuffer[Int]();
val invalidIds = new ListBuffer[Int]();
list.foreach( item => {
try {
if (item.process) {
successIds ++ item.id
} else {
errorIds ++ item.id
}
} catch {
case e: Exception => invalidIds ++ item.id
}
})
JsonResult(
Map("success" -> successIds,
"failed" -> errorIds,
"invalid" -> invalidIds)
)
}
Problem is using Mutable data structures isn't very "Scala-y". I would prefer to build up these lists in some more functional way but I am quite new to scala. Any thoughts or hints as to how this might be done?
My though is using something like the flatMap method that takes a tuple of collections and collates them in the same way the flatMap method does for a single collection:
def process(list: List[Objects]) = {
val (success, error, invalid) = list.flatMap( item => {
try {
if (item.process) {
(List(item.id), List.empty, List.empty)
} else {
(List.empty, List(item.id), List.empty)
}
} catch {
case e: Exception =>
(List.empty, List.empty, List(item.id))
}
})
JsonResult(
Map("success" -> success,
"failed" -> error,
"invalid" -> invalid)
)
}
flatMap isn't what you need here - you need groupBy:
def process(list: List[Objects]) = {
def result(x: Objects) =
try if (x.process) "success" else "failed"
catch {case _ => "invalid"}
JsonResult(list groupBy result mapValues (_ map (_.id)))
}
There's always recursion:
class Ob(val id: Int) { def okay: Boolean = id < 5 }
#annotation.tailrec def process(
xs: List[Ob],
succ: List[Int] = Nil,
fail: List[Int] = Nil,
invalid: List[Int] = Nil
): (List[Int], List[Int], List[Int]) = xs match {
case Nil => (succ.reverse, fail.reverse, invalid.reverse)
case x :: more =>
val maybeOkay = try { Some(x.okay) } catch { case e: Exception => None }
if (!maybeOkay.isDefined) process(more, succ, fail, x.id :: invalid)
else if (maybeOkay.get) process(more, x.id :: succ, fail, invalid)
else process(more, succ, x.id :: fail, invalid)
}
Which works as one would hope (skip the reverses if you don't care about order):
scala> process(List(new Ob(1), new Ob(7), new Ob(2),
new Ob(4) { override def okay = throw new Exception("Broken") }))
res2: (List[Int], List[Int], List[Int]) = (List(1,2),List(7),List(4))
Adapted to make it compile without "Objects"
def procex (item: String): Boolean = ((9 / item.toInt) < 1)
def process (list: List[String]) = {
val li: List[(Option[String], Option[String], Option[String])] = list.map (item => {
try {
if (procex (item)) {
(Some (item), None, None)
} else {
(None, Some (item), None)
}
} catch {
case e: Exception =>
(None, None, Some (item))
}
})
li
}
// below 10 => failure
val in = (5 to 15).map (""+_).toList
// 0 to throw a little exception
val ps = process ("0" :: in)
val succeeders = ps.filter (p=> p._1 != None).map (p=>p._1)
val errors = ps.filter (p=> p._2 != None).map (p=>p._2)
val invalides = ps.filter (p=> p._3 != None).map (p=>p._3)
What doesn't work:
(1 to 3).map (i=> ps.filter (p=> p._i != None).map (p=>p._i))
_i doesn't work.