I am struggling with using the extractor pattern in a certain use case where it seems that it could be very powerful.
I start with an input of Map[String, String] coming from a web request. This is either a searchRequest or a countRequest to our api.
searchRequest has keys
query(required)
fromDate(optional-defaulted)
toDate(optional-defaulted)
nextToken(optional)
maxResults(optional-defaulted)
countRequest has keys
query(required)
fromDate(optional-defaulted)
toDate(optional-defaulted)
bucket(optional-defaulted)
Then, I want to convert both of these to a composition type structure like so
protected case class CommonQueryRequest(
originalQuery: String,
fromDate: DateTime,
toDate: DateTime
)
case class SearchQueryRequest(
commonRequest: CommonQueryRequest,
maxResults: Int,
nextToken: Option[Long])
case class CountRequest(commonRequest: CommonQueryRequest, bucket: String)
As you can see, I am sort of converting Strings to DateTimes and Int, Long, etc. My issue is that I really need errors for invalid fromDate vs. invalid toDate format vs. invalid maxResults vs. invalid next token IF available.
At the same time, I need to stick in defaults(which vary depending on if it is a search or count request).
Naturally, with the Map being passed in, you can tell search vs. count so in my first go at this, I added a key="type" with value of search or count so that I could match at least on that.
Am I even going down the correct path? I thought perhaps using matching could be cleaner than our existing implementation but the further I go down this path, it seems to be getting a bit uglier.
thanks,
Dean
I would suggest you to take a look at scalaz.Validation and ValidationNel. It's super nice way to collect validation errors, perfect fit for input request validation.
You can learn more about Validation here: http://eed3si9n.com/learning-scalaz/Validation.html. However in my example I use scalaz 7.1 and it can be a little bit different from what described in this article. However main idea remains the same.
Heres small example for your use case:
import java.util.NoSuchElementException
import org.joda.time.DateTime
import org.joda.time.format.DateTimeFormat
import scala.util.Try
import scalaz.ValidationNel
import scalaz.syntax.applicative._
import scalaz.syntax.validation._
type Input = Map[String, String]
type Error = String
case class CommonQueryRequest(originalQuery: String,
fromDate: DateTime,
toDate: DateTime)
case class SearchQueryRequest(commonRequest: CommonQueryRequest,
maxResults: Int,
nextToken: Option[Long])
case class CountRequest(commonRequest: CommonQueryRequest, bucket: String)
def stringField(field: String)(input: Input): ValidationNel[Error, String] =
input.get(field) match {
case None => s"Field $field is not defined".failureNel
case Some(value) => value.successNel
}
val dateTimeFormat = DateTimeFormat.fullTime()
def dateTimeField(field: String)(input: Input): ValidationNel[Error, DateTime] =
Try(dateTimeFormat.parseDateTime(input(field))) recover {
case _: NoSuchElementException => DateTime.now()
} match {
case scala.util.Success(dt) => dt.successNel
case scala.util.Failure(err) => err.toString.failureNel
}
def intField(field: String)(input: Input): ValidationNel[Error, Int] =
Try(input(field).toInt) match {
case scala.util.Success(i) => i.successNel
case scala.util.Failure(err) => err.toString.failureNel
}
def countRequest(input: Input): ValidationNel[Error, CountRequest] =
(
stringField ("query") (input) |#|
dateTimeField("fromDate")(input) |#|
dateTimeField("toDate") (input) |#|
stringField ("bucket") (input)
) { (query, from, to, bucket) =>
CountRequest(CommonQueryRequest(query, from, to), bucket)
}
val validCountReq = Map("query" -> "a", "bucket" -> "c")
val badCountReq = Map("fromDate" -> "invalid format", "bucket" -> "c")
println(countRequest(validCountReq))
println(countRequest(badCountReq))
scalactic looks pretty cool as well and I may go that route (though not sure if we can use that lib or not but I think I will just proceed forward until someone says no).
Related
I have a very large file of json lines, which I intend to read into a list of case classes. Due to the size of the file, rather than reading the entire file into a variable first and then filtering, I would like to filter within the json decoding pattern matching. Currently the code looks like this:
import io.circe.Decoder
import io.circe.generic.semiauto.deriveDecoder
import io.circe.parser.decode
case class Person(name: String, age: Int, country: String)
val personList: List[Person] =
Source.fromResource("Persons.json").getLines.toList.map { line =>
implicit val jsonDecoder: Decoder[Person] = deriveDecoder[Person]
val decoded = decode[Person](line)
decoded match {
case Right(decodedJson) =>
Person(
decodedJson.name,
decodedJson.age,
decodedJson.country
)
case Left(ex) => throw new RuntimeException(ex)
}
}
however, if I wanted to only include Person instances with a country of "us", what would be the best way to accomplish this? Should I have nested pattern matching, that will specifically look for Person(_, _, "us") (im not sure how I would accomplish this), or is there some way I can implement Option handling?
You could do something like this:
import io.circe.Decoder
import io.circe.generic.semiauto.deriveDecoder
import io.circe.parser.decode
case class Person(name: String, age: Int, country: String)
implicit val jsonDecoder: Decoder[Person] = deriveDecoder[Person]
val personList: List[Person] =
Source
.fromResource("Persons.json")
.getLines
.flatMap { line =>
val decoded = decode[Person](line)
decoded match {
case Right(person # Person(_, _, "us")) => Some(person)
case Right(_) => None
case Left(ex) =>
println(s"couldn't decode: $line, will skip (error: ${ex.getMessage})")
None
}
}
.toList
println(s"US people: $personList")
A few things to note:
I moved the .toList to the end. In your implementation, you called it right after .getLines which kind of loses the lazyness of the whole thing. Assuming there's only a few US people out of huge number of people in the JSON file, this can be beneficial for performance & efficiency.
Wrapping each iteration's result in an Option along with flatMap over the original Iterator we're running upon is very helpful to get this kind collection filtering.
I didn't throw an exception upon an error, but rather logged it and moved on with a None. You could also accumulate errors and do whatever you want with them after all iterations are done, if that's helpful to you.
The # in person # Person(_, _, "us") can be used for something like "match & bind" upon the whole object in question.
As the comment to the original question noted - no need to re-instantiate the implicit Decoder upon each iteration. You can just pull it one layer up, as I did in my example.
I am building the back-end of an app using http4s. In the app i receive a json response from an external api (not the one i am working on). The api response is of this pattern below.
json response:
`{
"datatable" : {
"data" : [["AAPl", "MRT", "2020-03-20", 123, 123, 12.4, 233, 3234],
["AAPl", "MRT", "2020-03-20", 123, 123, 12.4, 233, 3234]],
"meta": {
"next_date : null
}
}`
my question is?
can someone show me how to create an entity decoder and entity encoder that would decode the pattern. i can seem to get it to work.
currently i have:
object Stocks {
case class tickerInfo(ticker: String, dim:String, date: String, a: Int, b: Int, c: Float, d: Int, e: Int)
case class data(data: Seq[tickerInfo])
case class meta(next_date : Option[String])
case class table(data: data, meta:meta)
case class stockInfo(datatable:table)
object stockInfo {
implicit def stockInfoEntityDecoder[F[_]:Sync]: EntityDecoder [F, stockInfo] = jsonOf
implicit def stockInfoEntityEncoder[F[_] : Applicative]: EntityEncoder[F, stockInfo] = jsonEncoderOf
}
val decodedJson = C.expect[stockInfo](GET(Uri.uri("www.externalApi.com")
}
But this doesn't work. Please can someone tell me where i am going wrong.
I am getting a run time error *not a compile error). and its an http4s error that says - InvalidMessageBodyFailure.
Thanks
You have a couple of errors in your model, but the main problem is that circe won't be able to automatically decode an array of jsons into a case class.
If you can not modify the source of the data, you would need to create your own custom codec.
object Stocks {
final case class TickerInfo(ticker: String, dim: String, date: String, a: Int, b: Int, c: Float, d: Int, e: Int)
final case class Meta(next_date : Option[String])
final case class Table(data: List[TickerInfo], meta: Meta)
final case class StockInfo(datatable: Table)
object StockInfo {
implicit final val TickerInfoDecoder: Decoder[TickerInfo] = Decoder[List[Json]].emap {
case ticker :: dim :: date :: a :: b :: c :: d :: e :: Nil =>
(
ticker.as[String],
dim.as[String],
date.as[String],
a.as[Int],
b.as[Int],
c.as[Float],
d.as[Int],
e.as[Int]
).mapN(TickerInfo).left.map(_.toString)
case list =>
Left(s"Bad number of fields in: ${list.mkString("[", ", ", "]")}")
}
implicit final val MetaDecoder: Decoder[Meta] = deriveDecoder
implicit final val TableDecoder: Decoder[Table] = deriveDecoder
implicit final val StockInfoDecoder: Decoder[StockInfo] = deriveDecoder
}
}
(you can see it working here, I leave outside the http4s part as it would be tricky to mock, but it shouldn't matter)
The error reporting of that custom encoder could be improved by providing more useful messages for each field. That is left as an exercise for the reader.
You need to:
create a data model, presumably consisting of some case classes and possibly sealed traits. What this data model should look like depends on the external API that you're talking to.
create JSON decoders for this data model. These should go in the case classes' companion objects, because that will allow the compiler to find them when needed without having to import anything.
Use the http4s-circe library in order to integrate these decoders with http4s. https://http4s.org/v0.19/json/
If you did everything correctly, you should then be able to use the Http4s client to retrieve the data, e. g. httpClient.expect[YourModelClass](Uri.uri("http://somewhere.com/something/")).
Welcome! I'd probably start inside and work your way out:
So for elements in the data array, perhaps something like (I'm guessing on the domain):
case class Stock(
name: String,
mart: String,
datePosted: LocalDate,
a: Int,
b: Int,
price: Double,
c: Int,
d: Int
)
Using Circe's automatic derevation should handle this pretty well.
Data seems like an array of arrays of exploded elements? So you may have to write some extraction logic to convert to the inner Object manually if that is the case. Something like this in the Controller may help:
elem match {
case name :: mart :: date :: a :: b :: price :: c :: d :: Nil => Stock(name, mart, date, a, b, price, c, d)
case invalid # _ => log.warn(s"Invalid record: $invalid")
}
Hopefully the above snippet returns something useful like an Either[E, A], etc.
Finally, you'd need objects for the outer JSON. Something simple to capture, such as case class ExternalApiRequest(dataTable: T), where T is a type appropriate for the above case. List[String] in the worst case?
Hope this helped! Let me know if you have any specific errors you were running into
How to write a method called calculator that accepts three string parameters:
def calculator(operand1: String, operator: String, operand2: String): Unit
Converts the operands to Int;
Performs the desired mathematical operator (+, -, *, or /) on the two operands
Prints the result, or a generic error messages
Your question shows that you put little to no effort into finding the solution yourself.
When asking a question on StackOverflow next time, ask a question about existing code (e.g. "Why am I getting this exception?" or "Why doesn't my code compile?") and don't assume some internet code monkey will magically write your code.
Anyways, as you seem to be a new member of SO, def calculator would look something like this:
import scala.collection.immutable.StringOps._
import scala.util.{Try, Success, Failure}
def calculator(left: String, op: String, right: String): Unit = {
def parse(value: String) = Try(value.toDouble)
(parse(left), parse(right)) match {
case (Success(leftDouble), Success(rightDouble)) => {
op match {
case "/" => println(leftDouble / rightDouble)
case "*" => println(leftDouble * rightDouble)
case "+" => println(leftDouble + rightDouble)
case "-" => println(leftDouble - rightDouble)
case invalid: String => println(s"Invalid operator $invalid.")
}
}
case (Failure(e), _) => println(s"Could not parse $left.")
case(_, Failure(e)) => println(s"Could not parse $right.")
case(Failure(e1), Failure(e2)) => println(s"Could not parse $left and $right.")
}
}
Try it out!
If you need any explanation don't hesitate to drop a comment.
I hope this helps.
I'm new to Scala and Play framework. I try to query all the data for selected columns from a data table and save them as Excel file.
Selected columns usually have different types, such as Int, Str, Timestamp, etc.
I want to convert all value types, include null into String
(null convert to empty string "")
without knowing the actual type of a column, so the code can be used for any tables.
According to Play's document, I can write the implicit converter below, however, this cannot handle null. Googled this for long time, cannot find solution. Can someone please let me know how to handle null in the implicit converter?
Thanks in advance~
implicit def valueToString: anorm.Column[String] =
anorm.Column.nonNull1[String] { (value, meta) =>
val MetaDataItem(qualified, nullable, clazz) = meta
value match {
case s: String => Right(s) // Provided-default case
case i: Int => Right(i.toString()) // Int to String
case t: java.sql.Clob => Right(t.toString()) // Blob/Text to String
case d: java.sql.Timestamp => Right(d.toString()) // Datatime to String
case _ => Left(TypeDoesNotMatch(s"Cannot convert $value: ${value.asInstanceOf[AnyRef].getClass} to String for column $qualified"))
}
}
As indicated in the documentation, if there a Column[T], allowing to parse of column of type T, and if the column(s) can be null, then Option[T] should be asked, benefiting from the generic support as Option[T].
There it is a custom Column[String] (make sure the custom one is used, not the provided Column[String]), so Option[String] should be asked.
import myImplicitStrColumn
val parser = get[Option[String]]("col")
I have a pattern to process web service requests using chained partial functions (this is a chain of responsibility pattern, I think?). In my example, let's say there are two parameters for the request, a string Id and a date. There's a verification step involving the id, a verification step checking the date, and finally some business logic that use both. So I have them implemented like so:
object Controller {
val OK = 200
val BAD_REQUEST = 400
type ResponseGenerator = PartialFunction[(String, DateTime), (String, Int)]
val errorIfInvalidId:ResponseGenerator = {
case (id, _) if (id == "invalid") => ("Error, Invalid ID!", BAD_REQUEST)
}
val errorIfFutureDate:ResponseGenerator = {
case (_, date) if (date.isAfter(DateTime.now)) => ("Error, date in future!", BAD_REQUEST)
}
val businessLogic:ResponseGenerator = {
case (id, date) => {
// ... do stuff
("Success!", OK)
}
}
def handleRequest(id:String, date:DateTime) = {
val chained = errorIfInvalidId orElse errorIfFutureDate orElse businessLogic
val result: (String, Int) = chained(id, date)
// make some sort of a response out of the message and status code
// e.g. in the Play framework...
Status(result._2)(result._1)
}
}
I like this pattern because it's very expressive - you can easily grasp what the controller method logic is just by looking at the chained functions. And, I can easily mix and match different verification steps for different requests.
The problem is that as I try to expand this pattern it starts to break down. Suppose my next controller takes an id I want to validate, but does not have the date parameter, and maybe it has some new parameter of a third type that does need validation. I don't want to keep expanding that tuple to (String, DateTime, Other) and have to pass in a dummy DateTime or Other. I want to have partial functions that accept different types of arguments (they can still return the same type). But I can't figure out how to compose them.
For a concrete question - suppose the example validator methods are changed to look like this:
val errorIfInvalidId:PartialFunction[String, (String, Int)] = {
case id if (id == "invalid") => ("Error, Invalid ID!", BAD_REQUEST)
}
val errorIfInvalidDate:PartialFunction[DateTime, (String, Int)] = {
case date if (date.isAfter(DateTime.now)) => ("Error, date in future!", BAD_REQUEST)
}
Can I still chain them together? It seems like I should be able to map the tuples to them, but I can't figure out how.
I'm a big fan of using scalaz's Validation for things like this. It gives you quite a bit of control over what you want to do with errors and how to handle them. Here's an example using you're controller:
import scalaz._
import Scalaz._
object Controller {
val OK = 200
val BAD_REQUEST = 400
case class Response(response: String, status: Int)
def validateIfInvalidId(id: String) = (id == "invalid") ?
Response("Error, Invalid ID!", BAD_REQUEST).fail[String] |
id.success[Response]
def validateIfFutureDate(date: DateTime, currentDate: DateTime = DateTime.now) = (date.isAfter(currentDate)) ?
Response("Error, date in future!", BAD_REQUEST).fail[DateTime] |
date.success[Response]
def handleRequest(id: String, date: DateTime) = {
val response = for {
validatedId <- validateIfInvalidId(id)
validatedDate <- validateIfFutureDate(date)
} yield {
// ... do stuff
Response("Success!", OK)
}
// make some sort of a response out of the message and status code
// e.g. in the Play framework...
response.fold(
failure => Status(failure.response, failure.status),
success => Status(success.response, success.status)
)
}
}
You can move the different validation functions off into their own world and then compose them anytime you want with the for comprehension in scala.
Okay, I found a way to do this which seems not too bad. Originally I was thinking it might work to wrap the "base" version of the partial function in another partial function that takes the tuple. But I couldn't figure out how to do it, until I hit on the obvious-in-retrospect idea of using isDefined in a case guard statement. Like so:
// "base" version
val errorIfInvalidId:PartialFunction[String, (String, Int)] = {
case id if (id == "invalid") => ("Error, Invalid ID!", BAD_REQUEST)
}
// wrapped to take tuple as parameter
val wrappedErrorIfInvalidId:PartialFunction[(String, DateTime), (String, Int)] = {
case (id, _) if (errorIfInvalidId.isDefinedAt(id)) => errorIfInvalidId(id)
}
This approach is serviceable, though I still wonder if there isn't a more direct way of accomplishing it. (I also may switch over to the Scalaz validation suggested by Noah after I get a chance to play with it a bit.)
You can make PartialFunction more generic, making it PartialFunction[Any, (String, Int)]
Altho, mb it will be slower. Do not know matching mechanics under PartialFunction