I am parsing a Json using JSON.parseFull.
Before parsing, Json was like this
{
"response":
{
"status":"ok",
"userTier":"developer",
"total":1,
"content":
{
"id":"technology/2014/feb/18/doge-such-questions-very-answered",
"type":"article",
"sectionId":"technology",
"sectionName":"Technology",
"webPublicationDate":"2014-02-18T10:25:30Z",
"webTitle":"What is Doge?",
"webUrl":"https://www.theguardian.com/technology/2014/feb/18/doge-such-questions-very-answered",
"apiUrl":"https://content.guardianapis.com/technology/2014/feb/18/doge-such-questions-very-answered",
"isHosted":false
}
}
}
After parsing, it becomes this,
Map(response ->
Map(status -> ok,
userTier -> developer,
total -> 1.0,
content ->
Map(webUrl ->
https://www.theguardian.com/technology/2014/feb/18/doge-such-questions-very-answered,
webPublicationDate -> 2014-02-18T10:25:30Z,
webTitle -> What is Doge?,
sectionName -> Technology,
apiUrl -> https://content.guardianapis.com/technology/2014/feb/18/doge-such-questions-very-answered,
id -> technology/2014/feb/18/doge-such-questions-very-answered,
isHosted -> false,
sectionId -> technology,
type -> article
)
)
)
I need to get the values like webUrl and webtitle.
Anyone knows how to achieve that?
Scala Json library is a bit brittle, in the sense it doesn't strongly type your parsed Json. Therefore, my guess would be to drop asInstanceOf calls here and there. Example below
type JsonMap = Map[String, Any]
val maybeParsedMap: Option[JsonMap] = JSON.parseFull(jsonString).map(_.asInstanceOf[JsonMap])
val content: Option[JsonMap] = for {
parsedMap <- maybeParsedMap
response <- parsedMap.get("response")
content <- response.asInstanceOf[JsonMap].get("content")
} yield content.asInstanceOf[JsonMap]
val webUrl: Option[String] = content.asInstanceOf[JsonMap].get("webUrl").map(_.asInstanceOf[String])
Related
I have a List of Maps. One of the maps has another map inside it (2 level deep). I need to access some of keys from the inner most map and finally change the values. The issue I'm facing is to retrieve the keys from the inner most map. I'm very new to Scala and tried different things without luck.
I have flatten the List to Map and then tried to retrieve the key, values. The thing is, I can print the entire inner map, but not sure how to iterate thru that.
Below is the code: at a very basic, I would like to retrieve the values corresponding to keys from innermost map; say for the keys "isStudentLoankRequested", "schoolStructure".
object ListToMaps {
def main(args: Array[String]) {
val dataInputKeysListvar = List(Map("identityKeyName" -> "MY_ID", "identityKeyValue" -> "abcd-1234-xyz"),
Map("identityKeyName" -> "OUR_ID", "identityKeyValue" -> "1234567890",
"identityInformation" -> Map("writeFrequency" -> "NEVER", "studentStatus" -> "SEP", "annualValue" -> 0,
"schoolStructure" -> "OTHER", "studentType" -> "FTS", "occupationDescription" -> "other",
"studentAccountBalanceRange" -> "string", "isStudentLoankRequested" -> "N", "schoolName" -> "myschool",
"parentsIncome" -> 100)),
Map("identityKeyName" -> "FINAL_DECISION_KEY", "identityKeyValue" -> "0000-ABCD-4567-IJKL"))
val x = dataInputKeysListvar.flatten.toMap
val y = x("identityInformation")
if (x.contains("identityInformation")){
println("value of y is" + y)
}
}
}
As you can see from the print stmt, I can print the entire map of the inner most map, but need help in terms of iterating thru that.
If you know at compile time which fields and values you need to change.
You can hard code the logic, like this:
def fixData(data: List[Map[String, Any]]): List[Map[String, Any]] =
data.map { outerMap =>
outerMap.get("identityInformation") match {
case Some(innerMap) =>
// Put as many key pairs you want to change.
// Note: if the key does not exists it will be added!
val updatedInnerMap = innerMap.asInstanceOf[Map[String, String]] ++ Map(
"isStudentLoankRequested" -> "No"
)
outerMap + ("identityInformation" -> updatedInnerMap)
case None =>
outerMap
}
}
If the key-values to change are dynamic, and / or some inner keys may not exists, or if the level of nesting can go on.
You can do something like this:
def updateMap(map: Map[String, Any], updates: Map[String, Any]): Map[String, Any] =
map.map {
case (key, value: Map[_, _]) =>
updates.get(key) match {
case Some(innerUpdates : Map[_, _]) =>
key -> updateMap(
map = value.asInstanceOf[Map[String, Any]],
updates = innerUpdates.asInstanceOf[Map[String, Any]]
)
case Some(newValue) =>
key -> newValue
case None =>
key -> value
}
case (key, value) =>
key -> updates.getOrElse(key, default = value)
}
def fixData(data: List[Map[String, Any]], updates: Map[String, Any]): List[Map[String, Any]] =
data.map(outerMap => updateMap(outerMap, updates))
Note: The above snippets use "unsafe" techniques like asInstanceOf because we lost type safety the moment you got a Map[String, Any]. Always that I see such structure, I think of JSON. I would suggest you to use an appropriate library for managing such kind of data, like circe, instead of writing code as the above.
I am not currently able to run a Raw Command in ReactiveMongo 0.12.5 using the Play JSON Plugin. The documentation (Run a raw command) is not currently accessible but from a cached page in my browser I can see the following:
import scala.concurrent.{ ExecutionContext, Future }
import play.api.libs.json.{ JsObject, Json }
import reactivemongo.play.json._
import reactivemongo.api.commands.Command
def rawResult(db: reactivemongo.api.DefaultDB)(implicit ec: ExecutionContext): Future[JsObject] = {
val commandDoc = Json.obj(
"aggregate" -> "orders", // we aggregate on collection `orders`
"pipeline" -> List(
Json.obj("$match" -> Json.obj("status" -> "A")),
Json.obj(
"$group" -> Json.obj(
"_id" -> "$cust_id",
"total" -> Json.obj("$sum" -> "$amount"))),
Json.obj("$sort" -> Json.obj("total" -> -1))
)
)
val runner = Command.run(JSONSerializationPack) // run is since deprecated
runner.apply(db, runner.rawCommand(commandDoc)).one[JsObject] // one is since deprecated
}
However I am not looking to return a JsObject (or anything in fact) - I actually want to update all documents in another collection as this previous answer illustrates. My issue is that both methods contain deprecated functions and so I have put together a combination to (possibly) work with JSON Collections (as mentioned):
def bulkUpdateScoreBA(scoreBAs: List[ScoreBA]) = {
def singleUpdate(scoreBA: ScoreBA) = Json.obj(
("q" -> Json.obj("_id" ->
Json.obj("$oid" -> scoreBA.idAsString(scoreBA._id))
)),
("u" ->
Json.obj("$set" ->
Json.obj("scoreBA" -> scoreBA.scoreBA)
)
)
)
val commandJson = Json.obj(
"update" -> "rst",
"updates" -> Json.arr(scoreBAs.map(singleUpdate)),
"ordered" -> false,
"writeConcern" -> Json.obj("w" -> "majority", "wtimeout" -> 5000)
)
val runner = Command.CommandWithPackRunner(JSONSerializationPack)
runner.apply(db, runner.rawCommand(commandJson)) // ?? how to get a Future[Unit] here
}
However I need this to return a Future[Unit] so that I can call it from the controller but I cannot find how this is done or even if what I have done so far is the best way. Any help is appreciated!
The Scaladoc for bulk update is available (since 0.12.7), with example in tests.
I have a map like :
val programming = Map(("functional", 1) -> "scala", ("functional", 2) -> "perl", ("orientedObject", 1) -> "java", ("orientedObject", 2) -> "C++")
with the same first element of key appearing multiple times.
How to regroup all the values corresponding to the same first element of key ? Which would turn this map into :
Map("functional" -> List("scala","perl"), "orientedObject" -> List("java","C++"))
UPDATE: This answer is based upon your original question. If you need the more complex Map definition, using a tuple as the key, then the other answers will address your requirements. You may still find this approach simpler.
As has been pointed out, you can't actually have multiple keys with the same value in a map. In the REPL, you'll note that your declaration becomes:
scala> val programming = Map("functional" -> "scala", "functional" -> "perl", "orientedObject" -> "java", "orientedObject" -> "C++")
programming: scala.collection.immutable.Map[String,String] = Map(functional -> perl, orientedObject -> C++)
So you end up missing some values. If you make this a List instead, you can get what you want as follows:
scala> val programming = List("functional" -> "scala", "functional" -> "perl", "orientedObject" -> "java", "orientedObject" -> "C++")
programming: List[(String, String)] = List((functional,scala), (functional,perl), (orientedObject,java), (orientedObject,C++))
scala> programming.groupBy(_._1).map(p => p._1 -> p._2.map(_._2)).toMap
res0: scala.collection.immutable.Map[String,List[String]] = Map(functional -> List(scala, perl), orientedObject -> List(java, C++))
Based on your edit, you have a data structure that looks something like this
val programming = Map(("functional", 1) -> "scala", ("functional", 2) -> "perl",
("orientedObject", 1) -> "java", ("orientedObject", 2) -> "C++")
and you want to scrap the numerical indices and group by the string key. Fortunately, Scala provides a built-in that gets you close.
programming groupBy { case ((k, _), _) => k }
This will return a new map which contains submaps of the original, grouped by the key that we return from the "partial" function. But we want a map of lists, so let's ignore the keys in the submaps.
programming groupBy { case ((k, _), _) => k } mapValues { _.values }
This gets us a map of... some kind of Iterable. But we really want lists, so let's take the final step and convert to a list.
programming groupBy { case ((k, _), _) => k } mapValues { _.values.toList }
You should try the .groupBy method
programming.groupBy(_._1._1)
and you will get
scala> programming.groupBy(_._1._1)
res1: scala.collection.immutable.Map[String,scala.collection.immutable.Map[(String, Int),String]] = Map(functional -> Map((functional,1) -> scala, (functional,2) -> perl), orientedObject -> Map((orientedObject,1) -> java, (orientedObject,2) -> C++))
you can now "clean" by doing something like:
scala> res1.mapValues(m => m.values.toList)
res3: scala.collection.immutable.Map[String,List[String]] = Map(functional -> List(scala, perl), orientedObject -> List(java, C++))
Read the csv file and create a map that contains key and list of values.
val fileStream = getClass.getResourceAsStream("/keyvaluepair.csv")
val lines = Source.fromInputStream(fileStream).getLines
var mp = Seq[List[(String, String)]]();
var codeMap=List[(String, String)]();
var res = Map[String,List[String]]();
for(line <- lines )
{
val cols=line.split(",").map(_.trim())
codeMap ++= Map(cols(0)->cols(1))
}
res = codeMap.groupBy(_._1).map(p => p._1 -> p._2.map(_._2)).toMap
Since no one has put in the specific ordering he asked for:
programming.groupBy(_._1._1)
.mapValues(_.toSeq.map { case ((t, i), l) => (i, l) }.sortBy(_._1).map(_._2))
I have a project set up with playframework 2.2.0 and play2-reactivemongo 0.10.0-SNAPSHOT. I'd like to query for few documents by their ids, in a fashion similar to this:
def usersCollection = db.collection[JSONCollection]("users")
val ids: List[String] = /* fetched from somewhere else */
val query = ??
val users = usersCollection.find(query).cursor[User].collect[List]()
As a query I tried:
Json.obj("_id" -> Json.obj("$in" -> ids)) // 1
Json.obj("_id.$oid" -> Json.obj("$in" -> ids)) // 2
Json.obj("_id" -> Json.obj("$oid" -> Json.obj("$in" -> ids))) // 3
for which first and second return empty lists and the third fails with error assertion 10068 invalid operator: $oid.
NOTE: copy of my response on the ReactiveMongo mailing list.
First, sorry for the delay of my answer, I may have missed your question.
Play-ReactiveMongo cannot guess on its own that the values of a Json array are ObjectIds. That's why you have to make a Json object for each id that looks like this: {"$oid": "526fda0f9205b10c00c82e34"}. When the ReactiveMongo Play plugin sees an object which first field is $oid, it treats it as an ObjectId so that the driver can send the right type for this value (BSONObjectID in this case.)
This is a more general problem actually: the JSON format does not match exactly the BSON one. That's the case for numeric types (BSONInteger, BSONLong, BSONDouble), BSONRegex, BSONDateTime, and BSONObjectID. You may find more detailed information in the MongoDB documentation: http://docs.mongodb.org/manual/reference/mongodb-extended-json/ .
I managed to solve it with:
val objectIds = ids.map(id => Json.obj("$oid" -> id))
val query = Json.obj("_id" -> Json.obj("$in" -> objectIds))
usersCollection.find(query).cursor[User].collect[List]()
since play-reactivemongo format considers BSONObjectID only when "$oid" is followed by string
implicit object BSONObjectIDFormat extends PartialFormat[BSONObjectID] {
def partialReads: PartialFunction[JsValue, JsResult[BSONObjectID]] = {
case JsObject(("$oid", JsString(v)) +: Nil) => JsSuccess(BSONObjectID(v))
}
val partialWrites: PartialFunction[BSONValue, JsValue] = {
case oid: BSONObjectID => Json.obj("$oid" -> oid.stringify)
}
}
Still, I hope there is a cleaner solution. If not, I guess it makes it a nice pull request.
I'm wondering if transforming id to BSONObjectID isn't more secure this way :
val ids: List[String] = ???
val bsonObjectIds = ids.map(BSONObjectID.parse(_)).collect{case Success(t) => t}
this will only generate valid BSONObjectIDs (and discard invalid ones)
If you do it this way :
val objectIds = ids.map(id => Json.obj("$oid" -> id))
your objectIds may not be valid ones depending on string id really being the stringify version of a BSONObjectID or not
If you import play.modules.reactivemongo.json._ it work without any $oid formatters.
import play.modules.reactivemongo.json._
...
val ids: Seq[BSONObjectID] = ???
val selector = Json.obj("_id" -> Json.obj("$in" -> ids))
usersCollection.find(selector).cursor[User].collect[Seq]()
I tried with the following and it worked for me:
val listOfItems = BSONArray(51, 61)
val query = BSONDocument("_id" -> BSONDocument("$in" -> listOfItems))
val ruleListFuture = bsonFutureColl.flatMap(_.find(query, Option.empty[BSONDocument]).cursor[ResponseAccDataBean]().
collect[List](-1, Cursor.FailOnError[List[ResponseAccDataBean]]()))
In the play! framework, using scala,say that i have a form such as follows:
import play.api.data._
import play.api.data.Forms._
import play.api.data.validation.Constraints._
case class User(someStringField: String, someIntField: Int)
val userForm = Form(
mapping(
"someStringField" -> text,
"someIntField" -> number verifying(x => SomeMethodThatReceivesAnIntAndReturnsABoolean(x))
)(User.apply)(User.unapply)
)
where SomeMethodThatReceivesAnIntAndReturnsABoolean is a method that performs some logic on the int to validate it.
However, i would like to be able to consider the value of the someStringField when validating the someIntField, is there a way to achieve this in play framework's forms? I know that i can do something like:
val userForm = Form(
mapping(
"someStringField" -> text,
"someIntField" -> number
)(User.apply)(User.unapply)
.verifying(x => SomeFunctionThatReceivesAnUserAndReturnsABoolean(x))
and then i would have the entire user instance available passed to the validation function. The problem with that approach is that the resulting error would be associated with the entire form instead of being associated with the someIntField field.
Is there a way to get both things, validate a field using another field and maintain the error associated to the specific field i wish to validate, instead of the entire form?
I have the same requirements with adding validation to fields depending on the value of other fields. I´m not sure how this is done in idiomatic PLAY 2.2.1 but I came up with the following solution. In this usage I´m degrading the builtin "mapping" into a simple type converter and apply my "advanced inter field" validation in the "validateForm" method. The mapping:
val userForm = Form(
mapping(
"id" -> optional(longNumber),
"surename" -> text,
"forename" -> text,
"username" -> text,
"age" -> number
)(User.apply)(User.unapply)
)
private def validateForm(form:Form[User]) = {
if(form("username").value.get == "tom" || form("age").value.get == "38") {
form
.withError("forename", "tom - forename error")
.withError("surename", "tom - surename error")
}
else
form
}
def update = Action { implicit request =>
userForm.bindFromRequest.fold({
formWithErrors => BadRequest(users.edit(validateForm(formWithErrors)))
}, { user =>
val theForm = validateForm(userForm.fill(user))
if(theForm.hasErrors) {
BadRequest(users.edit(theForm))
} else {
Users.update(user)
Redirect(routes.UsersController.index).flashing("notice" -> s"${user.forename} updated!")
}
})
}
Even though it works I´m urgently searching for a more idiomatic version...
EDIT: Use a custom play.api.data.format.Formatter in idiomatic play, more on http://workwithplay.com/blog/2013/07/10/advanced-forms-techniques/ - this lets you programmatically add errors to a form. My Formatter looks like this:
val usernameFormatter = new Formatter[String] {
override def bind(key: String, data: Map[String, String]): Either[Seq[FormError], String] = {
// "data" lets you access all form data values
val age = data.get("age").get
val username = data.get("username").get
if(age == "66") {
Left(List(FormError("username", "invalid"), FormError("forename", "invalid")))
} else {
Right(username)
}
}
override def unbind(key: String, value: String): Map[String, String] = {
Map(key -> value)
}
}
}
Registered in the form mapping like this:
mapping(
[...]
"username" -> of(usernameFormatter),
[....]
I believe what you're looking for is play.api.data.validation.Constraint.
Say you have a RegisterForm with a list of predefined cities and an otherCity field and you need either the cities or otherCity to be supplied, i.e., otherCity should be validated if cities is not provided:
case class RegisterForm(
email: String,
password: String,
cities: Option[List[String]],
otherCity: Option[String]
)
You can write a custom Constraint around this:
val citiesCheckConstraint: Constraint[RegisterForm] = Constraint("constraints.citiescheck")({
registerForm =>
// you have access to all the fields in the form here and can
// write complex logic here
if (registerForm.cities.isDefined || registerForm.otherCity.isDefined) {
Valid
} else {
Invalid(Seq(ValidationError("City must be selected")))
}
})
And your form definition becomes:
val registerForm = Form(
mapping(
"email" -> nonEmptyText.verifying(emailCheckConstraint),
"password" -> nonEmptyText.verifying(passwordCheckConstraint),
"cities" -> optional(list(text)),
"other_city" -> optional(text)
)(RegisterForm.apply)(RegisterForm.unapply).verifying(citiesCheckConstraint)
)
In this example emailCheckConstraint and passwordCheckConstraint are additional custom constraints that I defined similar to citiesCheckConstraint. This works in Play 2.2.x.
UPDATE:
Works on Play 2.3.8 as well.
if you don't mind having a prefix for you params you can group the related params:
val aForm = Form(
mapping(
"prefix" -> tuple(
"someStringField" -> text,
"someIntField" -> number
) verifying (tup => your verification)
)(tup => User.apply(tup._1, tup._2)(User.unapply...)
I use something similar just without the surrounding mapping.
You will have to adjust the apply/unapply a little and pass the arguments manually for it to compile.
The error will be registered to the "prefix" group.
I also find it weird that you cannot register errors on any field you'd like using FormError when verifying the form...
Thanks to Tom Myer, Here what I used
class MatchConstraint[A](val targetField:String, val map:(String, Map[String, String]) => A, val unmap:A => String) extends Formatter[A] {
override def bind(key: String, data: Map[String, String]): Either[Seq[FormError], A] = {
val first = data.getOrElse(key, "")
val second = data.getOrElse(targetField, "")
if (first == "" || !first.equals(second)) {
Left(List(FormError(key, "Not Match!")))
}
else {
Right(map(key, data))
}
}
override def unbind(key: String, value: A): Map[String, String] = Map(key -> unmap(value))
}
And here what's my form look like
val registerForm = Form(
mapping(
"email" -> email.verifying(minLength(6)),
"password" -> text(minLength = 6),
"passwordConfirmation" -> of(new MatchConstraint[String]("password", (key, data) => data.getOrElse(key, ""), str => str))
)(RegisterData.apply)(RegisterData.unapply)
)
I guess that they map the scala-code to JSR-Validation. There it's definitely not possible. There are some arguments to do this. Mainly that a validation should be simple and not make complex logic. How ever I still miss this too. OVal from play1 was better for me.
In the documentation:
Playframework Documentation
You can see the following code:
val userFormConstraintsAdHoc = Form(
mapping(
"name" -> text,
"age" -> number
)(UserData.apply)(UserData.unapply) verifying("Failed form constraints!", fields => fields match {
case userData => validate(userData.name, userData.age).isDefined
})
)
Mainly just verify after the unapply and you have all the fields mapped, so you can make a more complete validation.