I want to parse province to case class, it throws mismatch
scala.MatchError: Some(USA) (of class scala.Some)
val result = EntityUtils.toString(entity,"UTF-8")
val address = JsonParser.parse(result).extract[Address]
val value.province = Option(address.province)
val value.city = Option(address.city)
case class Access(
device: String,
deviceType: String,
os: String,
event: String,
net: String,
channel: String,
uid: String,
nu: Int,
ip: String,
time: Long,
version: String,
province: Option[String],
city: Option[String],
product: Option[Product]
)
This:
val value.province = Option(address.province)
val value.city = Option(address.city)
doesn't do what you think it does. It tries to treat value.province and value.city as extractors (which don't match the type, thus scala.MatchError exception). It doesn't mutate value as I believe you intended (because value apparently doesn't have such setters).
Since value is (apparently) Access case class, it is immutable and you can only obtain an updated copy:
val value2 = value.copy(
province = Option(address.province),
city = Option(address.city)
)
Assuming the starting point:
val province: Option[String] = ???
You can get the string with simple pattern matching:
province match {
case Some(stringValue) => JsonParser.parse(stringValue).extract[Province] //use parser to go from string to a case class
case None => .. //maybe provide a default value, depends on your context
}
Note: Without knowing what extract[T] returns it's hard to recommend a follow-up
Related
I have a dataset with following case class type:
case class AddressRawData(
addressId: String,
customerId: String,
address: String
)
I want to convert it to:
case class AddressData(
addressId: String,
customerId: String,
address: String,
number: Option[Int], //i.e. it is optional
road: Option[String],
city: Option[String],
country: Option[String]
)
Using a parser function:
def addressParser(unparsedAddress: Seq[AddressData]): Seq[AddressData] = {
unparsedAddress.map(address => {
val split = address.address.split(", ")
address.copy(
number = Some(split(0).toInt),
road = Some(split(1)),
city = Some(split(2)),
country = Some(split(3))
)
}
)
}
I am new to scala and spark. Could anyone please let me know how can this be done?
You were on the right path! There are multiple ways of doing this of course. But as you're already on the way by making some case classes, and you've started making a parsing function an elegant solution is by using the Dataset's map function. From the docs, this map function signature is the following:
def map[U](func: (T) ⇒ U)(implicit arg0: Encoder[U]): Dataset[U]
Where T is the starting type (AddressRawData in your case) and U is the type you want to get to (AddressData in your case). So the input of this map function is a function that transforms a AddressRawData to a AddressData. That could perfectly be the addressParser you've started making!
Now, your current addressParser has the following signature:
def addressParser(unparsedAddress: Seq[AddressData]): Seq[AddressData]
In order to be able to feed it to that map function, we need to make this signature:
def newAddressParser(unparsedAddress: AddressRawData): AddressData
Knowing all of this, we can work further! An example would be the following:
import spark.implicits._
import scala.util.Try
// Your case classes
case class AddressRawData(addressId: String, customerId: String, address: String)
case class AddressData(
addressId: String,
customerId: String,
address: String,
number: Option[Int],
road: Option[String],
city: Option[String],
country: Option[String]
)
// Your addressParser function, adapted to be able to feed into the Dataset.map
// function
def addressParser(rawAddress: AddressRawData): AddressData = {
val addressArray = rawAddress.address.split(", ")
AddressData(
rawAddress.addressId,
rawAddress.customerId,
rawAddress.address,
Try(addressArray(0).toInt).toOption,
Try(addressArray(1)).toOption,
Try(addressArray(2)).toOption,
Try(addressArray(3)).toOption
)
}
// Creating a sample dataset
val rawDS = Seq(
AddressRawData("1", "1", "20, my super road, beautifulCity, someCountry"),
AddressRawData("1", "1", "badFormat, some road, cityButNoCountry")
).toDS
val parsedDS = rawDS.map(addressParser)
parsedDS.show
+---------+----------+--------------------+------+-------------+----------------+-----------+
|addressId|customerId| address|number| road| city| country|
+---------+----------+--------------------+------+-------------+----------------+-----------+
| 1| 1|20, my super road...| 20|my super road| beautifulCity|someCountry|
| 1| 1|badFormat, some r...| null| some road|cityButNoCountry| null|
+---------+----------+--------------------+------+-------------+----------------+-----------+
As you see, thanks to the fact that you had already foreseen that parsing can go wrong, it was easily possible to use scala.util.Try to try and get the pieces of that raw address and add some robustness in there (the second line contains some null values where it could not parse the address string.
Hope this helps!
I have a method, which need String type as an argument:
type Identity = String
case class RequireSmth(param: Identity) extends Something
Now I call this method in more complex order:
createMe(as[List])(arg =>{ parse(RequireSmth(getAction(name, surname).map(bool => getData(surname, bool).id))) })
Parse looks like:
def parse(ob: Something)
Where:
def getAction(name: String, surname: String): Future[Boolean] = {
someObject.get(name).map(_.getSomething(surname).isPossibleToTake.getOrElse(false)) //someObject is defined in constructor and does not matter here
}
def getData: (String, Boolean) => MyObject = {
case ("Doe", true) => possible
case _ => notPossible
}
MyObject, possible and notPossible definition:
case class MyObject(id : String, name: String, surname: String)
val possible = MyObject( id = "ok", name ="John", surname = "Doe")
val notPossible = MyObject( id = "not ok", name ="John", surname = "Doe")
The problem is, when I call RequireSmth method I got an error:
type mismatch;
found: scala.concurrent.Future[String]
required: com.my.smth.Identity (which expands to) String
How can I solve this problem to return Identity (or String) instead of Future[String]?
Keep the information inside the Future like this:
getAction(name, surname).map(bool => getData(surname, bool).id).map(RequireSmth)
Just keep chaining the operations together, keeping everything inside the Future:
getAction(name, surname)
.map(bool => getData(surname, bool).id)
.map(RequireSmth) // Or x => RequireSmth(x) if necessary
.map(parse)
At some point you will get to a method that has a side-effect and returns Unit, and that will be executed when all the actions in the Future are complete.
In the unlikely event that you actually need to get the value out of the Future, use Await.result. But in most cases this will not be necessary.
You need to flip the method calls:
val res: Future[???] =
getAction(name, surname)
.map(bool => getData(surname, bool).id)
.map(RequireSmth)
.map(parse)
Note that Future[String] is not a String, it's a computation that will yield a value in the future, and that means that the entire computation stack needs to return a Future[T] as well (unless you explicitly await, which blocks and is not recommended).
I'm having trouble with checking for equality on a case class that I've mapped to an apache tinkerpop graph, but I want to be able to check for equality after I query the graph.
#label("apartment")
case class Apartment(#id id: Option[Int], address: String, zip: Zip, rooms: Rooms, size: Size, price: Price, link: String, active: Boolean = true, created: Date = new Date()) {}
val ApartmentAddress = Key[String]("address")
Await.result(result, Duration.Inf).foreach(apt => {
val dbResult = graph.V.hasLabel[Apartment].has(ApartmentAddress, apt.address).head().toCC[Apartment]
println(dbResult == apt) //always false :(
})
My problem is that when I've created the object it has no id, and the time stamp on it is obviously different. I read that if you add a second parameter list, it is excluded from equals, so I changed it:
#label("apartment")
case class Apartment(address: String, zip: Zip, rooms: Rooms, size: Size, price: Price, link: String, active: Boolean = true)(#id implicit val id: Option[Int] = None, implicit val created: Date = new Date()) {}
val ApartmentAddress = Key[String]("address")
Await.result(result, Duration.Inf).foreach(apt => {
val dbResult = graph.V.hasLabel[Apartment].has(ApartmentAddress, apt.address).head().toCC[Apartment]
println(dbResult == apt) //true! yay! :D
})
I can now check for equality using ==, but the value from the database loses its id, and the "created" value gets reset. And, one other frustrating thing, they always need to be created with extra parenthesis at the end:
Apartment(address, zip, rooms, size, price, link)()
Is there a way to achieve this functionality without overloading equals? Or make the value from the database maintain the original values using this approach?
It seems in your case, you just need it only for one time comparison, so I would not play with equals and just modified value on comparison
case class Apartment(
#id id: Option[Int] = None,
address: String,
zip: Zip,
rooms: Rooms,
size: Size,
price: Price,
link: String,
active: Boolean = true,
created: Date = new Date(0)) {
}
println(dbResult.copy(id = None, created = new Date(0)) == apt) //true! yay! :D
or add extra function to the class
case class Apartment(
#id id: Option[Int] = None,
address: String,
zip: Zip,
rooms: Rooms,
size: Size,
price: Price,
link: String,
active: Boolean = true,
created: Date = new Date(0)) {
def equalsIgnoreIdAndCreated(other: Apartment) = {
this.equals(other.copy(id = id, created = created))
}
}
println(dbResult.equalsIgnoreIdAndCreated(apt))
You can look at good explanation for case classes in
http://www.alessandrolacava.com/blog/scala-case-classes-in-depth/ and reasons why you should not override equals from automatically generated, otherwise just override equals.
So, the problem is in the title, but here are the details.
I have two case classes:
case class JourneyGroup(id: Option[Int] = None,
key: UUID,
name: String,
data: Option[JsValue],
accountId: Int,
createdAt: DateTime = DateTime.now,
createdById: Int,
updatedAt: Option[DateTime] = None,
updatedById: Option[Int] = None,
deletedAt: Option[DateTime] = None,
deletedById: Option[Int] = None)
and
case class JourneyGroupApi(id: Option[Int] = None,
key: UUID,
name: String,
data: Option[JsValue],
accountId: Int,
createdAt: DateTime = DateTime.now,
createdById: Int,
updatedAt: Option[DateTime] = None,
updatedById: Option[Int] = None,
deletedAt: Option[DateTime] = None,
deletedById: Option[Int] = None,
parties: Seq[Party] = Seq.empty[Party])
Background: the reason for having these two separate classes is the fact that slick does not support collections, and I do need collections of related objects that I build manually. Bottom line, I could not make it work with a single class.
What I need is an easy way to convert from one to another.
At this point, to unblock myself, I created a manual conversion:
def toJourneyGroupApi(parties: Seq[Party]): JourneyGroupApi = JourneyGroupApi(
id = id,
key = key,
name = name,
data = data,
accountId = accountId,
createdAt = createdAt,
createdById = createdById,
updatedAt = updatedAt,
updatedById = updatedById,
deletedAt = deletedAt,
deletedById = deletedById,
parties = parties
)
Which is working, but extremely ugly and requires a lot of maintenance.
One thing that I tried doing is:
convert the source object to tuple
Add an element to that tuple using shapeless
and build a target object from resulting tuple
import shapeless._
import syntax.std.tuple._
val groupApi = (JourneyGroup.unapply(group).get :+ Seq.empty[Party])(JourneyGroupApi.tupled)
But, this thing is claiming, that the result of :+ is not tuple, even though in console:
Party.unapply(p).get :+ Seq.empty[Participant]
res0: (Option[Int], model.Parties.Type.Value, Int, Int, org.joda.time.DateTime, Int, Option[org.joda.time.DateTime], Option[Int], Option[org.joda.time.DateTime], Option[Int], Seq[model.Participant]) = (None,client,123,234,2016-11-12T03:55:24.006-08:00,987,None,None,None,None,List())
What am I doing wrong? Maybe there is another way of achieving this.
Could you consider Composition?
case class JourneyGroup(
...
)
case class JourneyGroupApi(
journeyGroup: JourneyGroup=JourneyGroup(),
parties: Seq[Party] = Seq()
)
Converting a journeyGroup would just be something like JourneyGroupApi(journeyGroup,parties) and "converting" a journeyGroupApi would be a matter of accessing journeyGroupApi.journeyGroup. You could perhaps come up with names that worked better for this case. Not sure if this approach would fit the rest of your code. In particular referencing journeyGroup attributes in a journeyGroupApi will be one extra level, e.g. journeyGroupApi.journeyGroup.accountId. (This could potentially be mitigated by "shortcut" definitions on journeyGroupApi like lazy val accountId = journeyGroup.accountId.)
Inheritance might also be an approach to consider with a base case class of JourneyGroup then a normal class (not case class) that extends it with parties as the extra attribute. This option is discussed further in this SO thread.
I've been working with this example from the Elastic4s manual. It is working fine until it attempts to retrieve a document that does not have a field specified in the case class.
In this example from the manual, let's say one result only had name and was missing the location field. It would yield this error:
java.util.NoSuchElementException: key not found: location
I'm looking for a good approach to deal with search results that have varying fields.
Code sample:
case class Character(name: String, location: String)
implicit object CharacterHitAs extends HitAs[Character] {
override def as(hit: RichSearchHit): Character = {
Character(hit.sourceAsMap("name").toString, hit.sourceAsMap("location").toString) }}
val resp = client.execute {
search in "gameofthrones" / "characters" query "kings landing"
}.await
val characters :Seq[Character] = resp.as[Character]
When developing a case class with optional parameters, use Option:
case class Character(name: String, location: Option[String])
Character("Tyrion Lannister", None)
Then all you have to do is modify your data extractor to pass a None Option if it doesn't find the data:
val tyrion = Map("location" -> "King's Landing", "name" -> "Cersei Lannister")
val cersei = Map("father" -> "Tywin Lannister?", "name" -> "Cersei Lannister")
val jaime = Map("father" -> "Tywin Lannister", "location" -> "Tower of the Hand")
val characters = List(tyrion, cersei, jaime)
case class Character(name: String, location: Option[String])
characters.map(x => Character(x.getOrElse("name", "A CHARACTER HAS NO NAME"), x.get("location")))
The result of characters.map(...) is this:
res0: List[Character] = List(
Character(Cersei Lannister,Some(King's Landing)),
Character(Cersei Lannister,None),
Character(A CHARACTER HAS NO NAME NAME,Some(Tower of the Hand)))
From the source code for RichSearchHit, sourceAsMap should return a Map object:
def sourceAsMap: Map[String, AnyRef] = if (java.sourceAsMap == null) Map.empty else java.sourceAsMap.asScala.toMap
Given that you're using a Map shorthand, you should be able to convert your code to:
case class Character(name: String, location: Option[String])
implicit object CharacterHitAs extends HitAs[Character] {
override def as(hit: RichSearchHit): Character = {
Character(hit.sourceAsMap.getOrElse("name", "A CHARACTER HAS NO NAME"), hit.sourceAsMap.get("location")) }}