I use kryo serializer as a serializer in project works with spark and written in scala. I register all the classes I use in my project and there is one class which is not serialize or desiralize which is linked hash map.
Linked hash map registration :
kryo.register(scala.collection.mutable.linkedHashMap[_, _])
I get my data from elastic and the runtime type of properties is linkedHashMap.
For example document look like that : person{age : 20, name : "yarden"}, map to linkedHashMap[String, Any] and if there is another object in person etc.
When I want to deserialize Data which is stored as linkedHashMap(for example collect RDD) the result is just empty object.
I try to use the MapSerializer(another parameter in register function) but it is fail because it is for java linkedHashMap.
I try to search for suitable serializer for scala linkedHashMap but I didn't found.
There is a solution of every time I get a LinkedHashMap convert it to Map and it is work but it is not a good practice.
I think about maybe there is a way to cause runtime type to be map and not linkedHashMap but I didn't find any solution.
I Believe the best practice is to found serializer which is suitable to linkedHashMap of scala but I didn't found any.
Any solution of the things I didn't succeed to solve will be welcomed.
Try using the TraversibleSerializer.
For example:
kryo.register(classOf[mutable.LinkedHashMap[Any, Any]],
new TraversableSerializer[(Any, Any), mutable.LinkedHashMap[Any, Any]](true))
Related
I have two java maps which map a String to a set of Strings.
I want to convert the java maps to Scala and "add" map1 to map2 such that if they both have the same keys the value in the resultant map will be the union of the 2 value sets. I found a nice solution for the map addition:
map1 ++ map2.map { case (k,v) => k -> (v ++ map1.getOrElse(k,v))}
The issue happens when I convert the java maps to Scala via the 'asScala' call. When we do this conversation we get:
After the conversion to Scala, I am no longer able to run the solution above on these wrapper collections. The ++ operation on the map is still defined. However the SetWrapper class does not define a ++ operation. As far as I can tell the SetWrapper does not define any operations at all! Here is the full definition from the docs:
SetWrapper seems to extend an abstract class and does not define any of the functionality.
How can I get around this issue? Is there another conversation step to a real Set object?
JMapWrapper wraps a Java Map in a Scala Map; SetWrapper wraps a Scala Set in a Java Set. So it has Java's methods for Set, not Scala's.
You don't provide types, but it appears you start with a java.util.Map[SomeKey, java.util.Set[SomeType]]. Calling asScala on it will only convert the outer Map, not its values. You can fix this by using javaMap.asScala.mapValues(_.asScala) (with the usual caveat that it will call asScala each time a value is accessed; use .map instead if this isn't acceptable).
Is there some way i can iterate over the properties of object and get the name and value both while iterating.
I know about the product iterator, but using that I only get the property value not the name of the property at the same time.
Unless you are building tooling for programmers developing scala, it is very probable that you should be using a Map instead of an object.
eg:
val myObject: Map[String, Any] =
Map("prop1" -> 1, "prop2" -> "string", "prop3" -> List(1, 2, 3))
for ((key, value) <- myObject) {
println(key, value)
}
If you are building tooling or you can't just swap out the object for a map, you can use reflection as mentioned in the other answers.
Just don't do it!
Use another design: Scala is not PHP or Python or Perl; it's a statically typed functional language with a very expressive type system; such reflection (runtime inspection) is not needed 99.9% of the time and should be avoided for reasons of safety, correctness and performance.
I think, there is no way to do it without using reflection. (See for example here: Getting public fields (and their respective values) of an Instance in Scala/Java)
An other option is using Apache FieldUtils: https://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/reflect/FieldUtils.html But again it uses reflection in background.
Scala newbie here,
I'm using stanford's topic modelling toolkit
and it has a lazy iterable of type LazyIterable[(String, Array[Double])]
How should i iterate through all the elements in this iterable say it to print all these values?
I tried doing this by
while(it.hasNext){
System.out.println(it.next())
}
Gives an error
error: value next is not a member of scalanlp.collection.LazyIterable[(String, Array[Double])]
This is the API source -> iterable_name ->
InferCVB0DocumentTopicDistributions in
http://nlp.stanford.edu/software/tmt/tmt-0.4/api/edu/stanford/nlp/tmt/stage/package.html
Based on its source code, I can see that the LazyIterable implements the standard Scala Iterable interface, which means you have access to all the standard higher-order functions that all Scala collections implement - such as map, flatMap, filter, etc.
The one you will be interested in for printing all the values is foreach. So try this (no need for the while-loop):
it.foreach(println)
Seems like method invocation problem, just check the source code of LazyIterable, look at line 46
override def iterator : Iterator[A]
when you get an instance of LazyIterable, invoke iterator method, then you can do what you want.
The traditional way to use spray-json seems to be to bind all your models to appropriate JsonFormats (built-in or custom) at compile time, with the formats all as implicits. Is there a way instead to look the formatters up at runtime? I'm trying to marshal a heterogeneous list of values, and the only ways I'm seeing to do it are
Write an explicit lookup (e.g. using pattern matching) that hard-codes which fomratter to use for which value type, or
Something insane using reflection to find all the implicits
I'm pretty new to Scala and spray-json both, so I worry I'm missing some simpler approach.
More context: I'm trying to write a custom serializer that writes out only a specified subset of (lazy) object fields. I'm looping over the list of specified fields (field names) at runtime and getting the values by reflection (actually it's more complicated than that, but close enough), and now for each one I need to find a JsonFormat that can serialize it.
In groovy one can do:
class Foo {
Integer a,b
}
Map map = [a:1,b:2]
def foo = new Foo(map) // map expanded, object created
I understand that Scala is not in any sense of the word, Groovy, but am wondering if map expansion in this context is supported
Simplistically, I tried and failed with:
case class Foo(a:Int, b:Int)
val map = Map("a"-> 1, "b"-> 2)
Foo(map: _*) // no dice, always applied to first property
A related thread that shows possible solutions to the problem.
Now, from what I've been able to dig up, as of Scala 2.9.1 at least, reflection in regard to case classes is basically a no-op. The net effect then appears to be that one is forced into some form of manual object creation, which, given the power of Scala, is somewhat ironic.
I should mention that the use case involves the servlet request parameters map. Specifically, using Lift, Play, Spray, Scalatra, etc., I would like to take the sanitized params map (filtered via routing layer) and bind it to a target case class instance without needing to manually create the object, nor specify its types. This would require "reliable" reflection and implicits like "str2Date" to handle type conversion errors.
Perhaps in 2.10 with the new reflection library, implementing the above will be cake. Only 2 months into Scala, so just scratching the surface; I do not see any straightforward way to pull this off right now (for seasoned Scala developers, maybe doable)
Well, the good news is that Scala's Product interface, implemented by all case classes, actually doesn't make this very hard to do. I'm the author of a Scala serialization library called Salat that supplies some utilities for using pickled Scala signatures to get typed field information
https://github.com/novus/salat - check out some of the utilities in the salat-util package.
Actually, I think this is something that Salat should do - what a good idea.
Re: D.C. Sobral's point about the impossibility of verifying params at compile time - point taken, but in practice this should work at runtime just like deserializing anything else with no guarantees about structure, like JSON or a Mongo DBObject. Also, Salat has utilities to leverage default args where supplied.
This is not possible, because it is impossible to verify at compile time that all parameters were passed in that map.