The documentation describes using the magnet pattern to get implicit conversion to BSON types. See on this page http://mongodb.github.io/mongo-java-driver/4.1/driver-scala/bson/scala-documents/. I have tried defining an implicit object that extends BsonTransformer, but it fails to find a codec for the type. Am I missing something / has someone got this working? Sample code below, assume the insert method is being called.
case class CustomType(specialString: String)
implicit object TransformCustomType extends BsonTransformer[CustomType] {
def apply(value: CustomType): BsonString =
BsonString(value.specialString)
}
lazy val db: MongoDatabase = client.getDatabase(dbName).withCodecRegistry(DEFAULT_CODEC_REGISTRY)
lazy val testCollection: MongoCollection[CustomType] = db.getCollection[CustomType](collectionName)
def insert: Future[Completed] = testCollection.insertOne(CustomType("a")).toFuture
Error -
org.bson.codecs.configuration.CodecConfigurationException: Can't find a codec for class com.bla.BlaClass$CustomType.
*note that I am aware this can be done with
val codecRegistry = fromRegistries(fromProviders(classOf[CustomType]))
but I am just using this example to ask to learn the magnet pattern for a messier case.
If there is an implicit BsonTransformer[T] (instance of type class BsonTransformer) in a scope then there are implicit conversions
T => CanBeBsonValue (CanBeBsonValue is a wrapper over BsonValue),
(String, T) => CanBeBsonElement (CanBeBsonElement is a wrapper over BsonElement, which is a "tuple" of String and BsonValue),
Iterable[(String, T)] => CanBeBsonElements (CanBeBsonElements is a wrapper over Iterable[BsonElement])
defined in BsonMagnets (CanBeBsonValue, CanBeBsonElement, CanBeBsonElements are magnets 1 2).
Then Document can be created via factory methods
def apply(elems: CanBeBsonElement*): Document
def apply(elem: CanBeBsonElements): Document
So try
val doc = Document("a" -> CustomType("bb"))
val testCollection: MongoCollection[Document] = ???
testCollection.insertOne(doc)
Related
I have a JSON string and I created a function which parses this JSON as an object using Scala case class. I wrote the below code to parse it in a generic way. However, It gave me an error:
def getJsonObj[T](jsonString:String): T = {
implicit val formats: DefaultFormats.type = DefaultFormats
parse(jsonString).extract[T]
}
Error can be found below:
Error:(19, 32) No Manifest available for T.
parse(jsonString).extract[T] Error:(19, 32) not enough arguments for
method extract: (implicit formats: org.json4s.Formats, implicit mf:
scala.reflect.Manifest[T])T. Unspecified value parameter mf.
parse(jsonString).extract[T]
I found this No Manifest available for Type But I don't know how to fix it in my code. Also, I found this,Spark Scala - How to construct Scala Map from nested JSON (Scala: "No manifest available for type T") But I need to pass the case class to the function in a generic way. It seems a common problem but I can't solve it using the available answers as I am new in Scala.
Another point, How can I add try-catch to see if it's correctly parsed or not?
I think this answer solves your question, Scala: “No manifest available for type T”. It easily solved by implicitly passing on the manifest for the type method. I add an example of the code and a simple function for error handling.
val jsonStr: String = """{"airports":[{"name":"sfo","score":1},{"name":"phx","score":1},{"name":"sjc","score":1}]}"""
case class AirPortScores(name: String, score: Double)
case class JsonRulesHandler(airports: List[AirPortScores])
val json: JsonRulesHandler = getJsonObj[JsonRulesHandler](jsonStr)
println(json)
def getJsonObj[T](jsonString:String)(implicit m: Manifest[T]): T = {
extractFrom(jsonString) match {
case Success(jsonParsed) ⇒
jsonParsed
case Failure(exc) ⇒
throw new IllegalArgumentException(exc)
}
}
private def extractFrom[T](jsonString:String)(implicit m: Manifest[T]): Try[T] = {
implicit val formats: DefaultFormats.type = DefaultFormats
Try {
parse(jsonString).extract[T]
}
}
I am working in Spark 1.6.3. Here are two functions that do the same thing:
def modelFromBytesCV(modelArray: Array[Byte]): CountVectorizerModel = {
val tempPath: Path = KAZOO_TEMP_DIR.resolve(s"model_${System.currentTimeMillis()}")
Files.write(tempPath, modelArray)
CountVectorizerModel.read.load(tempPath.toString)
}
def modelFromBytesIDF(modelArray: Array[Byte]): IDFModel = {
val tempPath: Path = KAZOO_TEMP_DIR.resolve(s"model_${System.currentTimeMillis()}")
Files.write(tempPath, modelArray)
IDFModel.read.load(tempPath.toString)
}
I would like to make these functions generic. What I am hung up on is that the common trait between the CountVectorizerModel object and IDFModel is MLReadable[T] which itself must take as a type either CountVectorizerModel or IDFModel. This is sort of a recursive parent class loop that I can't figure out a solution to.
By comparison, the generic model writer is easy, because MLWritable is a common trait extended by all the models I am interested in:
def modelToBytes[M <: MLWritable](model: M): Array[Byte] = {
val tempPath: Path = KAZOO_TEMP_DIR.resolve(s"model_${System.currentTimeMillis()}")
model.write.overwrite().save(tempPath.toString)
Files.readAllBytes(tempPath)
}
How can I make a generic reader that will turn turn a spark-ml model into a byte array?
To make it work you'll need access to a specific MlReadable object.
import org.apache.spark.ml.util.MLReadable
def modelFromBytes[M](obj: MLReadable[M], modelArray: Array[Byte]): M = {
val tempPath: Path = ???
...
obj.read.load(tempPath.toString)
}
which could be later used as:
val bytes: Array[Byte] = ???
modelFromBytes(CountVectorizerModel, bytes)
Note that, despite the first appearance, there is nothing recursive here - MLReadable[M] refers to companion object, not class as such. So for example CountVectorizerModel object is MLReadable, while CountVectorizeModel class isn't.
Internally, Spark MLReader handles this in a different way - it creates an instance of the class using reflection, and then sets its Params. However this path won't be very useful for you here*.
If compatibility with the current API is required, you can try making readable object implicit:
def modelFromBytes[M](modelArray: Array[Byte])(implicit obj: MLReadable[M]): M = {
...
}
and then
implicit val readable: MLReadable[CountVectorizerModel] = CountVectorizerModel
modelFromBytes[CountVectorizerModel](bytes)
* Technically speaking it is possible to get companion object via reflection
def modelFromBytesCV[M <: MLWritable](
modelArray: Array[Byte])(implicit ct: ClassTag[M]): M = {
val tempPath: Path = ???
...
val cls = Class.forName(ct.runtimeClass.getName + "$");
cls.getField("MODULE$").get(cls).asInstanceOf[MLReadable[M]]
.read.load(tempPath.toString))
}
but I don't think that is a path worth exploring here. In particular we cannot really provide strict type bounds here - using MLWritable is a hack to limit human errors, but is rather useless for compiler.
I'm new with Salat,Casbah and MongoDB. When I've been trying to make a simple method to get all users from db,
import DAL.Instances.User.{UserDAO, User}
import com.novus.salat._
import com.novus.salat.global._
import com.novus.salat.annotations._
import com.novus.salat.dao._
import com.mongodb.casbah.Imports._
import com.mongodb.casbah.MongoConnection
object UserRepository {
def getAllUsers() = {
val userList= UserDAO.find()
userList.isEmpty match {
case true => throw new Exception("None users in your db")
case false => userList
}
}
I faced with two errors:
Error:(29, 31) No implicit view available from Unit => com
.mongodb.DBObject.
val userList= UserDAO.find()
^
Error:(29, 31) not enough arguments for method find: (implicit evidence$2: Unit => com.mongodb.DBObject)com.novus.salat.dao.SalatMongoCursor[DAL.Instances.User.User].
Unspecified value parameter evidence$2.
val userList= UserDAO.find()
^
Here is my User code:
object User {
case class User( _id: ObjectId = new ObjectId, name:String, age:Int)
object UserDAO extends SalatDAO[User, ObjectId](collection = MongoConnection()("fulltestdb")("user"))
}
I'm not sure what version of Salat you are using but if you look at the signature for find it'll give you a clue as to what the issue is:
def find[A <% DBObject](ref: A): SalatMongoCursor[ObjectType]
You need to call find with a parameter that has a view bound so that this parameter may be viewed as a DBObject. This means that an implicit conversion from A => DBObject is expected to be in scope.
In your case you aren't passing any parameter. This is being treated as Unit and so the compiler tries to find an implicit conversion from Unit => DBObject. This can't be found so compilation fails.
To fix this you're best bet is to pass in an empty DBObject, you can achieve this with MongoDBObject.empty from casbah. You could add an implicit conversion from Unit => MongoDBObject but I'd probably lean towards making it explicit where possible.
I'm trying to create a generic HTTP client in Scala using spray. Here is the class definition:
object HttpClient extends HttpClient
class HttpClient {
implicit val system = ActorSystem("api-spray-client")
import system.dispatcher
val log = Logging(system, getClass)
def httpSaveGeneric[T1:Marshaller,T2:Unmarshaller](uri: String, model: T1, username: String, password: String): Future[T2] = {
val pipeline: HttpRequest => Future[T2] = logRequest(log) ~> sendReceive ~> logResponse(log) ~> unmarshal[T2]
pipeline(Post(uri, model))
}
val genericResult = httpSaveGeneric[Space,Either[Failure,Success]](
"http://", Space("123", IdName("456", "parent"), "my name", "short_name", Updated("", 0)), "user", "password")
}
An object utils.AllJsonFormats has the following declaration. It contains all the model formats. The same class is used on the "other end" i.e. I also wrote the API and used the same formatters there with spray-can and spray-json.
object AllJsonFormats
extends DefaultJsonProtocol with SprayJsonSupport with MetaMarshallers with MetaToResponseMarshallers with NullOptions {
Of course that object has definitions for the serialization of the models.api.Space, models.api.Failure and models.api.Success.
The Space type seems fine, i.e. when I tell the generic method that it will be receiving and returning a Space, no errors. But once I bring an Either into the method call, I get the following compiler error:
could not find implicit value for evidence parameter of type
spray.httpx.unmarshalling.Unmarshaller[Either[models.api.Failure,models.api.Success]].
My expectation was that the either implicit in spray.json.DefaultJsonProtocol, i.e. in spray.json.StandardFormts, would have me covered.
The following is my HttpClient class trying it's best to be generic:
Update: Clearer/Repeatable Code Sample
object TestHttpFormats
extends DefaultJsonProtocol {
// space formats
implicit val idNameFormat = jsonFormat2(IdName)
implicit val updatedByFormat = jsonFormat2(Updated)
implicit val spaceFormat = jsonFormat17(Space)
// either formats
implicit val successFormat = jsonFormat1(Success)
implicit val failureFormat = jsonFormat2(Failure)
}
object TestHttpClient
extends SprayJsonSupport {
import TestHttpFormats._
import DefaultJsonProtocol.{eitherFormat => _, _ }
val genericResult = HttpClient.httpSaveGeneric[Space,Either[Failure,Success]](
"https://api.com/space", Space("123", IdName("456", "parent"), "my name", "short_name", Updated("", 0)), "user", "password")
}
With the above, the problem still occurs where the unmarshaller is unresolved. Help would be greatly appreciated..
Thanks.
Spray defines a default marshaller for Either[A,B] if a Marshaller[A] and Marshaller[B] are in defined scope inside the MetaMarshallers trait. But, going the other direction requires an Unmarshaller. You will need to define an in-scope Unmarshaller for Either[Failure, Success]. This cannot be coded without specific knowledge of the expected response and what the strategy will be for choosing whether to unmarshall a response as a Left or as a Right. For example, let's say you want to return a Failure on a non-200 response and a Success from a 200 json response body:
type ResultT = Either[Failure,Success]
implicit val resultUnmarshaller: FromResponseUnmarshaller[ResultT] =
new FromResponseUnmarshaller[ResultT] {
def apply(response: HttpResponse): Deserialized[ResultT] = response.status match {
case StatusCodes.Success(200) =>
Unmarshaller.unmarshal[Success](response.entity).right.map(Right(_))
case _ => Right(Left(Failure(...)))
}
}
Update
Looking deeper into this, the problem appears to be that the default eitherFormat in spray.json.StandardFormats is not a RootJsonFormat, which is required by the default JSON unmarshaller defined in spray.httpx.SprayJsonSupport. Defining the following implicit method should solve the issue:
implicit def rootEitherFormat[A : RootJsonFormat, B : RootJsonFormat] = new RootJsonFormat[Either[A, B]] {
val format = DefaultJsonProtocol.eitherFormat[A, B]
def write(either: Either[A, B]) = format.write(either)
def read(value: JsValue) = format.read(value)
}
I have an working example gist that hopefully explains how you would use this. https://gist.github.com/mikemckibben/fad4328de85a79a06bf3
I'm trying to create a data structure that has a PriorityQueue in it. I've succeeded in making a non-generic version of it. I can tell it works because it solves the A.I. problem I have.
Here is a snippet of it:
class ProntoPriorityQueue { //TODO make generic
implicit def orderedNode(node: Node): Ordered[Node] = new Ordered[Node] {
def compare(other: Node) = node.compare(other)
}
val hashSet = new HashSet[Node]
val priorityQueue = new PriorityQueue[Node]()
...
I'm trying to make it generic, but if I use this version it stops solving the problem:
class PQ[T <% Ordered[T]] {
//[T]()(implicit val ord: T => Ordered[T]) {
//[T]()(implicit val ord: Ordering[T] {
val hashSet = new HashSet[T]
val priorityQueue = new PriorityQueue[T]
...
I've also tried what's commented out instead of using [T <% Ordered[T]]
Here is the code that calls PQ:
//the following def is commented out while using ProntoPriorityQueue
implicit def orderedNode(node: Node): Ordered[Node] = new Ordered[Node] {
def compare(other: Node) = node.compare(other)
} //I've also tried making this return an Ordering[Node]
val frontier = new PQ[Node] //new ProntoPriorityQueue
//have also tried (not together):
val frontier = new PQ[Node]()(orderedNode)
I've also tried moving the implicit def into the Node object (and importing it), but essentially the same problem.
What am I doing wrong in the generic version? Where should I put the implicit?
Solution
The problem was not with my implicit definition. The problem was the implicit ordering was being picked up by a Set that was automatically generating in a for(...) yield(...) statement. This caused a problem where the yielded set only contained one state.
What's wrong with simply defining an Ordering on your Node (Ordering[Node]) and using the already-generic Scala PriorityQueue?
As general rule, it's better to work with Ordering[T] than T <: Ordered[T] or T <% Ordered[T]. Conceptually, Ordered[T] is an intrinsic (inherited or implemented) property of the type itself. Notably, a type can have only one intrinsic ordering relationship defined this way. Ordering[T] is an external specification of the ordering relationship. There can any be any number of different Ordering[T].
Also, if you're not already aware, you should know that the difference between T <: U and T <% U is that while the former includes only nominal subtype relations (actual inheritance), the latter also includes the application of implicit conversions that yield a value conforming to the type bound.
So if you want to use Node <% Ordered[Node] and you don't have a compare method defined in the class, an implicit conversion will be applied every time a comparison needs to be made. Additionally, if your type has its own compare, the implicit conversion will never be applied and you'll be stuck with that "built-in" ordering.
Addendum
I'll give a few examples based on a class, call it CIString that simply encapsulates a String and implements ordering as case-invariant.
/* Here's how it would be with direct implementation of `Ordered` */
class CIString1(val s: String)
extends Ordered[CIString1]
{
private val lowerS = s.toLowerCase
def compare(other: CIString1) = lowerS.compareTo(other.lowerS)
}
/* An uninteresting, empty ordered set of CIString1
(fails without the `extends` clause) */
val os1 = TreeSet[CIString1]()
/* Here's how it would look with ordering external to `CIString2`
using an implicit conversion to `Ordered` */
class CIString2(val s: String) {
val lowerS = s.toLowerCase
}
class CIString2O(ciS: CIString2)
extends Ordered[CIString2]
{
def compare(other: CIString2) = ciS.lowerS.compareTo(other.lowerS)
}
implicit def cis2ciso(ciS: CIString2) = new CIString2O(ciS)
/* An uninteresting, empty ordered set of CIString2
(fails without the implicit conversion) */
val os2 = TreeSet[CIString2]()
/* Here's how it would look with ordering external to `CIString3`
using an `Ordering` */
class CIString3(val s: String) {
val lowerS = s.toLowerCase
}
/* The implicit object could be replaced by
a class and an implicit val of that class */
implicit
object CIString3Ordering
extends Ordering[CIString3]
{
def compare(a: CIString3, b: CIString3): Int = a.lowerS.compareTo(b.lowerS)
}
/* An uninteresting, empty ordered set of CIString3
(fails without the implicit object) */
val os3 = TreeSet[CIString3]()
Well, one possible problem is that your Ordered[Node] is not a Node:
implicit def orderedNode(node: Node): Ordered[Node] = new Ordered[Node] {
def compare(other: Node) = node.compare(other)
}
I'd try with an Ordering[Node] instead, which you say you tried but there isn't much more information about. PQ would be declared as PQ[T : Ordering].