How to serialize object to AST using json4s? - scala

I am writing a Customer Serializer. In that Serializer I would like to somehow say: "and this thing you already know how to serialize".
My current approach looks like that:
import org.json4s.native.Serialization._
import org.json4s.JsonDSL.WithBigDecimal._
object WindowSerializer extends CustomSerializer[Window](format =>
( [omitted],
{
case Window(frame, size) =>
( "size" -> size ) ~
( "frame" -> parse(write(frame)) )
}))
That parse(write(frame)) things is both ugly and inefficient. How to fix that?

You can call Extraction.decompose(a: Any)(implicit formats: Formats): JValue which produces a JValue from some value using runtime reflection.
import org.json4s._
import org.json4s.jackson.JsonMethods._
import org.json4s.JsonDSL._
import java.util.UUID
case class Thing(name: String)
case class Box(id: String, thing: Thing)
class BoxSerializer extends CustomSerializer[Box](format => ({
case jv: JValue =>
val id = (jv \ "id").extract[String]
val thing = (jv \ "thing").extract[Thing]
Box(id, thing)
}, {
case b: Box =>
("token" -> UUID.randomUUID().toString()) ~
("id" -> box.id) ~
("thing" -> Extraction.decompose(box.thing))
}))
implicit val formats = DefaultFormats + new BoxSerializer
val box = Box("1000", Thing("a thing"))
// decompose the value to JSON
val json = Extraction.decompose(box)
println(pretty(json))
// {
// "token" : "d9bd49dc-11b4-4380-ab10-f6df005a384c",
// "id" : "1000",
// "thing" : {
// "name" : "a thing"
// }
// }
// and read a value of type Box back from the JSON
println(json.extract[Box])
// Box(1000,Thing(a thing))

Related

"graph-for-funcs.sc" script compilation error

I am trying to reproduce the devign experiment. When joern is updated, some error messages appear in the graph-for-funcs.sc script that parses the bin file into JSON. I modified some, and there are still some errors as shown in the figure below. Did you encounter similar errors and how did you solve them?
graph-for-funcs.sc
import scala.jdk.CollectionConverters._
import io.circe.syntax._
import io.circe.generic.semiauto._
import io.circe.{Encoder, Json}
import io.shiftleft.semanticcpg.language.types.expressions.generalizations.CfgNode
import io.shiftleft.codepropertygraph.generated.EdgeTypes
import io.shiftleft.codepropertygraph.generated.NodeTypes
import io.shiftleft.codepropertygraph.generated.nodes
import io.shiftleft.dataflowengineoss.language._
import io.shiftleft.semanticcpg.language._
import io.shiftleft.semanticcpg.language.types.expressions.Call
import io.shiftleft.semanticcpg.language.types.structure.Local
import io.shiftleft.codepropertygraph.generated.nodes.MethodParameterIn
import overflowdb._
import overflowdb.traversal._
final case class GraphForFuncsFunction(function: String,
file: String,
id: String,
AST: List[nodes.AstNode],
CFG: List[nodes.AstNode],
PDG: List[nodes.AstNode])
final case class GraphForFuncsResult(functions: List[GraphForFuncsFunction])
implicit val encodeEdge: Encoder[Edge] =
(edge: Edge) =>
Json.obj(
("id", Json.fromString(edge.toString)),
("in", Json.fromString(edge.inNode.toString)),
("out", Json.fromString(edge.outNode.toString))
)
implicit val encodeNode: Encoder[nodes.AstNode] =
(node: nodes.AstNode) =>
Json.obj(
("id", Json.fromString(node.toString)),
("edges",
Json.fromValues((node.inE("AST", "CFG").l ++ node.outE("AST", "CFG").l).map(_.asJson))),
("properties", Json.fromValues(node.propertyMap.asScala.toList.map { case (key, value) =>
Json.obj(
("key", Json.fromString(key)),
("value", Json.fromString(value.toString))
)
}))
)
implicit val encodeFuncFunction: Encoder[GraphForFuncsFunction] = deriveEncoder
implicit val encodeFuncResult: Encoder[GraphForFuncsResult] = deriveEncoder
#main def main(): Json = {
GraphForFuncsResult(
cpg.method.map { method =>
val methodName = method.fullName
val methodId = method.toString
val methodFile = method.location.filename
val astChildren = method.astMinusRoot.l
val cfgChildren = method.out(EdgeTypes.CONTAINS).asScala.collect { case node: nodes.CfgNode => node }.toList
val local = new NodeSteps(
method
.out(EdgeTypes.CONTAINS)
.hasLabel(NodeTypes.BLOCK)
.out(EdgeTypes.AST)
.hasLabel(NodeTypes.LOCAL)
.cast[nodes.Local])
val sink = local.evalType(".*").referencingIdentifiers.dedup
val source = new NodeSteps(method.out(EdgeTypes.CONTAINS).hasLabel(NodeTypes.CALL).cast[nodes.Call]).nameNot("<operator>.*").dedup
val pdgChildren = sink
.reachableByFlows(source)
.l
.flatMap { path =>
path.elements
.map {
case trackingPoint # (_: MethodParameterIn) => trackingPoint.start.method.head
case trackingPoint => trackingPoint.cfgNode
}
}
.filter(_.toString != methodId)
GraphForFuncsFunction(methodName, methodFile, methodId, astChildren, cfgChildren, pdgChildren.distinct)
}.l
).asJson
}
error
graph-for-funcs.sc:92: value evalType is not a member of io.shiftleft.semanticcpg.language.NodeSteps[io.shiftleft.codepropertygraph.generated.nodes.Local]
val sink = local.evalType(".*").referencingIdentifiers.dedup
^
graph-for-funcs.sc:93: value nameNot is not a member of io.shiftleft.semanticcpg.language.NodeSteps[io.shiftleft.codepropertygraph.generated.nodes.Call]
val source = new NodeSteps(method.out(EdgeTypes.CONTAINS).hasLabel(NodeTypes.CALL).cast[nodes.Call]).nameNot("<operator>.*").dedup
^
java.lang.RuntimeException: Compilation Failed
io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:50)
cats.effect.internals.IORunLoop$.liftedTree3$1(IORunLoop.scala:229)
cats.effect.internals.IORunLoop$.step(IORunLoop.scala:229)
cats.effect.IO.unsafeRunTimed(IO.scala:320)
cats.effect.IO.unsafeRunSync(IO.scala:239)
io.shiftleft.console.scripting.ScriptManager.runScript(ScriptManager.scala:130)
io.shiftleft.console.scripting.ScriptManager$CpgScriptRunner.runScript(ScriptManager.scala:64)
io.shiftleft.console.scripting.ScriptManager$CpgScriptRunner.runScript(ScriptManager.scala:54)
ammonite.$sess.cmd8$.<clinit>(cmd8.sc:1)

How do I turn a Scala case class into a mongo Document

I'd like to build a generic method for transforming Scala Case Classes to Mongo Documents.
A promising Document constructor is
fromSeq(ts: Seq[(String, BsonValue)]): Document
I can turn a case class into a Map[String -> Any], but then I've lost the type information I need to use the implicit conversions to BsonValues. Maybe TypeTags can help with this?
Here's what I've tried:
import org.mongodb.scala.bson.BsonTransformer
import org.mongodb.scala.bson.collection.immutable.Document
import org.mongodb.scala.bson.BsonValue
case class Person(age: Int, name: String)
//transform scala values into BsonValues
def transform[T](v: T)(implicit transformer: BsonTransformer[T]): BsonValue = transformer(v)
// turn any case class into a Map[String, Any]
def caseClassToMap(cc: Product) = {
val values = cc.productIterator
cc.getClass.getDeclaredFields.map( _.getName -> values.next).toMap
}
// transform a Person into a Document
def personToDocument(person: Person): Document = {
val map = caseClassToMap(person)
val bsonValues = map.toSeq.map { case (key, value) =>
(key, transform(value))
}
Document.fromSeq(bsonValues)
}
<console>:24: error: No bson implicit transformer found for type Any. Implement or import an implicit BsonTransformer for this type.
(key, transform(value))
def personToDocument(person: Person): Document = {
Document("age" -> person.age, "name" -> person.name)
}
Below code works without manual conversion of an object.
import reactivemongo.api.bson.{BSON, BSONDocument, Macros}
case class Person(name:String = "SomeName", age:Int = 20)
implicit val personHandler = Macros.handler[Person]
val bsonPerson = BSON.writeDocument[Person](Person())
println(s"${BSONDocument.pretty(bsonPerson.getOrElse(BSONDocument.empty))}")
You can use Salat https://github.com/salat/salat. A nice example can be found here - https://gist.github.com/bhameyie/8276017. This is the piece of code that will help you -
import salat._
val dBObject = grater[Artist].asDBObject(artist)
artistsCollection.save(dBObject, WriteConcern.Safe)
I was able to serialize a case class to a BsonDocument using the org.bson.BsonDocumentWriter. The below code runs using scala 2.12 and mongo-scala-driver_2.12 version 2.6.0
My quest for this solution was aided by this answer (where they are trying to serialize in the opposite direction): Serialize to object using scala mongo driver?
import org.mongodb.scala.bson.codecs.Macros
import org.mongodb.scala.bson.codecs.DEFAULT_CODEC_REGISTRY
import org.bson.codecs.configuration.CodecRegistries.{fromRegistries, fromProviders}
import org.bson.codecs.EncoderContext
import org.bson.BsonDocumentWriter
import org.mongodb.scala.bson.BsonDocument
import org.bson.codecs.configuration.CodecRegistry
import org.bson.codecs.Codec
case class Animal(name : String, species: String, genus: String, weight: Int)
object TempApp {
def main(args: Array[String]) {
val jaguar = Animal("Jenny", "Jaguar", "Panthera", 190)
val codecProvider = Macros.createCodecProvider[Animal]()
val codecRegistry: CodecRegistry = fromRegistries(fromProviders(codecProvider), DEFAULT_CODEC_REGISTRY)
val codec = Macros.createCodec[Animal](codecRegistry)
val encoderContext = EncoderContext.builder.isEncodingCollectibleDocument(true).build()
var doc = BsonDocument()
val writr = new BsonDocumentWriter(doc) // need to call new since Java lib w/o companion object
codec.encode(writr, jaguar, encoderContext)
print(doc)
}
};

How can I find a play form type (for handling post requests in controller) in order to map a class containing BSONObjectID type?

I'm working on a web application using Play Framework (2.2.6) / scala / mongoDB, and I have a problem with reactivemongo.bson.BSONObjectID. (I'm a beginner in both ReactiveMongo and Scala)
My controller contains this code :
val actForm = Form(tuple(
"name" -> optional(of[String]),
"shortcode" -> optional(of[String]),
"ccam" -> mapping(
"code" -> optional(of[String]),
"description" -> optional(of[String]),
"_id" -> optional(of[BSONObjectID])
)(CCAMAct.apply)(CCAMAct.unapply)
));
def addAct = AsyncStack(AuthorityKey -> Secretary) { implicit request =>
val user = loggedIn
actForm.bindFromRequest.fold(
errors => Future.successful(BadRequest(errors.errorsAsJson)), {
case (name, shortcode, ccam) =>
val newact = Json.obj(
"id" -> BSONObjectID.generate,
"name" -> name,
"shortcode" -> shortcode,
"ccam" -> ccam
)
settings.update(
Json.obj("practiceId" -> user.practiceId.get),
Json.obj("$addToSet" -> Json.obj("acts" -> Json.obj("acte" -> newact)))
).map { lastError => Ok(Json.toJson(newact)) }
})
}
The CCAMAct class is defined like this :
import models.db.Indexable
import play.api.libs.json._
import reactivemongo.bson.BSONObjectID
import reactivemongo.api.indexes.{Index, IndexType}
import models.db.{MongoModel, Indexable}
import scala.concurrent.Future
import play.modules.reactivemongo.json.BSONFormats._
import models.practice.Practice
import play.api.libs.functional.syntax._
case class CCAMAct(code:Option[String],
description:Option[String],
_id: Option[BSONObjectID] = None) extends MongoModel {}
object CCAMAct extends Indexable {
private val logger = play.api.Logger(classOf[CommonSetting]).logger
import play.api.Play.current
import play.modules.reactivemongo.ReactiveMongoPlugin._
import play.modules.reactivemongo.json.collection.JSONCollection
import scala.concurrent.ExecutionContext.Implicits.global
def ccam: JSONCollection = db.collection("ccam")
implicit val ccamFormat = Json.format[CCAMAct]
def index() = Future.sequence(
List (
Index(Seq("description" -> IndexType.Text))
).map(ccam.indexesManager.ensure)
).onComplete { indexes => logger.info("Text index on CCAM ends") }
}
Thus the compiler throws me this error :
Cannot find Formatter type class for reactivemongo.bson.BSONObjectID. Perhaps you will need to import play.api.data.format.Formats._
"_id" -> optional(of[BSONObjectID])
^
(Of course I have already imported "play.api.data.format.Formats._")
I also tried to add a custom Formatter following answers from similar posts on the web.
object Format extends Format[BSONObjectID] {
def writes(objectId: BSONObjectID): JsValue = JsString(objectId.stringify)
def reads(json: JsValue): JsResult[BSONObjectID] = json match {
case JsString(x) => {
val maybeOID: Try[BSONObjectID] = BSONObjectID.parse(x)
if(maybeOID.isSuccess)
JsSuccess(maybeOID.get)
else {
JsError("Expected BSONObjectID as JsString")
}
}
case _ => JsError("Expected BSONObjectID as JsString")
}
}
...without any success.
[UPDATED POST]
Finally I'm not able to find a play form type (for handling POST requests in controller) in order to map a class containing BSONObjectID type...
Anyone knows a clean solution to resolve this problem?
The JSON Format for the BSON types from ReactiveMongo are not provided by Play itself in Formats._, but by the ReactiveMongo plugin for Play.
import play.modules.reactivemongo.json.BSONFormats._

How to return model query result as JSON in scala play framework

I am using Play framework 2.1.1 with scala.I query a database table return to controller as list and then convert list to string and return to ajax call from javascript code.
How to return query result as json and return to ajax call throught controller?
Application.scala
import play.api._
import play.api.mvc._
import play.api.data._
import views.html._
import models._
object Application extends Controller {
def index = Action {
Ok(views.html.index())
}
def getSummaryTable = Action{
var sum="Summary Table";
Ok(ajax_result.render((Timesheet.getAll).mkString("\n")))
}
def javascriptRoutes = Action { implicit request =>
import routes.javascript._
Ok(
Routes.javascriptRouter("jsRoutes")(
// Routes
controllers.routes.javascript.Application.getSummaryTable
)
).as("text/javascript")
}
}
TimeSheet.scala
// Use PostgresDriver to connect to a Postgres database
import scala.slick.driver.PostgresDriver.simple._
import scala.slick.lifted.{MappedTypeMapper,BaseTypeMapper,TypeMapperDelegate}
import scala.slick.driver.BasicProfile
import scala.slick.session.{PositionedParameters,PositionedResult}
// Use the implicit threadLocalSession
import Database.threadLocalSession
import java.sql.Date
import java.sql.Time
case class Timesheet(ID: Int, dateVal: String, entryTime: Time, exitTime: Time, someVal: String)
object Timesheet {
//Definition of Timesheet table
// object TS extends Table[(Int,String,Time,Time,String)]("timesheet"){
val TSTable = new Table[Timesheet]("timesheet"){
def ID = column[Int]("id")
def dateVal = column[String]("date")
def entryTime = column[Time]("entry_time")
def exitTime = column[Time]("exit_time")
def someVal = column[String]("someval")
def * = ID ~ dateVal ~ entryTime ~ exitTime ~ someVal <> (Timesheet.apply _, Timesheet.unapply _)
}
def getAll: Seq[Timesheet] = {
Database.forURL("jdbc:postgresql://localhost:5432/my_db", "postgres", "password",null, driver="org.postgresql.Driver") withSession{
val q = Query(TSTable)
val qFiltered = q.filter(_.ID === 41 )
val qDateFilter = qFiltered.filter(_.dateVal === "01/03/2013")
val qSorted = qDateFilter.sortBy(_.entryTime)
qSorted.list
}
}
}
Also, don't forget to provide an implicit (or not) Json deserializer for your model, otherwise, Scala compiler will yell at you :-). You can do something like :
def allTimesheet = Action {
val timesheetWrites = Json.writes[Timesheet] // here it's the deserializer
val listofTimeSheet = Timesheet.getAll
Ok( Json.toJson( listofTimeSheet )( timesheetWrites ) )
}
or you can use implicits like :
def allTimesheet = Action {
implicit val timesheetWrites = Json.writes[Timesheet] // here it's the deserializer
val listofTimeSheet = Timesheet.getAll
Ok( Json.toJson( listofTimeSheet ) )
}
and even declare your deserializer in your model companion object like :
companion object
object Timesheet {
implicit val timesheetWrites = Json.writes[Timesheet] // here it's the deserializer
....
}
and in the controller
import models.Timesheet.timesheetWrites
def allTimesheet = Action {
val listofTimeSheet = Timesheet.getAll
Ok( Json.toJson( listofTimeSheet ) )
}
I recommend you use play.api.libs.Json.toJson.
Here's an example:
object Products extends Controller {
def list = Action {
val productCodes = Product.findAll.map(_.ean)
Ok(Json.toJson(productCodes))
}
Json.toJson returns a JsValue for which Play will automatically add a application/json header.
See Play For Scala chapter 8.

How to handle optional fields in JSON parsing in play 2.1

I'm using the new Play 2.1-RC1 framework and I have a class that has an Option[] field, something like this:
import play.api.libs.json._
import play.api.libs.json.util._
import play.api.libs.json.Reads._
import play.api.libs.json.Writes._
import play.api.libs.json.Format._
import play.api.libs.functional.syntax._
case class Test(name: String, value: Option[String])
object Test {
implicit val testFormat = (
(__ \ "name").format[String] and
(__ \ "value").format[Option[String]]
)(Test.apply, unlift(Test.unapply))
def fromJson(js: String): Test = {
Json.fromJson[Test](Json.parse(js)).fold(
valid = { t => t},
invalid = { e => {
val missingField = (e(0)._1).toString.substring(1)
val badJs = js.trim
val newJs = badJs.substring(0, badJs.length()-1)+",\""+missingField+"\":null}"
fromJson(newJs)
}}
)
}
}
I want to be able to handle JSON strings that omit the optional "value" data, e.g.
val y = """{"name":"someone"}"""
(edited question)
I can rewrite the json string (rather clumsily) as shown in the validation step, but
is there a simpler pattern I can use to supply None for missing Optional fields? Note that this rewrite does not work with nested structures, or anywhere where I can't simply append the missing field.
You can simply do this:
import play.api.libs.json._
import play.api.libs.functional.syntax._
case class Test(name: String, value: Option[String])
implicit val testFormat = Json.format[Test]
def hoge = Action(Json.parse.json) { request =>
Json.fromJson[Test](request.body)
...
}
OK ... so the answer is very simple. Use
fomatOpt()
for optional fields. So the test formatter now looks like this:
import play.api.libs.json._
import play.api.libs.json.util._
import play.api.libs.json.Reads._
import play.api.libs.json.Writes._
import play.api.libs.json.Format._
import play.api.libs.functional.syntax._
case class Test(name: String, value: Option[String])
object Test {
implicit val testFormat = (
(__ \ "name").format[String] and
(__ \ "value").formatOpt[String]
)(Test.apply, unlift(Test.unapply))
def fromJson(js: String): Test = {
Json.fromJson[Test](Json.parse(js)).fold(
valid = { t => t},
invalid = { e => {
println("BAD JSON!")
null
}}
)
}
}