playframework scala slick how many attributes are allowed - scala

In my Playframework scala application I have the following model:
case class ProcessTemplatesModel(
id: Option[Int] = None,
title: String,
version: String,
createdat: Option[String],
updatedat: Option[String],
deadline: Option[Date],
status: Option[String],
comment: Option[String],
checked: Option[Boolean],
checkedat: Option[Date],
approved: Option[Boolean],
approvedat: Option[Date],
deleted: Boolean,
approveprocess: Int,
trainingsprocess: Option[Int],
previousVersion: Option[Int],
originTemplate: Option[Int],
client: Int,
approveProcessInstance: Option[Int],
responsible: Option[Seq[UserModel]],
accountable: Option[Seq[UserModel]],
consulted: Option[Seq[UserModel]],
informed: Option[Seq[UserModel]])
object ProcessTemplatesModel {
implicit val processFormat = Json.format[ProcessTemplatesModel]
}
Today I added the approveProcessInstance: Option[Int],
Now I got this error while it compiles: No unapply or unapplySeq function found ... on this line: implicit val processFormat = Json.format[ProcessTemplatesModel]
Why does this fail in this case?

22 values is a max in play JSON, you can use 3rd party libraries to increase the number.
Here the issue thread in the Play source:
https://github.com/playframework/playframework/issues/3174
One of the possible solution:
https://github.com/xdotai/play-json-extensions
Example from my build.sbt
libraryDependencies ++= Seq(
cache,
filters,
ws,
// More than 22 fields in Json
"ai.x" %% "play-json-extensions" % "0.8.0"
)
For Play 2.6 you need to use version 10:
"ai.x" %% "play-json-extensions" % "0.10.0"
Then, in the file with JSON:
import ai.x.play.json.Jsonx
implicit val processFormat = Jsonx.formatCaseClass[ProcessTemplatesModel]
More details: https://github.com/xdotai/play-json-extensions#create-explicit-formatter

One other idea would be to decompose the model into fine grained models and have a composition of it. This way, you do not have to import yet another library!

Related

Get a Stream of Entity from an Http4s Response with Circe

I'm trying to retrieve a Stream[IO, Job] from an http4s Response, but the Scala compiler warns me that it cannot find any suitable Decoder:
Cannot decode into a value of type fs2.Stream[IO,Job], because no EntityDecoder[[+A]IO[A], fs2.Stream[IO,Job]] instance could be found.
[error] retrieved <- response.as[fs2.Stream[IO, Job]]
The code that generates the above error is the following:
import io.circe.generic.auto._
import org.http4s.circe.CirceEntityCodec._
// Many other imports
"should return the stream of all jobs" in {
for {
response <- jobsRoutes.orNotFound.run(
Request(
method = Method.GET,
uri = uri"/jobs",
headers = Headers(Accept(MediaType.`text/event-stream`))
)
)
retrieved <- response.as[fs2.Stream[IO, Job]]
} yield {
response.status shouldBe Status.Ok
}
}
In the build.sbt file, I have the following dependencies:
// Many other omitted dependencies
"org.http4s" %% "http4s-circe" % "0.23.14",
"io.circe" %% "circe-generic" % "0.14.2",
"io.circe" %% "circe-fs2" % "0.14.0",
The definition of the Job entity is:
final case class Job(
id: UUID,
date: Long,
salaryLo: Option[Int],
salaryHi: Option[Int],
currency: Option[String],
location: String,
tags: List[String],
description: String,
localUrl: Option[String],
externalUrl: Option[String]
image: Option[String],
country: Option[String],
title: String,
company: String,
seniority: Option[String],
other: Option[String]
)
I cannot understand what's going on.

Flink AvroRuntimeException: Not a Specific class

I try to use the confluent schema registry in an interactive Flink Scala shell to get started with the current 0.10.1 version of Flink.
Further context is available here https://github.com/geoHeil/streaming-reference/tree/5-basic-flink-setup
My problem is trying to initialize the serializer from the ConfluentRegistryAvroDeserializationSchema fails:
val serializer = ConfluentRegistryAvroDeserializationSchema.forSpecific[Tweet](classOf[Tweet], schemaRegistryUrl)
error: type arguments [Tweet] conform to the bounds of none of the overloaded alternatives of
value forSpecific: [T <: org.apache.avro.specific.SpecificRecord](x$1: Class[T], x$2: String, x$3: Int)org.apache.flink.formats.avro.registry.confluent.ConfluentRegistryAvroDeserializationSchema[T] <and> [T <: org.apache.avro.specific.SpecificRecord](x$1: Class[T], x$2: String)org.apache.flink.formats.avro.registry.confluent.ConfluentRegistryAvroDeserializationSchema[T]
The shell is set up like (i.e. additional JARs for avro or schema registry support are added as follows below:):
wget https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka_2.11/1.10.1/flink-connector-kafka_2.11-1.10.1.jar -P lib/
wget https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka-base_2.11/1.10.1/flink-connector-kafka-base_2.11-1.10.1.jar -P lib/
wget https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/0.10.2.1/kafka-clients-0.10.2.1.jar -P lib/
wget https://repo1.maven.org/maven2/org/apache/flink/flink-avro-confluent-registry/1.10.1/flink-avro-confluent-registry-1.10.1.jar -P lib/
wget https://repo1.maven.org/maven2/org/apache/flink/flink-avro/1.10.1/flink-avro-1.10.1.jar -P lib/
wget https://repo1.maven.org/maven2/org/apache/flink/force-shading/1.10.1/force-shading-1.10.1.jar -P lib/
wget https://repo1.maven.org/maven2/org/apache/avro/avro/1.8.2/avro-1.8.2.jar -P lib/
export TERM=xterm-color
./bin/start-scala-shell.sh local
I try to execute the following snippet:
import org.apache.flink.streaming.connectors.kafka.{
FlinkKafkaConsumer,
FlinkKafkaProducer
}
import org.apache.flink.formats.avro.registry.confluent.ConfluentRegistryAvroDeserializationSchema
import java.util.Properties
senv.enableCheckpointing(5000)
final case class Tweet(tweet_id: Option[String], text: Option[String], source: Option[String], geo: Option[String], place: Option[String], lang: Option[String], created_at: Option[String], timestamp_ms: Option[String], coordinates: Option[String], user_id: Option[Long], user_name: Option[String], screen_name: Option[String], user_created_at: Option[String], followers_count: Option[Long], friends_count: Option[Long], user_lang: Option[String], user_location: Option[String], hashtags: Option[Seq[String]])
val properties = new Properties()
properties.setProperty("bootstrap.servers", "localhost:9092")
properties.setProperty("group.id", "test")
val schemaRegistryUrl = "http://localhost:8081"
val serializer = ConfluentRegistryAvroDeserializationSchema.forSpecific[Tweet](classOf[Tweet], schemaRegistryUrl)
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/no-subject-td36269.html and http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Avro-from-avrohugger-still-invalid-td36274.html are links to Flinks mailing list
edit
The first hint I have stumbled upon is: https://github.com/zladovan/gradle-avrohugger-plugin I need to change the case classes to one of Avro Specfic or generic record when generating the classes. But I am also struggling there to get it working.
The case class Tweet example from above was generated from https://github.com/geoHeil/streaming-reference/blob/5-basic-flink-setup/common/models/src/main/avro/Tweet.avsc using https://github.com/zladovan/gradle-avrohugger-plugin in standard (i.e. case class) mode.
However, it needs to be moved to SpecificRecord format https://github.com/zladovan/gradle-avrohugger-plugin#source-formats to have a Tweet class which is compatible. This one is rather lengthy. For completeness it is available at https://gist.github.com/geoHeil/8b15d44d07e11c32a461b78365e0c158
The job is now still failing with:
Caused by: org.apache.avro.AvroRuntimeException: Not a Specific class: class com.github.geoheil.streamingreference.Tweet
even for an arguably compatible class. So this is not (yet) a full solution. Even though according to https://issues.apache.org/jira/browse/FLINK-12501 it should already work:
org.apache.avro.AvroRuntimeException: avro.shaded.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.avro.AvroRuntimeException: Not a Specific class: class com.github.geoheil.streamingreference.Tweet
at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:227)
at org.apache.flink.formats.avro.AvroDeserializationSchema.checkAvroInitialized(AvroDeserializationSchema.java:147)
at org.apache.flink.formats.avro.RegistryAvroDeserializationSchema.checkAvroInitialized(RegistryAvroDeserializationSchema.java:79)
at org.apache.flink.formats.avro.RegistryAvroDeserializationSchema.deserialize(RegistryAvroDeserializationSchema.java:64)
at org.apache.flink.streaming.connectors.kafka.internals.KafkaDeserializationSchemaWrapper.deserialize(KafkaDeserializationSchemaWrapper.java:45)
at org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.runFetchLoop(KafkaFetcher.java:140)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:718)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:200)
Caused by: avro.shaded.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.avro.AvroRuntimeException: Not a Specific class: class com.github.geoheil.streamingreference.Tweet
at avro.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2234)
at avro.shaded.com.google.common.cache.LocalCache.get(LocalCache.java:3965)
at avro.shaded.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
at avro.shaded.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:225)
... 9 more
Caused by: org.apache.avro.AvroRuntimeException: Not a Specific class: class com.github.geoheil.streamingreference.Tweet
at org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:285)
at org.apache.avro.specific.SpecificData$2.load(SpecificData.java:218)
at org.apache.avro.specific.SpecificData$2.load(SpecificData.java:215)
at avro.shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at avro.shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at avro.shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at avro.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
... 13 more

Consuming RESTful API and converting to Dataframe in Apache Spark

I am trying to convert output of url directly from RESTful api to Dataframe conversion in following way:
package trials
import org.apache.spark.sql.SparkSession
import org.json4s.jackson.JsonMethods.parse
import scala.io.Source.fromURL
object DEF {
implicit val formats = org.json4s.DefaultFormats
case class Result(success: Boolean,
message: String,
result: Array[Markets])
case class Markets(
MarketCurrency:String,
BaseCurrency:String,
MarketCurrencyLong:String,
BaseCurrencyLong:String,
MinTradeSize:Double,
MarketName:String,
IsActive:Boolean,
Created:String,
Notice:String,
IsSponsored:String,
LogoUrl:String
)
def main(args: Array[String]): Unit = {
val spark = SparkSession
.builder()
.appName(s"${this.getClass.getSimpleName}")
.config("spark.sql.shuffle.partitions", "4")
.master("local[*]")
.getOrCreate()
import spark.implicits._
val parsedData = parse(fromURL("https://bittrex.com/api/v1.1/public/getmarkets").mkString).extract[Array[Result]]
val mySourceDataset = spark.createDataset(parsedData)
mySourceDataset.printSchema
mySourceDataset.show()
}
}
The error is as follows and it repeats for every record:
Caused by: org.json4s.package$MappingException: Expected collection but got JObject(List((success,JBool(true)), (message,JString()), (result,JArray(List(JObject(List((MarketCurrency,JString(LTC)), (BaseCurrency,JString(BTC)), (MarketCurrencyLong,JString(Litecoin)), (BaseCurrencyLong,JString(Bitcoin)), (MinTradeSize,JDouble(0.01435906)), (MarketName,JString(BTC-LTC)), (IsActive,JBool(true)), (Created,JString(2014-02-13T00:00:00)), (Notice,JNull), (IsSponsored,JNull), (LogoUrl,JString(https://bittrexblobstorage.blob.core.windows.net/public/6defbc41-582d-47a6-bb2e-d0fa88663524.png))))))))) and mapping Result[][Result, Result]
at org.json4s.reflect.package$.fail(package.scala:96)
The structure of the JSON returned from this URL is:
{
"success": boolean,
"message": string,
"result": [ ... ]
}
So Result class should be aligned with this structure:
case class Result(success: Boolean,
message: String,
result: List[Markets])
Update
And I also refined slightly the Markets class:
case class Markets(
MarketCurrency: String,
BaseCurrency: String,
MarketCurrencyLong: String,
BaseCurrencyLong: String,
MinTradeSize: Double,
MarketName: String,
IsActive: Boolean,
Created: String,
Notice: Option[String],
IsSponsored: Option[Boolean],
LogoUrl: String
)
End-of-update
But the main issue is in the extraction of the main data part from the parsed JSON:
val parsedData = parse(fromURL("{url}").mkString).extract[Array[Result]]
The root of the returned structure is not an array, but corresponds to Result. So it should be:
val parsedData = parse(fromURL("{url}").mkString).extract[Result]
Then, I suppose that you need not to load the wrapper in the DataFrame, but rather the Markets that are inside. That is why it should be loaded like this:
val mySourceDataset = spark.createDataset(parsedData.result)
And it finally produces the DataFrame:
+--------------+------------+------------------+----------------+------------+----------+--------+-------------------+------+-----------+--------------------+
|MarketCurrency|BaseCurrency|MarketCurrencyLong|BaseCurrencyLong|MinTradeSize|MarketName|IsActive| Created|Notice|IsSponsored| LogoUrl|
+--------------+------------+------------------+----------------+------------+----------+--------+-------------------+------+-----------+--------------------+
| LTC| BTC| Litecoin| Bitcoin| 0.01435906| BTC-LTC| true|2014-02-13T00:00:00| null| null|https://bittrexbl...|
| DOGE| BTC| Dogecoin| Bitcoin|396.82539683| BTC-DOGE| true|2014-02-13T00:00:00| null| null|https://bittrexbl...|

Swagger models Option[Int] to Object while Option[String] is modelled correctly as string

I have the following case class
#ApiModel("StationResponse") case class StationResponse (id: Option[String],
#ApiModelProperty(dataType = "double", required = false)
longitude: Option[Double])
The longitude field is modelled as "Object" instead of "double"
I tried to override the dataType with #ApiModelProperty but no luck. Thanks for your comments.
I am using Swagger 1.3.12 with Scala 2.11.6
I solved the issue by adding #field annotation along with ApiModelProperty like below:
#ApiModel("StationResponse")
case class StationResponse (id: Option[String],
#(ApiModelProperty#field)(dataType = "double", required = false)
longitude: Option[Double])
I think it's now a bit simpler:
#ApiModelProperty(dataType = "Long") timingId: Option[Long]
workes just fine for me with: "io.swagger" %% "swagger-play2" % "1.6.0"
If you are using Springfox - per the documentation you can do the following:
#ApiModel("StationResponse")
case class StationResponse (id: Option[String], longitude: Option[Double]) {
#ApiModelProperty(dataType = "double", required = false)
def getLongitude: Option[Double] = longitude
}
Might be worth a shot on other Swagger generation libs

Reads[T] validator error using ReactiveMongo

I have a case class like this
case class Wish(_id: Option[String], title : String, text :String, cash: Int, created_at: Option[DateTime], updated_at : Option[DateTime])
Also I have defined a implicit reads validator as below
implicit val wishFormat = Json.format[Wish]
I am trying to read a Mongodb document into my wish class and I get error like below
scala> val js = "{\"_id\":{\"$oid\":\"5259c384dd8251bb085adfb4\"},\"title\":\"Shrikar\",\"text\":\"test\",\"cash\":12.0,\"created_at\":1381614468235,\"updated_at\":1381614468235}"
js: String = {"_id":{"$oid":"5259c384dd8251bb085adfb4"},"title":"Shrikar","text":"test","cash":12.0,"created_at":1381614468235,"updated_at":1381614468235}
scala> val test = Json.parse(js)
test: play.api.libs.json.JsValue = {"_id":{"$oid":"5259c384dd8251bb085adfb4"},"title":"Shrikar","text":"test","cash":12.0,"created_at":1381614468235,"updated_at":1381614468235}
scala> test.validate[Wish]
res11: play.api.libs.json.JsResult[Wish] = JsError(List((/_id,List(ValidationError(validate.error.expected.jsstring,WrappedArray())))))
Could you please help me resolve this issue?
Your ID must be a BSONObjectID.
case class Wish(
_id: Option[BSONObjectID],
title: String,
text: String,
cash: Int,
created_at: Option[DateTime],
updated_at: Option[DateTime]
)
Then you must import the format from the Play-ReactiveMongo plugin:
import play.modules.reactivemongo.json.BSONFormats._
The reads format converts a play.api.libs.json.JsValue to an object. Reads is expecting a json value that looks like this:
import play.api.libs.json.{Json, JsValue}
val js: JsValue = Json.obj(
"_id" -> "5259c384dd8251bb085adfb4",
"title" -> "Shrikar",
"text" -> "test",
"cash" -> 12.0,
"created_at" -> 1381614468235,
"updated_at" -> 1381614468235
)
You should start by taking a look at the Playframework working with JSON documentation. If you are using a library such as ReactiveMongo, then mongodb queries should return a BSONDocument or JsValue. According to the Reactive Mongo docs:
With Play2-ReactiveMongo, you can use directly the embedded JSON
library in Play >= 2.1. There is a specialized collection called
JSONCollection that deals naturally with JSValue and JSObject instead
of ReactiveMongo's BSONDocument.
If you can only receive mongodb query results as String, then you'll need to create a function to parse it into a JsValue.
I hope this helps!