I have written this code
import io.circe._, io.circe.generic.auto._, io.circe.parser._, io.circe.syntax._, io.circe.generic.extras._
implicit val config: Configuration = Configuration.default
case class Company(name: String)
case class Quote(average: Double)
case class Stats(price: Double)
#ConfiguredJsonCodec case class Bulk(company: Company, #JsonKey("advanced-stats") stats: Stats, quote: Quote)
val input = """{"AAPL": {"company": {"name": "Apple"},"advanced-stats": {"price":10},"quote": {"average":10}}}"""
val parsed = decode[Map[String, Bulk]](input)
When I try to execute this inside of ammonite I get an error
cmd5.sc:1: macro annotation could not be expanded (you cannot use a macro annotation in the same compilation run that defines it)
#ConfiguredJsonCodec case class Bulk(company: Company, #JsonKey("advanced-stats") stats: Stats, quote: Quote)
When I copy paste this code into a file and try to compile it then it gives compilation error
could not find Lazy implicit value of type io.circe.generic.extras.codec.ConfiguredAsObjectCodec
Edit:: Thanks to the answer below the code started to work on ammonite. it still doesn't compile when I copy paste it into a Scala file. I googled and changed the code to
object DefaultValues {
implicit val useDefaultValues = Configuration.default.withDefaults
}
import DefaultValues._
#ConfiguredJsonCodec
case class Bulk(
company: Company,
#JsonKey("advanced-stats") stats: Stats,
quote: Quote
)
but it still says
could not find Lazy implicit value of type io.circe.generic.extras.codec.ConfiguredAsObjectCodec[Bulk]
Have you enabled macro annotations? Scala 2.13 require flag that you can enable in Ammonite with:
interp.configureCompiler(_.settings.YmacroAnnotations.value = true)
in earlier versions of Ammonite which used Scala 2.12 and earlier you have to use
// replace with Scala version appropriate for your Ammonite
// \/
import $plugin.$ivy.`org.scalamacros:paradise_2.12.11:2.1.1`
Related
i want to writing a macro to get property names of a class.
but can not use Symbol module in quoted statement. i receive blow error...
inline def getProps(inline className: String): Iterable[String] = ${ getPropsImpl('className) }
private def getPropsImpl(className: Expr[String])(using Quotes): Expr[Iterable[String]] = {
import quotes.reflect.*
val props = '{
Symbol.classSymbol($className).fieldMembers.map(_.name) // error access to parameter x$2 from
} wrong staging level:
props - the definition is at level 0,
} - but the access is at level 1.
There are compile time and runtime of macros. And there are compile time and runtime of main code. The runtime of macros is the compile time of main code.
def getPropsImpl... =
'{ Symbol.classSymbol($className).fieldMembers.map(_.name) }
...
is incorrect because what Scala 3 macros do is transforming trees into trees (i.e. Exprs into Exprs, Expr is a wrapper over a tree) (*). The tree
Symbol.classSymbol($className).fieldMembers.map(_.name)
will make no sense inside the scope of application site. Symbol, Symbol.classSymbol etc. make sense here, inside the scope of macro.
def getPropsImpl... =
Symbol.classSymbol(className).fieldMembers.map(_.name)
...
would be also incorrect because className as a value doesn't exist yet, it's just a tree now.
I guess correct is with .valueOrAbort
import scala.quoted.*
inline def getProps(inline className: String): Iterable[String] = ${getPropsImpl('className)}
def getPropsImpl(className: Expr[String])(using Quotes): Expr[Iterable[String]] = {
import quotes.reflect.*
Expr.ofSeq(
Symbol.classSymbol(className.valueOrAbort).fieldMembers.map(s =>
Literal(StringConstant(s.name)).asExprOf[String]
)
)
}
Usage:
// in other file
getProps("mypackage.App.A") //ArraySeq(s, i)
// in other subproject
package mypackage
object App {
case class A(i: Int, s: String)
}
(*) Scala 2 macros can do more with c.eval. In Scala 3 there is similar thing staging.run but it's forbidden in macros.
Actually, c.eval (or forbidden staging.run) can be emulated in Scala 3 too
get annotations from class in scala 3 macros
Spark 2.3.0 with Scala 2.11. I'm implementing a custom Aggregator according to the docs here. The aggregator requires 3 types for input, buffer, and output.
My aggregator has to act upon all previous rows in the window so I declared it like this:
case class Foo(...)
object MyAggregator extends Aggregator[Foo, ListBuffer[Foo], Boolean] {
// other override methods
override def bufferEncoder: Encoder[ListBuffer[Mod]] = ???
}
One of the override methods is supposed to return the encoder for the buffer type, which in this case is a ListBuffer. I can't find any suitable encoder for org.apache.spark.sql.Encoders nor any other way to encode this so I don't know what to return here.
I thought of creating a new case class which has a single property of type ListBuffer[Foo] and using that as my buffer class, and then using Encoders.product on that, but I am not sure if that is necessary or if there is something else I am missing. Thanks for any tips.
You should just let Spark SQL do its work and find the proper encoder using ExpressionEncoder as follows:
scala> spark.version
res0: String = 2.3.0
case class Mod(id: Long)
import org.apache.spark.sql.Encoder
import scala.collection.mutable.ListBuffer
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
scala> val enc: Encoder[ListBuffer[Mod]] = ExpressionEncoder()
enc: org.apache.spark.sql.Encoder[scala.collection.mutable.ListBuffer[Mod]] = class[value[0]: array<struct<id:bigint>>]
I cannot see anything in org.apache.spark.sql.Encoders that could be used to directly encode a ListBuffer, or for that matter even a List
One option seems to be going with putting it in a case class, as you suggested:
import org.apache.spark.sql.Encoders
case class Foo(field: String)
case class Wrapper(lb: scala.collection.mutable.ListBuffer[Foo])
Encoders.product[Wrapper]
Another option could be to use kryo:
Encoders.kryo[scala.collection.mutable.ListBuffer[Foo]]
Or finally you could look at ExpressionEncoders, which extend Encoder:
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
ExpressionEncoder[scala.collection.mutable.ListBuffer[Foo]]
This is the best solution, as it keeps everything transparent to catalyst and therefore allows it to do all of its wonderful optimisations.
One thing I noticed whilst having a play:
ExpressionEncoder[scala.collection.mutable.ListBuffer[Foo]].schema == ExpressionEncoder[List[Foo]].schema
I haven't tested any of the above whilst executing aggregations, so there may be runtime issues. Hope this is helpful.
When encoding to Json with circe we really want the type field to show e.g.
scala> val fooJson = foo.asJson
fooJson: io.circe.Json =
{
"this_is_a_string" : "abc",
"another_field" : 123,
"type" : "Foo"
}
This is taken from the release notes which previously mentions that you can configure the encoding like this:
implicit val customConfig: Configuration =
Configuration.default.withSnakeCaseKeys.withDefaults.withDiscriminator("type")
Also other information about circe here suggests that without any configuration you should get some class type information in the encoding json.
Am I missing something? How do you get the class type to show?
UPDATE 30/03/2017: Follow up to OP's comment
I was able to make this work, as shown in the linked release notes.
Preparation step 1: add additional dependency to build.sbt
libraryDependencies += "io.circe" %% "circe-generic-extras" % "0.7.0"
Preparation step 2: setup dummy sealed trait hierarchy
import io.circe.{ Decoder, Encoder }
import io.circe.parser._, io.circe.syntax._
import io.circe.generic.extras.Configuration
import io.circe.generic.extras.auto._
import io.circe.generic.{ semiauto => boring } // <- This is the default generic derivation behaviour
import io.circe.generic.extras.{ semiauto => fancy } // <- This is the new generic derivation behaviour
implicit val customConfig: Configuration = Configuration.default.withDefaults.withDiscriminator("type")
sealed trait Stuff
case class Foo(thisIsAString: String, anotherField: Int = 13) extends Stuff
case class Bar(thisIsAString: String, anotherField: Int = 13) extends Stuff
object Foo {
implicit val decodeBar: Decoder[Bar] = fancy.deriveDecoder
implicit val encodeBar: Encoder[Bar] = fancy.deriveEncoder
}
object Bar {
implicit val decodeBar: Decoder[Bar] = boring.deriveDecoder
implicit val encodeBar: Encoder[Bar] = boring.deriveEncoder
}
Actual code using this:
val foo: Stuff = Foo("abc", 123)
val bar: Stuff = Bar("xyz", 987)
val fooString = foo.asJson.noSpaces
// fooString: String = {"thisIsAString":"abc","anotherField":123,"type":"Foo"}
val barString = bar.asJson.noSpaces
// barString: String = {"thisIsAString":"xyz","anotherField":987,"type":"Bar"}
val bar2 = for{
json <- parse(barString)
bar2 <- json.as[Stuff]
} yield bar2
// bar2: scala.util.Either[io.circe.Error,Stuff] = Right(Bar(xyz,987))
val foo2 = for{
json <- parse(fooString)
foo2 <- json.as[Stuff]
} yield foo2
// foo2: scala.util.Either[io.circe.Error,Stuff] = Right(Foo(abc,123))
So, provided you import the extra dependency (which is where Configuration comes from), it looks like it works.
Finally, as a sidenote, it does seem that there is some disconnection between Circe's DESIGN.md and practice, for which I am actually happy.
Original answer:
I am not sure this is supposed to be supported, by design.
Taken from Circe's DESIGN.md:
Implicit scope should not be used for configuration. Lots of people have asked for a way to configure generic codec derivation to use e.g. a type field as the discriminator for sealed trait hierarchies, or to use snake case for member names. argonaut-shapeless supports this quite straightforwardly with a JsonCoproductCodec type that the user can provide implicitly.
I don't want to criticize this approach—it's entirely idiomatic Scala, and it often works well in practice—but I personally don't like using implicit values for configuration, and I'd like to avoid it in circe until I am 100% convinced that there's no alternative way to provide this functionality.
What this means concretely: You'll probably never see an implicit argument that isn't a type class instance—i.e. that isn't a type constructor applied to a type in your model—in circe, and configuration of generic codec derivation is going to be relatively limited (compared to e.g. argonaut-shapeless) until we find a nice way to do this kind of thing with type tags or something similar.
In particular, customConfig: Configuration seems to be exactly the type of argument that the last paragraph refers to (e.g. an implicit argument that isn't a type class instance)
I am sure that #travis-brown or any other Circe's main contributors could shed some more light on this, in case there was in fact a way of doing this - and I would be very happy to know it! :)
I'm trying to convert JSON from Ajax request to Case class in Scala.js using scala-json https://github.com/MediaMath/scala-json
Here is my classes:
sealed trait Result
sealed trait Error extends Result
sealed trait Msg extends Result
case class MsgData(msg: Seq[String], args: Seq[Int]) extends Msg
case class CommentError(#name("obj.comment") comment: Seq[MsgData]) extends Error
Here is how I'm trying to convert:
import json._
implicit val msgDataAcc = ObjectAccessor.create[MsgData]
implicit val commentErrorAcc = ObjectAccessor.create[CommentError]
println("here2")
val errors = JValue.fromString(req.responseText).toObject[CommentError]
println("here3")
This code just silently dies on string with conversion and "here3" never printed to console.
Here is my JSON from server:
{"obj.comment":[{"msg":["error.minLength"],"args":[10]}],"obj.name":[{"msg":["error.path.missing"],"args":[]}]}
What I'm doing wrong? How to fix this?
So I'm guessing this is scala-js. Any exceptions that happen at the top level (entry point) of a scala-js application aren't always echoed out correctly (depending on environment/browser), if you wrap the whole thing in a Try and print out the stack trace during the catch, you should successfully see exception being thrown.
Main issue above is that you need to define 'accessors' for the case classes. There's 2 ways of doing this, one works out of the box by adding an implicit for each type, the other way requires macro-paradise and gives you a much simpler way of defining accessors for case classes.
Here is the normal non-macro-paradise way:
case class MsgData(msg: Seq[String], args: Seq[Int]) extends Msg
object MsgData {
implicit val acc = ObjectAccessor.create[MsgData]
}
case class CommentError(#name("obj.comment") comment: Seq[MsgData]) extends Error
object CommentError {
implicit val acc = ObjectAccessor.create[CommentError]
}
The implicits can be placed anywhere (following the general rule for scala implicits). Placing them in the companion object is the best way to guarantee the implicit can be found anywhere, without special imports or anything needed.
This is 'less magical' than other libs like circe that use shapeless to automatically derive factories, sometimes in a bloated way. scala-json aims to keep the accessor visible for extension purposes, but this does lead to some explicit boilerplate.
This can be reduced down using macro-paradise:
#accessor case class MsgData(msg: Seq[String], args: Seq[Int]) extends Msg
#accessor case class CommentError(#name("obj.comment") comment: Seq[MsgData]) extends Error
This does the exact same thing as the above code, we just leverage macro-paradise to add the implicit 'acc' field to the companion object automatically.
The following code executes fine in a scala shell given snakeyaml version 1.17
import org.yaml.snakeyaml.Yaml
import org.yaml.snakeyaml.constructor.Constructor
import scala.collection.mutable.ListBuffer
import scala.beans.BeanProperty
class EmailAccount {
#scala.beans.BeanProperty var accountName: String = null
override def toString: String = {
return s"acct ($accountName)"
}
}
val text = """accountName: Ymail Account"""
val yaml = new Yaml(new Constructor(classOf[EmailAccount]))
val e = yaml.load(text).asInstanceOf[EmailAccount]
println(e)
However when running in spark (2.0.0 in this case) the resulting error is:
org.yaml.snakeyaml.constructor.ConstructorException: Can't construct a java object for tag:yaml.org,2002:EmailAccount; exception=java.lang.NoSuchMethodException: EmailAccount.<init>()
in 'string', line 1, column 1:
accountName: Ymail Account
^
at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:350)
at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:182)
at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:141)
at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:127)
at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:450)
at org.yaml.snakeyaml.Yaml.load(Yaml.java:369)
... 48 elided
Caused by: org.yaml.snakeyaml.error.YAMLException: java.lang.NoSuchMethodException: EmailAccount.<init>()
at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.createEmptyJavaBean(Constructor.java:220)
at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:190)
at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:346)
... 53 more
Caused by: java.lang.NoSuchMethodException: EmailAccount.<init>()
at java.lang.Class.getConstructor0(Class.java:2810)
at java.lang.Class.getDeclaredConstructor(Class.java:2053)
at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.createEmptyJavaBean(Constructor.java:216)
... 55 more
I launched the scala shell with
scala -classpath "/home/placey/snakeyaml-1.17.jar"
I launched the spark shell with
/home/placey/Downloads/spark-2.0.0-bin-hadoop2.7/bin/spark-shell --master local --jars /home/placey/snakeyaml-1.17.jar
Solution
Create a self-contained application and run it using spark-submit instead of using spark-shell.
I've created a minimal project for you as a gist here. All you need to do is put both files (build.sbt and Main.scala) in some directory, then run:
sbt package
in order to create a JAR. The JAR will be in target/scala-2.11/sparksnakeyamltest_2.11-1.0.jar or a similar location. You can get SBT from here if you haven't used it yet. Finally, you can run the project:
/home/placey/Downloads/spark-2.0.0-bin-hadoop2.7/bin/spark-submit --class "Main" --master local --jars /home/placey/snakeyaml-1.17.jar target/scala-2.11/sparksnakeyamltest_2.11-1.0.jar
The output should be:
[many lines of Spark's log)]
acct (Ymail Account)
[more lines of Spark's log)]
Explanation
Spark's shell (REPL) transforms all classes you define in it by adding $iw parameter to your constructors. I've explained it here. SnakeYAML expects a zero-parameter constructor for JavaBean-like classes, but there isn't one, so it fails.
You can try this yourself:
scala> class Foo() {}
defined class Foo
scala> classOf[Foo].getConstructors()
res0: Array[java.lang.reflect.Constructor[_]] = Array(public Foo($iw))
scala> classOf[Foo].getConstructors()(0).getParameterCount
res1: Int = 1
As you can see, Spark transforms the constructor by adding a parameter of type $iw.
Alternative solutions
Define your own Constructor
If you really need to get it working in the shell, you could define your own class implementing org.yaml.snakeyaml.constructor.BaseConstructor and make sure that $iw gets passed to constructors, but this is a lot of work (I actually wrote my own Constructor in Scala for security reasons some time ago, so I have some experience with this).
You could also define a custom Constructor hard-coded to instantiate a specific class (EmailAccount in your case) similar to the DiceConstructor shown in SnakeYAML's documentation. This is much easier, but requires writing code for each class you want to support.
Example:
case class EmailAccount(accountName: String)
class EmailAccountConstructor extends org.yaml.snakeyaml.constructor.Constructor {
val emailAccountTag = new org.yaml.snakeyaml.nodes.Tag("!emailAccount")
this.rootTag = emailAccountTag
this.yamlConstructors.put(emailAccountTag, new ConstructEmailAccount)
private class ConstructEmailAccount extends org.yaml.snakeyaml.constructor.AbstractConstruct {
def construct(node: org.yaml.snakeyaml.nodes.Node): Object = {
// TODO: This is fine for quick prototyping in a REPL, but in a real
// application you should probably add type checks.
val mnode = node.asInstanceOf[org.yaml.snakeyaml.nodes.MappingNode]
val mapping = constructMapping(mnode)
val name = mapping.get("accountName").asInstanceOf[String]
new EmailAccount(name)
}
}
}
You can save this as a file and load it in the REPL using :load filename.scala.
Bonus advantage of this solution is that it can create immutable case class instances directly. Unfortunately Scala REPL seems to have issues with imports, so I've used fully qualified names.
Don't use JavaBeans
You can also just parse YAML documents as simple Java maps:
scala> val yaml2 = new Yaml()
yaml2: org.yaml.snakeyaml.Yaml = Yaml:1141996301
scala> val e2 = yaml2.load(text)
e2: Object = {accountName=Ymail Account}
scala> val map = e2.asInstanceOf[java.util.Map[String, Any]]
map: java.util.Map[String,Any] = {accountName=Ymail Account}
scala> map.get("accountName")
res4: Any = Ymail Account
This way SnakeYAML won't need to use reflection.
However, since you're using Scala, I recommend trying
MoultingYAML, which is a Scala wrapper for SnakeYAML. It parses YAML documents to simple Java types and then maps them to Scala types (even your own types like EmailAccount).