I'm trying to map a column which is a Map with frozen type
My column family has a field
batsmen_data map<text, frozen<bat_card>>
bat_card has two fields
bat_id int,
bat_name text,
Map the column field
object batsmenData extends MapColumn[ScoreCardData, ScoreCard, String ,Batting](this) {
override lazy val name="batsmen_data"
}
This is not an ideal way to do it. Because MapColumn Supports only primitive types. Can any one help me out on how to create a UDT Columns
I've figured out a way to map the UDT using phantom. But not sure this is right way.
I have a column
batsmen_data map<text, frozen<bat_card>>
To map this, we write the below code
object batsmenData extends MapColumn[ScoreCardData, ScoreCard, String, Batting](this) with CustomPrimitives {
override lazy val name="batsmen_data"
}
When you compile, error will be shown No Primitive found for class Batting
This is because Phantom has defined Primitives for native types like String, Int etc. To avoid you have to define primitive for class Batting as shown below(You have to extend CustomPrimitives trait in the class, else you will get the same error).
trait CustomPrimitives extends Config {
implicit object BattingPrimitive extends Primitive[Batting]{
override type PrimitiveType = Batting
override def clz: Class[Batting] = classOf[Batting]
override def cassandraType: String = {
Connector.session.getCluster.getMetadata.getKeyspace(cassandraKeyspace).getUserType("bat_card").toString()
}
override def fromRow(column: String, row: dsl.Row): Try[Batting] = ???
override def fromString(value: String): Batting = ???
override def asCql(value: Batting): String = ???
}
After this one more error will be shown saying Codec not found for requested operation: [frozen <-> Batting]. This is because cassandra expects a codec for the custom UDT type to serialize and deserialze the data. To Avoid this you have to write a CodecClass which will help to deserialize(as i need only deserializing) the UDT value into custom object.
public class BattingCodec extends TypeCodec<Batting>{
protected BattingCodec(TypeCodec<UDTValue> innerCodec, Class<Batting> javaType) {
super(innerCodec.getCqlType(), javaType);
}
#Override
public ByteBuffer serialize(Batting value, ProtocolVersion protocolVersion) throws InvalidTypeException {
return null;
}
#Override
public Batting deserialize(ByteBuffer bytes, ProtocolVersion protocolVersion) throws InvalidTypeException {
return null;
}
#Override
public Batting parse(String value) throws InvalidTypeException {
return null;
}
#Override
public String format(Batting value) throws InvalidTypeException {
return null;
}
}
Once the codec is defined last step is to register this codec into codec registry.
val codecRegistry = CodecRegistry.DEFAULT_INSTANCE
val bat_card = Connector.session.getCluster.getMetadata.getKeyspace(cassandraKeyspace).getUserType("bat_card")
val batCodec = new BattingCodec(TypeCodec.userType(bat_card), classOf[Batting])
codecRegistry.register(batCodec)
Now using deserialize function in BattingCodec, we can map the bytes to required object.
This method is working fine. But i'm not sure this is the ideal way to achieve the UDT functionality using Phantom
Related
I am trying to deserialize a scala enumeration from integer value.
object TestEnum extends Enumeration {
type TestEnum = Value
val None = Value(0)
val One = Value(1)
val Two = Value(2)
val Four = Value(4) // scalastyle:ignore
#JsonCreator
def forValue(value: Int): TestEnum = {
TestEnum.Value(value)
}
}
class TestEnum extends TypeReference[TestEnum.type]
When I try to deserialize this field, it throws an error as - Cannot deserialize value of type com.example.TestEnum$from Integer value (tokenJsonToken.VALUE_NUMBER_INT)
I see the jackson doc suggests to use the JsonCreator for the same in java, however nothing mentioned for scala enum.
I am using the defaultScalaMapper without any customization here.
I was able to solve it with jsonCreator in the class using the enum. as follows -
class Example(testEnum: TestEnum) {
#JsonCreator
def this(testEnumNum: Int) = {
this(TestEnum.forValue(testEnumNum))
}
}
However I need to do this in every class, I was hoping if there can be a better solution for this.
I have the unfortunate task of converting a networking library that I wrote couple years back from scala to java due to lack of offshore scala resources.
One of the tricker areas : converting the package object and its type aliases and case classes. Here is an excerpt:
package object xfer {
type RawData = Array[Byte]
type DataPtr = String
type PackedData = RawData
// type PackedData = (DataPtr, RawData, RawData)
// type UnpackedData = (DataPtr, Any, RawData)
type UnpackedData = Any
case class TaggedEntry(tag: String, data: Array[Byte])
case class TypedEntry[T](tag: String, t: T)
case class XferWriteParams(tag: String, config: XferConfig, data: RawData, md5: RawData) {
override def toString: DataPtr = s"XferWriteParams: config=$config datalen=${data.length}} md5len=${md5.length}}"
}
As an example the RawData has 32 usages. I suppose that one approach could be to do simple Find/Replace of all 32 instances with byte[]: but is there a more elegant way?
For the case class'es .. I'm leery of creating another few top level files in this package for each of them - and likewise another few top level files in each of a dozen other packages .. but is there any alternative?
ADT-esque trait-case-class clusters like
trait T
case class C1 extends T
case class C2 extends T
could be converted to an abstract base class T, with nested static classes C1, C2:
abstract class T {
static class C1 extends T { ... }
static class C2 extends T { ... }
}
This would at least eliminate the need to explode each such enumeration into thousand top-level classes in separate files.
For type-aliases, you might consider promoting them to full fledged wrapper classes, but you would have to be very careful whenever instantiating classes like RawData, using ==/equals, hashCode, or putting them in HashSets or HashMaps. Something like this might work:
class RawData {
private final byte[] value;
public RawData(byte[] v) { ... }
public byte[] getValue() { ... }
public static RawData of(byte[] value) {
return new RawData(value);
}
#Override public int hashCode() {
return value.hashCode();
}
#Override public boolean equals(Object other) {
if (other instanceof RawData) {
return value.equals(((RawData) other).value);
} else {
return false;
}
}
#Override public String toString() {
return String.valueOf(value);
}
}
This would keep the signatures similar, and might even enhance type-safety to some extent. In cases where it is really performance critical, I'd propose to just find/replace all occurrences by byte[].
I'm building an AWS lambda function with their POJO handler, but abstracting over the RequestHandler interface results in an erased type. When this happens AWS can't cast to the input type of my lambda function:
java.util.LinkedHashMap cannot be cast to com.amazonaws.services.lambda.runtime.events.SNSEvent: java.lang.ClassCastException
java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to com.amazonaws.services.lambda.runtime.events.SNSEvent
The following code works when uploaded to AWS:
import com.amazonaws.services.lambda.runtime._
import com.amazonaws.services.lambda.runtime.events.SNSEvent
// Only working version
class PojoTest1 extends Handler1[SNSEvent]{
override def handleRequest(input: SNSEvent, context: Context): Unit =
println(s"message: ${input.getRecords.get(0).getSNS.getMessage}")
}
trait Handler1[Event] extends RequestHandler[Event, Unit]{
override def handleRequest(input: Event, context: Context): Unit
}
Now because I'm using Scala I'm abstracting away the Java RequestHandler with generic traits. The following is a minified example of what doesn't work:
// Doesn't work
class PojoTest2 extends Handler2[SNSEvent]{
override def act(input: SNSEvent): Unit =
println(s"message: ${input.getRecords.get(0).getSNS.getMessage}")
}
trait Handler2[Event] extends RequestHandler[Event, Unit]{
def act(input: Event): Unit
override def handleRequest(input: Event, context: Context): Unit = act(input)
}
When I run javap PojoTest1.class this is the method that makes everything work:
public void handleRequest(com.amazonaws.services.lambda.runtime.events.SNSEvent, com.amazonaws.services.lambda.runtime.Context);
When I run javap PojoTest2.class You can see from this signature that the type of SNSEvent has been erased to Object:
public void handleRequest(java.lang.Object, com.amazonaws.services.lambda.runtime.Context);
This looks exactly like the issue described in SI-8905. Unfortunately the posted workaround doesn't seem to work either:
// Doesn't work
abstract class Handler3[T] extends Handler2[T]
class PojoTest3 extends Handler3[SNSEvent]{
override def act(input: SNSEvent): Unit =
println(s"message: ${input.getRecords.get(0).getSNS.getMessage}")
}
Even extending an abstract class directly doesn't yield better results:
// Doesn't work
class PojoTest4 extends Handler4[SNSEvent]{
override def act(input: SNSEvent): Unit =
println(s"message: ${input.getRecords.get(0).getSNS.getMessage}")
}
abstract class Handler4[Event] extends RequestHandler[Event, Unit] {
def act(input: Event): Unit
override def handleRequest(input: Event, context: Context): Unit = act(input)
}
When I use javap on any of the classes that don't work I still get the same method signature with the erased type.
I'm using Scala 2.12.7, sbt 1.1.2, and sbt-assembly 0.14.8.
I'm looking for any sort of work around to deal with this.
Note: I don't work for Amazon or Sun/Oracle so parts of the answer is a speculation.
I think there is a fundamental conflict between JVM type erasure, how AWS tries to work it around and what you are trying to do. I also don't think that the bug you referenced is relevant. I think the behavior is the same for Java.
AFAIU from the AWS point of view the problem looks like this: there is a stream of events of different types and a bunch of handlers. You need to decide which events a given handler can handle. The obvious solution is to look at the signature of the handleRequest method and use the argument's type. Unfortunately JVM type system doesn't really supports generics so you have to look for the most specific method (see further) and assume that that method is the real deal.
Now assume you develop a compiler that targets JVM (Scala or Java, further examples will be in Java to show that this is not a Scala-specific issue). Since JVM doesn't support generics you have to erasure your types. And you want to erase them to the most narrow type that covers all possible arguments so you are still type-safe at the JVM level.
For the RequestHandler.handleRequest
public O handleRequest(I input, Context context);
the only valid type erasure is
public Object handleRequest(Object input, Context context);
because I and O are unbound.
Now assume you do
public class PojoTest1 implements RequestHandler<SNSEvent, Void> {
#Override
public Void handleRequest(SNSEvent input, Context context) {
// whatever
return null;
}
}
At this point you say that you have a handleRequest method with this non-generic signature and the compiler has to respect it. But at the same time it has to respect your implements RequestHandler as well. So what the compiler has to do is to add a "bridge method" i.e. to produce a code logically equivalent to
public class PojoTest1 implements RequestHandler {
// bridge-method
#Override
public Object handleRequest(Object input, Context context) {
// call the real method casting the argument
return handleRequest((SNSEvent)input, context);
}
// your original method
public Void handleRequest(SNSEvent input, Context context) {
// whatever
return null;
}
}
Note how your handleRequest is not really an override of the RequestHandler.handleRequest. The fact that you also have Handler1 doesn't change anything. What is really important is that you have an override in your non-generic class so the compiler has to generate a non-generic method (i.e. a method with not erased types) in your final class. Now you have two methods and AWS can understand that the one that takes SNSEvent is the most specific one so it is represents your real bound.
Now assume you do add your generic intermediate class Handler2:
public abstract class Handler2<E> implements RequestHandler<E, Void> {
protected abstract void act(E input);
#Override
public Void handleRequest(E input, Context context) {
act(input);
return null;
}
}
At this point the return type is fixed but the argument is still an unbound generic. So compiler has to produce something like this:
public abstract class Handler2 implements RequestHandler {
protected abstract void act(Object input);
// bridge-method
#Override
public Object handleRequest(Object input, Context context) {
// In Java or Scala you can't distinguish between methods basing
// only on return type but JVM can easily do it. This is again
// call of the other ("your") handleRequest method
return handleRequest(input, context);
}
public Void handleRequest(Object input, Context context) {
act(input);
return null;
}
}
So now when we come to
public class PojoTest2 extends Handler2<SNSEvent> {
#Override
protected void act(SNSEvent input) {
// whatever
}
}
you have overridden act but not handleRequest. Thus the compiler doesn't have to generate a specific handleRequest method and it doesn't. It only generates a specific act. So the generated code looks like this:
public class PojoTest2 extends Handler2 {
// Bridge-method
#Override
protected void act(Object input) {
act((SNSEvent)input); // call the "real" method
}
protected void act(SNSEvent input) {
// whatever
}
}
Or if you flatten the tree and show all (relevant) methods in PojoTest2, it looks like this:
public class PojoTest2 extends Handler2 {
// bridge-method
#Override
public Object handleRequest(Object input, Context context) {
// In Java or Scala you can't distinguish between methods basing
// only on return type but JVM can easily do it. This is again
// call of the other ("your") handleRequest method
return handleRequest(input, context);
}
public Void handleRequest(Object input, Context context) {
act(input);
return null;
}
// Bridge-method
#Override
protected void act(Object input) {
act((SNSEvent)input); // call the "real" method
}
protected void act(SNSEvent input) {
// whatever
}
}
Both of the handleRequest methods accept just Object as a parameter and this is what AWS has to assume. Since you didn't override the handleRequest method in PojoTest2 (and not having to do so is the whole point of your inheritance hierarchy), the compiler didn't produce a more specific method for it.
Unfortunately I don't see any good workaround for this problem. If you want AWS to recognize the bound of the I generic parameter, you have to override handleRequest at the place in hierarchy where this bound becomes really known.
You may try do something like this:
// Your _non-generic_ sub-class has to have the following implementation of handleRequest:
// def handleRequestImpl(input: EventType, context: Context): Unit = handleRequestImpl(input, context)
trait UnitHandler[Event] extends RequestHandler[Event, Unit]{
def act(input: Event): Unit
protected def handleRequestImpl(input: Event, context: Context): Unit = act(input)
}
The benefit of this approach is that you can still put some additional wrapping logic (such as logging) into your handleRequestImpl. But still this will work only by convention. I see no way to force developers to use this code in the correct way.
If the whole point of your Handler2 is just bind the output type O to Unit without adding any wrapping logic, you can just do this without renaming the method to act:
trait UnitHandler[Event] extends RequestHandler[Event, Unit]{
override def handleRequest(input: Event, context: Context): Unit
}
In such way your sub-classes still will have to implement handleRequest with specific types bound to Event and compiler will have to produce specific methods there so the issue will not happen.
As #SergGr said, there are no real generics in the JVM. All types are replaced with their bounds or objects.
This answer has a different take on how to achieve the creation of custom abstract handlers which doesn't involve using the AWS RequestHandler.
The way I have solved this is by using context bounds and ClassTag like this:
abstract class LambdaHandler[TEvent: ClassTag, TResponse<: Any] {
def lambdaHandler(inputStream: InputStream, outputStream: OutputStream, context: Context): Unit = {
val json = Source.fromInputStream(inputStream).mkString
log.debug(json)
val event = decodeEvent(json)
val response = handleRequest(event, context)
// do things with the response ...
outputStream.close()
}
def decodeEvent(json: String): TEvent = jsonDecode[TEvent](json)
}
where jsonDecode is a function that turns the String event to the expected TEvent. In the following example I use json4s but you can use any de/serialization method you want:
def jsonDecode[TEvent: ClassTag](json: String): TEvent = {
val mapper = Mapper.default
jsonDecode(mapper)
}
In the end, you will be able to write functions like this
// AwsProxyRequest and AwsProxyResponse are classes from the com.amazonaws.serverless aws-serverless-java-container-core package
class Function extends LambdaHandler[AwsProxyRequest, AwsProxyResponse] {
def handleRequest(request: AwsProxyRequest, context: Context): AwsProxyResponse = {
// handle request and retun an AwsProxyResponse
}
}
Or custom SNS handlers where TEvent is the custom type of the SNS message:
// SNSEvent is a class from the com.amazonaws aws-lambda-java-events package
abstract class SnsHandler[TEvent: ClassTag] extends LambdaHandler[TEvent, Unit]{
override def decodeEvent(json: String): TEvent = {
val event: SNSEvent = jsonDecode[SNSEvent](json)
val message: String = event.getRecords.get(0).getSNS.getMessage
jsonDecode[TEvent](message)
}
}
If you use this method, straight out of the box, you will quickly realize that there are a large number of edge cases deserializing the JSON payloads because there are inconsistencies in the
types that you get from AWS events. Therefore, you will have to fine tune the jsonDecode method to suit your needs.
Alternatively, use an existing library that takes care of these steps for you. There is one library that I know of for Scala (but have not used) called aws-lambda-scala
or you can take a look at the full implementation of my LambdaHandler in GitHub
You can solve this in a way that doesn't require you to have every sub class call a base class method. Basically the base class needs to be told what the concrete type of the request is at runtime before it can convert it into an object of that type. So create an abstract method that gives it that information:
public abstract class BaseHandler<RequestType, ResponseType> implements RequestHandler<Map<String, Object>, ResponseType> {
#Override
public ResponseType handleRequest(Map<String, Object> request, Context context) {
// Convert the Map to Request Object for the sub classes
final ObjectMapper mapper = new ObjectMapper();
final RequestType requestPojo = mapper.convertValue(request, this.getRequestType());
return this.handle(requestPojo, context);
}
protected abstract ResponseType handle(RequestType request, Context context);
protected abstract Class<RequestType> getRequestType();
}
Then each base class just needs to implement the abstract methods. It isn't exactly ideal as the getRequestType is a strange method to have to have, but I think it is a little cleaner than the other answers here.
Example base class:
public class SubHandler extends BaseHandler<SNSEvent, Void> {
#Override
protected Void handle(SNSEvent input, Context context) {
// Handler implementation goes here
}
#Override
protected Class<SNSEvent> getRequestType() {
// strange method needed, but fairly obvious implementation since the generic requires the typing to be correct.
return SNSEvent.class;
}
}
Related:
Converting a Map to Object Java
Converting a Map to Object Scala
Trying to create new table, where each row can have extra data called context, which is key/value map. Key is string and value can be few bytes (bytes array).
The following definition doesn't compile, with the following error:
Error:(55, 30) Cannot find primitive implementation for class Array
object context extends MapColumn[String, Array[Byte]]
Here is my code:
case class UserJourny(
id: Long,
time: Int,
activity_type: Int,
context: Map[String, Array[Byte]]
)
abstract class UserJournyModel extends Table[UserJournyModel, UserJourny] {
override def tableName: String = "user_journy"
object dyid extends BigIntColumn with PartitionKey {
override lazy val name = "id"
}
object time extends IntColumn with ClusteringOrder with Descending {
override lazy val name = "time"
}
object activity_type extends IntColumn
object context extends MapColumn[String, Array[Byte]]
}
How should I do it right?
Use MapColumn[String, ByteBuffer] instead, which Cassandra natively understands. Then you need some basic fun stuff to encode a byte array to a buffer.
val buf = ByteBuffer.wrap(Array[Byte](1, 2, 3, 4, 5))
You can also derive a primitive to make your life easier overall.
implicit val byteArrayPrimitive = Primitive.derive[
Array[Byte],
ByteBuffer
](ByteBuffer.wrap)(_.array)
Now you can successfully use MapColumn[String, Array[Byte]], and Cassandra will store it as a buffer anyway, it will have map<text, buffer> type inside.
(Essentially I need some kind of a synthesis of these two questions (1, 2), but I'm not smart enough to combine them myself.)
I have a set of JAXB representations in Scala like this:
abstract class Representation {
def marshalToXml(): String = {
val context = JAXBContext.newInstance(this.getClass())
val writer = new StringWriter
context.createMarshaller.marshal(this, writer)
writer.toString()
}
}
class Order extends Representation {
#BeanProperty
var name: String = _
...
}
class Invoice extends Representation { ... }
The problem I have is with my unmarshalling "constructor" methods:
def unmarshalFromJson(marshalledData: String): {{My Representation Subclass}} = {
val mapper = new ObjectMapper()
mapper.getDeserializationConfig().withAnnotationIntrospector(new JaxbAnnotationIntrospector())
mapper.readValue(marshalledData, this.getClass())
}
def unmarshalFromXml(marshalledData: String): {{My Representation Subclass}} = {
val context = JAXBContext.newInstance(this.getClass())
val representation = context.createUnmarshaller().unmarshal(
new StringReader(marshalledData)
).asInstanceOf[{{Type of My Representation Subclass}}]
representation // Return the representation
}
Specifically, I can't figure out how to attach these unmarshalling methods in a typesafe and DRY way to each of my classes, and then to call them from Scala (and hopefully sometimes by using only abstract type information). In other words, I would like to do this:
val newOrder = Order.unmarshalFromJson(someJson)
And more ambitiously:
class Resource[R <: Representation] {
getRepresentation(marshalledData: String): R =
{{R's Singleton}}.unmarshalFromXml(marshalledData)
}
In terms of my particular stumbling blocks:
I can't figure out whether I should define my unmarshalFrom*() constructors once in the Representation class, or in a singleton Representation object - if the latter, I don't see how I can automatically inherit that down through the class hierarchy of Order, Invoice etc.
I can't get this.type (as per this answer) to work as a way of self-typing unmarshalFromJson() - I get a compile error type mismatch; found: ?0 where type ?0 required: Representation.this.type on the readValue() call
I can't figure out how to use the implicit Default[A] pattern (as per this answer) to work down my Representation class hierarchy to call the singleton unmarshalling constructors using type information only
I know this is a bit of a mammoth question touching on various different (but related) issues - any help gratefully received!
Alex
The key is to not try and attach the method to the class but rather pass it in as a parameter. To indicate the type you are expecting and let the type system handle passing it in. I tried to make the unmarshal invocation something that reads a little DSL like.
val order = UnMarshalXml( xml ).toRepresentation[Order]
The following is a fully testable code snippet
abstract class Representation {
def marshalToXml(): String = {
val context = JAXBContext.newInstance(this.getClass)
val writer = new StringWriter
context.createMarshaller.marshal(this, writer)
writer.toString
}
}
#XmlRootElement
class Order extends Representation {
#BeanProperty
var name: String = _
}
case class UnMarshalXml( xml: String ) {
def toRepresentation[T <: Representation](implicit m:Manifest[T]): T = {
JAXBContext.newInstance(m.erasure).createUnmarshaller().unmarshal(
new StringReader(xml)
).asInstanceOf[T]
}
}
object test {
def main( args: Array[String] ) {
val order = new Order
order.name = "my order"
val xml = order.marshalToXml()
println("marshalled: " + xml )
val received = UnMarshalXml( xml ).toRepresentation[Order]
println("received order named: " + received.getName )
}
}
You should see the following output if you run test.main
marshalled: <?xml version="1.0" encoding="UTF-8" standalone="yes"?><order><name>my order</name></order>
received name: my order
Here's the updated version of Neil's code which I used to support the second use case as well as the first:
case class UnmarshalXml(xml: String) {
def toRepresentation[T <: Representation](implicit m: Manifest[T]): T =
toRepresentation[T](m.erasure.asInstanceOf[Class[T]])
def toRepresentation[T <: Representation](typeT: Class[T]): T =
JAXBContext.newInstance(typeT).createUnmarshaller().unmarshal(
new StringReader(xml)
).asInstanceOf[T]
}
This supports simple examples like so:
val order = UnmarshalXml(xml).toRepresentation[Order]
But also for abstract type based usage, you can use like this:
val order = UnmarshalXml(xml).toRepresentation[T](typeOfT)
(Where you have grabbed and stored typeOfT using another implicit Manifest at the point of declaring T.)