I have a SpecificRecord object from which I want to read a property from. Lets assume that I do not have access to the generated class of this SpecificRecord object (since these are dynamically chosen at runtime from a pool of classes via Reflection). Every generated class does have a common property "EXTRACT_DT_TM" (and a getter method for it) that I want to be able to read from this SpecificRecord object (Which is what I am having issues with).
So far I have tried:
public String map(SpecificRecord record) {
System.out.println("genericData to String = " +
GenericData.get().deepCopy(record.getSchema(), record).toString());
GenericRecord genericRecord = (GenericRecord)
GenericData.get().deepCopy(record.getSchema(), record);
System.out.println("genericRecord to String = " + genericRecord.toString());
Long extractionTime = (Long) genericRecord.get("EXTRACT_DT_TM");
But I get the following exception:
2021-08-22 01:14:43,280 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record cannot be cast to org.apache.avro.specific.SpecificRecord
at com.x.y.analytics.ods.z.ExtractionDatePartitionFn.map(ExtractionDatePartitionFn.java:49)
at com.x.y.analytics.ods.wolfe.ExtractionDatePartitionFn.map(ExtractionDatePartitionFn.java:22)
at org.apache.crunch.fn.ExtractKeyFn.map(ExtractKeyFn.java:64)
at org.apache.crunch.fn.ExtractKeyFn.map(ExtractKeyFn.java:29)
at org.apache.crunch.MapFn.process(MapFn.java:34)
Looking for any thoughts on how to read this value from the SpecificRecord.. In the above approach, I was trying to convert a SpecificRecord to a GenericRecord and then read the value from it..
For now since the position of the property is fixed across all generated classes, I was able to use the following. I would still like to know if anyone has a better answer that is not tied to the index.
Long extractionTime = (Long) record.get(2); // EXTRACT_DT_TM field
Related
Why does this code:
class _SequentialTextPageState {
String jsonTextPref = 'seqtext';
int jsonTextSuff = 10;
String jsonText = jsonTextPref + jsonTextSuff.toString();
}
generate these errors?
Error: The instance member 'jsonTextPref' can't be accessed in an initializer.
Error: The instance member 'jsonTextSuff' can't be accessed in an initializer.
It seems to me that concatenation between String and int is correct?
Dart initializes objects in multiple phases. Initializing members directly ("field initializers") occurs early in object initialization, before this becomes valid, so that phase cannot initialize members that depend on other parts of the object.
Dart provides multiple ways to initialize members, so if one member needs to depend on another, you can initialize it in a later phase by using a different mechanism. For example, you could do one of:
Add the late keyword to make the dependent member lazily initialized.
Move initialization of the dependent member into the constructor body.
In the case of a Flutter State subtype, you could initialize the dependent member in its initState method, which in some cases is more appropriate.
Note that in some cases you additionally can consider replacing the member variable with a read-only getter instead. For example, in your case, perhaps you could use:
String get jsonText => jsonTextPref + jsonTextSuff.toString();
That would be appropriate if jsonText should always depend on jsonTextPref and jsonTextSuff, would never need to have an independent value, and if it's acceptable for jsonText to return a new object every time it's accessed.
Dart does not allow field initializers to refer to the object itself. Fields must always be fully initialized before any access is given to the object begin created.
The initializers can only access static and top-level variables, not any instance variables on the object itself.
With null safety, you will be allowed to write late String jsonText = this.something + this.other;. That field will then not be initialized until it's first read or written, which is necessarily after the object itself has been created.
You can only use constant expressions as initializers. x=this.y is not constant.
The error was displayed when I did following:
class MyClass {
String id;
String[] imagePaths;
}
It will mark the String in the line String id; as error, but the error is in the next line, it should be List<String> imagePaths; instead of String[] imagePaths; then the error in the line above also disappears. This can be very confusing if you have a big class and the actual error is many lines underneath the first marked line (talking from experience...)
I'm parsing some JSON to read stored data into custom objects. I've implemented this simply using jsonDecode from dart:convert and writing a fromJson() constructor method on each of my custom objects. They're nested - so a TopicList object has a property which is a List<Topic>. I've written the fromJson() constructor on both TopicList and Topic, and I'm trying to use map to take the decoded JSON string and create an object from it. But it's failing because the list that map returns is of the wrong type, despite it being created from constructor methods.
class TopicList {
List<Topic> topics;
TopicList() {}
TopicList.fromJson(String jsonTopicList) {
Map decoded = jsonDecode(jsonTopicList);
// this *doesn't* work
topics = decoded['topics'].map((t) => Topic.fromJson(t)).toList();
// this does work
topics = [];
for (Map<String,dynamic> t in decoded['topics']) {
Topic newT = Topic.fromJson(t);
topics.add(newT);
}
}
}
class Topic {
String topic;
Topic({this.topic});
Topic.fromJson(Map<String, dynamic> t) {
this.topic = t['topic'];
}
}
When I try/catch the map method it tells me that the map().toList() call is returning a List<dynamic> rather than List<Topic>.
The reality is more complex than this with a further level (another List of other objects belonging to Topic) and with other properties. But this is the core of what is going on.
I'm not interested in workarounds - I've got one of those, which is working in the code above. But I want to understand why seemingly an object's constructor - in this example, it's Topic.fromJson() - is returning a dynamic object rather than a Topic.
Give a type argument to .map().
topics = decoded['topics'].map<Topic>((t) => Topic.fromJson(t)).toList();
Change
topics = decoded['topics'].map((t) => Topic.fromJson(t)).toList();
to
topics = decoded['topics'].map<Topic>((t) => Topic.fromJson(t)).toList();
That way, map will return an iterator of Topic which toList will convert to a list.
I am trying to compare two jsons, expected and the API Response using Javers, as part of testing. I want the comparison to exclude the ID parameters that are dynamically generated by response.
My VO is like
public class expectedResponse{
#DiffIgnore
private String id;
private String name;
}
Both my expectedResponse- which is read from excel file and the actual response from API are deserialized into this format and then both the responses are compared.
JsonNode expectedOutput = mapper.readTree(expected.toString());
JsonNode apiResponse = mapper.readTree(actual.toString());
diff=javers.compare(expectedOutput, apiResponse);
But this comparison doesn't exclude/ignore the ID field. Any Idea how I can get it to work? I want only the ID field excluded in comparison results, diff in name should be listed.
Also question 2> I am trying to list the changes from diff
if (diff.hasChanges())
{
List<ValueChange> changes=diff.getChangesByType(ValueChange.class);
for (ValueChange change : changes)
{
logger.info(change.getPropertyName()+ "||" +change.getLeft().toString() + "||" +change.getRight().toString());
change.getPropertyName()- doesnt print the property's name but simply prints "_value" as its value.
Can you pls help in identifying what is going wrong with the code and how can I get this fixed? I am not finding much useful documentations about Javers anywhere in google. Any help is appreciated.
You should compare you domain object instead of object with JsonNode class, look that #DiffIgnore annotation is present only in your domain class and there is no connection between JsonNode and ExpectedResponse, thats why Javers doesn't know to ignore this field.
To summarise, your code should looks like this:
ExpectedResponse expectedOutput = ...
ExpectedResponse apiResponse = ...
diff=javers.compare(expectedOutput, apiResponse);
I'm having some issues with a Wicket (8.0.0-M4) NumberTextField in Kotlin (1.1.0).
My stripped-down form looks like this:
class Test : AbstractWebPage() {
val housenumberModel: Model<Int> = Model<Int>()
val housenumber = NumberTextField<Int>("housenumberModel", housenumberModel)
val form: Form<Unit> = object : Form<Unit>("adressForm") {}
override fun onInitialize() {
super.onInitialize()
form.add(housenumber.setRequired(false))
form.add(object : SubmitLink("submit") {
override fun onSubmit() {
super.onSubmit()
println(housenumberModel.`object`) // this is line 28
}
})
add(form)
}
}
After submitting the form I get the following stacktrace:
java.lang.ClassCastException: java.lang.String cannot be cast to
java.lang.Number
at com.mycompany.test.pages.Test$onInitialize$1.onSubmit(Test.kt:28)
at org.apache.wicket.markup.html.form.Form.delegateSubmit(Form.java:1312)
at org.apache.wicket.markup.html.form.Form.process(Form.java:979)
at org.apache.wicket.markup.html.form.Form.onFormSubmitted(Form.java:802)
at org.apache.wicket.markup.html.form.Form.onRequest(Form.java:715)
at org.apache.wicket.core.request.handler.ListenerRequestHandler.internalInvoke(ListenerRequestHandler.java:301)
at org.apache.wicket.core.request.handler.ListenerRequestHandler.invoke(ListenerRequestHandler.java:250)
at org.apache.wicket.core.request.handler.ListenerRequestHandler.invokeListener(ListenerRequestHandler.java:210)
at org.apache.wicket.core.request.handler.ListenerRequestHandler.respond(ListenerRequestHandler.java:203)
at org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:912)
at org.apache.wicket.request.RequestHandlerExecutor.execute(RequestHandlerExecutor.java:65)
at org.apache.wicket.request.cycle.RequestCycle.execute(RequestCycle.java:283)
at org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:253)
at org.apache.wicket.request.cycle.RequestCycle.processRequestAndDetach(RequestCycle.java:221)
at org.apache.wicket.protocol.http.WicketFilter.processRequestCycle(WicketFilter.java:262)
at org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:204)
at org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:286)
[...]
If I use
val housenumberModel: Model<Int> = Model.of(0)
instead of
val housenumberModel: Model<Int> = Model<Int>()
everything works fine. But since my NumberTextField is optional I don't want to have it pre-initialized with 0.
Me and my colleagues were trying to change the type signature of the Model in every way we could imagine but came to no solution. A co-worker suggested to write a custom Wicket converter since Kotlins Int is represendeted as a primitive type (From the docs: "On the JVM, non-nullable values of this type are represented as values of the primitive type int.") Even though I don't know yet if this would work it seems like an overkill for me.
Another hack I could think of: writing some JavaScript to delete the zero from the input field. Also not really something I would want to do.
Question: Is there a simple solution to my problem?
(And as a bonus-question: has already anyone written a larger Wicket application in Kotlin and could tell me if this combination is ready for prime time to develop a critical project with this stack or is my problem just the tip of the iceberg?)
[edit]
Solution as pointed out by svenmeier:
Using
val housenumber = NumberTextField<Int>("housenumberModel", housenumberModel, Int::class.java)
works.
Or as an alternative:
val housenumbervalue: Int? = null
val housenumberModel: IModel<Int> = PropertyModel<Int>(this, "housenumbervalue")
val housenumber = NumberTextField<Int>("housenumberModel", housenumberModel)
Because of type erasure your NumberTextField cannot detect the generic type parameter of your model. Since your model object is null, it cannot be used to derive the type either.
In this case Wicket assumes a String model object type :/.
Either provide the type to the NumberTextField explicitly, or use a model that keeps its generic information, e.g. a PropertyModel.
There is a way to tell wicket about the type you want, it is by adding the type in the constructor. More here.
In Java it looks like this:
new NumberTextField<Integer>("housenumberModel", housenumberModel, Integer.class);
I'm trying to serialize data into / from my classes, derived from MonoBehaviour, which cannot be created from client code (e.g., with the new keyword), but rather must be created by a Unity3D-specific method, GameObject.AddComponent<T>(). How can I use the YamlDotNet framework to populate my classes with values without having to create an adapter for each one? Is there some sort of built-in adapter that I can configure, such that YamlDotNet doesn't instantiate the class it's trying to serialize to?
A typical file might contain a mapping of items, e.g.,
%YAML 1.1
%TAG !invt! _PathwaysEngine.Inventory.
%TAG !intf! _PathwaysEngine.Adventure.
---
Backpack_01: !invt!Item+yml
mass: 2
desc:
nouns: /^bag|(back)?pack|sack|container$/
description: |
Your backpack is only slightly worn, and...
rand_descriptions:
- "It's flaps twirl in the breeze."
- "You stare at it. You feel enriched."
MagLite_LR05: !invt!Lamp+yml
cost: 56
mass: 2
time: 5760
desc:
nouns: /^light|flashlight|maglite|lr_05$/
description: |
On the side of this flashlight is a label...
(Type "light" to turn it on and off.)
...
Where the tags are the fully specified class names of my Items, e.g., PathwaysEngine.Inventory.Lamp+yml, PathwaysEngine is the namespace I use for my game engine code, Inventory deals with items & whatnot, and Lamp+yml is how the compiler denotes a nested class, yml inside Lamp. Lamp+yml might look like this:
public partial class Lamp : Item, IWearable {
public new class yml : Item.yml {
public float time {get;set;}
public void Deserialize(Lamp o) {
base.Deserialize((Item) o);
o.time = time;
}
}
}
I call Deserialize() on all objects that derive from Thing from Awake(), i.e., once the MonoBehaviour classes exist in the game. Elsewhere, I've already created a pretty complicated Dictionary filled with objects of type Someclass+yml, and then Deserialize takes an instance of the real, runtime class Someclass and populates it with values. There's got to be a cleaner way to do this, right?
How can I:
Tell the Deserializer what my classes are?
See the second edit for a good solution for the above issue
Get the data without it attempting to create my MonoBehaviour-derived classes?
Edit: I've since worked at the problem, and have found out a good way of dealing with custom data (in my particular case of trying to parse regexes out of my data, and having them not be considered strings & therefore, un-castable to regex) is to use a IYamlTypeConverter for that particular string. Using YamlDotNet with Unity3D MonoBehaviours, however, is still an issue.
Another Edit: The above examples use a pretty ugly way of determining types. In my case, the best thing to do was to register the tags first with the deserializer, e.g.,
var pre = "tag:yaml.org,2002:";
var tags = new Dictionary<string,Type> {
{ "regex", typeof(Regex) },
{ "date", typeof(DateTime) },
{ "item", typeof(Item) }};
foreach (var tag in tags)
deserializer.RegisterTagMapping(
pre+tag.Key, tag.Value);
Then, I use the !!tag notation in the *.yml file, e.g.,
%YAML 1.1
---
Special Item: !!item
nouns: /thing|item|object/
someBoolean: true
Start Date: !!date 2015-12-17
some regex: !!regex /matches\s+whatever/
...
You can pass a custom implementation of IObjectFactory to the constructor of the Deserializer class. Every time the deserializer needs to create an instance of an object, it will use the IObjectFactory to create it.
Notice that your factory will be responsible for creating instances of every type that is deserialized. The easiest way to implement it is to create a decorator around DefaultObjectFactory, such as:
class UnityObjectFactory : IObjectFactory
{
private readonly DefaultObjectFactory DefaultFactory =
new DefaultObjectFactory();
public object Create(Type type)
{
// You can use specific types manually
if (type == typeof(MyCustomType))
{
return GameObject.AddComponent<MyCustomType>();
}
// Or use a marker interface
else if (typeof(IMyMarkerInterface).IsAssignableFrom(type))
{
return typeof(GameObject)
.GetMethod("AddComponent")
.MakeGenericMethod(type)
.Invoke();
}
// Delegate unknown types to the default factory
else
{
return DefaultFactory(type);
}
}
}