I'm reverse engineering a file format that stores each field as TLV blocks (type, length, value).
The fields do not have to be in order, or even present at all. Their presence is denoted with a sentinel, which is a 16-bit type identifier and a 32-bit end offset. There are hundreds of unique identifiers, but a decent chunk of those are just single primitive values. aside from denoting the type, they can also identify what field the data should be stored in.
It is also worth noting that there will never be a duplicate id on a parent structure. The only time is can occur is if there are multiple of the same object type in an array/list.
I have successfully written a Kaitai definition for one of them:
meta:
id: struct_02ea
endian: le
seq:
- id: unk_00
type: s4
- id: fields
type: field_block
repeat: eos
types:
sentinel:
seq:
- id: id
type: u2
- id: end_offset
type: u4
field_block:
seq:
- id: sentinel
type: sentinel
- id: value
type:
switch-on: sentinel.id
cases:
0xF0: u1
0xF1: u1
0xF2: u1
0xF3: u1
0xF4: u4
0xF5: u4
size: sentinel.end_offset - _root._io.pos
Handling things this way does work, and I could likely map out the entire format like this. However, when it comes time to compiling this definition into another format, things get nasty.
Since I am wrapping each field in a field_block, the generated code stores these values in that type of object. This is incredibly inefficient when half of the generated field_block objects store a single integer. It would also require the consuming code to iterate through a list of each field block in order to get the actual field's value.
Ideally, I would like to define this structure so that the sentinels are only parsed while Kaitai is reading the data, and each value would be mapped to a field on the parent structure.
Is this possible? This technology is really cool, and I'd love to use it in my project, but I feel like the overhead that this is generating is a lot more trouble than it's worth.
Here's an example of the definition when compiled into C#:
using System.Collections.Generic;
namespace Kaitai
{
public partial class Struct02ea : KaitaiStruct
{
public static Struct02ea FromFile(string fileName)
{
return new Struct02ea(new KaitaiStream(fileName));
}
public Struct02ea(KaitaiStream p__io, KaitaiStruct p__parent = null, Struct02ea p__root = null) : base(p__io)
{
m_parent = p__parent;
m_root = p__root ?? this;
_read();
}
private void _read()
{
_unk00 = m_io.ReadS4le();
_fields = new List<FieldBlock>();
{
var i = 0;
while (!m_io.IsEof) {
_fields.Add(new FieldBlock(m_io, this, m_root));
i++;
}
}
}
public partial class Sentinel : KaitaiStruct
{
public static Sentinel FromFile(string fileName)
{
return new Sentinel(new KaitaiStream(fileName));
}
public Sentinel(KaitaiStream p__io, Struct02ea.FieldBlock p__parent = null, Struct02ea p__root = null) : base(p__io)
{
m_parent = p__parent;
m_root = p__root;
_read();
}
private void _read()
{
_id = m_io.ReadU2le();
_endOffset = m_io.ReadU4le();
}
private ushort _id;
private uint _endOffset;
private Struct02ea m_root;
private Struct02ea.FieldBlock m_parent;
public ushort Id { get { return _id; } }
public uint EndOffset { get { return _endOffset; } }
public Struct02ea M_Root { get { return m_root; } }
public Struct02ea.FieldBlock M_Parent { get { return m_parent; } }
}
public partial class FieldBlock : KaitaiStruct
{
public static FieldBlock FromFile(string fileName)
{
return new FieldBlock(new KaitaiStream(fileName));
}
public FieldBlock(KaitaiStream p__io, Struct02ea p__parent = null, Struct02ea p__root = null) : base(p__io)
{
m_parent = p__parent;
m_root = p__root;
_read();
}
private void _read()
{
_sentinel = new Sentinel(m_io, this, m_root);
switch (Sentinel.Id) {
case 243: {
_value = m_io.ReadU1();
break;
}
case 244: {
_value = m_io.ReadU4le();
break;
}
case 245: {
_value = m_io.ReadU4le();
break;
}
case 241: {
_value = m_io.ReadU1();
break;
}
case 240: {
_value = m_io.ReadU1();
break;
}
case 242: {
_value = m_io.ReadU1();
break;
}
default: {
_value = m_io.ReadBytes((Sentinel.EndOffset - M_Root.M_Io.Pos));
break;
}
}
}
private Sentinel _sentinel;
private object _value;
private Struct02ea m_root;
private Struct02ea m_parent;
public Sentinel Sentinel { get { return _sentinel; } }
public object Value { get { return _value; } }
public Struct02ea M_Root { get { return m_root; } }
public Struct02ea M_Parent { get { return m_parent; } }
}
private int _unk00;
private List<FieldBlock> _fields;
private Struct02ea m_root;
private KaitaiStruct m_parent;
public int Unk00 { get { return _unk00; } }
public List<FieldBlock> Fields { get { return _fields; } }
public Struct02ea M_Root { get { return m_root; } }
public KaitaiStruct M_Parent { get { return m_parent; } }
}
}
Affiliate disclaimer: I'm a Kaitai Struct maintainer (see my GitHub profile).
Since I am wrapping each field in a field_block, the generated code stores these values in that type of object. This is incredibly inefficient when half of the generated field_block objects store a single integer. It would also require the consuming code to iterate through a list of each field block in order to get the actual field's value.
I think that rather than trying to describe the entire format with an ultimate Kaitai Struct specification, it's better for you not to let the generated code parse all the fields automatically. Move the parsing control to your application code, where you use the type Struct02ea.FieldBlock that represents the individual field and basically replicate the "repeat until end of stream" loop that the generated code that you posted was doing:
_fields = new List<FieldBlock>();
{
var i = 0;
while (!m_io.IsEof) {
_fields.Add(new FieldBlock(m_io, this, m_root));
i++;
}
}
The advantage of doing so is that you can adjust the loop to fit your needs. To avoid the overhead you describe, you'll probably want to keep the Struct02ea.FieldBlock object in a local variable inside the loop body, pull only the values you care about (save them in your compact, consumer-friendly output structures) and let it leave the scope after the loop iteration ends. This will allow each original FieldBlock object to get garbage-collected once you process it, so the overhead they have will be limited to a single instance and not multiplied by the number of fields in the file.
The most straightforward and seamless way to prevent the Kaitai Struct-generated code parse fields (but otherwise keep everything the same) is to add if: false in the KSY specification, as #webbnh suggested in a GitHub issue:
seq:
- id: unk_00
type: s4
- id: fields
type: field_block
repeat: eos
if: false # add this
The if: false works better than omitting it from seq entirely, because the kaitai-struct-compiler has occasional troubles with unused types (when compiling the KSY spec with unused types, you may get an error "Unable to derive _parent type in ..." due to a compiler bug). But with this if: false trick, you can't run into them because the field_block type is no longer unused.
My receiving Viewmodel (QuestionsPageViewModel) is not receiving the TopidId after passing it through Shell navigation as shown in the code below. I have placed the breakpoint at the LoadQuestions method in the QuestionsPageViewModel. When it is called, TopicId is null.
What am I missing?
HomePageViewModel
//This is in a command executed after clicking a button. And this is working fine
await Shell.Current.GoToAsync($"{nameof(QuestionsPage)}?TopicId={pack.TopicId}");
QuestionsPageViewModel
[INotifyPropertyChanged]
[QueryProperty(nameof(TopicId), nameof(TopicId))]
public partial class QuestionsPageViewModel
{
public ObservableRangeCollection<Question> QuestionsList { get; set; } = new();
[ObservableProperty]
string? title;
[ObservableProperty]
public string topicId;
public QuestionsPageViewModel()
{
LoadQuestions();
}
async void LoadQuestions()
{
Title = Utilities.ConvertTopicIdToString(short.Parse(TopicId));
try
{
using (var context = new DataContext())
{
QuestionPack questionPack = context.QuestionPacks
.First(x => x.TopicId == short.Parse(TopicId));
var questions = JsonConvert.DeserializeObject<List<Question>>(questionPack.Questions);
QuestionsList.AddRange(questions);
}
}
catch (Exception ex)
{
await Shell.Current.DisplayAlert("Error", $"Something went wrong: {ex}", "Cancel");
}
}
}
}
First of all, your field topicId should be private. CommumityToolkit.Mvvm will generate for you the public property.
Secondly, topicId is null because you're checking its value in a function called in the constructor. While you're executing the constructor, the shell navigation parameters are not initialized yet.
If you want to be sure that LoadQuestions() will be called after topicId is initialized, CommumityToolkit.Mvvm since version 8.0.0 should generate a partial method that can be used to execute some code after an ObservableProperty changes its value. In your case the name of this method should be OnTopicIdChanged(string value).
Try adding in your viewmodel this method and remove the function call from the constructor:
partial void OnTopicIdChanged(string value)
{
LoadQuestions();
}
I have a string property
public string XYZ
{
get => // do stuff
set => // do stuff which handles null
}
because I'm hoping it will get called....
But will it really?
(EF6.4)
It appears it will. If you implement the property with a backing field it's easy to test this by putting a breakpoint in the setter. EG
private string xyz;
public string XYZ
{
get
{
return xyz;
}
set
{
xyz = value;
}
}
And I think it would have to, as EF does not know whether your entities have a non-standard default value for a property. eg you could write
private string xyz = "none";
public string XYZ
{
get
{
return xyz;
}
set
{
xyz = value;
}
}
And so the hydration code would need to run the setter to get a correct result.
Should I make my own getters and setters in Swift? Im confused by the built in getters ad setters...is this even needed?
//properties for the resident
private var name: String!
var apartmentNumber: String!
var email : String!
var phoneNumber : String!
public func getName()->String{
return self.name
}
public func setName(name : String){
self.name = name
}
}
I've written an article for exactly this. I'll paste it here.
Stop writing getters and setters in Swift
I see this time and time again, and it's about time I write an article in one place to consolidate all my thoughts. If you find yourself writing code that looks like this, listen up:
public class C {
private var _i: Int = 0
public var i: Int {
get {
return self._i
}
set {
self._i = newValue
}
}
}
This pattern* is completely pointless in Swift, and I'll explain why, but firstly we need to take a short detour through Java land. Why Java? Because most of the people I run into who write Swift like this have some sort of Java background, either
because it was taught in their computer sceince courses, or
because they're coming over to iOS development, from Android
What's the point of getters and setters?
Suppose we have the following class in Java:
public class WeatherReport {
public String cityName;
public double temperatureF;
public WeatherReport(String cityName, double temperatureF) {
this.cityName = cityName;
this.temperatureF = temperatureF;
}
}
If you showed this class to any CS prof, they're surely going to bark at you for breaking encapsulation. But what does that really mean? Well, imagine how a class like this would be used. Someone would write some code that looks something like this:
WeatherReport weatherReport = weatherAPI.fetchWeatherReport();
weatherDisplayUI.updateTemperatureF(weatherReport.temperatureF);
Now suppose you wanted to upgrade your class to store data in a more sensible temperature unit (beating the imperial system dead horse, am I funny yet?) like Celcius or Kelvin. What happens when you update your class to look like this:
public class WeatherReport {
public String cityName;
public double temperatureC;
public WeatherReport(String cityName, double temperatureC) {
this.cityName = cityName;
this.temperatureC = temperatureC;
}
}
You've changed the implementation details of your WeatherReport class, but you've also made an API breaking change. Because temperatureF was public, it was part of this class' API. Now that you've removed it, you're going to cause compilation errors in every consumer that depended on the exitense of the temperatureF instance variable.
Even worse, you've changed the semantics of the second double argument of your constructor, which won't cause compilation errors, but behavioural errors at runtime (as people's old Farenheit based values are attemped to be used as if they were celcius values). However, that's not an issue I'll be discussing in this article.
The issue here is that consumers of this class will be strongly coupled to the implementation details of your class. To fix this, you introduce a layer of seperation between your implementation details and your interface. Suppose the Farenheit version of our class was implemented like so:
public class WeatherReport {
private String cityName;
private double temperatureF;
public WeatherReport(String cityName, double temperatureF) {
this.cityName = cityName;
this.temperatureF = temperatureF;
}
public String getCityName() {
return this.cityName;
}
public void setCityName(String cityName) {
this.cityName = cityName;
}
public double getTemperatureF() {
return this.temperatureF;
}
public void setTemperatureF(double temperatureF) {
this.temperatureF = temperatureF;
}
}
The getters and setters are really basic methods that access or update our instance variables. Notice how this time, our instance variables are private, and only our getters and setters are public. A consumer would use this code, as so:
WeatherReport weatherReport = weatherAPI.fetchWeatherReport();
weatherDisplayUI.updateTemperatureF(weatherReport.getTemperatureF());
This time, when we make the upgrade to celcius, we have the freedom to change our instance variables, and tweak our class to keep it backwards compatible:
public class WeatherReport {
private String cityName;
private double temperatureC;
public WeatherReport(String cityName, double getTemperatureC) {
this.cityName = cityName;
this.temperatureC = temperatureC;
}
public String getCityName() {
return this.cityName;
}
public void setCityName(String cityName) {
this.cityName = cityName;
}
// Updated getTemperatureF is no longer a simple getter, but instead a function that derives
// its Farenheit value from the Celcius value that actuallyed stored in an instance variable.
public double getTemperatureF() {
return this.getTemperatureC() * 9.0/5.0 + 32.0;
}
// Updated getTemperatureF is no longer a simple setter, but instead a function
// that updates the celcius value stored in the instance variable by first converting from Farenheit
public void setTemperatureF(double temperatureF) {
this.setTemperatureC((temperatureF - 32.0) * 5.0/9.0);
}
// Mew getter, for the new temperatureC instance variable
public double getTemperatureC() {
return this.temperatureC;
}
// New setter, for the new temperatureC instance variable
public void setTemperatureC(double temperatureC) {
this.temperatureC = temperatureC;
}
}
We've added new getters and setters so that new consumers can deal with temperatures in Celcius. But importantly, we've re-implemented the methods that used to be getters and setters for temperatureF (which no longer exists), to do the appropraite conversions and forward on to the Celcius getters and setters. Because these methods still exist, and behave identically as before, we've successfully made out implementation change (storing F to storing C), without breaking our API. Consumers of this API won't notice a difference.
So why doesn't this translate into Swift?
It does. But simply put, it's already done for you. You see, stored properties in Swift are not instance variables. In fact, Swift does not provide a way for you to create or directly access instance variables.
To understand this, we need to have a fuller understanding of what properties are. There are two types, stored and computed, and neither of them are "instance variables".
Stored properties: Are a combination of a comiler-synthesized instance variable (which you never get to see, hear, touch, taste, or smell), and the getter and setter that you use to interact with them.
Computed proepties: Are just a getter and setter, without any instance variable to act as backing storage. Really, they just behave as functions with type () -> T, and (T) -> Void, but have a pleasant dot notation syntax:
print(weatherReport.temperatureC)
weatherReport.temperatureC = 100
rather than a function calling synax:
print(weatherReport.getTemperatureC())
weatherReport.setTemperatureC(100)
So in fact, when you write:
class C {
var i: Int
}
i is the name of the getter and setter for an instance variable the compiler created for you. Let's call the instance variable $i (which is not an otherwise legal Swift identifier). There is no way to directly access $i. You can only get its value by calling the getter i, or update its value by calling its setter i.
So lets see how the WeatherReport migration problem looks like in Swift. Our initial type would look like this:
public struct WeatherReport {
public let cityName: String
public let temperatureF: Double
}
Consumers would access the temperature with weatherReport.temperatureF. Now, this looks like a direct access of an isntance variable, but remember, that's simply not possible in Swift. Instead, this code calls the compiler-syntehsized getter temperatureF, which is what accesses the instance variable $temperatureF.
Now let's do our upgrade to Celcius. We will first update our stored property:
public struct WeatherReport {
public let cityName: String
public let temperatureC: Double
}
This has broken our API. New consumers can use temperatureC, but old consumers who depended on temperatureF will no longer work. To support them, we simply add in a new computed property, that does the conversions between Celcius and Fahenheit:
public struct WeatherReport {
public let cityName: String
public let temperatureC: Double
public var temperatureF: Double {
get { return temperatureC * 9/5 + 32 }
set { temperatureC = (newValue - 32) * 5/9 }
}
}
Because our WeatherReport type still has a getter called temperatureF, consumers will behave just as before. They can't tell whether a property that they access is a getter for a stored property, or a computed property that derives its value in some other way.
So lets look at the original "bad" code. What's so bad about it?
public class C {
private var _i: Int = 0
public var i: Int {
get {
return self._i
}
set {
self._i = newValue
}
}
}
When you call c.i, the following happens:
You access the getter i.
The getter i accesses self._i, which is yet another getter
The getter _i access the "hidden" instance variable $i
And it's similar for the setter. You have two layers of "getterness". See what that would look like in Java:
public class C {
private int i;
public C(int i) {
this.i = i;
}
public int getI1() {
return this.i;
}
public void setI1(int i) {
this.i = i;
}
public int getI2() {
return this.getI1();
}
public void setI2(int i) {
this.setI1(i);
}
}
It's silly!
But what if I want a private setter?
Rather than writing this:
public class C {
private var _i: Int = 0
public var i: Int {
get {
return self._i
}
}
}
You can use this nifty syntax, to specify a seperate access level for the setter:
public class C {
public private(set) var i: Int = 0
}
Now isn't that clean?
There is no need to create setters and getters for stored properties in Swift and you shouldn't create them either.
You can control the accessibility of getters/setters separately when you declare a property.
public private(set) var name: String // public getter, private setter
If you want to implement some custom logic in your setter, you should use property observer, i.e. didSet/willSet.
var name: String {
didSet {
// This is called every time `name` is set, so you can do your custom logic here
}
}
You will hardly come across the need to create your own getters and setters.
Swift's computed property lets you use getters and setters in a very simple way.
Eg: below defined is a computed property circleArea that returns area of circle depending on radius.
var radius: Float = 10
var circleArea: Float {
get {
return .pi * powf(radius, 2)
}
set {
radius = sqrtf(newValue / .pi)
}
}
While you can observe a stored value and perform some task using property observers:
var radius: Float = 10 {
willSet {
print("before setting the value: \(value)")
}
didSet {
print("after the value is set: \(value)")
}
}
radius += 1
// before setting the value: 10.0
// after the value is set: 11.0
However if you feel like using getter setter, you can define appropriate functions for that. Below defined is an extension on Integer to get and set properties value.
extension Int {
func getValue() -> Int {
return self
}
mutating func setValue(_ val: Int) {
self = val
}
}
var aInt: Int = 29
aInt.getValue()
aInt.setValue(45)
print(aInt)
// aInt = 45
I have collection with thousands of documents, in document there's field named Rate, problem is currently its type is string, so when it's not available, the old developer set it to "N/A". For now I want to change the type of this field to numeric in C# (set it to 0 when n/a), but if I do so I can't load the past data.
Can we customize the deserialization so it will convert N/A to 0?
You need to create an IBsonSerializer or SerializerBase<> and attach it to the property you wish to serialize using the BsonSerializerAttribute. Something like the following:
public class BsonStringNumericSerializer : SerializerBase<double>
{
public override double Deserialize(BsonDeserializationContext context, BsonDeserializationArgs args)
{
var type = context.Reader.GetCurrentBsonType();
if (type == BsonType.String)
{
var s = context.Reader.ReadString();
if (s.Equals("N/A", StringComparison.InvariantCultureIgnoreCase))
{
return 0.0;
}
else
{
return double.Parse(s);
}
}
else if (type == BsonType.Double)
{
return context.Reader.ReadDouble();
}
// Add any other types you need to handle
else
{
return 0.0;
}
}
}
public class YourClass
{
[BsonSerializer(typeof(BsonStringNumericSerializer))]
public double YourDouble { get; set; }
}
If you don't want to use attributes you can create an IBsonSerializationProvider and register it using BsonSerializer.RegisterSerializationProvider.
Full documentation of MongoDB C# Bson serialization can be found here