SpecificAvroDeserializer and "specifc.avro.reader" - apache-kafka

I have a question about SpecificAvroDeserializer because of class name, I always assumed, SpecificAvroSerializer will produce SpecificRecord and SpecificAvroDeserializer will be able to consume it.
To my surprise, if I don't set the 'specific.avro.reader' property, SpecificAvroDeserializer tries to read byte array as GenericRecord because of the following line of code at the class "AbstractKafkaAvroDeserializer" (https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/serializers/AbstractKafkaAvroDeserializer.java)..
private DatumReader getDatumReader(Schema writerSchema, Schema readerSchema) {
boolean writerSchemaIsPrimitive =
AvroSchemaUtils.getPrimitiveSchemas().values().contains(writerSchema);
// do not use SpecificDatumReader if writerSchema is a primitive
if (useSpecificAvroReader && !writerSchemaIsPrimitive) {
if (readerSchema == null) {
readerSchema = getReaderSchema(writerSchema);
}
return new SpecificDatumReader(writerSchema, readerSchema);
} else {
if (readerSchema == null) {
return new GenericDatumReader(writerSchema);
}
return new GenericDatumReader(writerSchema, readerSchema);
}
}
I thought the constructor of the SpecificAvroDeserializer should set this but it does not.. (https://github.com/confluentinc/schema-registry/blob/master/avro-serde/src/main/java/io/confluent/kafka/streams/serdes/avro/SpecificAvroDeserializer.java)
So what am I missing here?

Related

Managing PDF usage rights in iText7

I'm using PdfReader.HasUsageRights() and PdfReader.RemoveUsageRights() in iTextSharp v5. Can't seem to find similar functionality in iText7?
There is probably no direct alternative but it's easy to implement those methods yourself:
public boolean hasUsageRights(PdfDocument pdfDocument) {
PdfDictionary perms = pdfDocument.getCatalog().getPdfObject().getAsDictionary(PdfName.Perms);
if (perms == null) {
return false;
}
return perms.containsKey(new PdfName("UR")) || perms.containsKey(PdfName.UR3);
}
public void removeUsageRights(PdfDocument pdfDocument) {
PdfDictionary perms = pdfDocument.getCatalog().getPdfObject().getAsDictionary(PdfName.Perms);
if (perms == null) {
return;
}
perms.remove(new PdfName("UR"));
perms.remove(PdfName.UR3);
if (perms.size() == 0) {
pdfDocument.getCatalog().remove(PdfName.Perms);
}
}
If you need the first method then you can pass either a document created with PdfDocument(PdfReader, PdfWriter) constructor or with PdfDocument(PdfReader) one. If you need the second method then you need to pass a document created in stamping mode, i.e. with PdfDocument(PdfReader, PdfWriter) constructor

Setting Field Within Class using Reflection

This should be easy but it's managed to confound me a few times. I'm trying to set a Value within a class using Reflection.
public class EngineeringValueClass<T> {
public T Value { get { } set { } }
}
Then in a calling class I have:
public class MyClass {
EngineeringValueClass<double> Value1;
EngineeringValueClass<double> Value2;
// Along with many others.
public void SetValueByName(object FieldName,object FieldValue) {
// Get the "Value" field of a generic EngineeringValueClass<double>
PropertyInfo MyValuePropRef =
typeof(EngineeringValueClass<double>).GetProperty("Value");
// Get the field within this class I want to set.
FieldInfo MyNameFieldRef = typeof(MyClass).GetField(FieldName.ToString());
MyValuePropRef.SetValue(MyNameFieldRef.GetValue,
FieldValue,
null);
}
}
My goal is to have
SetValueByName("Value1",2.3);
set Value1's "Value" using the set accessor. I presume my problem is that MyNameFieldRef.GetValue doesn't return an object reference but rather a "Value", but I'm not sure how to work around that. I don't want to pass "this" because that's not right either.
Okay, I finally figured this out:
public void SetValueByName(object FieldName, object FieldValue) {
Type t = typeof(MyClass);
FieldInfo PrimaryField = t.GetField(FieldName.ToString());
object ValueField = PrimaryField.GetValue(this);
// To get the type of Value
MethodInfo TypeValueField = ValueField.GetType().GetMethod("GetValueType");
Type ValueType = (Type) TypeValueField.Invoke(ValueField, null);
// I added a "GetValueType () { return typeof(T); } to EngineeringValueClass
if (ValueType == typeof(Int16))
{
((EngineeringValueClass<Int16>)ValueField).Value = Int16.Parse(FieldValue.ToString());
}...
}

Is this a good pattern for PATCH

I am implementing a REST style API that allows an object to be PATCH'ed. The intention of the PATCH operation is to allow one or more properties in a class to be updated without touching an of the other properties that may be set.
The are examples of partial updates on the ServiceStack OrmLite page, but it seems to need hardcoding to indicate which fields will be partially updated. In my scenario it is upto the depend application to decide which fields to send.
I also have to cope with the scenario that the object may not have been persisted yet.
To get around this I have implemented the following:
public object Patch(Myclass request)
{
HttpStatusCode SuccessCode;
try
{
var result = (MyClass)Get(request);
if (result != null)
{
request.PopulateWithNonDefaultValues(result);
dbFactory.Run(dbCmd => dbCmd.UpdateNonDefaults(request, r => r.myId == request.myId));
}
else
{
dbFactory.Run(dbCmd => dbCmd.Save(request));
}
SuccessCode = HttpStatusCode.Accepted;
}
catch (Exception e)
{
log.Error(e);
SuccessCode = HttpStatusCode.InternalServerError;
}
return new HttpResult()
{
StatusCode = SuccessCode
};
}
It works, but something doesn't feel right, I'm sure there must be a better way?
That looks ok although you're code will be a lot shorter if you just throw let it throw C# Exceptions when there's an error and if you're inheriting from ServiceStack's New API base Service class you can use the already available Db property, e.g:
public object Patch(Myclass request)
{
var result = (MyClass)Get(request);
if (result != null)
{
request.PopulateWithNonDefaultValues(result);
Db.UpdateNonDefaults(request, r => r.myId == request.myId);
}
else
{
Db.Save(request);
}
return new HttpResult
{
StatusCode = HttpStatusCode.Accepted
};
}

How do I set multiple error messages for different scenarios in a Custom validation attribute?

I'm just getting to grips with custom validation attributes, and I'm trying to write a custom validation attirbute which will be placed at class level to validate against multiple properties of my model.
I can access all properties on my model, and I want to be able to check for multiple conditions in my IsValid overload, and report on them, having different error messages as follows (simplistic example).
public override bool IsValid(object value)
{
var model = (MyObject) value;
//if this value is set, I don't want to do anything other checks
if (model.Prop3)
{
return true;
}
if (model.Prop1 == "blah" && model.Prop2 == 1)
{
ErrorMessage = "you can't enter blah if prop 2 equals 1";
return false;
}
if(model.Prop1 == "blah blah" && model.Prop2 == 2)
{
ErrorMessage = "you can't enter blah blah if prop 2 equals 2";
return false;
}
return true;
}
But when I do this I get an exception on the first time ErrorMessage is referenced "Cannot set property more than once.
Now I could split up my custom attribute into multiple custom attributes, but hoped there would be a way to do it in one, otherwise, I'll be repeating my "catch all" in each
//if this value is set, I don't want to do anything other checks
if (model.Prop3)
{
return true;
}
I've had a search already, but couldn't find anything, so apologies if I am missing anything obvious.
thanks in advance!
In MVC4 you can override IsValid to return different messages as the ValidationResult
public class StrongPasswordAttribute : ValidationAttribute
{
protected override ValidationResult IsValid(object value, ValidationContext context)
{
if (value == null)
return new ValidationResult("Password is required");
var val = value.ToString();
if (!Regex.Match(val, #"^(?=.*[a-z]).{0,}$").Success)
{
return new ValidationResult("Password must contain at least one lower case letter");
}
if (!Regex.Match(val, #"^(?=.*[A-Z]).{0,}$").Success)
{
return new ValidationResult("Password must contain at least one UPPER case letter");
}
if (!Regex.Match(val, #"^(?=.*\d).{0,}$").Success)
{
return new ValidationResult("Password must contain at least one number");
}
return ValidationResult.Success;
}
}
Interesting question! I can think of two work-arounds to this. So not proper solutions based on what you want but they might help to re-use your code. Cant you create a CustomAttribute abstract class called MyCustomAttribute (or something) that overrides IsValid in the following way:
public override bool IsValid(object value)
{
var model = (MyObject) value;
//if this value is set, I don't want to do anything other checks
if (model.Prop3)
{
return true;
}
CustomValidate(model);
}
CustomValidate(MyObject model) is your abstract method then, you can write multiple custom attribute classes that extend MyCustomAttribute and purely need to implement the validation logic for A particular scenario.
So you can have two classes:
public class BlahCustomAttribute : MyCustomAttribute
{
public override Boolean CustomValidate(MyObject obj)
{
if (model.Prop1 == "blah" && model.Prop2 == 1)
{
ErrorMessage = "you can't enter blah if prop 2 equals 1";
return false;
}
}
}
public class BlahBlahCustomAttribute : MyCustomAttribute
{
public override Boolean CustomValidate(MyObject obj)
{
if (model.Prop1 == "blah" && model.Prop2 == 1)
{
ErrorMessage = "you can't enter blah blah if prop 2 equals 1";
return false;
}
}
}
Hope this helps - not exactly what you wanted but will do the job and its clean as well.
The other solution is to comma-separate the error messages in the ErrorMessage property and handle it in the front-end (but I would go with the first approach).

Serializing Entity Framework problems

Like several other people, I'm having problems serializing Entity Framework objects, so that I can send the data over AJAX in a JSON format.
I've got the following server-side method, which I'm attempting to call using AJAX through jQuery
[WebMethod]
public static IEnumerable<Message> GetAllMessages(int officerId)
{
SIBSv2Entities db = new SIBSv2Entities();
return (from m in db.MessageRecipients
where m.OfficerId == officerId
select m.Message).AsEnumerable<Message>();
}
Calling this via AJAX results in this error:
A circular reference was detected while serializing an object of type \u0027System.Data.Metadata.Edm.AssociationType
Which is because of the way the Entity Framework creates circular references to keep all the objects related and accessible server side.
I came across the following code from (http://hellowebapps.com/2010-09-26/producing-json-from-entity-framework-4-0-generated-classes/) which claims to get around this problem by capping the maximum depth for references. I've added the code below, because I had to tweak it slightly to get it work (All angled brackets are missing from the code on the website)
using System.Web.Script.Serialization;
using System.Collections.Generic;
using System.Collections;
using System.Linq;
using System;
public class EFObjectConverter : JavaScriptConverter
{
private int _currentDepth = 1;
private readonly int _maxDepth = 2;
private readonly List<int> _processedObjects = new List<int>();
private readonly Type[] _builtInTypes = new[]{
typeof(bool),
typeof(byte),
typeof(sbyte),
typeof(char),
typeof(decimal),
typeof(double),
typeof(float),
typeof(int),
typeof(uint),
typeof(long),
typeof(ulong),
typeof(short),
typeof(ushort),
typeof(string),
typeof(DateTime),
typeof(Guid)
};
public EFObjectConverter( int maxDepth = 2,
EFObjectConverter parent = null)
{
_maxDepth = maxDepth;
if (parent != null)
{
_currentDepth += parent._currentDepth;
}
}
public override object Deserialize( IDictionary<string,object> dictionary, Type type, JavaScriptSerializer serializer)
{
return null;
}
public override IDictionary<string,object> Serialize(object obj, JavaScriptSerializer serializer)
{
_processedObjects.Add(obj.GetHashCode());
Type type = obj.GetType();
var properties = from p in type.GetProperties()
where p.CanWrite &&
p.CanWrite &&
_builtInTypes.Contains(p.PropertyType)
select p;
var result = properties.ToDictionary(
property => property.Name,
property => (Object)(property.GetValue(obj, null)
== null
? ""
: property.GetValue(obj, null).ToString().Trim())
);
if (_maxDepth >= _currentDepth)
{
var complexProperties = from p in type.GetProperties()
where p.CanWrite &&
p.CanRead &&
!_builtInTypes.Contains(p.PropertyType) &&
!_processedObjects.Contains(p.GetValue(obj, null)
== null
? 0
: p.GetValue(obj, null).GetHashCode())
select p;
foreach (var property in complexProperties)
{
var js = new JavaScriptSerializer();
js.RegisterConverters(new List<JavaScriptConverter> { new EFObjectConverter(_maxDepth - _currentDepth, this) });
result.Add(property.Name, js.Serialize(property.GetValue(obj, null)));
}
}
return result;
}
public override IEnumerable<System.Type> SupportedTypes
{
get
{
return GetType().Assembly.GetTypes();
}
}
}
However even when using that code, in the following way:
var js = new System.Web.Script.Serialization.JavaScriptSerializer();
js.RegisterConverters(new List<System.Web.Script.Serialization.JavaScriptConverter> { new EFObjectConverter(2) });
return js.Serialize(messages);
I'm still seeing the A circular reference was detected... exception being thrown!
I solved these issues with the following classes:
public class EFJavaScriptSerializer : JavaScriptSerializer
{
public EFJavaScriptSerializer()
{
RegisterConverters(new List<JavaScriptConverter>{new EFJavaScriptConverter()});
}
}
and
public class EFJavaScriptConverter : JavaScriptConverter
{
private int _currentDepth = 1;
private readonly int _maxDepth = 1;
private readonly List<object> _processedObjects = new List<object>();
private readonly Type[] _builtInTypes = new[]
{
typeof(int?),
typeof(double?),
typeof(bool?),
typeof(bool),
typeof(byte),
typeof(sbyte),
typeof(char),
typeof(decimal),
typeof(double),
typeof(float),
typeof(int),
typeof(uint),
typeof(long),
typeof(ulong),
typeof(short),
typeof(ushort),
typeof(string),
typeof(DateTime),
typeof(DateTime?),
typeof(Guid)
};
public EFJavaScriptConverter() : this(1, null) { }
public EFJavaScriptConverter(int maxDepth = 1, EFJavaScriptConverter parent = null)
{
_maxDepth = maxDepth;
if (parent != null)
{
_currentDepth += parent._currentDepth;
}
}
public override object Deserialize(IDictionary<string, object> dictionary, Type type, JavaScriptSerializer serializer)
{
return null;
}
public override IDictionary<string, object> Serialize(object obj, JavaScriptSerializer serializer)
{
_processedObjects.Add(obj.GetHashCode());
var type = obj.GetType();
var properties = from p in type.GetProperties()
where p.CanRead && p.GetIndexParameters().Count() == 0 &&
_builtInTypes.Contains(p.PropertyType)
select p;
var result = properties.ToDictionary(
p => p.Name,
p => (Object)TryGetStringValue(p, obj));
if (_maxDepth >= _currentDepth)
{
var complexProperties = from p in type.GetProperties()
where p.CanRead &&
p.GetIndexParameters().Count() == 0 &&
!_builtInTypes.Contains(p.PropertyType) &&
p.Name != "RelationshipManager" &&
!AllreadyAdded(p, obj)
select p;
foreach (var property in complexProperties)
{
var complexValue = TryGetValue(property, obj);
if(complexValue != null)
{
var js = new EFJavaScriptConverter(_maxDepth - _currentDepth, this);
result.Add(property.Name, js.Serialize(complexValue, new EFJavaScriptSerializer()));
}
}
}
return result;
}
private bool AllreadyAdded(PropertyInfo p, object obj)
{
var val = TryGetValue(p, obj);
return _processedObjects.Contains(val == null ? 0 : val.GetHashCode());
}
private static object TryGetValue(PropertyInfo p, object obj)
{
var parameters = p.GetIndexParameters();
if (parameters.Length == 0)
{
return p.GetValue(obj, null);
}
else
{
//cant serialize these
return null;
}
}
private static object TryGetStringValue(PropertyInfo p, object obj)
{
if (p.GetIndexParameters().Length == 0)
{
var val = p.GetValue(obj, null);
return val;
}
else
{
return string.Empty;
}
}
public override IEnumerable<Type> SupportedTypes
{
get
{
var types = new List<Type>();
//ef types
types.AddRange(Assembly.GetAssembly(typeof(DbContext)).GetTypes());
//model types
types.AddRange(Assembly.GetAssembly(typeof(BaseViewModel)).GetTypes());
return types;
}
}
}
You can now safely make a call like new EFJavaScriptSerializer().Serialize(obj)
Update : since version Telerik v1.3+ you can now override the GridActionAttribute.CreateActionResult method and hence you can easily integrate this Serializer into specific controller methods by applying your custom [GridAction] attribute:
[Grid]
public ActionResult _GetOrders(int id)
{
return new GridModel(Service.GetOrders(id));
}
and
public class GridAttribute : GridActionAttribute, IActionFilter
{
/// <summary>
/// Determines the depth that the serializer will traverse
/// </summary>
public int SerializationDepth { get; set; }
/// <summary>
/// Initializes a new instance of the <see cref="GridActionAttribute"/> class.
/// </summary>
public GridAttribute()
: base()
{
ActionParameterName = "command";
SerializationDepth = 1;
}
protected override ActionResult CreateActionResult(object model)
{
return new EFJsonResult
{
Data = model,
JsonRequestBehavior = JsonRequestBehavior.AllowGet,
MaxSerializationDepth = SerializationDepth
};
}
}
and finally..
public class EFJsonResult : JsonResult
{
const string JsonRequest_GetNotAllowed = "This request has been blocked because sensitive information could be disclosed to third party web sites when this is used in a GET request. To allow GET requests, set JsonRequestBehavior to AllowGet.";
public EFJsonResult()
{
MaxJsonLength = 1024000000;
RecursionLimit = 10;
MaxSerializationDepth = 1;
}
public int MaxJsonLength { get; set; }
public int RecursionLimit { get; set; }
public int MaxSerializationDepth { get; set; }
public override void ExecuteResult(ControllerContext context)
{
if (context == null)
{
throw new ArgumentNullException("context");
}
if (JsonRequestBehavior == JsonRequestBehavior.DenyGet &&
String.Equals(context.HttpContext.Request.HttpMethod, "GET", StringComparison.OrdinalIgnoreCase))
{
throw new InvalidOperationException(JsonRequest_GetNotAllowed);
}
var response = context.HttpContext.Response;
if (!String.IsNullOrEmpty(ContentType))
{
response.ContentType = ContentType;
}
else
{
response.ContentType = "application/json";
}
if (ContentEncoding != null)
{
response.ContentEncoding = ContentEncoding;
}
if (Data != null)
{
var serializer = new JavaScriptSerializer
{
MaxJsonLength = MaxJsonLength,
RecursionLimit = RecursionLimit
};
serializer.RegisterConverters(new List<JavaScriptConverter> { new EFJsonConverter(MaxSerializationDepth) });
response.Write(serializer.Serialize(Data));
}
}
You can also detach the object from the context and it will remove the navigation properties so that it can be serialized. For my data repository classes that are used with Json i use something like this.
public DataModel.Page GetPage(Guid idPage, bool detach = false)
{
var results = from p in DataContext.Pages
where p.idPage == idPage
select p;
if (results.Count() == 0)
return null;
else
{
var result = results.First();
if (detach)
DataContext.Detach(result);
return result;
}
}
By default the returned object will have all of the complex/navigation properties, but by setting detach = true it will remove those properties and return the base object only. For a list of objects the implementation looks like this
public List<DataModel.Page> GetPageList(Guid idSite, bool detach = false)
{
var results = from p in DataContext.Pages
where p.idSite == idSite
select p;
if (results.Count() > 0)
{
if (detach)
{
List<DataModel.Page> retValue = new List<DataModel.Page>();
foreach (var result in results)
{
DataContext.Detach(result);
retValue.Add(result);
}
return retValue;
}
else
return results.ToList();
}
else
return new List<DataModel.Page>();
}
I have just successfully tested this code.
It may be that in your case your Message object is in a different assembly? The overriden Property SupportedTypes is returning everything ONLY in its own Assembly so when serialize is called the JavaScriptSerializer defaults to the standard JavaScriptConverter.
You should be able to verify this debugging.
Your error occured due to some "Reference" classes generated by EF for some entities with 1:1 relations and that the JavaScriptSerializer failed to serialize.
I've used a workaround by adding a new condition :
!p.Name.EndsWith("Reference")
The code to get the complex properties looks like this :
var complexProperties = from p in type.GetProperties()
where p.CanWrite &&
p.CanRead &&
!p.Name.EndsWith("Reference") &&
!_builtInTypes.Contains(p.PropertyType) &&
!_processedObjects.Contains(p.GetValue(obj, null)
== null
? 0
: p.GetValue(obj, null).GetHashCode())
select p;
Hope this help you.
I had a similar problem with pushing my view via Ajax to UI components.
I also found and tried to use that code sample you provided. Some problems I had with that code:
SupportedTypes wasn't grabbing the types I needed, so the converter wasn't being called
If the maximum depth is hit, the serialization would be truncated
It threw out any other converters I had on the existing serializer by creating its own new JavaScriptSerializer
Here are the fixes I implemented for those issues:
Reusing the same serializer
I simply reused the existing serializer that is passed into Serialize to solve this problem. This broke the depth hack though.
Truncating on already-visited, rather than on depth
Instead of truncating on depth, I created a HashSet<object> of already seen instances (with a custom IEqualityComparer that checked reference equality). I simply didn't recurse if I found an instance I'd already seen. This is the same detection mechanism built into the JavaScriptSerializer itself, so worked quite well.
The only problem with this solution is that the serialization output isn't very deterministic. The order of truncation is strongly dependent on the order that reflections finds the properties. You could solve this (with a perf hit) by sorting before recursing.
SupportedTypes needed the right types
My JavaScriptConverter couldn't live in the same assembly as my model. If you plan to reuse this converter code, you'll probably run into the same problem.
To solve this I had to pre-traverse the object tree, keeping a HashSet<Type> of already seen types (to avoid my own infinite recursion), and pass that to the JavaScriptConverter before registering it.
Looking back on my solution, I would now use code generation templates to create a list of the entity types. This would be much more foolproof (it uses simple iteration), and have much better perf since it would produce a list at compile time. I'd still pass this to the converter so it could be reused between models.
My final solution
I threw out that code and tried again :)
I simply wrote code to project onto new types ("ViewModel" types - in your case, it would be service contract types) before doing my serialization. The intention of my code was made more explicit, it allowed me to serialize just the data I wanted, and it didn't have the potential of slipping in queries on accident (e.g. serializing my whole DB).
My types were fairly simple, and I didn't need most of them for my view. I might look into AutoMapper to do some of this projection in the future.