How to convert SinkRecord to JSON string? - apache-kafka

Imagine myAPICreate requires a JSON string.
public void put(Collection<SinkRecord> collection) {
for (SinkRecord record : collection) {
JSONObject recordJson = toJSON(record.value());
String recordJsonString = recordJson.toString();
myAPICreate(recordJsonString);
}
}
toJSON is a helper I have defined which just takes the record and returns a JSONObject.
JSONObject json = new JSONObject()
.put("a", record.getString("a"))
.put("b", record.getString("b"))
.put("c", record.getString("c"));
I feel like I might be doing a lot of redundant work here. Is it necessary to have the code in put convert it to JSON or is there a way to use the converters so that record already comes in as JSON or a JSON string? Then I can just pass myAPICreate(record.value().toString()) without having to manually do it?

When you create a SinkRecord, you have a key & value schema w/ a key and value Object. Those objects should be Struct instances that must be created with the matching Schema
In the Connector configuration, you would then use JSONConverter (or other converter) to get the serialized output

Related

Deserialize request body to specific class instead of JsonObject

Say we have this:
Router router = Router.router(vertx);
router.put("/products/:productID").handler(this::handleAddProduct);
and this:
private void handleAddProduct(RoutingContext ctx) {
String productID = ctx.request().getParam("productID");
HttpServerResponse response = ctx.response();
JsonObject product = ctx.getBodyAsJson();
products.put(productID, product);
response.end();
}
my question is - how can we deserialize ctx.getBodyAsJson() to a specific Java class instead of the generic JsonObject class?
you can use JsonObject.mapTo(Class), e.g.:
JsonObject product = ctx.getBodyAsJson();
Product instance = product.mapTo(Product.class);
UPDATE
you can customize the (de)serialization behavior by manipulating the ObjectMapper instance(s) associated with the Json class. here are some examples:
// only serialize non-null values
Json.mapper.setSerializationInclusion(JsonInclude.Include.NON_NULL);
// ignore values that don't map to a known field on the target type
Json.mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
keep in mind Json holds a reference to two different ObjectMappers:
mapper, and
prettyMapper

Use GuidRepresentation.Standard with MongoDB

I am implementing a custom IBsonSerializer with the official MongoDB driver (C#). I am in the situation where I must serialize and deserialize a Guid.
If I implement the Serialize method as follow, it works:
public void Serialize(BsonWriter bsonWriter, Type nominalType, object value, IBsonSerializationOptions options)
{
BsonBinaryData data = new BsonBinaryData(value, GuidRepresentation.CSharpLegacy);
bsonWriter.WriteBinaryData(data);
}
However I don't want the Guid representation to be CSharpLegacy, I want to use the standard representation. But if I change the Guid representation in that code, I get the following error:
MongoDB.Bson.BsonSerializationException: The GuidRepresentation for the writer is CSharpLegacy, which requires the subType argument to be UuidLegacy, not UuidStandard.
How do I serialize a Guid value using the standard representation?
Old question but in case someone finds it on google like I did...
Do this once:
BsonDefaults.GuidRepresentation = GuidRepresentation.Standard;
For example, in a Web Application/Web API, your Global.asax.cs file is best place to add it once
public class WebApiApplication : System.Web.HttpApplication
{
protected void Application_Start()
{
BsonDefaults.GuidRepresentation = GuidRepresentation.Standard;
//Other code...below
}
}
If you don't want to modify the global setting BsonDefaults.GuidRepresentation (and you shouldn't, because modifying globals is a bad pattern), you can specify the setting when you create your collection:
IMongoDatabase db = ???;
string collectionName = ???;
var collectionSettings = new MongoCollectionSettings {
GuidRepresentation = GuidRepresentation.Standard
};
var collection = db.GetCollection<BsonDocument>(collectionName, collectionSettings);
Then any GUIDs written to the collection will be in the standard format.
Note that when you read records from the database, you will get a System.FormatException if the GUID format in the database is different from the format in your collection settings.
It looks like what's happening is when you are not explicitly passing the GuidRepresentation to BsonBinaryData constructor, it defaults to passing GuidRepresentation.Unspecified and that ultimately maps to GuidRepresentation.Legacy (see this line in the source)
So you need to explicitly pass the guidRepresentation as a third argument to BsonBinaryData set to GuidRepresentation.Standard.
edit: As was later pointed out, you can set BsonDefaults.GuidRepresentation = GuidRepresentation.Standard if that's what you always want to use.

EF Code First - Map Dictionary or custom type as an nvarchar

I want to use EF Code first for a database that I'm currently accessing using plain old ADO.NET with stored procedures.
In my database I have some nvarchar(MAX) columns that should be mapped to and from a Dictionary<string, string>.
When saved to database, it is an XML formatted string. I use this technique to allow internationalization of e.g. a name of a product in an online store. I don't know how many languages any given user want to translate to so I can't have a Name column for each language.
I also wanted to avoid storing the values in a seperate table, so I ended up with the Dictionary - XML approach.
The way I do it now, is to just treat any of these columns as a string whenever I interact with the database. I have a custom mapper function that can turn the XML into a Dictionary, and back to XML.
But I can't seem to find a way to do this with EF Code first? Any ideas?
You can add a property that will return your Dictionary<,> as a XML string and then remove the mapping for your Dictionary<,> property.
[NotMapped]
public Dictionary<string,string> MyDictionary
{
get; set;
}
public string DictionaryAsXml
{
get
{
return ToXml(MyDictionary);
}
set
{
MyDictionary = FromXml(value);
}
}
If you don't want to expose your DictionaryAsXml property have a look at this blog post. It shows how you can persist private and protected properties.
I had some difficulties with the xml conversion in VB.NET. Therefore I took advantage of newtonsoft.json to serialize the dictionary to a JSON string and back.
Public Property JsonDict As String
Get
If MyDict Is Nothing Then
Return Nothing
Else
Return JsonConvert.SerializeObject(MyDict)
End If
End Get
Set(value As String)
If value Is Nothing Then
MyDict = Nothing
Else
MyDict = JsonConvert.DeserializeObject(Of Dictionary(Of Single, Single))(value)
End If
End Set
End Property
<NotMapped>
Public Property MyDict As Dictionary(Of Single, Single)

Saving a Hashtable using Mongo & the Play framework?

I've got a model defined like the following...
#MongoEntity
public class Ent extends MongoModel{
public Hashtable<Integer, CustomType> fil;
public int ID;
public Ent(){
fil = new Hashtable<Integer, CustomType>();
}
}
CustomType is a datatype I've created which basically holds a list of items (among other things). At some point in my web application I update the hashtable from a controller and then read back the size of the item I just updated. Like the following...
public static void addToHash(CustomType type, int ID, int key){
//First I add an element to the list I'm storing in custom type.
Ent ent = Ent.find("byID",ID).first();
CustomType element = user.fil.get(key);
if(element == null) element = new CustomType();
element.add(type);
ent.save();
//Next I reset the variables and read back the value I just stored..
ent = null;
ent = User.find("byID",ID).first();
element = ent.fil.get(ID);
System.out.println("SIZE = " + element.size()); //null pointer here
}
As you can see by my above example I add the element, save the model and then attempt to read back what I have just added and it has not been saved. The above model Ent is a minimal version of the entire Model I'm actually using. All other values in the model including List's, String's, Integer's etc. update correctly when they're updated but this Hashtable I'm storing isn't. Why would this be happening and how could I correct it?
You should probably post on the play framework forum for better help..
Alternatives for a mongodb framework are morphia and springdata which have good documentation.
Not sure how Play maps a hash table to a document value, but it seems it cannot update just the hash table using a mongo operator.
You should be able to mark the whole document for update which would work but slower.

Return raw string from REST service method

I have a REST service method written in C#, defined as below:
[WebGet(UriTemplate = "/{par1}/{par2}/{par3}")]
public string ProcessGet(string par1, string par2, string par3)
{
return Execute(...);
}
It should return result as XML or JSON, based on one parameter (I generate the json and XML serialization)
How can I make this method to return the RAW string, just as I created it, without HTMLEncoding it?
Thank you
Return it as a Stream - that causes the "raw" mode to be used and WCF will not touch your response. You can find more information at http://blogs.msdn.com/b/carlosfigueira/archive/2008/04/17/wcf-raw-programming-model-web.aspx.