Inner destructured property not received by ELK stack as destructured, but as escaped string - azure-service-fabric

I'm not sure if this is a Serilog, ELK, Service Fabric, code or config issue.
I am writing a Service Fabric Stateless Service. My logging configuration line is this:
Logger = new LoggerConfiguration()
.WriteTo.EventFlow(loggingOptions.DiagnosticPipeline)
.Destructure.With<JsonNetDestructuringPolicy>()
.Enrich.FromLogContext()
.CreateLogger()
.ForContext(properties);
...where properties is an array of PropertyEnricher and the Policy is from Destructurama's JsonNetDestructuringPolicy.
I have a custom object base class that is successfully being destructured, so if I call it Data, then in ELK's JSON tab I see:
"payload": {
"Data": {
"Property1": "Prop Value",
"Property2": "Prop2 Value"
}
}
However, when one of the inner objects is also destructured, it is sent as an escaped JSON string instead of being destructured, with no quotes around the property names:
"payload": {
"Data": {
"Property1": "Prop Value",
"DestructuredProp": "{InnerProperty: \"Inner Value\"}"
}
}
What I expected was:
"payload": {
"Data": {
"Property1": "Prop Value",
"DestructuredProp": {
"InnerProperty": "Inner Value"
}
}
}
I don't know why the inner property names are not given quotes, or why the entire value is being escaped and quoted instead of destructured.
I have verified that my destructuring code is being executed. I can manually add quotes around the property name, for example, but it just results in more escaped quotes in the inner value.
My own code was destructuring it directly from C#. I thought it might have been a bug in my destructuring code, so I looked around some more and found Destructurama's JsonNetDestructuringPolicy, so I tried that, converting my object with JObject.fromObject(), but the same thing happens with it.
I'm pretty sure I should be able to do this with Serilog. I don't think there would be a depth limit setting if it couldn't do more than one layer deep. Why doesn't this work? I have tried refreshing the field index in Kibana, but the JSON view shows the escaped string, so I'm pretty sure it is being sent incorrectly, and isn't an ELK issue.
---EDIT---
Here is a destructuring policy I tried. My initial object is JsonEvent, and it has a Dictionary that is not destructuring, even though the Dictionayr policy is successfully being invoked.
public class JsonEventDestructuringPolicy : IDestructuringPolicy
{
public bool TryDestructure(object value, ILogEventPropertyValueFactory propertyValueFactory, out LogEventPropertyValue result)
{
if (value is JsonEvent jsonEvent)
{
var properties = new List<LogEventProperty>();
foreach (var property in value.GetType().GetProperties())
{
var propertyValue = property.GetValue(value);
var isCollection = propertyValue is ICollection<Dictionary<string,string>>;
var isDictionary = propertyValue is Dictionary<string,string>;
if (isCollection)
LoggingContext.Message("Found collection of dictionary: " + property.Name);
else if (isDictionary)
LoggingContext.Message("Found dictionary: " + property.Name);
else if (property.Name.Equals("Parameters"))
LoggingContext.Message("Found Parameters: " + propertyValue.GetType());
if (propertyValue != null)
properties.Add(new LogEventProperty(property.Name, propertyValueFactory.CreatePropertyValue(propertyValue, isCollection || isDictionary)));
}
result = new StructureValue(properties);
return true;
}
if (value is Dictionary<string, string> dictionary)
{
var properties = new List<LogEventProperty>();
foreach (var kvp in dictionary)
{
if (!string.IsNullOrWhiteSpace(kvp.Value))
properties.Add(new LogEventProperty("\"" + kvp.Key + "\"", propertyValueFactory.CreatePropertyValue(kvp.Value)));
}
result = new StructureValue(properties);
return true;
}
result = null;
return false;
}
}
It is being invoked like this:
public static void Message(JsonEvent message)
{
Logger.ForContext(GetEnrichers(message))
.Information(message.Event);
}
private static IEnumerable<ILogEventEnricher> GetEnrichers(JsonEvent message)
{
return new List<ILogEventEnricher>()
.Add("Data", message, true)
.Add("CorrelationId", ServiceTracingContext.CorrelationId)
.Add("CorrelationDateTime", ServiceTracingContext.CorrelationDateTime)
.Add("RouteTemplate", ServiceTracingContext.RouteTemplate)
.ToArray();
}

Related

Resolve a standalone string/file using typesafe config

I'm looking for a way to say:
val c: Config = ConfigFactory.parseString("a=fox,b=dog")
val s: String = """This is a "quick" brown ${a}.\nThat is a lazy, lazy ${b}."""
println(c.resolveString(s))
// Should print:
// > This is a "quick" brown fox.
// > That is a lazy lazy dog.
My two ideas:
Just find the placeholders with regex and replace from config one by one
convert s to config with single value and use resolveWith - but it seems quoting can be really tricky
Maybe there is an easier way?
A naive solution:
class Resolver(vars: Config) {
private lazy val placeholderRegex = "(?<=\\$\\{).*?(?=\\})".r
def resolveString(s: String): String = {
placeholderRegex.findAllIn(s).foldLeft(s) { (str, v) =>
if (vars.hasPath(v)) str.replaceAll("\\Q${" + v + "}\\E", vars.getString(v)) else str
}
}
It should be fine if the string is not huge and there are no insane numbers of distinct placeholders in it.
I have a similar situation where I automatically push elastic search index definitions before
I launch the job which uses those index definitions.
In my case the string which contains variable references is the
JSON index/template definition, which ALSO comes from the typesafe config.
(see es.template_or_index.json, below.)
I resolve these refs using the utility method I wrote on top
of apache.commons StrSubstitutor.
(see: VariableReferenceResolver.resolveReferences(String template), below)
Below is a sample of my config. Note how the property 'es.animal' is injected into the index/template json.
This can't be done with out-of-the-box features of the typesafe config library (but I think this would
be a wonderful feature for them to add !)
es {
// user and password credentials should not be checked in to git. deployer is expected to
// set these parameters into the their environment in whatever way is convenient -- .bashrc or whatever.
user = dummy
user = ${?ES_USER}
password = dummy.passwd
password = ${?ES_PASSWORD}
hostPort = "localhost:9200"
hostPort = ${?ES_HOST_PORT}
protocol = http
protocol = ${?ES_PROTOCOL}
animal = horse // This gets injected into the configuration property es.template_or_index.json
// Note: for template json there is a second round of interpolation performed so the json can reference any defined
// property of this configuration file (or anything it includes)
template_or_index {
name = test_template
json = """
{
"template": "${es.animal}_sql-*",
"settings": {
"number_of_shards": 50,
"number_of_replicas": 2
},
"mappings": {
"test_results" : {
"date_detection": false,
"properties" : {
"timestamp" : { "type" : "date"},
"yyyymmdd" : { "type" : "string", "index" : "not_analyzed"}
}
}
}
}
"""
}
}
package com.foo
import com.typesafe.config.Config;
import org.apache.commons.lang.text.StrLookup;
import org.apache.commons.lang.text.StrSubstitutor;
public class VariableReferenceResolver {
final StrSubstitutor substitutor;
static class ConfigStrLookup extends StrLookup {
private final Config config;
ConfigStrLookup(Config config) {
this.config = config;
}
public String lookup(String key) {
return config.getString(key);
}
}
public VariableReferenceResolver (Config config) {
substitutor=new StrSubstitutor(new ConfigStrLookup(config));
}
public String resolveReferences(String template) {
return substitutor.replace(template);
}
}
public class OtherClass {
private static void getIndexConfiguration(String path) throws IOException {
System.setProperty("config.file", path);
Config config = ConfigFactory.load();
String user = config.getString("es.user");
String password = config.getString("es.password");
String protocol = config.getString("es.protocol");
String hostPort = config.getString("es.hostPort");
String indexOrTemplateJson = config.getString("es.template_or_index.json");
String indexOrTemplateName = config.getString("es.template_or_index.name");
VariableReferenceResolver resolver = new VariableReferenceResolver(config);
String resolvedIndexOrTemplateJson = resolver.resolveReferences(indexOrTemplateJson);
File jsonFile = File.createTempFile("index-or-template-json", indexOrTemplateName);
Files.write(Paths.get(jsonFile.getAbsolutePath()), resolvedIndexOrTemplateJson.getBytes());
curlIndexOrTemplateCreateCommand =
String.format(
"curl -XPUT -k -u %s:%s %s://%s/_template/%s -d #%s",
user, password, protocol, hostPort, indexOrTemplateName, jsonFile.getAbsolutePath());
}
....
}

Build dynamic LINQ queries from a string - Use Reflection?

I have some word templates(maybe thousands). Each template has merge fields which will be filled from database. I don`t like writing separate code for every template and then build the application and deploy it whenever a template is changed or a field on the template is added!
Instead, I'm trying to define all merge fields in a separate xml file and for each field I want to write the "query" which will be called when needed. EX:
mergefield1 will call query "Case.Parties.FirstOrDefault.NameEn"
mergefield2 will call query "Case.CaseNumber"
mergefield3 will call query "Case.Documents.FirstOrDefault.DocumentContent.DocumentType"
Etc,
So, for a particular template I scan its merge fields, and for each merge field I take it`s "query definition" and make that request to database using EntityFramework and LINQ. Ex. it works for these queries: "TimeSlots.FirstOrDefault.StartDateTime" or
"Case.CaseNumber"
This will be an engine which will generate word documents and fill it with merge fields from xml. In addition, it will work for any new template or new merge field.
Now, I have worked a version using reflection.
public string GetColumnValueByObjectByName(Expression<Func<TEntity, bool>> filter = null, string objectName = "", string dllName = "", string objectID = "", string propertyName = "")
{
string objectDllName = objectName + ", " + dllName;
Type type = Type.GetType(objectDllName);
Guid oID = new Guid(objectID);
dynamic Entity = context.Set(type).Find(oID); // get Object by Type and ObjectID
string value = ""; //the value which will be filled with data from database
IEnumerable<string> linqMethods = typeof(System.Linq.Enumerable).GetMethods(BindingFlags.Static | BindingFlags.Public).Select(s => s.Name).ToList(); //get all linq methods and save them as list of strings
if (propertyName.Contains('.'))
{
string[] properies = propertyName.Split('.');
dynamic object1 = Entity;
IEnumerable<dynamic> Child = new List<dynamic>();
for (int i = 0; i < properies.Length; i++)
{
if (i < properies.Length - 1 && linqMethods.Contains(properies[i + 1]))
{
Child = type.GetProperty(properies[i]).GetValue(object1, null);
}
else if (linqMethods.Contains(properies[i]))
{
object1 = Child.Cast<object>().FirstOrDefault(); //for now works only with FirstOrDefault - Later it will be changed to work with ToList or other linq methods
type = object1.GetType();
}
else
{
if (linqMethods.Contains(properies[i]))
{
object1 = type.GetProperty(properies[i + 1]).GetValue(object1, null);
}
else
{
object1 = type.GetProperty(properies[i]).GetValue(object1, null);
}
type = object1.GetType();
}
}
value = object1.ToString(); //.StartDateTime.ToString();
}
return value;
}
I`m not sure if this is the best approach. Does anyone have a better suggestion, or maybe someone has already done something like this?
To shorten it: The idea is to make generic linq queries to database from a string like: "Case.Parties.FirstOrDefault.NameEn".
Your approach is very good. I have no doubt that it already works.
Another approach is using Expression Tree like #Egorikas have suggested.
Disclaimer: I'm the owner of the project Eval-Expression.NET
In short, this library allows you to evaluate almost any C# code at runtime (What you exactly want to do).
I would suggest you use my library instead. To keep the code:
More readable
Easier to support
Add some flexibility
Example
public string GetColumnValueByObjectByName(Expression<Func<TEntity, bool>> filter = null, string objectName = "", string dllName = "", string objectID = "", string propertyName = "")
{
string objectDllName = objectName + ", " + dllName;
Type type = Type.GetType(objectDllName);
Guid oID = new Guid(objectID);
object Entity = context.Set(type).Find(oID); // get Object by Type and ObjectID
var value = Eval.Execute("x." + propertyName, new { x = entity });
return value.ToString();
}
The library also allow you to use dynamic string with IQueryable
Wiki: LINQ-Dynamic

Relational Queries In parse.com (Unity)

I found example in parse.com. I have 2 objects : Post and Comment, in the Comment objects have a collumn: "parent" pointer to Post obj and I want to join them:
var query = ParseObject.GetQuery ("Comment");
// Include the post data with each comment
query = query.Include("parent");
query.FindAsync().ContinueWith(t => {
IEnumerable<ParseObject> comments = t.Result;
// Comments now contains the last ten comments, and the "post" field
// contains an object that has already been fetched. For example:
foreach (var comment in comments)
{
// This does not require a network access.
string o= comment.Get<string>("content");
Debug.Log(o);
try {
string post = comment.Get<ParseObject>("parent").Get<string>("title");
Debug.Log(post);
} catch (Exception ex) {
Debug.Log(ex);
}
}
});
It worked!
And then, I have 2 objects: User and Gamescore, in the Gamescore objects have a collumn: "playerName" pointer to Post obj I want join them too:
var query = ParseObject.GetQuery ("GameScore");
query.Include ("playerName");
query.FindAsync ().ContinueWith (t =>{
IEnumerable<ParseObject> result = t.Result;
foreach (var item in result) {
Debug.Log("List score: ");
int score = item.Get<int>("score");
Debug.Log(score);
try {
var obj = item.Get<ParseUser>("playerName");
string name = obj.Get<string>("profile");
//string name = item.Get<ParseUser>("playerName").Get<string>("profile");
Debug.Log(name);
} catch (Exception ex) {
Debug.Log(ex);
}
}
});
but It isn't working, Please help me!
Why didn't you do the following like you did your first example:
query = query.Include ("playerName");
you just have -
query.Include ("playerName");
One solution would be to ensure that your ParseUser object is properly fetched. ie:
var obj = item.Get<ParseUser>("playerName");
Task t = obj.FetchIfNeededAsync();
while (!t.IsCompleted) yield return null;
Then you can do this without worrying:
string name = obj.Get<string>("profile");
But that will be another potential request to Parse, which is unfortunate. It seems that query.Include ("playerName") isn't properly working in the Unity version of Parse?
I believe you're supposed to use multi-level includes for this, like .Include("parent.playerName") in your first query.

What is it that should be done here?

I have been following this tutorial to come up with a simple source code editor. (The feature that I want the most is keyword highlighting.) What I do not understand is the last part:
class Scanner extends RuleBasedScanner {
public Scanner() {
WordRule rule = new WordRule(new IWordDetector() {
public boolean isWordStart(char c) {
return Character.isJavaIdentifierStart(c);
}
public boolean isWordPart(char c) {
return Character.isJavaIdentifierPart(c);
}
});
Token keyword = new Token(new TextAttribute(Editor.KEYWORD, null, SWT.BOLD));
Token comment = new Token(new TextAttribute(Editor.COMMENT));
Token string = new Token(new TextAttribute(Editor.STRING));
//add tokens for each reserved word
for (int n = 0; n < Parser.KEYWORDS.length; n++) {
rule.addWord(Parser.KEYWORDS[n], keyword);
}
setRules(new IRule[] {
rule,
new SingleLineRule("#", null, comment),
new SingleLineRule("\"", "\"", string, '\\'),
new SingleLineRule("'", "'", string, '\\'),
new WhitespaceRule(new IWhitespaceDetector() {
public boolean isWhitespace(char c) {
return Character.isWhitespace(c);
}
}),
});
}
}
The instruction is as follows:
For each of the keywords in our little language, we define a word entry in our WordRule. We pass our keyword detector, together with rules for recognizing comments, strings, and white spaces to the scanner. With this simple set of rules, the scanner can segment a stream of bytes into sections and then use the underlying rules to color the sections.
Shed me some light please? I do not know what it is I have to do to set the desired keywords..

tinymce.dom.replace throws an exception concerning parentNode

I'm writing a tinyMce plugin which contains a section of code, replacing one element for another. I'm using the editor's dom instance to create the node I want to insert, and I'm using the same instance to do the replacement.
My code is as follows:
var nodeData =
{
"data-widgetId": data.widget.widgetKey(),
"data-instanceKey": "instance1",
src: "/content/images/icon48/cog.png",
class: "widgetPlaceholder",
title: data.widget.getInfo().name
};
var nodeToInsert = ed.dom.create("img", nodeData);
// Insert this content into the editor window
if (data.mode == 'add') {
tinymce.DOM.add(ed.getBody(), nodeToInsert);
}
else if (data.mode == 'edit' && data.selected != null) {
var instanceKey = $(data.selected).attr("data-instancekey");
var elementToReplace = tinymce.DOM.select("[data-instancekey=" + instanceKey + "]");
if (elementToReplace.length === 1) {
ed.dom.replace(elementToReplace[0], nodeToInsert);
}
else {
throw new "No element to replace with that instance key";
}
}
TinyMCE breaks during the replace, here:
replace : function(n, o, k) {
var t = this;
if (is(o, 'array'))
n = n.cloneNode(true);
return t.run(o, function(o) {
if (k) {
each(tinymce.grep(o.childNodes), function(c) {
n.appendChild(c);
});
}
return o.parentNode.replaceChild(n, o);
});
},
..with the error Cannot call method 'replaceChild' of null.
I've verified that the two argument's being passed into replace() are not null and that their parentNode fields are instantiated. I've also taken care to make sure that the elements are being created and replace using the same document instance (I understand I.E has an issue with this).
I've done all this development in Google Chrome, but I receive the same errors in Firefox 4 and IE8 also. Has anyone else come across this?
Thanks in advance
As it turns out, I was simply passing in the arguments in the wrong order. I should have been passing the node I wanted to insert first, and the node I wanted to replace second.