ServiceStack Ormlite - Postgres serializing Date property with MaxDate to JsonB - postgresql

I have a complex object which I save to a JsonB field in postgres using Ormlite.
One of the property is a DateTime and is set to DateTime.Max.
Retrieving the object from Postgres the DateTime property value is set to DateTime.Min value
01/01/0001 00:00:00
Not sure if this is a bug with Ormlite or the json serializer.
Code snippet to replicate
class Program
{
static void Main(string[] args)
{
var item = new LicenseCheckTemp();
item.Body = new CheckHistory();
item.Body.List.Add(new ItemHistory() {AddedOn = DateTime.MaxValue, Note = "Test"});
var factory = GetFactory(ConfigurationManager.AppSettings["PostgresConnectionString"]);
using (var db = factory.OpenDbConnection())
{
db.CreateTableIfNotExists<LicenseCheckTemp>();
db.Save(item);
}
using (var db = factory.OpenDbConnection())
{
var items = db.Select<LicenseCheckTemp>();
foreach (var licenseCheck in items.OrderBy(x=>x.Id))
{
if (licenseCheck.Body != null && licenseCheck.Body.List.Any())
{
foreach (var itemHistory in licenseCheck.Body.List)
{
Console.WriteLine($"{itemHistory.AddedOn} : Note {itemHistory.Note}");
}
}
}
}
Console.ReadKey();
}
public static IDbConnectionFactory GetFactory(string connection)
{
var factory = new OrmLiteConnectionFactory(connection,
PostgreSqlDialect.Provider);
factory.DialectProvider.NamingStrategy = new OrmLiteNamingStrategyBase();
return factory;
}
}
public class LicenseCheckTemp
{
[AutoIncrement]
public int Id { get; set; }
[CustomField("json")]
public CheckHistory Body { get; set; }
}
public class CheckHistory
{
public List<ItemHistory> List { get; set; } = new List<ItemHistory>();
}
public class ItemHistory
{
public string Note { get; set; }
public DateTime AddedOn { get; set; }
}

Whilst OrmLite doesn't have explicit support for PostgreSQL JSON DataTypes, OrmLite's existing JSON serialization of ComplexType properties should allow this to work naturally as seen below:
I've added an example of this test in this commit:
OrmLiteConfig.DialectProvider.NamingStrategy = new OrmLiteNamingStrategyBase();
var item = new LicenseCheckTemp();
item.Body = new CheckHistory();
item.Body.List.Add(new ItemHistory { AddedOn = DateTime.MaxValue, Note = "Test" });
using (var db = OpenDbConnection())
{
db.DropAndCreateTable<LicenseCheckTemp>();
db.GetLastSql().Print();
db.Save(item);
}
using (var db = OpenDbConnection())
{
var items = db.Select<LicenseCheckTemp>();
items.PrintDump();
foreach (var licenseCheck in items.OrderBy(x => x.Id))
{
if (licenseCheck.Body != null && licenseCheck.Body.List.Any())
{
foreach (var itemHistory in licenseCheck.Body.List)
{
$"{itemHistory.AddedOn} : Note {itemHistory.Note}".Print();
}
}
}
}
Which is working as expected, i.e. it Prints out:
CREATE TABLE "LicenseCheckTemp"
(
"Id" INTEGER PRIMARY KEY AUTOINCREMENT,
"Body" json NULL
);
[
{
Id: 1,
Body:
{
List:
[
{
Note: Test,
AddedOn: 9999-12-31T23:59:59.9999999-05:00
}
]
}
}
]
12/31/9999 11:59:59 PM : Note Test
Showing CreateTable creating a "json" type for Body property with the row being serialized and returned fully populated.
Can't repro why it's not working for you, are you using the latest v4.0.54 release of OrmLite? Does it work with a smaller DateTime? (perhaps Max DateTime in your TimeZone exceeds what your PgSql configured instance supports).

Related

How can get a list data of Google Sheets based on column names in Entity Framework

I'm modeling data search in Google Sheets using API (EF). I am currently connected to Google Sheets data. I also wrote a search based on RowId it's ok. Everything works fine. However I can't find data based on Id. Everything I have:
ItemGoogleSheet.cs
public class ItemGoogleSheet
{
public string Id { get; set; }
public string Name { get; set; }
}
ItemsGoogleSheetMapper.cs
public class ItemsGoogleSheetMapper
{
public static List<ItemGoogleSheet> MapFromRangeData(IList<IList<object>> values)
{
var items = new List<ItemGoogleSheet>();
foreach (var value in values)
{
ItemGoogleSheet item = new()
{
Id = value[0].ToString(),
Name = value[1].ToString(),
};
items.Add(item);
}
return items;
}
public static IList<IList<object>> MapToRangeData(ItemGoogleSheet item)
{
var objectList = new List<object>() { item.Id, item.Name };
var rangeData = new List<IList<object>> { objectList };
return rangeData;
}
}
ItemsGoogleSheetVATController.cs
public class ItemsGoogleSheetVATController : ControllerBase
{
const string SPREADSHEET_ID = "xxxx";
const string SHEET_NAME = "xx";
SpreadsheetsResource.ValuesResource _googleSheetValues;
public ItemsGoogleSheetVATController(GoogleSheetsHelper googleSheetsHelper)
{
_googleSheetValues = googleSheetsHelper.Service.Spreadsheets.Values;
}
[HttpGet("{rowId}")]
public IActionResult GetRowID(int rowId)
{
var range = $"{SHEET_NAME}!A{rowId}:AG{rowId}";
var request = _googleSheetValues.Get(SPREADSHEET_ID, range);
var response = request.Execute();
var values = response.Values;
return Ok(ItemsGoogleSheetMapper.MapFromRangeData(values).FirstOrDefault());
}
[HttpGet]
public IActionResult GetID(string id)
{
//How to get Data from Id
//return Ok();
}
}
My Google Sheets Data:
As in my description. I want to find Id = 0102 then it will output a list of results of: 0102, 01022101, 01022102
How can I get list of data based on Id column. Asking for any solutions from everyone. Thank you!
I have solved the problem. Thank you!

How to write an audit log entry per changed property with Audit.NET EntityFramework.Core

I'm trying to get the Audit:NET EntityFramework.Core extension to write an AuditLog entry per changed property.
For this purpose I've overidden the EntityFrameworkDataProvider.InsertEvent with a custom DataProvider.
The problem is, using DbContextHelper.Core.CreateAuditEvent to create a new EntityFrameworkEvent returns null.
The reason seems to be, at this point in the code execution DbContextHelper.GetModifiedEntries determines all EF Entries have State.Unmodified, even if they are clearly included in the EventEntry changes.
I'm trying to circumvent CreateAuditEvent by manually creating the contents is impossible due to private/internal properties.
Maybe there is an alternative solution to this problem I'm not seeing, i'm open to all suggestions.
Audit entity class
public class AuditLog
{
public int Id { get; set; }
public string Description { get; set; }
public string OldValue { get; set; }
public string NewValue { get; set; }
public string PropertyName { get; set; }
public DateTime AuditDateTime { get; set; }
public Guid? AuditIssuerUserId { get; set; }
public string AuditAction { get; set; }
public string TableName { get; set; }
public int TablePK { get; set; }
}
Startup configuration
Audit.Core.Configuration.Setup()
.UseCustomProvider(new CustomEntityFrameworkDataProvider(x => x
.AuditEntityAction<AuditLog>((ev, ent, auditEntity) =>
{
auditEntity.AuditDateTime = DateTime.Now;
auditEntity.AuditAction = ent.Action;
foreach(var change in ent.Changes)
{
auditEntity.OldValue = change.OriginalValue.ToString();
auditEntity.NewValue = change.NewValue.ToString();
auditEntity.PropertyName = change.ColumnName;
}
}
Custom data provider class
public class CustomEntityFrameworkDataProvider : EntityFrameworkDataProvider
{
public override object InsertEvent(AuditEvent auditEvent)
{
var auditEventEf = auditEvent as AuditEventEntityFramework;
if (auditEventEf == null)
return null;
object result = null;
foreach (var entry in auditEventEf.EntityFrameworkEvent.Entries)
{
if (entry.Changes == null || entry.Changes.Count == 0)
continue;
foreach (var change in entry.Changes)
{
var contextHelper = new DbContextHelper();
var newEfEvent = contextHelper.CreateAuditEvent((IAuditDbContext)auditEventEf.EntityFrameworkEvent.GetDbContext());
if (newEfEvent == null)
continue;
newEfEvent.Entries = new List<EventEntry>() { entry };
entry.Changes = new List<EventEntryChange> { change };
auditEventEf.EntityFrameworkEvent = newEfEvent;
result = base.InsertEvent(auditEvent);
}
}
return result;
}
}
Check my answer here https://github.com/thepirat000/Audit.NET/issues/323#issuecomment-673007204
You don't need to call CreateAuditEvent() you should be able to iterate over the Changes list on the original event and call base.InsertEvent() for each change, like this:
public override object InsertEvent(AuditEvent auditEvent)
{
var auditEventEf = auditEvent as AuditEventEntityFramework;
if (auditEventEf == null)
return null;
object result = null;
foreach (var entry in auditEventEf.EntityFrameworkEvent.Entries)
{
if (entry.Changes == null || entry.Changes.Count == 0)
continue;
// Call base.InsertEvent for each change
var originalChanges = entry.Changes;
foreach (var change in originalChanges)
{
entry.Changes = new List<EventEntryChange>() { change };
result = base.InsertEvent(auditEvent);
}
entry.Changes = originalChanges;
}
return result;
}
Notes:
This could impact performance, since it will trigger an insert to the database for each column change.
If you plan to use async calls to DbContext.SaveChangesAsync, you should also implement the InsertEventAsync method on your CustomDataProvider
The Changes property is only available for Updates, so if you also want to audit Inserts and Deletes, you'll need to add the logic to get the column values from the ColumnValues property on the event

The entity or complex type cannot be constructed in a LINQ to Entities query [duplicate]

This question already has answers here:
The entity cannot be constructed in a LINQ to Entities query
(14 answers)
Closed 10 years ago.
I have two functions that look exactly the same except they create lists of two different objects. The two different objects look very much alike, but when I try to run one of the functions on one of the objects, I get the error message, "The entity or complex type cannot be constructed in a LINQ to Entities query.". Can someone explain to me what is happening in very simple terms? Also, can you tell me how to change my code so that it works? Thanks, Allan.
Function 1 (works):
public static List<ChartApp> ListChartApplications()
{
using (var db = new LatencyDBContext())
{
var appNames = db.LoginApplications.Select(item => new ChartApp()
{
LoginApplicationID = item.LoginApplicationID,
LoginAppName = item.LoginAppName,
}).OrderBy(item => item.LoginAppName);
return appNames.ToList();
}
}
Function 2 (throws error on "return appNames.ToList();"):
public static List<LoginApplication> ListApplications()
{
using (var db = new LatencyDBContext())
{
var appNames = db.LoginApplications.Select(item => new LoginApplication()
{
LoginApplicationID = item.LoginApplicationID,
LoginAppName = item.LoginAppName,
}).OrderBy(item => item.LoginAppName);
return appNames.ToList();
}
}
Classes:
public class ChartApp
{
public ChartApp()
{
this.LoginHistories = new List<ChartHist>();
}
public int? LoginApplicationID { get; set; }
public string LoginAppName { get; set; }
public virtual ICollection<ChartHist> LoginHistories { get; set; }
public int Step { get; set; }
}
public class LoginApplication
{
public LoginApplication()
{
this.LoginHistories = new List<LoginHistory>();
}
public int LoginApplicationID { get; set; }
public string LoginAppName { get; set; }
public virtual ICollection<LoginHistory> LoginHistories { get; set; }
}
Edit: Could the difference possibly be that one of the objects are mapped to the database?
public class LoginApplicationMap : EntityTypeConfiguration<LoginApplication>
{
public LoginApplicationMap()
{
// Primary Key
this.HasKey(t => t.LoginApplicationID);
// Properties
this.Property(t => t.LoginAppName)
.HasMaxLength(500);
// Table & Column Mappings
this.ToTable("LoginApplication");
this.Property(t => t.LoginApplicationID).HasColumnName("LoginApplicationID");
this.Property(t => t.LoginAppName).HasColumnName("LoginAppName");
}
}
My solution in this case was to just delete the non-working function and use the working one in all places. For, similar functions that are mapped, I use the following function to return values.
public static List<LoginEnvironment> ListEnvironments(bool allSelection)
{
using (var db = new LatencyDBContext())
{
//GET ALL THE ENVIRONMENT NAMES
var envNames = from e in db.LoginEnvironments
orderby e.LoginEnvName
select e;
//PUT ALL THE ENVIRONMENTS INTO A LOCAL LIST
var listEnv = new List<LoginEnvironment>();
if (allSelection)
{
var defaultAll = new LoginEnvironment();
defaultAll.LoginEnvironmentID = 0;
defaultAll.LoginEnvName = "All";
listEnv.Add(defaultAll);
}
foreach (var item in envNames)
{
var localEnv = new LoginEnvironment();
localEnv.LoginEnvironmentID = item.LoginEnvironmentID;
localEnv.LoginEnvName = item.LoginEnvName;
listEnv.Add(localEnv);
}
return listEnv;
}
}

How to correctly model loosely-typed properties in RavenDB

I am new to RavenDB and looking for guidance on the correct way to store loosely-typed data. I have a type with a list of key/value pairs. The type of the value property isn't known at design time.
public class DescriptiveValue
{
public string Key { get; set; }
public object Value { get; set; }
}
When I query a DescriptiveValue that was saved with a DateTime or Guid Value, the deserialized data type is string. Numeric values appear to retain their data types.
Is there an elegant solution to retain the data type or should I simply store all values as strings? If I go the string route, will this limit me when I later want to sort and filter this data (likely via indexes?)
I hoping this is a common problem that is easily solved and I'm just thinking about the problem incorrectly. Your help is much appreciated!
UPDATE:
The output of this unit test is: Assert.AreEqual failed. Expected:<2/2/2012 10:00:01 AM (System.DateTime)>. Actual:<2012-02-02T10:00:01.9047999 (System.String)>.
[TestMethod]
public void Store_WithDateTime_IsPersistedCorrectly()
{
AssertValueIsPersisted<DateTime>(DateTime.Now);
}
private void AssertValueIsPersisted<T>(T value)
{
ObjectValuedAttribute expected = new ObjectValuedAttribute() { Value = value };
using (var session = this.NewSession())
{
session.Store(expected);
session.SaveChanges();
}
TestDataFactory.ResetRavenDbConnection();
using (var session = this.NewSession())
{
ObjectValuedAttribute actual = session.Query<ObjectValuedAttribute>().Single();
Assert.AreEqual(expected.Value, actual.Value);
}
}
I would expect actual to be a DateTime value.
Absolutely - that's one of the strength of schema-less document databases. See here: http://ravendb.net/docs/client-api/advanced/dynamic-fields
The problem is that RavenDB server has no notion of the type of Value. When sending your object to the server, Value is persisted as a string, and when you later query that document, the deserializer does not know about the original type, so Value is deserialized as a string.
You can solve this by adding the original type information to ObjectValuedAttribute:
public class ObjectValuedAttribute {
private object _value;
public string Key { get; set; }
public object Value {
get {
// convert the value back to the original type
if (ValueType != null && _value.GetType() != ValueType) {
_value = TypeDescriptor
.GetConverter(ValueType).ConvertFrom(_value);
}
return _value;
}
set {
_value = value;
ValueType = value.GetType();
}
}
public Type ValueType { get; private set; }
}
In the setter of Value we also store the type of it. Later, when getting back the value, we convert it back to its original type.
Following test passes:
public class CodeChef : LocalClientTest {
public class ObjectValuedAttribute {
private object _value;
public string Key { get; set; }
public object Value {
get {
// convert value back to the original type
if (ValueType != null && _value.GetType() != ValueType) {
_value = TypeDescriptor
.GetConverter(ValueType).ConvertFrom(_value);
}
return _value;
}
set {
_value = value;
ValueType = value.GetType();
}
}
public Type ValueType { get; private set; }
}
[Fact]
public void Store_WithDateTime_IsPersistedCorrectly() {
AssertValueIsPersisted(DateTime.Now);
}
private void AssertValueIsPersisted<T>(T value) {
using (var store = NewDocumentStore()) {
var expected = new ObjectValuedAttribute { Value = value };
using (var session = store.OpenSession()) {
session.Store(expected);
session.SaveChanges();
}
using (var session = store.OpenSession()) {
var actual = session
.Query<ObjectValuedAttribute>()
.Customize(x => x.WaitForNonStaleResults())
.Single();
Assert.Equal(expected.Value, actual.Value);
}
}
}
}

How to decorate a class item to be an index and get the same as using ensureIndex?

I'd like to define in class declaration which items are index, something like:
public class MyClass {
public int SomeNum { get; set; }
[THISISANINDEX]
public string SomeProperty { get; set; }
}
so to have the same effect as ensureIndex("SomeProperty")
Is this possible?
I think this is a nice idea, but you have to do this yourself, there's no built-in support for it. If you have an access layer you can do it in there. You'd need an attribute class, something like this;
public enum IndexConstraints
{
Normal = 0x00000001, // Ascending, non-indexed
Descending = 0x00000010,
Unique = 0x00000100,
Sparse = 0x00001000, // allows nulls in the indexed fields
}
// Applied to a member
[AttributeUsage(AttributeTargets.Property | AttributeTargets.Field)]
public class EnsureIndexAttribute : EnsureIndexes
{
public EnsureIndex(IndexConstraints ic = IndexConstraints.Normal) : base(ic) { }
}
// Applied to a class
[AttributeUsage(AttributeTargets.Class)]
public class EnsureIndexesAttribute : Attribute
{
public bool Descending { get; private set; }
public bool Unique { get; private set; }
public bool Sparse { get; private set; }
public string[] Keys { get; private set; }
public EnsureIndexes(params string[] keys) : this(IndexConstraints.Normal, keys) {}
public EnsureIndexes(IndexConstraints ic, params string[] keys)
{
this.Descending = ((ic & IndexConstraints.Descending) != 0);
this.Unique = ((ic & IndexConstraints.Unique) != 0); ;
this.Sparse = ((ic & IndexConstraints.Sparse) != 0); ;
this.Keys = keys;
}
}//class EnsureIndexes
You could then apply attributes at either the class or member level as follows. I found that adding at member level was less likely to get out of sync with the schema compared to adding at the class level. You need to make sure of course that you get the actual element name as opposed to the C# member name;
[CollectionName("People")]
//[EnsureIndexes("k")]// doing it here would allow for multi-key configs
public class Person
{
[BsonElement("k")] // name mapping in the DB schema
[BsonIgnoreIfNull]
[EnsureIndex(IndexConstraints.Unique|IndexConstraints.Sparse)] // name is implicit here
public string userId{ get; protected set; }
// other properties go here
}
and then in your DB access implementation (or repository), you need something like this;
private void AssureIndexesNotInlinable()
{
// We can only index a collection if there's at least one element, otherwise it does nothing
if (this.collection.Count() > 0)
{
// Check for EnsureIndex Attribute
var theClass = typeof(T);
// Walk the members of the class to see if there are any directly attached index directives
foreach (var m in theClass.GetProperties(BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.FlattenHierarchy))
{
List<string> elementNameOverride = new List<string>(1);
EnsureIndexes indexAttr = null;
// For each members attribs
foreach (Attribute attr in m.GetCustomAttributes())
{
if (attr.GetType() == typeof(EnsureIndex))
indexAttr = (EnsureIndex)attr;
if (attr.GetType() == typeof(RepoElementAttribute))
elementNameOverride.Add(((RepoElementAttribute)attr).ElementName);
if ((indexAttr != null) && (elementNameOverride.Count != 0))
break;
}
// Index
if (indexAttr != null)
{
if (elementNameOverride.Count() > 0)
EnsureIndexesAsDeclared(indexAttr, elementNameOverride);
else
EnsureIndexesAsDeclared(indexAttr);
}
}
// Walk the atributes on the class itself. WARNING: We don't validate the member names here, we just create the indexes
// so if you create a unique index and don't have a field to match you'll get an exception as you try to add the second
// item with a null value on that key
foreach (Attribute attr in theClass.GetCustomAttributes(true))
{
if (attr.GetType() == typeof(EnsureIndexes))
EnsureIndexesAsDeclared((EnsureIndexes)attr);
}//foreach
}//if this.collection.count
}//AssureIndexesNotInlinable()
EnsureIndexes then looks like this;
private void EnsureIndexesAsDeclared(EnsureIndexes attr, List<string> indexFields = null)
{
var eia = attr as EnsureIndexes;
if (indexFields == null)
indexFields = eia.Keys.ToList();
// use driver specific methods to actually create this index on the collection
var db = GetRepositoryManager(); // if you have a repository or some other method of your own
db.EnsureIndexes(indexFields, attr.Descending, attr.Unique, attr.Sparse);
}//EnsureIndexes()
Note that you'll place this after each and every update because if you forget somewhere your indexes may not get created. It's important to ensure therefore that you optimise the call so that it returns quickly if there's no indexing to do before going through all that reflection code. Ideally, you'd do this just once, or at the very least, once per application startup. So one way would be to use a static flag to track whether you've already done so, and you'd need additional lock protection around that, but over-simplistically, it looks something like this;
void AssureIndexes()
{
if (_requiresIndexing)
AssureIndexesInit();
}
So that's the method you'll want in each and every DB update you make, which, if you're lucky would get inlined by the JIT optimizer as well.
See below for a naive implementation which could do with some brains to take the indexing advice from the MongoDb documentation into consideration. Creating indexes based on queries used within the application instead of adding custom attributes to properties might be another option.
using System;
using System.Reflection;
using MongoDB.Bson.Serialization.Attributes;
using MongoDB.Driver;
using NUnit.Framework;
using SharpTestsEx;
namespace Mongeek
{
[TestFixture]
class TestDecorateToEnsureIndex
{
[Test]
public void ShouldIndexPropertyWithEnsureIndexAttribute()
{
var server = MongoServer.Create("mongodb://localhost");
var db = server.GetDatabase("IndexTest");
var boatCollection = db.GetCollection<Boat>("Boats");
boatCollection.DropAllIndexes();
var indexer = new Indexer();
indexer.EnsureThat(boatCollection).HasIndexesNeededBy<Boat>();
boatCollection.IndexExists(new[] { "Name" }).Should().Be.True();
}
}
internal class Indexer
{
private MongoCollection _mongoCollection;
public Indexer EnsureThat(MongoCollection mongoCollection)
{
_mongoCollection = mongoCollection;
return this;
}
public Indexer HasIndexesNeededBy<T>()
{
Type t = typeof (T);
foreach(PropertyInfo prop in t.GetProperties() )
{
if (Attribute.IsDefined(prop, typeof (EnsureIndexAttribute)))
{
_mongoCollection.EnsureIndex(new[] {prop.Name});
}
}
return this;
}
}
internal class Boat
{
public Boat(Guid id)
{
Id = id;
}
[BsonId]
public Guid Id { get; private set; }
public int Length { get; set; }
[EnsureIndex]
public string Name { get; set; }
}
internal class EnsureIndexAttribute : Attribute
{
}
}