I have this Linq statement that runs as expected in LinqPad (using Linq to SQL) but when I bring it into my C# app using EF 6, I get an error saying that cd.LogTimestamp.Date (in the GroupBy statement) is not supported. I'm attempting to convert a datetime2 value to a date value for group by purposes.
helper.GetDbContext().SiteLog
.Where( sl => sl.SiteId == 3 )
.GroupBy( cd => new { cd.SiteId, cd.LogTimestamp.Date } )
.Select( g => new DailyTraffic()
{
SiteId = g.Key.SiteId,
TrafficDate = g.Key.Date,
NewUsers = g.Count( row => row.IsNewUser ),
ReturningUsers = g.Count( row => row.IsReturningUser ),
TotalUsers = g.Count( row => row.IsNewUser ) + g.Count( row => row.IsReturningUser ),
PageViews = g.Count( row => row.IsPageView )
} )
.OrderBy( g => g.SiteId ).ThenBy( g => g.TrafficDate )
.ToList();
Is there a preferred way to convert a datetime2 value to a date value in EF 6
Using this QA ( how to use entity framework to group by date not date with time ) as a guide, you'll want to use EntityFunctions so you can use built-in SQL Server functions directly, this does of course mean your application won't be immediately portable to other databases.
.GroupBy( cd => new { cd.SiteId, EntityFunctions.TruncateTime( cd.LogTimestamp ) } )
Related
I’m trying to groupby using JuliaDB , everything works well except when i want to apply a function that takes two columns in one time.
JuliaDB.groupby(
(
TOTAL_PAID = :PRICE => sum,
FREQUENCY_IN_DAYS = :ORDER_DATE => handle_dates,
RECENCY_IN_DAYS = :ORDER_DATE => (x -> Dates.value(Date(now()) - maximum(x))),
COUNT_ORDERS_LAST_MONTH = :ORDER_DATE => countOnlyLastMonth,
TOTAL_PAID_LAST_MONTH = [:ORDER_DATE, :PRICE] => countOnlyLastMonth,
COUNT_ORDERS = length
),
t, :USER_KEY)
It’s this line that doesn’t work
TOTAL_PAID_LAST_MONTH = [:ORDER_DATE, :PRICE] => countOnlyLastMonth,
With DataFrames.groupby you would do it as following :
[:ORDER_DATE, :PRICE] => countOnlyLastMonth => :TOTAL_PAID_LAST_MONTH,
How can i do that with juliaDB ? Thank you
I need to get top 10 rows for each group in a table with entity framework.
Based on other solution on SO, I tried 2 things:
var sendDocuments = await context.Set<DbDocument>
.Where(t => partnerIds.Contains(t.SenderId))
.GroupBy(t => t.SenderId)
.Select(t => new
{
t.Key,
Documents = t.OrderByDescending(t2 => t2.InsertedDateTime).Take(10)
})
.ToArrayAsync();
error:
System.InvalidOperationException: 'The LINQ expression
'(GroupByShaperExpression: KeySelector: (d.SenderId),
ElementSelector:(EntityShaperExpression:
EntityType: DbDocument
ValueBufferExpression:
(ProjectionBindingExpression: EmptyProjectionMember)
IsNullable: False ) )
.OrderByDescending(t2 => t2.InsertedDateTime)' could not be translated. Either rewrite the query in a form that can be translated,
> or switch to client evaluation explicitly by inserting a call to
> either AsEnumerable(), AsAsyncEnumerable(), ToList(), or
> ToListAsync().
and
var sendDocuments2 = await context.Set<DbDocument>
.Where(t => partnerIds.Contains(t.SenderId))
.GroupBy(t => t.SenderId)
.SelectMany(t => t.OrderByDescending(t2 => t2.InsertedDateTime).Take(10))
.ToArrayAsync();
error:
System.InvalidOperationException: 'Processing of the LINQ expression
't => t
.OrderByDescending(t2 => t2.InsertedDateTime)
.AsQueryable()
.Take(10)' by 'NavigationExpandingExpressionVisitor' failed. This may indicate either a bug or a limitation in EF Core.
Any other idea?
Update (EF Core 6.0):
EF Core 6.0 added support for translating GroupBy result set projection, so the original code for taking (key, items) now works as it should, i.e.
var query = context.Set<DbDocument>()
.Where(e => partnerIds.Contains(e.SenderId))
.GroupBy(e => e.SenderId)
.Select(g => new
{
g.Key,
Documents = g.OrderByDescending(e => e.InsertedDateTime).Take(10)
});
However flattening (via SelectMany) is still unsupported, so you have to use the below workaround if you need such query shape.
Original (EF Core 3.0/3.1/5.0):
This is a common problem, unfortunately not supported by EF Core 3.0/3.1/5.0 query translator specifically for GroupBy.
The workaround is to do the groping manually by correlating 2 subqueries - one for keys and one for corresponding data.
Applying it to your examples would be something like this.
If you need (key, items) pairs:
var query = context.Set<DbDocument>()
.Where(t => partnerIds.Contains(t.SenderId))
.Select(t => t.SenderId).Distinct() // <--
.Select(key => new
{
Key = key,
Documents =
context.Set<DbDocument>().Where(t => t.SenderId == key) // <--
.OrderByDescending(t => t.InsertedDateTime).Take(10)
.ToList() // <--
});
If you need just flat result set containing top N items per key:
var query = context.Set<DbDocument>()
.Where(t => partnerIds.Contains(t.SenderId))
.Select(t => t.SenderId).Distinct() // <--
.SelectMany(key => context.Set<DbDocument>().Where(t => t.SenderId == key) // <--
.OrderByDescending(t => t.InsertedDateTime).Take(10)
);
I need to get top 10 rows for each group in a table with entity framework.
Based on other solution on SO, I tried 2 things:
var sendDocuments = await context.Set<DbDocument>
.Where(t => partnerIds.Contains(t.SenderId))
.GroupBy(t => t.SenderId)
.Select(t => new
{
t.Key,
Documents = t.OrderByDescending(t2 => t2.InsertedDateTime).Take(10)
})
.ToArrayAsync();
error:
System.InvalidOperationException: 'The LINQ expression
'(GroupByShaperExpression: KeySelector: (d.SenderId),
ElementSelector:(EntityShaperExpression:
EntityType: DbDocument
ValueBufferExpression:
(ProjectionBindingExpression: EmptyProjectionMember)
IsNullable: False ) )
.OrderByDescending(t2 => t2.InsertedDateTime)' could not be translated. Either rewrite the query in a form that can be translated,
> or switch to client evaluation explicitly by inserting a call to
> either AsEnumerable(), AsAsyncEnumerable(), ToList(), or
> ToListAsync().
and
var sendDocuments2 = await context.Set<DbDocument>
.Where(t => partnerIds.Contains(t.SenderId))
.GroupBy(t => t.SenderId)
.SelectMany(t => t.OrderByDescending(t2 => t2.InsertedDateTime).Take(10))
.ToArrayAsync();
error:
System.InvalidOperationException: 'Processing of the LINQ expression
't => t
.OrderByDescending(t2 => t2.InsertedDateTime)
.AsQueryable()
.Take(10)' by 'NavigationExpandingExpressionVisitor' failed. This may indicate either a bug or a limitation in EF Core.
Any other idea?
Update (EF Core 6.0):
EF Core 6.0 added support for translating GroupBy result set projection, so the original code for taking (key, items) now works as it should, i.e.
var query = context.Set<DbDocument>()
.Where(e => partnerIds.Contains(e.SenderId))
.GroupBy(e => e.SenderId)
.Select(g => new
{
g.Key,
Documents = g.OrderByDescending(e => e.InsertedDateTime).Take(10)
});
However flattening (via SelectMany) is still unsupported, so you have to use the below workaround if you need such query shape.
Original (EF Core 3.0/3.1/5.0):
This is a common problem, unfortunately not supported by EF Core 3.0/3.1/5.0 query translator specifically for GroupBy.
The workaround is to do the groping manually by correlating 2 subqueries - one for keys and one for corresponding data.
Applying it to your examples would be something like this.
If you need (key, items) pairs:
var query = context.Set<DbDocument>()
.Where(t => partnerIds.Contains(t.SenderId))
.Select(t => t.SenderId).Distinct() // <--
.Select(key => new
{
Key = key,
Documents =
context.Set<DbDocument>().Where(t => t.SenderId == key) // <--
.OrderByDescending(t => t.InsertedDateTime).Take(10)
.ToList() // <--
});
If you need just flat result set containing top N items per key:
var query = context.Set<DbDocument>()
.Where(t => partnerIds.Contains(t.SenderId))
.Select(t => t.SenderId).Distinct() // <--
.SelectMany(key => context.Set<DbDocument>().Where(t => t.SenderId == key) // <--
.OrderByDescending(t => t.InsertedDateTime).Take(10)
);
I am facing this linq query for entity framework core (2.0).
var result = await dbContext.Table1DbSet
.Where(t1e => t1e.Id == id && t1e.Id2 == id2)
.Select
(
t1e =>
t1e.Table2NavPropICollection.Select
(
t2e => new
{
singleObject = t2e.Table3NavPropObject.TargetObject,
enumerable1 = t2e.Table3NavPropObject.Table4NavPropObject.Table5NavPropICollection.Select(t5e => t5e.TargetObject),
enumerable2 = t2e.Table3NavPropObject.Table6NavPropObject.Table7NavPropICollection.Select(t7e => t7e.TargetObject),
enumerable3 = t2e.Table3NavPropObject.Table8NavPropICollection.SelectMany(t8e => t8e.Table9NavPropICollection.Select(t9e => t9e.TargetObject))
}
)
)
.ToListAsync();
The goal is to query all the referenced instances of TargetObject which is referenced accros ~ 10 different tables.
Currently it returns an IEnumerable where the anonymous object contains the properties singleObject, enumerable1, enumerable2, enumerable3. All those properties are of type TargetObject or IEnumerable.
Can I, and how, rewrite the query to not return anonymous objects but just an IEnumerable containing all the values?
For some reason the compiler won't let me iterate over the anonymous collection and flatten it manually.
This should do the trick if you want one array per t1e row.
var result = await dbContext.Table1DbSet
.Where(t1e => t1e.Id == id && t1e.Id2 == id2)
.Select
(
t1e =>
t1e.Table2NavPropICollection.Select
(
t2e => new[] {t2e.Table3NavPropObject.TargetObject}.Concat(
t2e.Table3NavPropObject.Table4NavPropObject.Table5NavPropICollection.Select(t5e => t5e.TargetObject)).Concat(
t2e.Table3NavPropObject.Table6NavPropObject.Table7NavPropICollection.Select(t7e => t7e.TargetObject)).Concat(
t2e.Table3NavPropObject.Table8NavPropICollection.SelectMany(t8e => t8e.Table9NavPropICollection.Select(t9e => t9e.TargetObject)))
)
)
.ToListAsync();
If you want it completely flat, you'll want to switch that Select to a SelectMany().
var result = await dbContext.Table1DbSet
.Where(t1e => t1e.Id == id && t1e.Id2 == id2)
.SelectMany
(
t1e =>
t1e.Table2NavPropICollection.Select
(
t2e => new[] {t2e.Table3NavPropObject.TargetObject}.Concat(
t2e.Table3NavPropObject.Table4NavPropObject.Table5NavPropICollection.Select(t5e => t5e.TargetObject)).Concat(
t2e.Table3NavPropObject.Table6NavPropObject.Table7NavPropICollection.Select(t7e => t7e.TargetObject)).Concat(
t2e.Table3NavPropObject.Table8NavPropICollection.SelectMany(t8e => t8e.Table9NavPropICollection.Select(t9e => t9e.TargetObject)))
)
)
.ToListAsync();
Is Add and/or Concat a possible solution?
PoC:
var a = new [] { 1,2,3 };
var result = new {
firstA = a.First(),
otherAs = a,
backwards = a.Reverse()
};
var final = new List<int>();
final.Add(result.firstA);
final.Concat(result.otherAs.Concat(result.backwards))
.Dump();
I use a MySQL database in production and a SQLite database for running tests. One part of my application is used to gather monthly statistics for a year. I've successfully done this, however it came at a cost of not being able to automate tests because I'm using MySQL specific functions when querying for the data:
my $criteria = {
status => ['Complete'],
'YEAR(completed_on)' => DateTime->now()->year(),
};
my $attributes = {
select => [ { count => 'title' }, 'completed_on' ],
as => [qw/num_completed datetime/],
group_by => [qw/MONTH(completed_on)/],
};
Notice I'm using YEAR and MONTH MySQL functions.
I know one way I can substitute the where clause to eliminate the use of MySQLs YEAR function, something like this:
my $dtf = $schema->storage->datetime_parser;
my $begin_date = DateTime->from_day_of_year( year => DateTime->now()->year(), day_of_year => 1 ); #inception o_O
my $end_date = DateTime->from_day_of_year( year => DateTime->now()->year(), day_of_year => 36[56] );
my $criteria = {
status => ['Complete'],
completed_on =>
-between => [
$dtf->format_datetime($begin_date),
$dtf->format_datetime($end_date),
]
};
Using the recommended way to query date fields using DBIC
But I'm stumped as to what to do with the group_by clause and how to make the grouping of this fields date value by month database agnostic as well. Wondering if anyone has any ideas?
Thanks!
Sometimes you will have to make engine specific code in DBIx::Class if you're trying to do special things. You can use $schema->storage->sqlt_type to make different SQL.
Note you can also use substr(completed_on,1,4) to get the year in SQLite.
This will solve your problem:
my $type = $schema->storage->sqlt_type;
my $criteria;
my $attributes;
if ($type eq 'MySQL') {
$criteria = {
status => ['Complete'],
'YEAR(completed_on)' => DateTime->now()->year(),
};
$attributes = {
select => [ { count => 'title' }, 'completed_on' ],
as => [qw/num_completed datetime/],
group_by => [qw/MONTH(completed_on)/],
};
}
elsif ($type eq 'SQLite') {
my $dtf = $schema->storage->datetime_parser;
my $begin_date = DateTime->from_day_of_year( year => DateTime->now()->year(), day_of_year => 1 ); #inception o_O
my $end_date = DateTime->from_day_of_year( year => DateTime->now()->year() + 1, day_of_year => 1 )->add( seconds => -1 );
$criteria = {
status => ['Complete'],
completed_on => {
-between => [
$dtf->format_datetime($begin_date),
$dtf->format_datetime($end_date),
]
}
};
$attributes = {
select => [ { count => 'title' }, 'completed_on' ],
as => [qw/num_completed datetime/],
group_by => ['substr(completed_on,6,2)'],
};
}