How to avoid writing duplicate queries when needing counts and then items - entity-framework

I have constructed a LINQ query that joins about a half dozen tables. The problem is, for paging purposes, I want to get a count first of how many items will be returned. So the issue I'm running into is having to write the exact same query twice: one to get the item count then another to build my collection of items.
Example:
using (var context = new DbContext())
{
var items = from i in context.Table1
join a in context.TableA on i.SomeProperty equals a.SomeProperty
join b in context.TableB on i.SomeOtherProperty equals b.SomeProperty
join c in context.TableC on i.AnotherProperty equals c.SomeProperty
etc.
etc.
select i;
count = items.Count();
}
return count;
.
.
.
using (var context = new DbContext())
{
var items = from i in context.Table1
join a in context.TableA on i.SomeProperty equals a.SomeProperty
join b in context.TableB on i.SomeOtherProperty equals b.SomeProperty
join c in context.TableC on i.AnotherProperty equals c.SomeProperty
etc.
etc.
select new
{
DynamicProp1 = i.SomeProperty,
DyanmicProp2 = a.SomeProperty,
DyanmicProp3 = b.SomePropery,
etc.
etc.
}
... do some stuff with 'items'...
}
I cannot think of any way to avoid this duplicate query. I need access to all the joined tables in order to build my collection. I would appreciate any tips or suggestions.

You can create method which get context and return IQueryable of items with all needed entities:
class Holder
{
TableAItem A{get;set;}
TableBItem B{get;set;}
...
}
IQueryable<Holder> GetQuery(DbContext context)
{
return from i in context.Table1
join a in context.TableA on i.SomeProperty equals a.SomeProperty
join b in context.TableB on i.SomeOtherProperty equals b.SomeProperty
join c in context.TableC on i.AnotherProperty equals c.SomeProperty
...
select new Holder
{
A = i,
B = b
....
};
}
using (var context = new DbContext())
{
var items = GetQuery(context);
count = items.Count();
}
return count;
using (var context = new DbContext())
{
var items = from r in GetQuery(context)
select new
{
DynamicProp1 = r.a.SomeProperty,
DyanmicProp2 = r.a.SomeProperty,
DyanmicProp3 = r.b.SomePropery,
etc.
etc.
}
... do some stuff with 'items'...
}

Remember that making a query doesn't execute it, this is called deferred execution. So why not make the query and then pass it around as an IQueryable<> object. For example, consider this code:
Just a simple method to return the last char from a string, but it also writes out what it's doing:
public char GetLastChar(string input)
{
Console.WriteLine("GetLastChar from {0}", input);
return input.Last();
}
Now this code using the method:
var listOfStuff = new List<string> { "string1", "string2", "string3" };
Console.WriteLine("Making the query");
var results = from s in listOfStuff
select GetLastChar(s);
Console.WriteLine("Before getting count");
var count = results.Count();
Console.WriteLine("Now enumerating the query");
foreach(var s in results)
{
Console.WriteLine(s);
}
You will see the output as follows:
Making the query
Before getting count
GetLastChar from string1
GetLastChar from string2
GetLastChar from string3
3
Now enumerating the query
GetLastChar from string1
1
GetLastChar from string2
2
GetLastChar from string3
3

Related

Split One Row into many based on splitting string in multiple cells

Trying to split one row into many based on string in two cells. it is similar to the question
LINQ to separate column value of a row to different rows in .net
but i need to split based on Product & Cost Columns rather than product column only
SNo.
Product
Cost
1
colgate,closeup,pepsodent
50,100,150
2
rin,surf
100
into
SNo.
Product
Cost
1
colgate
50
1
closeup
100
1
pepsodent
150
2
rin
100
2
surf
100
I'm using Linq to Object with Entity Framework
Try the following. Since you have not presented any model it can be inaccurate in names.
var loaded = ctx.Products.ToList();
var query =
from p in loaded
from sp in p.Product.Split(',').Zip(p.Cost.Split(','), (p, c) => (p, c))
select new
{
Sno = p.Sno,
Product = sp.p,
Cost = sp.c
};
var splitted = query.ToList();
Using #SvyatoslavDanyliv naming, here is an answer:
var loaded = ctx.Products.ToList();
var query =
from p in loaded
from sp in p.Product.Split(',').Zip(p.Cost.Split(','), (p, c) => (p, c))
select new
{
Sno = p.Sno,
Product = sp.p,
Cost = sp.c
};
var splitted = query.ToList();
It feels a bit complicated to me. I would prefer using an extension method to create a variant of Zip that repeats the last element of a shorter sequence to match the longer sequence:
public static class EnumerableExt {
public static IEnumerable<(T1 First,T2 Second)> ZipExtend<T1,T2>(this IEnumerable<T1> s1, IEnumerable<T2> s2) {
var s1e = s1.GetEnumerator();
var s2e = s2.GetEnumerator();
T1 s1eLast = default;
T2 s2eLast = default;
bool has_s2 = false;
if (s1e.MoveNext()) {
do {
s1eLast = s1e.Current;
if (s2e.MoveNext()) {
s2eLast = s2e.Current;
has_s2 = true;
}
else if (!has_s2)
yield break;
yield return (s1eLast, s2eLast);
} while (s1e.MoveNext());
if (has_s2)
while (s2e.MoveNext())
yield return (s1eLast, s2e.Current);
}
yield break;
}
}
Then the answer is:
var query =
from p in loaded
from pr in p.Product.Split(',').ZipExtend(p.Cost.Split(','))
select new
{
Sno = p.Sno,
Product = pr.First,
Cost = pr.Second
};
var splitted = query.ToList();

Saving changes if adding to context

EF Core 2.2.x
The question Im trying to answer is if I can limit my call of SaveChanges to only once or do I need it each time I add an entity, since in the next lower level I need to query the DbContext to see if the item exists.
Am trying to "merge" some data between two sets of similar tables. So in the process of adding records and drilling down the hierarchy, I have to first see if a record exists and if not, then add it.
/*
List<StagedOrder> sorders;
StagedOrder
Id
Number
InsertDate
Notes
StagedOrderLine
Id
ItemNo
Qty
StagedOrderKey
*/
foreach( StagedOrder s in sorders) {
// get order
Order o = (from a in ctx.Orders where a.number = s.Number select a).FirstOrDefault();
if(o == null) {
o = new Order() {
Number = s.Number,
InsertDate = DateTime.Now,
Notes = "imported order"
};
ctx.Orders.Add(o);
// ctx.SaveChanges() ???
}
// get lines for orders
OrderLines ol = (from a in ctx.DetailLines where a.OrderKey = o.Id select a).ToList();
if(ol == null) {
foreach(StagedOrderLine l in s.StagedOrderLines) {
ol = new DetailLine() {ItemNo = l.ItemNo, Qty = l.Qty, OrderKey = l.Id}
ctx.DetailLines.Add(ol);
ctx.SaveChanges();
}
}
}
// can I limit my call to savechanges only once here?
// ctx.SaveChanges();
Also, should I create a new DetailLine using an Order.Id or an entity reference, eg.
ol = new DetailLine() {ItemNo = l.ItemNo, Qty = l.Qty, Order = o}

How to improve query with two contexts - MVC5, LINQ n EF

I have two contexts. In one of them i have two views of which i get cods related to an entity from the another context. This query is taking too long time. How to improve it?
var negociacoes = _db.Negociacoes.Include(o=> o.User).ToArray();
var produtos = _oriDb.Vw_Produtos.ToArray();
var clientesVendedor = _oriDb.Vw_ClientesVendedores.ToArray();
var query = from n in negociacoes
join p in produtos on n.ProdutoId equals p.ProdutoId
join c in clientesVendedor on n.ClienteId equals c.codigo_entidade
select new NegociacaoView
{
NegociacaoId = n.NegociacaoId,
ProdutoId = n.ProdutoId,
Produto = p.descricao,
ClienteId = n.ClienteId,
Cliente = c.razao_social,
Rca = n.Rca,
Quantidade = n.Quantidade,
Preco = n.Preco,
Situacao = n.Situacao,
UserId = n.User.UserName,
Atendente = n.Atendente,
CondicaoId = n.CondicaoId,
DataCriacao = n.DataCriacao,
DataLiberacao = n.DataLiberacao,
Observacao = n.Observacao,
User = n.User
};
return query.ToList();
There are a couple of ways to speed this up:
It helps to run the smallest most efficient query first, then use those results to constrain the following queries.
Defining a select list so the database doesn't have to materialize every column will speed things up and use less memory.
Unfortunately, no matter how you do it in LINQ, you will end up with sql that uses large IN statements. A sproc would give you access to temp tables and joins that would be even better.
var negociacoes = _db.Negociacoes.Include(o=> o.User).ToArray();
//Use results of first query to constrain the second two. You could maybe combine the second two into one query.
var clientIds = negociacoes.Select(x => x.ClienteId);
var productIds = negociacoes.Select(x => x.ProdutoId);
var produtos = _oriDb.Vw_Produtos
.Where(x => productIds.Contains(x.ProdutoId))
//add a select. You're only using two columns from this table.
//.Select(x => new { })
.ToArray();
var clientesVendedor = _oriDb.Vw_ClientesVendedores
.Where(x => clientIds.Contains(x.codigo_entidade))
//add a select. You're only using two columns from this table.
//.Select(x => new { })
.ToArray();
var query = from n in negociacoes
join p in produtos on n.ProdutoId equals p.ProdutoId
join c in clientesVendedor on n.ClienteId equals c.codigo_entidade
select new NegociacaoView
{
NegociacaoId = n.NegociacaoId,
ProdutoId = n.ProdutoId,
Produto = p.descricao,
ClienteId = n.ClienteId,
Cliente = c.razao_social,
Rca = n.Rca,
Quantidade = n.Quantidade,
Preco = n.Preco,
Situacao = n.Situacao,
UserId = n.User.UserName,
Atendente = n.Atendente,
CondicaoId = n.CondicaoId,
DataCriacao = n.DataCriacao,
DataLiberacao = n.DataLiberacao,
Observacao = n.Observacao,
User = n.User
};
return query.ToList();

JPA Custom Query

I need your help. Basically I want to create a custom query for a view I made that contains most of the data needed by the client. The tricky part here is that the client can specify which columns to include in the search. A sample query would be like:
SELECT distinct s.empno FROM SesdbAllView s
WHERE s.lastname IN :lname AND s.examTaken IN :exam AND
s.training IN :train AND s.trainingFrom BETWEEN :from AND :to AND
s.eligibility IN :elig AND s.profession IN :prof
So I tried translating this to Criteria API but still stuck on how to do it especially in the BETWEEN keywords (where I check a range of a Date and also another for a Integer). When it comes to the IN keywords I'm not sure if I did it correctly as well.
My current code now is:
CriteriaBuilder cb = em.getCriteriaBuilder();
CriteriaQuery<Tuple> cq = cb.createTupleQuery();
Root<SesdbAllView> r = cq.from(SesdbAllView.class);
Predicate p = cb.conjunction();
for (Map.Entry<String, Object> param : parameters.entrySet()) {
if (param.getValue() instanceof List) {
Expression<String> exp = r.get(param.getKey());
p = cb.and(p, exp.in((List<String>)param.getValue()));
} else if (param.getValue() instanceof DateFromTo) {
DateFromTo fromTo = (DateFromTo) param.getValue();
p = cb.between(r.get(param.getKey()).as(Date.class),fromTo.getFrom(),fromTo.getTo());
} else if (param.getValue() instanceof IntegerFromTo) {
IntegerFromTo fromTo = (IntegerFromTo) param.getValue();
p = cb.between(r.get(param.getKey()).as(Integer.class),fromTo.getFrom(),fromTo.getTo());
} else {
p = cb.and(p, cb.equal(r.get(param.getKey()), param.getValue()));
}
}
cq.distinct(true);
cq.multiselect(r.get("empNo"))
.where(p);
List<Tuple> result = em.createQuery(cq).getResultList();

using the TSqlParser

I'm attempting to parse SQL using the TSql100Parser provided by microsoft. Right now I'm having a little trouble using it the way it seems to be intended to be used. Also, the lack of documentation doesn't help. (example: http://msdn.microsoft.com/en-us/library/microsoft.data.schema.scriptdom.sql.tsql100parser.aspx )
When I run a simple SELECT statement through the parser it returns a collection of TSqlStatements which contains a SELECT statement.
Trouble is, the TSqlSelect statement doesn't contain attributes such as a WHERE clause, even though the clause is implemented as a class. http://msdn.microsoft.com/en-us/library/microsoft.data.schema.scriptdom.sql.whereclause.aspx
The parser does recognise the WHERE clause as such, looking at the token stream.
So, my question is, am I using the parser correctly? Right now the token stream seems to be the most useful feature of the parser...
My Test project:
public static void Main(string[] args)
{
var parser = new TSql100Parser(false);
IList<ParseError> Errors;
IScriptFragment result = parser.Parse(
new StringReader("Select col from T1 where 1 = 1 group by 1;" +
"select col2 from T2;" +
"select col1 from tbl1 where id in (select id from tbl);"),
out Errors);
var Script = result as TSqlScript;
foreach (var ts in Script.Batches)
{
Console.WriteLine("new batch");
foreach (var st in ts.Statements)
{
IterateStatement(st);
}
}
}
static void IterateStatement(TSqlStatement statement)
{
Console.WriteLine("New Statement");
if (statement is SelectStatement)
{
PrintStatement(sstmnt);
}
}
Yes, you are using the parser correctly.
As Damien_The_Unbeliever points out, within the SelectStatement there is a QueryExpression property which will be a QuerySpecification object for your third select statement (with the WHERE clause).
This represents the 'real' SELECT bit of the query (whereas the outer SelectStatement object you are looking at has just got the 'WITH' clause (for CTEs), 'FOR' clause (for XML), 'ORDER BY' and other bits)
The QuerySpecification object is the object with the FromClauses, WhereClause, GroupByClause etc.
So you can get to your WHERE Clause by using:
((QuerySpecification)((SelectStatement)statement).QueryExpression).WhereClause
which has a SearchCondition property etc. etc.
Quick glance around would indicate that it contains a QueryExpression, which could be a QuerySpecification, which does have the Where clause attached to it.
if someone lands here and wants to know how to get the whole elements of a select statement the following code explain that:
QuerySpecification spec = (QuerySpecification)(((SelectStatement)st).QueryExpression);
StringBuilder sb = new StringBuilder();
sb.AppendLine("Select Elements");
foreach (var elm in spec.SelectElements)
sb.Append(((Identifier)((Column)((SelectColumn)elm).Expression).Identifiers[0]).Value);
sb.AppendLine();
sb.AppendLine("From Elements");
foreach (var elm in spec.FromClauses)
sb.Append(((SchemaObjectTableSource)elm).SchemaObject.BaseIdentifier.Value);
sb.AppendLine();
sb.AppendLine("Where Elements");
BinaryExpression binaryexp = (BinaryExpression)spec.WhereClause.SearchCondition;
sb.Append("operator is " + binaryexp.BinaryExpressionType);
if (binaryexp.FirstExpression is Column)
sb.Append(" First exp is " + ((Identifier)((Column)binaryexp.FirstExpression).Identifiers[0]).Value);
if (binaryexp.SecondExpression is Literal)
sb.Append(" Second exp is " + ((Literal)binaryexp.SecondExpression).Value);
I had to split a SELECT statement into pieces. My goal was to COUNT how many record a query will return. My first solution was to build a sub query such as
SELECT COUNT(*) FROM (select id, name from T where cat='A' order by id) as QUERY
The problem was that in this case the order clause raises the error "The ORDER BY clause is not valid in views, inline functions, derived tables, sub-queries, and common table expressions, unless TOP or FOR XML is also specified"
So I built a parser that split a SELECT statment into fragments using the TSql100Parser class.
using Microsoft.Data.Schema.ScriptDom.Sql;
using Microsoft.Data.Schema.ScriptDom;
using System.IO;
...
public class SelectParser
{
public string Parse(string sqlSelect, out string fields, out string from, out string groupby, out string where, out string having, out string orderby)
{
TSql100Parser parser = new TSql100Parser(false);
TextReader rd = new StringReader(sqlSelect);
IList<ParseError> errors;
var fragments = parser.Parse(rd, out errors);
fields = string.Empty;
from = string.Empty;
groupby = string.Empty;
where = string.Empty;
orderby = string.Empty;
having = string.Empty;
if (errors.Count > 0)
{
var retMessage = string.Empty;
foreach (var error in errors)
{
retMessage += error.Identifier + " - " + error.Message + " - position: " + error.Offset + "; ";
}
return retMessage;
}
try
{
// Extract the query assuming it is a SelectStatement
var query = ((fragments as TSqlScript).Batches[0].Statements[0] as SelectStatement).QueryExpression;
// Constructs the From clause with the optional joins
from = (query as QuerySpecification).FromClauses[0].GetString();
// Extract the where clause
where = (query as QuerySpecification).WhereClause.GetString();
// Get the field list
var fieldList = new List<string>();
foreach (var f in (query as QuerySpecification).SelectElements)
fieldList.Add((f as SelectColumn).GetString());
fields = string.Join(", ", fieldList.ToArray());
// Get The group by clause
groupby = (query as QuerySpecification).GroupByClause.GetString();
// Get the having clause of the query
having = (query as QuerySpecification).HavingClause.GetString();
// Get the order by clause
orderby = ((fragments as TSqlScript).Batches[0].Statements[0] as SelectStatement).OrderByClause.GetString();
}
catch (Exception ex)
{
return ex.ToString();
}
return string.Empty;
}
}
public static class Extension
{
/// <summary>
/// Get a string representing the SQL source fragment
/// </summary>
/// <param name="statement">The SQL Statement to get the string from, can be any derived class</param>
/// <returns>The SQL that represents the object</returns>
public static string GetString(this TSqlFragment statement)
{
string s = string.Empty;
if (statement == null) return string.Empty;
for (int i = statement.FirstTokenIndex; i <= statement.LastTokenIndex; i++)
{
s += statement.ScriptTokenStream[i].Text;
}
return s;
}
}
And to use this class simply:
string fields, from, groupby, where, having, orderby;
SelectParser selectParser = new SelectParser();
var retMessage = selectParser.Parse("SELECT * FROM T where cat='A' Order by Id desc",
out fields, out from, out groupby, out where, out having, out orderby);