mvc entity framework select with dynamic where clause - entity-framework

I am doing a asp.net-mvc with entity-framework App.
I have a select instruction, where I need to have a dymanic where condition.
It is a common case where you have a filter composed by a string like "aaaa bbbb cccc". I need to bring all data that contains all the filter string or part of it.
I doing part ot it witt Split function, but it is difficult to get all posible combinations.
I Would rather use a Store Procedure. But the porpose of it, is to use Entity Framework.
As far I did this.
public IEnumerable<UploadSearch> GetUploadsBySearch(string search)
{
IEnumerable<UploadSearch> viewModel = (from uploads in _db.Uploads
.Where(p => p.ProcessState_id == Security.APPROVED && p.Finder.Contains(search))
.OrderByDescending(p => p.UploadDate)
select new UploadSearch
{
User_id = uploads.User_id,
UserName = uploads.Users.Name,
UserLastName = uploads.Users.LastName,
});
And I Add a for instruction to loop throu the string,
string[] param = search.Replace(" "," ").Split(' ');
string _param = "";
int large = param.Length;
for (int i=0;i<large-1;i++)
{
_param +=param[i] ' ' + param[i + 1];
IEnumerable<UploadSearch> _viewModel = (from uploads in _db.Uploads
.Where(p => p.ProcessState_id == Security.APPROVED && p.Finder.Contains(_param))
.OrderByDescending(p => p.UploadDate)
select new UploadSearch
{
User_id = uploads.User_id,
UserName = uploads.Users.Name,
UserLastName = uploads.Users.LastName,
});
viewModel = viewModel.Union(_viewModel);
}
And I use a Union clause.
But There has to be another way to do it.
Any Ideas?

You should use LinqKit, it has a PredicateBuilder class that allows you to dynamically build queries
https://github.com/scottksmith95/LINQKit#predicatebuilder
IQueryable<Product> SearchProducts (params string[] keywords)
{
var predicate = PredicateBuilder.New<Product>();
foreach (string keyword in keywords)
{
string temp = keyword;
predicate = predicate.Or (p => p.Description.Contains (temp));
}
return dataContext.Products.Where (predicate);
}

Related

Is there a way to make this linq2sql more DRY?

I would like to remove repetitions in this paragraph of code. Is it possible to write a function to cleanup any table nvarchar columns with a specified array of strings (or characters)? That query will be executed with ef core at the end.
I tried VS2022 refactoring tool to Extract Method, but it's extracting the whole Where(). I would like to "build" the Replaces chain. Is that possible?
var request = GetRequestValuesFromApi();
var query = _context.Persons.AsQueryable();
if (!string.IsNullOrWhiteSpace(request.Firstname))
{
string firstname = request.Firstname.Replace("-", "").Replace(" ", "").Replace("’", "").Replace("'", "");
query = query.Where(x => EF.Functions.Collate(x.Firstname!, "binary_ai").Replace("-", "").Replace(" ", "").Replace("’", "").Replace("'", "").Contains(firstname));
}
if (!string.IsNullOrWhiteSpace(request.Lastname))
{
string lastname = request.Lastname.Replace("-", "").Replace(" ", "").Replace("’", "").Replace("'", "");
query = query.Where(x => EF.Functions.Collate(x.Lastname!, "binary_ai").Replace("-", "").Replace(" ", "").Replace("’", "").Replace("'", "").Contains(lastname));
}
// ...
var results = query.ToList();
If you are willing to use LINQKit, you can embed Expression variables in queries and have them expanded in place before the query is executed:
var query = _context.Persons.AsQueryable().AsExpandable();
Expression<Func<string,string>> FixNameEFn = (string name) => name.Replace("-", "").Replace(" ", "").Replace("’", "").Replace("'", "");
var FixNameFn = FixNameEFn.Compile();
if (!string.IsNullOrWhiteSpace(request.Firstname)) {
string firstname = FixNameFn(request.Firstname);
query = query.Where(x => FixNameEFn.Invoke(EF.Functions.Collate(x.Firstname!, "binary_ai")).Contains(firstname));
}
if (!string.IsNullOrWhiteSpace(request.Lastname)) {
string lastname = FixNameFn(request.Lastname);
query = query.Where(x => FixNameEFn.Invoke(EF.Functions.Collate(x.Lastname!, "binary_ai")).Contains(lastname));
}
You can do this without LINQKit, but it would involve pushing the special operations up into a custom Where method (e.g., like WhereNameContains) which then builds or modifies an Expression<Func<>> and uses it in a regular Where call. But this would allow you to remove the EF.Functions.Collate repetition as well.

EF Core 5.0 - The linq expression could not be translated. either rewrite the query

This question has been asked before and answered a few times, but not in a generalized way. Rather, the answers are specific to the asker (which makes sense)...but I use this pattern a lot, and I'm sure others do as well, so I'm looking for a more general answer. I know why this is happening, but I'm not sure what to do about it exactly. the offending line below is
RoleId = ur.First(xu => xu.UserId == x.Id).RoleId,
My understanding is that I have to convert this in some way before the projection, but that's where I'm stuck. Or use Contains() instead of First() (which is an implicit Where() filter), but I'm not sure how to rewrite it so the server will do the query, instead of the client. What is the best way forward?
var roles = await _roleManager.Roles.ToListAsync();
var rolesList = roles.Select(x => new {x.Id, x.Name}).ToList();
var rid = ur.First(xu => xu.UserId == 4).RoleId;
var ur = await _context.UserRoles.ToListAsync();
var appUsers = await _context.Users
.Select(x => new AppUserViewModel
{
Id = x.Id,
StaffId = x.StaffId,
Email = x.Email,
UserName = x.UserName,
PhoneNumber = x.PhoneNumber,
RoleId = ur.First(xu => xu.UserId == x.Id).RoleId,
RoleSelectListItems = rolesList.Select(yy => new SelectListItem
{
Value = yy.Id.ToString(),
Text = yy.Name
}).ToList()
})
.ToListAsync();
Actually you should work with IQueryable, not lists.
var roles = _roleManager.Roles.AsQueryable();
var ur = _context.UserRoles.AsQueryable();
var appUsers = await _context.Users
.Select(x => new AppUserViewModel
{
Id = x.Id,
StaffId = x.StaffId,
Email = x.Email,
UserName = x.UserName,
PhoneNumber = x.PhoneNumber,
RoleId = ur.FirstOrDefault(xu => xu.UserId == x.Id).RoleId,
RoleSelectListItems = roles.Select(yy => new SelectListItem
{
Value = yy.Id.ToString(),
Text = yy.Name
}).ToList()
})
.ToListAsync();
I got an answer to this. Linq sends itself down to the database, but certain expressions can't be interpreted by the translator which is why it throws this error. So if, for example, your Linq expression references a method to return a value, well, that method can't be passed down to your db, so you have to get that value before you use it in your Linq expression.
So in the example above, RoleId = ur.First(xu => xu.UserId == x.Id).RoleId, it's really two expressions, and the translator doesn't know what to do with that. The first part is "return to me the first object in the userRoles list where the id equals the id of the user" and the second part is "and now give me the RoleId". But you can't send that down - it can't be translated. So I'd have to get the RoleId from the userRoles list as a separate pair of transactions.
Get the role object, then separate out the RoleId value from that object.

Build dynamic LINQ queries from a string - Use Reflection?

I have some word templates(maybe thousands). Each template has merge fields which will be filled from database. I don`t like writing separate code for every template and then build the application and deploy it whenever a template is changed or a field on the template is added!
Instead, I'm trying to define all merge fields in a separate xml file and for each field I want to write the "query" which will be called when needed. EX:
mergefield1 will call query "Case.Parties.FirstOrDefault.NameEn"
mergefield2 will call query "Case.CaseNumber"
mergefield3 will call query "Case.Documents.FirstOrDefault.DocumentContent.DocumentType"
Etc,
So, for a particular template I scan its merge fields, and for each merge field I take it`s "query definition" and make that request to database using EntityFramework and LINQ. Ex. it works for these queries: "TimeSlots.FirstOrDefault.StartDateTime" or
"Case.CaseNumber"
This will be an engine which will generate word documents and fill it with merge fields from xml. In addition, it will work for any new template or new merge field.
Now, I have worked a version using reflection.
public string GetColumnValueByObjectByName(Expression<Func<TEntity, bool>> filter = null, string objectName = "", string dllName = "", string objectID = "", string propertyName = "")
{
string objectDllName = objectName + ", " + dllName;
Type type = Type.GetType(objectDllName);
Guid oID = new Guid(objectID);
dynamic Entity = context.Set(type).Find(oID); // get Object by Type and ObjectID
string value = ""; //the value which will be filled with data from database
IEnumerable<string> linqMethods = typeof(System.Linq.Enumerable).GetMethods(BindingFlags.Static | BindingFlags.Public).Select(s => s.Name).ToList(); //get all linq methods and save them as list of strings
if (propertyName.Contains('.'))
{
string[] properies = propertyName.Split('.');
dynamic object1 = Entity;
IEnumerable<dynamic> Child = new List<dynamic>();
for (int i = 0; i < properies.Length; i++)
{
if (i < properies.Length - 1 && linqMethods.Contains(properies[i + 1]))
{
Child = type.GetProperty(properies[i]).GetValue(object1, null);
}
else if (linqMethods.Contains(properies[i]))
{
object1 = Child.Cast<object>().FirstOrDefault(); //for now works only with FirstOrDefault - Later it will be changed to work with ToList or other linq methods
type = object1.GetType();
}
else
{
if (linqMethods.Contains(properies[i]))
{
object1 = type.GetProperty(properies[i + 1]).GetValue(object1, null);
}
else
{
object1 = type.GetProperty(properies[i]).GetValue(object1, null);
}
type = object1.GetType();
}
}
value = object1.ToString(); //.StartDateTime.ToString();
}
return value;
}
I`m not sure if this is the best approach. Does anyone have a better suggestion, or maybe someone has already done something like this?
To shorten it: The idea is to make generic linq queries to database from a string like: "Case.Parties.FirstOrDefault.NameEn".
Your approach is very good. I have no doubt that it already works.
Another approach is using Expression Tree like #Egorikas have suggested.
Disclaimer: I'm the owner of the project Eval-Expression.NET
In short, this library allows you to evaluate almost any C# code at runtime (What you exactly want to do).
I would suggest you use my library instead. To keep the code:
More readable
Easier to support
Add some flexibility
Example
public string GetColumnValueByObjectByName(Expression<Func<TEntity, bool>> filter = null, string objectName = "", string dllName = "", string objectID = "", string propertyName = "")
{
string objectDllName = objectName + ", " + dllName;
Type type = Type.GetType(objectDllName);
Guid oID = new Guid(objectID);
object Entity = context.Set(type).Find(oID); // get Object by Type and ObjectID
var value = Eval.Execute("x." + propertyName, new { x = entity });
return value.ToString();
}
The library also allow you to use dynamic string with IQueryable
Wiki: LINQ-Dynamic

EF Append to an IQueryable a Where that generates an OR

I'm trying to achieve dynamic filtering on a table. My UI has filters that can be enabled or disabled on demand, and as you can imagine, my query should be able to know when to add filters to the query.
What I have so far is that I check if the filter object has a value, and if it does it adds a where clause to it. Example:
var q1 = DBContext.Table1
if (!string.IsNullOrEmpty(filterModel.SubjectContains))
q1 = q1.Where(i => i.Subject.Contains(filterModel.SubjectContains));
if (filterModel.EnvironmentId != null)
q1 = q1.Where(i => i.EnvironmentId == filterModel.EnvironmentId);
if (filterModel.CreatedBy != null)
q1 = q1.Where(i => i.CreatedByUserId == filterModel.CreatedBy);
var final = q1.Select(i => new
{
IssuesId = i.IssuesId,
Subject = i.Subject,
EnvironmentId = i.EnvironmentId,
CreatedBy = i.CreatedByUser.FullName,
});
return final.ToList();
The code above generates T-SQL that contains a WHERE clause for each field that uses AND to combine the conditions. This is fine, and will work for most cases.
Something like:
Select
IssueId, Subject, EnvironmentId, CreatedById
From
Table1
Where
(Subject like '%stackoverflow%')
and (EnvironmentId = 1)
and (CreatedById = 123)
But then I have a filter that explicitly needs an IssueId. I'm trying to figure out how the EF Where clause can generate an OR for me. I'm looking something that should generate a Tsql that looks like this:
Select
IssueId, Subject, EnvironmentId, CreatedById
From
Table1
Where
(Subject like '%stackoverflow%')
and (EnvironmentId = 1)
and (CreatedById = 123)
or (IssueId = 10001)
Found a solution for this that doesn't have to do multiple database call and works for me.
//filterModel.StaticIssueIds is of type List<Int32>
if (filterModel.StaticIssueIds != null)
{
//Get all ids declared in filterModel.StaticIssueIds
var qStaticIssues = DBContext.Table1.Where(i => filterModel.StaticIssueIds.Contains(i.IssuesId));
//Let's get all Issues that isn't declared in filterModel.StaticIssueIds from the original IQueryable
//we have to do this to ensure that there isn't any duplicate records.
q1 = q1.Where(i => !filterModel.StaticIssueIds.Contains(i.IssuesId));
//We then concatenate q1 and the qStaticIssues.
q1 = q1.Concat(qStaticIssues);
}
var final = q1.Select(i => new
{
IssuesId = i.IssuesId,
Subject = i.Subject,
EnvironmentId = i.EnvironmentId,
CreatedBy = i.CreatedByUser.FullName,
});
return final.ToList();

using the TSqlParser

I'm attempting to parse SQL using the TSql100Parser provided by microsoft. Right now I'm having a little trouble using it the way it seems to be intended to be used. Also, the lack of documentation doesn't help. (example: http://msdn.microsoft.com/en-us/library/microsoft.data.schema.scriptdom.sql.tsql100parser.aspx )
When I run a simple SELECT statement through the parser it returns a collection of TSqlStatements which contains a SELECT statement.
Trouble is, the TSqlSelect statement doesn't contain attributes such as a WHERE clause, even though the clause is implemented as a class. http://msdn.microsoft.com/en-us/library/microsoft.data.schema.scriptdom.sql.whereclause.aspx
The parser does recognise the WHERE clause as such, looking at the token stream.
So, my question is, am I using the parser correctly? Right now the token stream seems to be the most useful feature of the parser...
My Test project:
public static void Main(string[] args)
{
var parser = new TSql100Parser(false);
IList<ParseError> Errors;
IScriptFragment result = parser.Parse(
new StringReader("Select col from T1 where 1 = 1 group by 1;" +
"select col2 from T2;" +
"select col1 from tbl1 where id in (select id from tbl);"),
out Errors);
var Script = result as TSqlScript;
foreach (var ts in Script.Batches)
{
Console.WriteLine("new batch");
foreach (var st in ts.Statements)
{
IterateStatement(st);
}
}
}
static void IterateStatement(TSqlStatement statement)
{
Console.WriteLine("New Statement");
if (statement is SelectStatement)
{
PrintStatement(sstmnt);
}
}
Yes, you are using the parser correctly.
As Damien_The_Unbeliever points out, within the SelectStatement there is a QueryExpression property which will be a QuerySpecification object for your third select statement (with the WHERE clause).
This represents the 'real' SELECT bit of the query (whereas the outer SelectStatement object you are looking at has just got the 'WITH' clause (for CTEs), 'FOR' clause (for XML), 'ORDER BY' and other bits)
The QuerySpecification object is the object with the FromClauses, WhereClause, GroupByClause etc.
So you can get to your WHERE Clause by using:
((QuerySpecification)((SelectStatement)statement).QueryExpression).WhereClause
which has a SearchCondition property etc. etc.
Quick glance around would indicate that it contains a QueryExpression, which could be a QuerySpecification, which does have the Where clause attached to it.
if someone lands here and wants to know how to get the whole elements of a select statement the following code explain that:
QuerySpecification spec = (QuerySpecification)(((SelectStatement)st).QueryExpression);
StringBuilder sb = new StringBuilder();
sb.AppendLine("Select Elements");
foreach (var elm in spec.SelectElements)
sb.Append(((Identifier)((Column)((SelectColumn)elm).Expression).Identifiers[0]).Value);
sb.AppendLine();
sb.AppendLine("From Elements");
foreach (var elm in spec.FromClauses)
sb.Append(((SchemaObjectTableSource)elm).SchemaObject.BaseIdentifier.Value);
sb.AppendLine();
sb.AppendLine("Where Elements");
BinaryExpression binaryexp = (BinaryExpression)spec.WhereClause.SearchCondition;
sb.Append("operator is " + binaryexp.BinaryExpressionType);
if (binaryexp.FirstExpression is Column)
sb.Append(" First exp is " + ((Identifier)((Column)binaryexp.FirstExpression).Identifiers[0]).Value);
if (binaryexp.SecondExpression is Literal)
sb.Append(" Second exp is " + ((Literal)binaryexp.SecondExpression).Value);
I had to split a SELECT statement into pieces. My goal was to COUNT how many record a query will return. My first solution was to build a sub query such as
SELECT COUNT(*) FROM (select id, name from T where cat='A' order by id) as QUERY
The problem was that in this case the order clause raises the error "The ORDER BY clause is not valid in views, inline functions, derived tables, sub-queries, and common table expressions, unless TOP or FOR XML is also specified"
So I built a parser that split a SELECT statment into fragments using the TSql100Parser class.
using Microsoft.Data.Schema.ScriptDom.Sql;
using Microsoft.Data.Schema.ScriptDom;
using System.IO;
...
public class SelectParser
{
public string Parse(string sqlSelect, out string fields, out string from, out string groupby, out string where, out string having, out string orderby)
{
TSql100Parser parser = new TSql100Parser(false);
TextReader rd = new StringReader(sqlSelect);
IList<ParseError> errors;
var fragments = parser.Parse(rd, out errors);
fields = string.Empty;
from = string.Empty;
groupby = string.Empty;
where = string.Empty;
orderby = string.Empty;
having = string.Empty;
if (errors.Count > 0)
{
var retMessage = string.Empty;
foreach (var error in errors)
{
retMessage += error.Identifier + " - " + error.Message + " - position: " + error.Offset + "; ";
}
return retMessage;
}
try
{
// Extract the query assuming it is a SelectStatement
var query = ((fragments as TSqlScript).Batches[0].Statements[0] as SelectStatement).QueryExpression;
// Constructs the From clause with the optional joins
from = (query as QuerySpecification).FromClauses[0].GetString();
// Extract the where clause
where = (query as QuerySpecification).WhereClause.GetString();
// Get the field list
var fieldList = new List<string>();
foreach (var f in (query as QuerySpecification).SelectElements)
fieldList.Add((f as SelectColumn).GetString());
fields = string.Join(", ", fieldList.ToArray());
// Get The group by clause
groupby = (query as QuerySpecification).GroupByClause.GetString();
// Get the having clause of the query
having = (query as QuerySpecification).HavingClause.GetString();
// Get the order by clause
orderby = ((fragments as TSqlScript).Batches[0].Statements[0] as SelectStatement).OrderByClause.GetString();
}
catch (Exception ex)
{
return ex.ToString();
}
return string.Empty;
}
}
public static class Extension
{
/// <summary>
/// Get a string representing the SQL source fragment
/// </summary>
/// <param name="statement">The SQL Statement to get the string from, can be any derived class</param>
/// <returns>The SQL that represents the object</returns>
public static string GetString(this TSqlFragment statement)
{
string s = string.Empty;
if (statement == null) return string.Empty;
for (int i = statement.FirstTokenIndex; i <= statement.LastTokenIndex; i++)
{
s += statement.ScriptTokenStream[i].Text;
}
return s;
}
}
And to use this class simply:
string fields, from, groupby, where, having, orderby;
SelectParser selectParser = new SelectParser();
var retMessage = selectParser.Parse("SELECT * FROM T where cat='A' Order by Id desc",
out fields, out from, out groupby, out where, out having, out orderby);