I am writing a RIA service, which is also exposed using SOAP.
One of its methods needs to read data from a very big table.
At the beginning I was doing something like:
public IQueryable<MyItem> GetMyItems()
{
return this.ObjectContext.MyItems.Where(x => x.StartDate >= start && x.EndDate <= end);
}
But then I stopped because I was worried about the performance.
As far as I understand MyItemsis fully loaded and "Where" just filters the elements that were loaded at the first access of the property MyItems. Because MyItemswill have really lots of rows, I don't think this is the right approach.
I tried to google a bit the question but no interesting results came up.
So, I was thinking I could create a new instance of the context inside the GetMyItems method and load MyItems selectively. Something like:
public IQueryable<MyItems> GetMyItems(string Username, DateTime Start, DateTime End)
{
using (MyEntities ctx = new MyEntities ())
{
var objQuery = ctx.CreateQuery<MyItems>(
"SELECT * FROM MyItems WHERE Username = #Username AND Timestamp >= #Start AND Timestamp <= #End",
new ObjectParameter("#Username", Username),
new ObjectParameter("#Start", Start),
new ObjectParameter("#End", End));
return objQuery.AsQueryable();
}
}
But I am not sure at all this is the correct way to do it.
Could you please assist me and point out the right approach to do this?
Thanks in advance,
Cheers,
Gianluca.
As far as I understand MyItemsis fully loaded and "Where" just filters the elements that were loaded at the first access of the property MyItems.
No. That's entirely wrong. Don't fix "performance problems" until you actually have them. The code you already have is likely to perform better than the code you propose replacing it with. It certainly won't behave in the way you describe. But don't take my word for it. Use the performance profiler. Use SQL Profiler. And test!
Related
I want to create queries using EF FromSqlInterpolated or FromSqlRaw that allows me to use Like clauses, but I don't know what is the right way to do it without opening the application to SqlInjection attacks.
One first approach has took me to the following code
var results = _context.Categories.FromSqlInterpolated(
$"Select * from Category where name like {"%" + partialName + "%"}");
First test worked fine, it returns results when providing expected strings, and returns nothing when i provide something like ';select * from Category Where name='Notes'--%';
Still I don't know much about SqlInjection, at least not enough to feel safe with the query shown before.
Does someone know if the query is safe, or if there is a right way to do it?
Thanks
From this document
The FromSqlInterpolated and ExecuteSqlInterpolated methods allow using
string interpolation syntax in a way that protects against SQL injection attacks.
var results = _context.Categories.FromSqlInterpolated(
$"Select * from Category where name like {"%" + partialName + "%"}");
Or you can also change your query to Linq-to-Entity like this way
var results = _context.Categories.Where(p => p.name.Contains(partialName ));
I was having some performance issues using SharpRepository, and after playing around the SQL Query Profiler I found the reason.
With EF I can do stuff like this:
var books = db.Books.Where(item => item.Year == '2016');
if (!string.IsNullorEmpty(search_author))
books = books.Where(item => item.Author.Contains(search_author);
return (books.ToList());
EF will not really do anything until books is used (last line) and then it will compile a query that will select only the small set of data matching year and author from the db.
But SharpRepository evaluates books at once, so this:
var books = book_repo.Books.FindAll(item => item.Year == '2016');
if (!string.IsNullorEmpty(search_author))
books = books.Where(item => item.Author.Contains(search_author);
return (books.ToList());
will compile a query like "select * from Books where Year == '2016'" at the first line, and get ALL those records from the database! Then at the second line it will make a search for the author within the C# code... That behaviour can be a major difference in performance when using large databases, and it explains why my queries timed out...
I tried using repo.GetAll().Where() instead of repo.FindAll().... but it worked the same way.
Am I misunderstanding something here, and is there a way around this issue?
You can use repo.AsQueryable() but by doing that you lose some of the functionality that SharpRepository can provide, like caching or and aspects/hooks you are using. It basically takes you out of the generic repo layer and lets you use the underlying LINQ provider. It has it's benefits for sure but in your case you can just build the Predicate conditionally and pass that in to the FindAll method.
You can do this by building an Expression predicate or using Specifications. Working with the Linq expressions does not always feel clean, but you can do it. Or you can use the Specification pattern built into SharpRepository.
ISpecification<Book> spec = new Specification<Book>(x => x.Year == 2016);
if (!string.IsNullorEmpty(search_author))
{
spec = spec.And(x => x.Author.Contains(search_author));
}
return repo.FindAll(spec);
For more info on Specifications you can look here: https://github.com/SharpRepository/SharpRepository/blob/develop/SharpRepository.Samples/HowToUseSpecifications.cs
Ivan Stoev provided this answer:
"The problem is that most of the repository methods return IEnumerable. Try repo.AsQueryable(). "
I have an existing app that I am trying to upgrade from MVC5/EF6 to MVC6/EF7. We dynamically create some of our SQL tables, and as a result, have made use of the
System.Data.Entity.Database.SqlQuery
method to automatically map to entities that we use throughout our application.
This method seems to have gone away (i.e. not part of
Microsoft.Data.Entity.Infrastructure.Database ) in EF7 (or is not yet implemented). Are there plans to re-implement this method in EF7 or is there another way to accomplish this? Our project is kind of dead in the water until we figure this out.
Edited on May 20, 2015
I've been trying to make this work with FromSql, since that's what's available in Beta4, but no matter what combination of concatenated string, parameters I try, I keep getting different versions of an "Incorrect Syntax near #xxxvariable" message.
var results = Set<AssessmentResult>().FromSql("dbo.GetAssessmentResults #FieldA='Data1', #FieldB='Data2', #UserId = 2303");
var results2 = Set<AssessmentResult>().FromSql("dbo.GetAssessmentResults #FieldA= {0}", intData);
Both of these calls result in
"Incorrect syntax near '#FieldA'"
Any ideas?
We recently introduced the .FromSql() extension method on DbSet. It has the added benefit that you can continue composing LINQ on top of it.
var customers = db.Customers
.FromSql("SELECT * FROM Customer")
.Where(c => c.Name.StartsWith("A"));
I am using AspecJ to capture the query being executed in each of the form and show the time each query takes to execute. We are using spring jdbc and my aspect look as below:
#Aspect
public class QueryProfilerAspect {
#Pointcut("call(* org.springframework.jdbc.core.simple.SimpleJdbcTemplate.query* (..))")
public void profileQuery() {
}
#Around("profileQuery()")
public Object profile(ProceedingJoinPoint thisJoinPoint) throws Throwable {
// System.out.println("Inside join point execution");
SimpleJdbcTemplate template = (SimpleJdbcTemplate) thisJoinPoint
.getTarget();
JdbcTemplate jdbcTemplate = (JdbcTemplate) template
.getNamedParameterJdbcOperations().getJdbcOperations();
DataSource ds = jdbcTemplate.getDataSource();
// System.out.println("Datasource name URL =="
// + ds.getConnection().getMetaData().getURL());
// System.out.println("Datasource name ==" + schemaName);
String sqlQuery = thisJoinPoint.getArgs()[0].toString();
final long start, end;
start = System.nanoTime();
Object ll = thisJoinPoint.proceed();
end = System.nanoTime();
long executionTime = ((end - start) / 1000) / 1000;
System.out.println("execution_time=" +executionTime + sqlquery="+sqlQuery );
return ll;
}
Functionality wise this works however if i put this i my application it makes the application too slow. I am using compile time weaving. And the aspect finds 1683 query* method calls within the application.
Is there anything I can do to optimize this. Any suggestion/help will be really appreciated.
First I would advice against using System.nanoTime(). It's precision using cold like that is atrocious and pretty much useless for measuring timespans. Certainly not better than System.currentTimeMillies() if you divide the result anyway.
What probably slows you down most though, are the the String operations performed at various places in your aspect. If you have to concatenate Strings, at least use a StringBuilder to do so before outputting it. That might be done by the optimizer already, but you can never be too sure. ;)
And... Sysout isn't exactly the way to go if you want logging - looking into one of the various logging implementations (slf4j with logback is my personal favourite, but there are others as well) will be worth your time.
Especially if you want to use the fact that Spring has the feature you are trying to build (, as asked and answered here before: Seeing the underlying SQL in the Spring JdbcTemplate?
(Edit: I know it's only the query, not the time measuring, but not having to worry about that should shave some time off your overhead as well.)
I'm using Symfony 1.2 with Doctrine. I have a Place model with translations in two languages. This Place model has also a nested set behaviour.
I'm having problems now creating a new place that belongs to another node. I've tried two options but both of them fail:
1 option
$this->mergeForm(new PlaceTranslationForm($this->object->Translation[$lang->getCurrentCulture()]));
If I merge the form, what happens is that the value of the place_id field id an array. I suppose is because it is waiting a real object with an id. If I try to set place_id='' there is another error.
2 option
$this->mergeI18n(array($lang->getCurrentCulture()));
public function mergeI18n($cultures, $decorator = null)
{
if (!$this->isI18n())
{
throw new sfException(sprintf('The model "%s" is not internationalized.', $this->getModelName()));
}
$class = $this->getI18nFormClass();
foreach ($cultures as $culture)
{
$i18nObject = $this->object->Translation[$culture];
$i18n = new $class($i18nObject);
unset($i18n['id']);
$i18n->widgetSchema['lang'] = new sfWidgetFormInputHidden();
$this->mergeForm($i18n); // pass $culture too
}
}
Now the error is:
Couldn't hydrate. Found non-unique key mapping named 'lang'.
Looking at the sql, the id is not defined; so it can't be a duplicate record (I have a unique key (id, lang))
Any idea of what can be happening?
thanks!
It looks like the issues you are having are related to embedding forms within each other, which can be tricky. You will likely need to do things in the updateObject/bind methods of the parent form to get it to pass its values correctly to its child forms.
This article is worth a read:
http://www.blogs.uni-osnabrueck.de/rotapken/2009/03/13/symfony-merge-embedded-form/comment-page-1/
It gives some good info on how embedding (and mergeing) forms work. The technique the article uses will probably work for you, but I've not used I18n in sf before, so it may well be that there is a more elegant solution built in?