The datagrid that I use on the client is based on SQL row number; it also requires a total number of pages for its paging. I also use the PagedList on the server.
SQL Profiler shows that the PagedList makes 2 db calls - the first to get the total number of records and the second to get the current page. The thing is that I can't find a way to extract that total number of records from the PagedList. Therefore, currently I have to make an extra call to get that total which creates 3 calls in total for each request, 2 of which are absolutely identical. I understand that I probably won't be able to rid of the call to get the totals but I hate to call it twice. Here is an extract from my code, I'd really appreciate any help in this:
var t = from c in myDb.MyTypes.Filter<MyType>(filterXml) select c;
response.Total = t.Count(); // my first call to get the total
double d = uiRowNumber / uiRecordsPerPage;
int page = (int)Math.Ceiling(d) + 1;
var q = from c in myDb.MyTypes.Filter<MyType>(filterXml).OrderBy(someOrderString)
select new ReturnType
{
Something = c.Something
};
response.Items = q.ToPagedList(page, uiRecordsPerPage);
PagedList has a .TotalItemCount property which reflects the total number of records in the set (not the number in a particular page). Thus response.Items.TotalItemCount should do the trick.
Related
I'm estimating last mile delivery costs in an large urban network using by-route distances. I have over 8000 customer agents and over 100 retail store agents plotted in a GIS map using lat/long coordinates. Each customer receives deliveries from its nearest store (by route). The goal is to get two distance measures in this network for each store:
d0_bar: the average distance from a store to all of its assigned customers
d1_bar: the average distance between all customers common to a single store
I've written a startup function with a simple foreach loop to assign each customer to a store based on by-route distance (customers have a parameter, "customer.pStore" of Store type). This function also adds, in turn, each customer to the store agent's collection of customers ("store.colCusts"; it's an array list with Customer type elements).
Next, I have a function that iterates through the store agent population and calculates the two average distance measures above (d0_bar & d1_bar) and writes the results to a txt file (see code below). The code works, fortunately. However, the problem is that with such a massive dataset, the process of iterating through all customers/stores and retrieving distances via the openstreetmap.org API takes forever. It's been initializing ("Please wait...") for about 12 hours. What can I do to make this code more efficient? Or, is there a better way in AnyLogic of getting these two distance measures for each store in my network?
Thanks in advance.
//for each store, record all customers assigned to it
for (Store store : stores)
{
distancesStore.print(store.storeCode + "," + store.colCusts.size() + "," + store.colCusts.size()*(store.colCusts.size()-1)/2 + ",");
//calculates average distance from store j to customer nodes that belong to store j
double sumFirstDistByStore = 0.0;
int h = 0;
while (h < store.colCusts.size())
{
sumFirstDistByStore += store.distanceByRoute(store.colCusts.get(h));
h++;
}
distancesStore.print((sumFirstDistByStore/store.colCusts.size())/1609.34 + ",");
//calculates average of distances between all customer nodes belonging to store j
double custDistSumPerStore = 0.0;
int loopLimit = store.colCusts.size();
int i = 0;
while (i < loopLimit - 1)
{
int j = 1;
while (j < loopLimit)
{
custDistSumPerStore += store.colCusts.get(i).distanceByRoute(store.colCusts.get(j));
j++;
}
i++;
}
distancesStore.print((custDistSumPerStore/(loopLimit*(loopLimit-1)/2))/1609.34);
distancesStore.println();
}
Firstly a few simple comments:
Have you tried timing a single distanceByRoute call? E.g. can you try running store.distanceByRoute(store.colCusts.get(0)); just to see how long a single call takes on your system. Routing is generally pretty slow, but it would be good to know what the speed limit is.
The first simple change is to use java parallelism. Instead of using this:
for (Store store : stores)
{ ...
use this:
stores.parallelStream().forEach(store -> {
...
});
this will process stores entries in parallel using standard Java streams API.
It also looks like the second loop - where avg distance between customers is calculated doesn't take account of mirroring. That is to say distance a->b is equal to b->a. Hence, for example, 4 customers will require 6 calculations: 1->2, 1->3, 1->4, 2->3, 2->4, 3->4. Whereas in case of 4 customers your second while loop will perform 9 calculations: i=0, j in {1,2,3}; i=1, j in {1,2,3}; i=2, j in {1,2,3}, which seems wrong unless I am misunderstanding your intention.
Generally, for long running operations it is a good idea to include some traceln to show progress with associated timing.
Please have a look at above and post results. With more information additional performance improvements may be possible.
I am trying to make some kind of paging. But, I need to do it on a grouped result, because every time I do a page. It is a requirement that all data for a given group is fetched.
Below code:
var erere = dbCtx.StatusViewList
.GroupBy(p => p.TurbineNumber)
.OrderBy(p => p.FirstOrDefault().TurbineNumber)
.Skip(0)
.Take(10)
.ToList();
I have 200k items and the statement above seems to be so slow the connection times out. My best bet is its the orderby that slows it down. Any suggestions how to do this, or how to speed the statement above up?
At your case, grouping on server side is not needed at all, because anyway you will get all data, but with additional overhead on server side. So try another approach:
var groupPage = dbCtx.StatusViewList.Select(x => TurbineNumber)
.Distinct().OrderBy(x => x.TurbineNumber).Skip(40).Take(20).ToList();
var data = dbCtx.StatusViewList.Where(x => groupPage.Contains(x.TurbineNumber))
.ToList().GroupBy(x => x.TurbineNumber).ToList();
The GroupBy needs to visit all elements to group all StatusViews into groups of StatusViews that have equal TurbineNumber.
After that, you take every group, from every group your take the first element and ask for its TurbineNumber, to sort by Turbine Number.
Apparently you take into account that a group of StatusViews might be empty (FirstOrDefault, instead of First), but then again, you assume that FirstOrDefault never returns null.
One of the things that could speed up your query is using the Key of your groups. The Key is the element on which you grouped, in your case the TurbineNumber: All elements in the a group have the same TurbineNumber.
var result = dbCtx.StatusViewList
.GroupBy(statusView => statusView.TurbineNumber)
.OrderBy(group => group.Key)
...
I think that will be a first step to improve performance.
However, you return a fixed number of Groups. Some Groups might be huge, 1000s of elements, some groups might be small: only one element. So the result of one page could be 10 groups, each with 1000 elements, having a total of 10000 elements. It could also be 10 groups, each with 1 element, a total of 10 elements. I'm not sure if this would be the result you want by paging.
Wouldn't you prefer a page that always has the same number of elements, preferably with the same TurbineNumber, If there are not many same TurbineNumbers fill the rest of your page with the next TurbineNumber. If there are too many StatusViews with this TurbineNumber divide them into several pages?
Something like:
TurbineNumber StatusView
4 A
4 B
4 F
5 D
5 K
6 C
6 Z
6 Q
6 W
7 E
To do this, don't GroupBy, use OrderBy and then Skip and Take
IEnumerable<StatusView> GetPage(int pageNr, int pageSize)
{
return dbCtx.StatusViewList
.Orderby(statusView => statusView.TurbineNumber)
.Skip(pageNr * pageSize)
.Take(pageSize)
}
If you create an extra index for TurbineNumber, this will be very fast:
In your DbContext.OnModelCreating(DbModelBuilder modelBuilder):
// Add an extra index on TurbineNumber:
var indexAttribute = new IndexAttribute("TurbineIndex", 0) {IsUnique = false}
var indexAnnotation =new IndexAnnotation(indexAttribute);
modelBuilder.Entity<Statusview>()
.Property(statusView => statusView.TurbineNumber)
.HasColumnAnnotation("MyIndexName", indexAnnotation);
I'm trying to get whether the product's stock status is instock/outofstock (Integers representing each state are fine. i don't necessarily need the "in stock"/"out of stock" strings per se).
I've tried various things to no avail.
1)
$inStock = $obj->get('Magento\CatalogInventory\Api\Data\StockItemInterface')->getisInStock()'
// Magento\CatalogInventory\Api\Data\StockItemInterface :: getisInStock returns true no matter what, even for 0qty products
// summary: not useful. How do you get the real one?
2)
$inStock = $obj->get('\Magento\CatalogInventory\Api\StockStateInterface')->verifyStock($_product->getId());
// test results for "verifyStock":
// a 0 qty product is in stock
// a 0 qty product is out of stock
// summary: fail. find correct method, with tests.
3)
$stockItemRepository = $obj->get('Magento\CatalogInventory\Model\Stock\StockItemRepository');
stockItem = $stockItemRepository->get($_product->getId());
$inStock = $stockItem->getIsInStock();
// Uncaught Magento\Framework\Exception\NoSuchEntityException: Stock Item with id "214"
// summmary: is stockitem not 1to1 with proudctid?
The weird thing is, getting stock quantities works just fine.
$availability = (String)$obj->get('\Magento\CatalogInventory\Api\StockStateInterface')->getStockQty($_product->getId(), $_product->getStore()->getWebsiteId());
So why isn't getIsInStock working?
This was one way I did it.
$stockItemResource = $obj->create('Magento\CatalogInventory\Model\ResourceModel\Stock\Item');
// grab ALL stock items (i.e. object that contains stock information)
$stockItemSelect = $stockItemResource->getConnection()->select()->from($stockItemResource->getMainTable());
$stockItems = $stockItemResource->getConnection()->fetchAll($stockItemSelect);
$inStock = null;
foreach($stockItems as $k => $item) {
if ($item['product_id'] == $_productId) {
$inStock = $item['is_in_stock'];
break; // not breaking properly. 'qz' still prints
}
}
Notes on efficiency:
I'm sure there are another ways to target the single item specifically, instead of getting all. Either through a method, or by adjusting the query passed in somehow.
But this method is probably more efficient for large n, avoiding the n+1 query problem.
You do still end up iterating through a lot, but perhaps theta(n) of iterating through a cached PHP variable is probably lower than n+1 querying the database. Haven't tested, just a hypothesis.
The returned structure is an array of arrays, where the sub-array (which also happens to be a stock item) has the product ID and the stock status value. And because the product ID and the stock status value is on the same level of nesting, we have no choice but to iterate through each sub-array to check the product_id, choose that sub-array, and grab the stock value. In short, we can't just utilize the hashmap, since the keys of the sub-array are not product IDs.
Ultimately, the efficiency of this depends on your use case. Rarely will you grab all stock items, unless doing mass exports. So the ultimate goal is to really just stay within the configured time limit is allowed for a request to persist.
I have a report that should read values from 2 dataset by Currency:
Dataset1: Production Total
Dataset2: Net Total
Ive tried to use:
Lookup(Fields!Currency_Type.Value,
Fields!Currency_Type1.Value,
Fields!Gross_Premium_Amount.Value,
"DataSet2")
This returns only the first amount from dataset 2.
I've tried Lookupset function as well but it didn't SUM the retrieved values.
Any help would be appreciated.
Thanks Jamie for the reply.
THis is what i have done and it worked perfect:
From Report Properties--> Code , write the below function:
Function SumLookup(ByVal items As Object()) As Decimal
If items Is Nothing Then
Return Nothing
End If
Dim suma As Decimal = New Decimal()
Dim ct as Integer = New Integer()
suma = 0
ct = 0
For Each item As Object In items
suma += Convert.ToDecimal(item)
Next
If (ct = 0) Then return 0 else return suma
End Function
Then you can call the function:
code.SumLookup(LookupSet(Fields!Currency_Type.Value, Fields!Currency_Type1.Value,Fields!Gross_Premium_Amount.Value, "DataSet2"))
Yes, Lookup will only return the first matching value. Three options come to mind:
Change your query, so that you only need to get one value: use a GROUP BY and SUM(...) to combine your two rows in the query. If you are using this query other places, then make a copy and change that.
Is there some difference in the rows? Such as one is for last year and one is for this year? If so, create an artificial lookup key and lookup the two values separately:
=Lookup(Fields!Currency_Type.Value & ","
& YEAR(DATEADD(DateInterval.Year,-1,today())),
Fields!Currency_Type1.Value & ","
& Fields!Year.Value,
Fields!Gross_Premium_Amount.Value,
"DataSet2")
+
Lookup(Fields!Currency_Type.Value & ","
& YEAR(today()),
Fields!Currency_Type1.Value & ","
& Fields!Year.Value,
Fields!Gross_Premium_Amount.Value,
"DataSet2")
Use the LookupSet function as mentioned. With this you'll get a collection of the values back, and then need to add those together. The easiest way to do this is with embedded code in the report. Add this function to the report's code:
Function AddList(ByVal items As Object()) As Double
If items Is Nothing Then
Return 0
End If
Dim Total as Double
Total = 0
For Each item As Object In items
Total = Total + CDbl(item)
Next
Return Total
End Function
Now call that with:
=Code.AddList(LookupSet(Fields!Currency_Type.Value,
Fields!Currency_Type1.Value,
Fields!Gross_Premium_Amount.Value,
"DataSet2"))
(Note: this code was not tested. I just composed it in the Stack Overflow edit window & I'm no fan of VB. But it should give you a good idea of what to do.)
I have a mongo collection with documents. There is one field in every document which is 0 OR 1. I need to random sample 1000 records from the database and count the number of documents who have that field as 1. I need to do this sampling 1000 times. How do i do it ?
For people coming to the answer, you should now use the new $sample aggregation function, new in 3.2.
https://docs.mongodb.org/manual/reference/operator/aggregation/sample/
db.collection_of_things.aggregate(
[ { $sample: { size: 15 } } ]
)
Then add another step to count up the 0s and 1s using $group to get the count. Here is an example from the MongoDB docs.
For MongoDB 3.0 and before, I use an old trick from SQL days (which I think Wikipedia use for their random page feature). I store a random number between 0 and 1 in every object I need to randomize, let's call that field "r". You then add an index on "r".
db.coll.ensureIndex(r: 1);
Now to get random x objects, you use:
var startVal = Math.random();
db.coll.find({r: {$gt: startVal}}).sort({r: 1}).limit(x);
This gives you random objects in a single find query. Depending on your needs, this may be overkill, but if you are going to be doing lots of sampling over time, this is a very efficient way without putting load on your backend.
Here's an example in the mongo shell .. assuming a collection of collname, and a value of interest in thefield:
var total = db.collname.count();
var count = 0;
var numSamples = 1000;
for (i = 0; i < numSamples; i++) {
var random = Math.floor(Math.random()*total);
var doc = db.collname.find().skip(random).limit(1).next();
if (doc.thefield) {
count += (doc.thefield == 1);
}
}
I was gonna edit my comment on #Stennies answer with this but you could also use a seprate auto incrementing ID index here as an alternative if you were to skip over HUGE amounts of record (talking huge here).
I wrote another answer to another question a lot like this one where some one was trying to find nth record of the collection:
php mongodb find nth entry in collection
The second half of my answer basically describes one potential method by which you could approach this problem. You would still need to loop 1000 times to get the random row of course.
If you are using mongoengine, you can use a SequenceField to generate an incremental counter.
class User(db.DynamicDocument):
counter = db.SequenceField(collection_name="user.counters")
Then to fetch a random list of say 100, do the following
def get_random_users(number_requested):
users_to_fetch = random.sample(range(1, User.objects.count() + 1), min(number_requested, User.objects.count()))
return User.objects(counter__in=users_to_fetch)
where you would call
get_random_users(100)