I have a requirement where we need to show around 24k records which has 84 cols in one go, as user wants filtering on entire set of data.
So can we have virtual scrolling mechanism with ag-grid without lazy loading?? If so could you please here. Any example are most welcome for reference.
Having tried this sort of thing with a similar number of rows and columns, I've found that it's just about impossible to get reasonable performance, especially if you are using things like "framework" renderers. And if you enable grouping, you're going to have a bad time.
What my team has done to enable filtering and sorting across an entire large dataset includes:
We used the client-side row model - the grid's simplest mode
We only load a "page" of data at a time. This involves trial and error with a reasonable sample of data and the actual features that you are using to arrive at the maximum page size that still allows the grid to perform well with respect to scrolling / rendering.
We implemented our own paging. This includes display of a paging control, and fetching the next/previous page from the server. This obviously requires server-side support. From an ag-grid point of view, it is only ever managing one page of data. Each page gets completely replaced with the next page via round-trip to the server.
We implemented sorting and filtering on the server side. When the user sorts or filters, we catch the event, and send the sort/filter parameters to the server, and get back a new page. When this happens, we revert to page 0 (or page 1 in user parlance).
This fits in nicely with support for non-grid filters that we have elsewhere in the page (in our case, a toolbar above the grid).
We only enable grouping when there is a single page of data, and encourage our users to filter their data to get down to one page of data so that they can group it. Depending on the data, page size might be as high as 1,000 rows. Again, you have to arrive at page size on a case-by-case basis.
So, in short, when we have the need to support filtering/sorting over a large dataset, we do all of the performance-intensive bits on the server side.
I'm sure that others will argue that ag-grid has a lot of advanced features that I'm suggesting that you not use. And they would be correct, for small-to-medium sized datasets, but when it comes to handling large datasets, I've found that ag-grid just can't handle it with reasonable performance.
Related
I know that from most of today's REST APIs, web calls responses have to be paginated.
But, I don't see on the web any insight on how to select the ideal size of a batch returned by an API call: should it be 10, 100, 1000. To be short: on what factors should we base the reflection of the size of an API response?
Some people state that it should be based on the number of elements displayed by the UI. I don't agree with this, as not all APIs are directly linked with an UI, and, in any cases, modern REST APIs allow to chose the number of items in output batch with a configurable parameter up to a certain amount.
So, how could we define the value for this "maximum number of elements returned by an HTTP request"?
Should it be based on the payload size? Internal architecture of the API? Should it be based on performance measurement?
Any insight on this? I'm not really looking for an explicit figure, but more some techniques that could help to find the answer. The best for me would be the process followed by some successful APIs. But, I cannot find any.
My general approach to this is to:
Try to avoid paging
Make REST clients resilient against paging changes, by using next and previous links, instead of using a ?page= attribute. A REST client knows another page is available strictly by the appearance of the link.
I don't use hard rules or measurements to figure out when paging is needed. Paging is generally painful, so my first approach would be to try to figure out what requirements drives the need for paging, and if that requirement can be removed.
Once I've determined it's not possible to remove this requirement in another way, I would set the cut-off of a page as large as reasonable, to remove the likelyhood clients need to do additional requests.
If it's a closed API (used only by clients you control), pick whatever the UI wants. It's trivial to change. If clients can choose from several options, you can include a pageSize parameter. Or, better..
If it's an open API (used by clients you don't control), then let clients control what size paging they want. Rather than support a pageNumber parameter, support offset and limit. Offset is how many records to skip before starting to return records, and limit is the maximum number of records to return. If the client is not happy with how their request performs, they can adjust the parameters to suit their needs. Any upper limit your API has should be driven by what your service can handle. It's neither possible nor desirable for you to try to figure out the Magic Maximum Page Size that makes all clients happy and no clients sad.
Also, please note that none of this has anything to do with ReST, which is silent when it comes to paging.
I usually make a rough performance measurement by hand. I want as few requests as necessary, but do not want to risk timeouts.
So we run a downline report. That gathers everyone in the downline of the person who is logged in. Some people of clients run this with no problem as it returns less than 100 records.
Some people of clients however returns 4,000 - 6,000 rows which comes out to be about 8 MB worth of information. I actually had to up my buffer limit on my development machine to handle the large request.
What are some of the best ways to store this large piece of data and help prevent it from being run multiple times consecutively?
Can it be stored in a cookie?
Session is out of the question as this would eat up way to much memory on the server.
I'm open to pretty much anything at this point, trying to better streamline the old process into a much quicker efficient one.
Right now what is done, is it loads the entire recordset, it loops through the recordset building out the data into return_value cells.
Would this be better to turn into a jquery/ajax call?
The only main requirements are:
classic asp
jquery/javascript
T-SQL
Why not change the report to be paged? Phase 1: run the entire query, but the page only displays the right set of rows based on selected page. Now your response buffer problem is fixed. Phase 2: move the paging into the query using Row_Number(), now your database usage problem is fixed. Phase 3: offer the user an option of "display to screen" (using above) or "export to csv" where you can most likely export all the data, since csv is nice and compact.
Using a cookie seems unwise, given the responses to the question What is the maximum size of a web browser's cookie's key?.
I would suggest using ASP to create a file on the Web server and writing the data to that file. When the user requests the report, you can then determine if "enough time" has passed for it to be worth running the report again, or if the cached version is sufficient. User's login details could presumably be used for naming the file, or the Session.SessionID, or you could store something new in the user's session. Advantage of using their login would be that your cache of the report can exist longer than a user's session.
Taking Brian's Answer further, query page count, which would be records returned / items per page rounded up. Then join the results of every page query on client side. Pages start at a offset provided through the query. Now you have the full amount on the client without overflowing your buffer. And it can be tailored to an interface and user option (display x per page).
I have an iOS app that presents content in a tableView. I've added a 'like/dislike' feature that interacts with my database (I use Parse.com). Every time someone likes/dislikes a piece of content, the specifics are sent to the Parse database. For each piece of content, I'd like to calculate and display the percentage of 'likes' over 'likes' + 'dislikes'. This is pretty simple math, but I can't wrap my head around the best way of designing my database table and the most efficient way to calculate the 'liked' percentage for each piece of content before the tableView physically appears.
As it is, I already have a loop in my tableView's viewDidLoad which compares the content from another database table to the 'like/dislike' table to restore the 'like/dislike' button state of the user (if they already liked/disliked a piece of content).
At first, I thought of creating an array in the initial viewDidLoadloop. However, using the whereKey: equalTo: type of query for each piece content to simply find the amount of likes/dislikes takes forever. As predicted, it is very slow in cellForRowAtIndexPath as well.
Worst case, I can make these calculations server-side and just pull the 'liked' percentage. However, I'd like to implement this in the app somehow. I'm a complete beginner, so I may be going about this all wrong.
Here is the basis of my database table:
Edit: I've managed to build a server-side program that calculates the percentage of users that 'like' pieces of content. My app pulls this percentage from the database at runtime. To make the percentage change more responsive when the user 'likes' something, I locally calculate an updated percentage. The problem here is when the user exits the app and reopens, the data reloads. If the server-side program had not run recently, the app will display an old 'liked' percentage (the most up to date % would not be calculated yet). The two solutions I see to fix this are:
Run the server-side program every 1-3 min
Post more data to the database when someone likes content (this would involve additional database queries for every single 'like').
I think both of these options are way too expensive for what I'm trying to accomplish.
I'd suggest leaving the calculations to the server side, and responding with the information to utilize in the app. This will save you from processing and parsing the incoming results.
You have greater processing power on a Server than on a device.
If I am using Varnish to cache my entire documents, by what mechanism would you advise I increment a page view count as well.
For example, lets supose that I have an auction listing, such as ebay, and I would like to cache the entire page since I know it is never going to change.
How would you then increase the page view count of this listing.
Lets say that my application is running from Zend Framework.
Would it be correct to make an ESI (Edge Side Include) to a node.js server which increments a page view count in Redis?
I'm looking for something that wil be 100% supported and will yielf accurate page view request numbers. (I'm not concerned about duplicate requests either, I"ll handle that in my application logic to prevent one IP from nuking the page view count).
I would separate your statistics logic from your application. Use a small piece of javascript that requests a resource with a unique timestamp (e.g. an image like /statistics?pageId=3&ts=234234249). You can cache your complete page (no need to bother with ESI) and have the statistics handled by a fast (multiplexing) server like node.js, netty, tornado.
If you need the pageCount in your page, request a small piece of javascript/json data instead of an image and update the DOM in javascript.
This way, you can log better statistics (e.g. dimensions of the page), you minimize traffic and keep statistics a separate concern.
We're using Crystal 11 through their webserver. When we run a report, it does the Sql query and displays the first page of the report in the Crystal web reportviewer.
When you hit the next page button, it reruns the Sql query and displays the next page.
How do we get the requerying of the data to stop?
We also have multiple people running the same reports at the same time (it is a web server after all), and we don't want to cache data between different instances of the same report, we only want to cache the data in each single instance of the report.
The reason to have pagination is not only a presentation concern. With pagination the single most important advantage is lazy loading of data - so that in theory, depending on given filters, you load only what you need.
Just imagine if you have millions of records in your db and you load all of them. First of all is gonna be a hell of a lot slower, second you're fetching a lot of stuff you don't really need. All the web models nowadays are based on lazy loading rather than bulk loading. Think about Google App Engine: you can't retrieve more than 1000 records in a given transaction from the Google Datastore - and you know that if you'll only try and display them your browser will die.
I'll close with a question - do you have a performance issue of any kind?
If so, you probably think you'll make it better but it's probably not the case, because you'll reduce the load on the server but each single query will be much more resource consuming.
If not my advice is to leave it alone! :)