So we run a downline report. That gathers everyone in the downline of the person who is logged in. Some people of clients run this with no problem as it returns less than 100 records.
Some people of clients however returns 4,000 - 6,000 rows which comes out to be about 8 MB worth of information. I actually had to up my buffer limit on my development machine to handle the large request.
What are some of the best ways to store this large piece of data and help prevent it from being run multiple times consecutively?
Can it be stored in a cookie?
Session is out of the question as this would eat up way to much memory on the server.
I'm open to pretty much anything at this point, trying to better streamline the old process into a much quicker efficient one.
Right now what is done, is it loads the entire recordset, it loops through the recordset building out the data into return_value cells.
Would this be better to turn into a jquery/ajax call?
The only main requirements are:
classic asp
jquery/javascript
T-SQL
Why not change the report to be paged? Phase 1: run the entire query, but the page only displays the right set of rows based on selected page. Now your response buffer problem is fixed. Phase 2: move the paging into the query using Row_Number(), now your database usage problem is fixed. Phase 3: offer the user an option of "display to screen" (using above) or "export to csv" where you can most likely export all the data, since csv is nice and compact.
Using a cookie seems unwise, given the responses to the question What is the maximum size of a web browser's cookie's key?.
I would suggest using ASP to create a file on the Web server and writing the data to that file. When the user requests the report, you can then determine if "enough time" has passed for it to be worth running the report again, or if the cached version is sufficient. User's login details could presumably be used for naming the file, or the Session.SessionID, or you could store something new in the user's session. Advantage of using their login would be that your cache of the report can exist longer than a user's session.
Taking Brian's Answer further, query page count, which would be records returned / items per page rounded up. Then join the results of every page query on client side. Pages start at a offset provided through the query. Now you have the full amount on the client without overflowing your buffer. And it can be tailored to an interface and user option (display x per page).
Related
Theres a ton of videos and websites trying to explain backend vs frontend, but unfortunately none of them explains it in a way that you know how to develop a backend - driven website (at least I haven't found anything good).
So, I wanted to ensure that I understood it and kindly ask you to confirm or correct me on this topic.
Example:
I wanted to build Mini - Google. I have a Database containing 1000 stored websites.
Assumption #1:
Everytime I type something into the search bar, the autofill suggestions change. This means, everytime i type, another website / API gets called returning the current autofill suggestions. On a developer site, this means the website e.g. is a Python script which gets called with the current word typed in as a Parameter and is returning all suggestions as e.g. JSON:
// Client Side Script
function ontype(input):
suggestions = get("https://api.googlemini.com/suggestions?q=" + str(input))
show(suggestions)
Assumption #2:
This also means I could manually call the website containing the Python script, providing a random word and it would always return a JSON containing the autofill suggestions for that word.
Question #1:
If A#1 turns out true but A#2 turns out false, how could I prevent a user from randomly accessing the "API" while still returning results when called by a script?
Assumption #3:
After pressing enter, my website googlemini.com/search?... would be called. As google.com/search reloads everytime searching for a new query (or going to page 2 etc.), I assume, instead of calling an API, when the server gets the client request, it first searches through its database, sorts the results and then returns a whole html as a static webpage:
// Server Side Script
#app.route("/search")
function oncall():
query = getparam("q")
results = searchdatabase(query)
html = buildhtml(results)
return html
Question #2:
Often, I hear (or at least understand it this way) that database and webserver are 2 seperate servers. How would that work? Wouldn't that mean the database server needs to be accessible to the web too (of course it would have security layers etc., but technically it would)? How could I access the database server from the webserver?
Question #3:
Are there, on a technical basis, any other ways to build backend services?
That's it. I would also appreciate any recommendations like videos, websites or others to learn how to technically setup and / or secure backend servers.
Thanks in advance.
For your first question you can yes there is a way to prevent miss use.
What you can do is add identifier to api like Auth token to identify a user and every time a user access the api you can save the count on the server n whenever the count has exceeded a limit within a time span you can reject the call. And the limit can be set in such a way that it doesn't trouble the honest user and punishes the wrong one. There are even more complex and effective methods but this is the basic idea.
For question number to let me explain you a simple concept a database is a very efficient, resourcefull and expensive data storage solution we never want it to be used in a general sense as varible store or something. We always want to access the database in call get the data process the data update the data. So we do it data way and its not necessary you make sepreate server for data base. The thing is we mostly make databse to be accessible to various platforms android, ios, windows. So its better to add some abstraction and keep data base as a separte entity.
For the last, I am not well aware about what you meant by other but I am listing some backend teechnologies, some of these might be used in isolation some of these not some other tools as well.
Django
FLask
Djnago rest
GraphQL
SQL
PHP
Node
Deno
I use a datasheet view of a query with aggregate sub queries attached as fields. Of course this is not editable and that is fine as its merely an overview listing of all the records along with some sum information from related tables. I have noticed that when a query is not editable the record selector lock information is not displayed. This made me wonder.
Is there is some event that can be captured to display in more or less real time when a record is locked or released by other users?
Alternatively is there any other way to display in my overview list or elsewhere what records are currently locked and if possible by what user?
Access 2010(x64)
For an updatable query, the locked status may be displayed on the left margin as you have noted. But that reflects record-locking by the query engine, not the same thing as whether a data result is updateable under normal circumstances.
For a read-only query, Access won't show a lock icon because in that context it isn't useful information (from most people's point of view).
You could use VBA to check the attribute of the query as a whole, and display a notification when the form is loaded. But that doesn't relate to the record-locking icon.
Is there is some event that can be captured to display in more or less real time when a record is locked or released by other users? -- I believe the simple answer is no.
Access 2007 saw the end of the JET Security model, so there is no way for you to manage user-level security in files created using 2007 or later.
The only alternative would be to use the Win API to register users by their NT ids, and to develop your own model which responded to activity. Clearly this would be no mean feat!
[Edit]
As for detecting record locks, it's possible you could implement this using an events handler class together with the ADO library:
http://msdn.microsoft.com/en-gb/library/windows/desktop/ms678373%28v=vs.85%29.aspx
If you don't mind getting your hands dirty with Class Modules (something some pundits never got to grips with), then you can find a lead-in here.
I have an iOS app that presents content in a tableView. I've added a 'like/dislike' feature that interacts with my database (I use Parse.com). Every time someone likes/dislikes a piece of content, the specifics are sent to the Parse database. For each piece of content, I'd like to calculate and display the percentage of 'likes' over 'likes' + 'dislikes'. This is pretty simple math, but I can't wrap my head around the best way of designing my database table and the most efficient way to calculate the 'liked' percentage for each piece of content before the tableView physically appears.
As it is, I already have a loop in my tableView's viewDidLoad which compares the content from another database table to the 'like/dislike' table to restore the 'like/dislike' button state of the user (if they already liked/disliked a piece of content).
At first, I thought of creating an array in the initial viewDidLoadloop. However, using the whereKey: equalTo: type of query for each piece content to simply find the amount of likes/dislikes takes forever. As predicted, it is very slow in cellForRowAtIndexPath as well.
Worst case, I can make these calculations server-side and just pull the 'liked' percentage. However, I'd like to implement this in the app somehow. I'm a complete beginner, so I may be going about this all wrong.
Here is the basis of my database table:
Edit: I've managed to build a server-side program that calculates the percentage of users that 'like' pieces of content. My app pulls this percentage from the database at runtime. To make the percentage change more responsive when the user 'likes' something, I locally calculate an updated percentage. The problem here is when the user exits the app and reopens, the data reloads. If the server-side program had not run recently, the app will display an old 'liked' percentage (the most up to date % would not be calculated yet). The two solutions I see to fix this are:
Run the server-side program every 1-3 min
Post more data to the database when someone likes content (this would involve additional database queries for every single 'like').
I think both of these options are way too expensive for what I'm trying to accomplish.
I'd suggest leaving the calculations to the server side, and responding with the information to utilize in the app. This will save you from processing and parsing the incoming results.
You have greater processing power on a Server than on a device.
I have a fairly long HTML form that the user fills out. After filling it out, the user is given a preview of the data they are submitting. From there they are able to commit the data into the system or go back and edit it. I'm wondering what the best approach to handling this preview step is. Some ideas I had are:
Store the form data in a cookie for
the preview
Store the form data in a
session
Put the data in the DB, with
a status column indicating it's a
preview
What do you usually do when creating a preview like this? Are there other issues to consider?
Put the data as hidden fields ().
Why not cookie or session?
- If the user decide to discard this data, he may just navigate to other page. When he return later and see the data intact, he maybe surprised.
Why not database?
- If the user just close the browser, who clean up the data in your db? ... I would rather not write a cron job for this.
I'm not sure if it's the best-practise, but when I did this task, I've put it in a session. I expected the user to preview and submit/reedit the data during just single session so the session was enough for me.
If you want you preview to persist on your users machine you should use a cookie - that means the user doesn't have to sumbit/reedit the preview during single session, but can close the browser between this operations, and than return back to the preview in next session. Using this aproach, you have to consider that user can deny cookies in his browser. That's why people usually combine sessions with cookies together.
Putting the data in a database (with a status column) is not necessary unless you want to track and store the previews and edit operations somehow. You can imagine the database as a drawer in your table - you put there papers with whatever you want to store and find later. If you're just drawing a preview draft, and after the result is submitted, only a final version is stored in a drawer/database and the preview is crumpled and thrown away, than you won't put this in database. But if for some reason you think you will later go through the drafts, then they have to be stored in a database.
I'm not sure if it's clear with my english, but I did my best :D
I'd gauge it based on how difficult the form was to fill out in the first place. If it's a lengthy process (like information for a mortgage or something) and you have user logins, you may want to provide them an opportunity to save the uncompleted form and come back to it later.
A session is only good for (depending on your setup) tasks that will take less than an hour. Manual input of data (like CD/DVD cataloging) that is easy to start and easy to finish will be perfect to store a session. Likewise, if the person has to stop and root around for some documents (again, in the case of a mortgage app, or an online tax form, etc.), you'll have a really irate person if the session times out and they have to retype information.
I'd avoid directly injecting content into a cookie, since the data is passed for subsequent requests and, assumedly, you already have access to basic session functionality.
If you go with a DB, you will need to timestamp the access (assuming you don't just leave it around with some saved name as determined by your user, like 'My 2008 Mortgage Documents') so you can clean it up later. If the user does save it mid-form, just leave it around until they complete the form or delete it.
We're using Crystal 11 through their webserver. When we run a report, it does the Sql query and displays the first page of the report in the Crystal web reportviewer.
When you hit the next page button, it reruns the Sql query and displays the next page.
How do we get the requerying of the data to stop?
We also have multiple people running the same reports at the same time (it is a web server after all), and we don't want to cache data between different instances of the same report, we only want to cache the data in each single instance of the report.
The reason to have pagination is not only a presentation concern. With pagination the single most important advantage is lazy loading of data - so that in theory, depending on given filters, you load only what you need.
Just imagine if you have millions of records in your db and you load all of them. First of all is gonna be a hell of a lot slower, second you're fetching a lot of stuff you don't really need. All the web models nowadays are based on lazy loading rather than bulk loading. Think about Google App Engine: you can't retrieve more than 1000 records in a given transaction from the Google Datastore - and you know that if you'll only try and display them your browser will die.
I'll close with a question - do you have a performance issue of any kind?
If so, you probably think you'll make it better but it's probably not the case, because you'll reduce the load on the server but each single query will be much more resource consuming.
If not my advice is to leave it alone! :)