When uploaded records from Admin Panel today and found a rare increase in the stats which cant able to understand. Have uploaded json with 113 records but in stats it displays as 298,000 !!! Cant able to understand why it is happening. Is it a problem in Algolia and is it advisable to trust Algolia Billing, as we cant able to find any Ticket support for Algolia too.
The information about the number of records should be good, if you click on the day view on the up right corner:
You should see a spike in the number of records. You can also check the number of operation done during the same period of time in the graph below, it should be higher to this number of record.
Best regards
oops problem is CSV file which was uploaded. Microsoft Excel generated CSV file with empty comma for more than a million lines (just comma alone without data). And when uploaded it seems Algolia confused and billed.
Related
we have some functionality where we search large amount of data using large sentence which user provide. provided sentence can have 1000+ words. now we're using text search to find the data, but it is taking a lot of time even it is getting crashed at some point. we can not use pagination because we need entire searched results in front end. so how can we fix this issue? any kind of help is welcome. Thanks in advance.
I am having an excessive amount of Firestore reads in the past few weeks. My system generally was processing about 60k reads per day. About 3 weeks ago it jumped to roughly 10 million a day and the past 2 days have hit over 40 million records in a single day! My user base has not grown, my code has not changed so there is no reason for this spike. I suspect an endpoint is being hit from someone outside the scope of my application that may be trying to penetrate or retrieve records. I have reached Firestore repeatedly for help with this as it becoming a huge loss every day this happens but they are unable to assist me.
Is there a way to trace an origin of read requests or more importantly see counts for which collections or documents are being read? This must be traceable somehow as Firestore bills you per read but I cannot seem to find it.
There is currently no source IP address tracking with Cloud Firestore. All reads fall under the same bucket, which is that "they happened".
If you're building a mobile app, now would be a good time to use security rules to limit which authenticated users can read and write what parts of your database, so that it's not just being accessed unlimited from anywhere on the internet.
I have been asked to develop a web application to be hosted on Sharepoint 2013 that has to work with lots of data. It basically consists in a huge form to be used to save and edit many information.
Unfortunately, due to some work restrictions, I do not have access to the backend, so it has to be done entirely client-side.
I am already aware on how to programmatically create sharepoint lists with site columns and save data on them with REST.
The problem is, I need to create a Sharepoint list (to be used as database) with at least 379 site columns (fields), of which 271 has to be single lines of text and 108 multiple lines of text, and by doing so I think I would exceed the threshold limit (too many site columns on a single list).
Is there any way I could make this work? Any other solution on how to save big amounts of data on Sharepoint, by only using client-side solutions (e.g. REST)? Maybe there is a way to save a XML or JSON file in any way on Sharepoint through REST?
I don't remember if there is actually some limit regarding columns in SP 2013. For sure there is a limit when You would be using lookup columns (up to 12 columns in one view), but since You are using only text columns this should not be blocking... and limit regarding number of rows that may be present in one view (5000 - normal user, 20 000 - admin user)
Best to check all here https://learn.microsoft.com/en-us/sharepoint/install/software-boundaries-and-limits
As described by MS:
The sum of all columns in a SharePoint list cannot exceed 8,000 bytes.
Also You may create one Note column and store all date in JSON structure which will be saved in this column or create Library not list and store JSON documents in list (just be aware that by default JSON format is blocked in SharePoint to be stored in lists and You need to change those settings in Central Admin for application pool -> link). Just be aware that with this approach You will be not able to use many OOB SharePoint features like column filters which might be handy
I have a script set up in Filemaker 11 which should import data from an excel sheet. There's a field containing a unique number in both the Filemaker database and the .xlsx file which is used to match already existing entries. "Update matching records in found set" and "Add remaining data as new record" are both enabled.
Unfortunately, Filemaker seems to behave completely arbitrarily here. Using the same script and the same .xlsx file several times in a row, the results are completely unpredictable. Sometimes already existing records are correctly skipped or updated sometimes they are added a second (or third or fifth …) time.
Is this a bug, maybe specific to version 11, which was sorted out later? Am I missing something about importing?
Here's my official answer to your question:
Imports into FileMaker databases are found set sensitive.
If you're importing records in a method that acts on existing records (update matching), FileMaker uses the found set showing in the layout on your active window to do the matching on rather than the entire table itself.
Regarding it's localness (per your comment above), it allows you to do more precise imports. In a case where you want to make sure you only match for specific records (e.g. you have a spreadsheet of data from employees at company A and you only want to update employee records for company A) you could perform a find and limit the found set to just your desired records before importing with matching turned on. This wat the import will ONLY look at the records in your found set to do it's evaluation. This means less processing because FM has to look at fewer records and also less risk that you're going to find a match that you didn't want (depending on what your match criteria is).
I'm having a hard time finding a good and up-to-date reference for you. All I can find is this one that is form FM10 days on Format. I would suggest bookmarking the FileMaker 13 help pages. It's the same set of help documents available when you use the Help menu in FileMaker Pro, but I find it much easier to search the docs via a browser.
I am working on a Website which is displaying all the apps from the App Store. I am getting AppStore data by their EPF Data Feeds through EPF Importer. In that database I get the pricing of each App for every store. There are dozen of rows in that set of data whose table structure is like:
application_price
The retail price of an application.
Name Key Description
export_date The date this application was exported, in milliseconds since the UNIX Epoch.
application_id Y Foreign key to the application table.
retail_price Retail price of the application, or null if the application is not available.
currency_code The ISO3A currency code.
storefront_id Y Foreign key to the storefront table.
This is the table I get now my problem is that I am not getting any way out that how I can calculate the price reduction of apps and the new free apps from this particular dataset. Can any one have idea how can I calculate it?
Any idea or answer will be highly appreciated.
I tried to store previous data and the current data and then tried to match it. Problem is the table is itself too large and comparing is causing JOIN operation which makes the query execution time to more than a hour which I cannot afford. there are approx 60, 000, 000 rows in the table
With these fields you can't directly determine price drops or new application. You'll have to insert these in your own database, and determine the differences from there. In a relational database like MySQL this isn't too complex:
To determine which applications are new, you can add your own column "first_seen", and then query your database to show all objects where the first_seen column is no longer then a day away.
To calculate price drops you'll have to calculate the difference between the retail_price of the current import, and the previous import.
Since you've edited your question, my edited answer:
It seems like you're having storage/performance issues, and you know what you want to achieve. To solve this you'll have to start measuring and debugging: with datasets this large you'll have to make sure you have the correct indexes. Profiling your queries should helping in finding out if they do.
And probably, your environment is "write once a day", and read "many times a minute". (I'm guessing you're creating a website). So you could speed up the frontend by processing the differences (price drops and new application) on import, rather than when displaying on the website.
If you still are unable to solve this, I suggest you open a more specific question, detailing your DBMS, queries, etc, so the real database administrators will be able to help you. 60 million rows are a lot, but with the correct indexes it should be no real trouble for a normal database system.
Compare the table with one you've downloaded the previous day, and note the differences.
Added:
For only 60 million items, and on a contemporary PC, you should be able to store a sorted array of the store id numbers and previous prices in memory, and do an array lookup faster than the data is arriving from the network feed. Mark any differences found and double-check them against the DB in post-processing.
Actually I also trying to play with these data, and I think best approach for you base on data from Apple.
You have 2 type of data : full and incremental (updated data daily). So within new data from incremental (not really big as full) you can compare only which record updated and insert them into another table to determine pricing has changed.
So you have a list of records (app, song, video...) updated daily with price has change, just get data from new table you created instead of compare or join them from various tables.
Cheers