I'd like to analyze our db and create better indices for it.
Because our app is very complex, and we don't know what are the most used parts of our app, I'd like to somehow see what are the most used read queries that we hit our db with.
That would make it very easy for me to analyze and create the right indices for them.
Any ideas on how to do that?
you can enable database profiling for this.
get the details here - https://docs.mongodb.com/v3.2/tutorial/manage-the-database-profiler/
alternatively a simpler way would be to use the mongostat (details here -https://docs.mongodb.com/v3.2/administration/monitoring/) which captures and returns the counts of database operations by type (e.g. insert, query, update, delete, etc.).
Related
I have about 50 users that needs to be removed. I can drop the users one after another by running drop command but I was wondering if I could get an answer that would significantly reduce the number of queries. Fairly new to psql, just wanted to know if there exists an efficient way or a function that would help me achieve what I'm looking for.
I am working on a project which uses graphql and PostgreSQL where we want to select data from the database with a value after a certain date. It is currently selecting all data from the database and then filtering it on the server:
.filter(({time}) => moment(time).isAfter(startTime))
However I would have thought it would be best to do this filtering in the database query as the full dataset is never used.
Is there any benefit to doing it on the server rather than in the database query?
Barring some unusual edge case -- such as other parts of your backend code really do need all the data for some reason -- it would definitely be more efficient to filter everything on the Postgres side via the SQL that is being used to fetch the data in the first place.
This is true for several reasons:
Assuming the table is properly indexed, the filtering will be able to occur much faster within the database.
The unneeded data will not need to be serialized and sent over the wire to the backend, only to then be discarded by the backend's own filtering.
The memory footprint should be reduced on both the Postgres and server end due to needing to process only a portion of the results.
I've not worked with GraphQL myself, but from doing a bit of poking around through its docs, it appears GraphQL often uses other mechanisms in different layers (outside of the database) to try to improve performance.
It would be worth seeing what the actual SQL is that your GraphQL query is generating (that may be possible via a function in GraphQL; it could also be done by enabling certain log settings on the Postgres server and correlating the log output to the query). That may lead to further optimization possibilities if you want to keep things purely GraphQL.
Jumping down to a raw query seems like it would be a good possibility though. Certainly that is something that is often done with ORMs like Django and ActiveRecord.
I'm currently building an OLAP database in postgres and want to compare the performance of a column-store vs row-store database. CitusDB open-sourced its columnar-store extension cstore_fdw so I'm comparing database performance with and without this extension.
The example shows how to make a test db and query it. I have that example running. But then I try to add indices to it to and get the error ERROR: cannot create index on foreign table "table_name". It makes sense that I can't add indices to a foreign table. Yet, I still need to index that table, or else there's no way it will do well slicing or drilling into the data. How do I do this?
cstore_fdw currently doesn't support PostgreSQL indexes. But it automatically stores some min/max statistics in skip indexes which makes execution of some queries much more efficient.
To learn more about how to use skip indexes please consult documentation.
We have an app that uses postgres database, that has about 50 tables. Each table contains about 3 Million records (on average). The tables get updated with new data every now and than. Now, we want to implement search feature in our app. The search needs to be performed on one table at a time (no joins needed).
I've read about postgres full text support and that looks promising. But it seems that Solr is Super fast in comparison to it. Can I use my existing postgres database with Solr? If tables get updated would I need to re-index everything again?
It is definitely worth giving Solr a try. We moved many MySQL queries involving JOINs on multiple tables with sorting on different fields to Solr. We are very happy with Solr's search speed, sort speed, faceting capabilities and highly configurable text analysis/tokenization options.
If tables get updated would I need to re-index everything again?
No, you can run delta imports to only re-index your new and updated documents. See https://wiki.apache.org/solr/DataImportHandler.
Get started with https://lucene.apache.org/solr/4_1_0/tutorial.html and all the links in there.
Since nobody has leapt in, I'll answer.
I'm afraid it all depends. It depends on (at least)
how big the text is in each "document"
how flexible you want your searching to be
how much integration you need between database and text-search
how fast is fast enough
how much experience you have with both
When I've had a database that needs some text searching, I've just used PG's built-in options. If I didn't have superuser access to the db, or was already running a big Java setup then Solr might well have appealed.
I'm committed along the route of using SQLite without core data.
I need to speed up a function which performs some database transactions after querying the database. I've created a dictionary for the rows with all the values I'll need.
I need to do this to avoid the database locking.
At the moment I'm calling my add record to database function, which opens and closes the database each time.
Obviously this is where the process is slow.
I was thinking that it's common for apps to be embedded with a database setup script, so it must be possible to run a batch of queries.
So I'm thinking if I can build up a string with all my queries I could just execute that.
But I'm not 100% this is the best approach or how to execute batch queries.
Can anyone advise me how to proceed?
For starters .. check out these links:
how-do-i-improve-the-performance-of-sqlite
ios-coredata-batch-insert (Yes I know that you said no core data - but it is worth a read)
fast-bulk-inserts-into-sqlite (Looks similar in content to the first link)
I was about to do the same - using plain SQLite instead of CoreData - but changed my mind later. In that process if found this link useful: Improve INSERT-per-second performance of SQLite? . Beyond the obvious (transaction,prepared statement,..) it uses some SQLite specific performance tweaks.