I have a basic Postgres instance in GCP with the following configuration:
https://i.stack.imgur.com/5FFZ6.png
I have a simple python script using pycopg2 that does inserts in a loop to the database connected through the sql auth proxy.
All the inserts are done in the same transaction.
The problem is that it is taking a couple of hours to insert around 200,000 records.
When I run the script on a local database it takes a couple of seconds.
What could be causing this huge difference?
Related
I have a chron job that runs on a stateless server. On this chron job, I am trying to take a snapshot of my Postgres GCP Sql db (PRODUCTION_DATABASE), save it to S3 and then upload it to my staging, qa-1, dev databases. The problem is one table, call it LARGE_TABLE, needs to be shrunk because the table size is growing rapidly, thus causing problems and exceeding timeouts. Does anyone have any advice on how to get this done?
I tried running the cloud_sql_proxy to run pg_dump but no go with that method. Is there a way I can truncate one table and make a backup?
I am about to move my production PostgreSQL DB to a managed DB from Digital Ocean. So, there is a possibility for a down time during the maintenance/back up procedures which may last up to 10 seconds. How to effectively handle this DB down time without getting the rails application crashed? I read about 'reconnect: true' in database.yml. But not sure it will work for PostgreSQL. Any suggestions?
I'm using a real Mongo instance for my integration tests.
On every test, I call mongorestore my_db_state --drop because I need a clean db to start the next test.
On my tiny db (with only one test user account) it takes 180 ms to run mongorestore my_db_state (without --drop) and 455 ms with --drop.
I don't understand why it's taking so long to drop the db, but how can I speed this up?
I may have a bunch of different connections on the mongo server because I never close them on my Flask server, but they probably should be closed after the test is over...just not sure if this is an issue to care about.
Is there a faster way to restore a snapshot to its original state for testing purposes?
Drop and recreate collections from your test runner using persistent MongoClient instances.
First of all i do not know if is possible what i want to achive, will describe below:
I have access of a remote PostgreSQL that holds the data i need (let's say Remote PostgreSQL 1)
have just credentials to read from database
What i want to achive is to create a localt PostgreSQL on my machine (let's say Local PostgreSQL 2)
Want to copy and check for missing data from Remote PostgreSQL 1 to Local PostgreSQL 2 in real time or at list to copy at the end of the day data
The scenario will work perfect with replication but the issue is that Remote PostgreSQL 1 is not owned by me and can not be used as real time DB, because of this trying to find a solution to get all the data from Remote PostgreSQL 1 to Local PostgreSQL 2.
Could be the following scenarios:
first time setup to downlaod all the database from Remote PostgreSQL 1 to Local PostgreSQL 2
after first time setup to check what data came new inside and add them in Local PostgreSQL 2
Would be great if this could be done on OS level on UBUNTU. My application is written in python 3 i could do scripts to do all this job but i speak of 100 millions of raws per table huge amount of data. Think will be problems to get everything from database and start to check everything what is missing and not.
Any ideas would be great.
If the owner of Remote Database 1 won't cooperate with you other than to give you read only access to the tables, then you don't have any efficient options. If the remote owner does or can be convinced to keep insertion/modification timestamp columns in all the tables (although then deletions would be a problem), or an in-database "audit" log for all the tables, you could use those. I think you have an organizational/political problem rather a programming problem.
i have written a wrong infinite-loop recursive query as i have been learning recursive postgres queries and affected the whole server postgres server
every database request takes inifinitely long time, giving me timeouts. so i have tried doing /etc/init.d/postgresql restart but while it seems to have helped a little, the postgres queries still run slowly
are the unfinished queries cached even after the postgtres server restart? how could i escape this sticky situation? :(
thank you ammoQ and Tometzky for your answers, it ended up the postgres process was cleaning up a lot of stuff. it gets back to normal after a couple of minutes