I am about to move my production PostgreSQL DB to a managed DB from Digital Ocean. So, there is a possibility for a down time during the maintenance/back up procedures which may last up to 10 seconds. How to effectively handle this DB down time without getting the rails application crashed? I read about 'reconnect: true' in database.yml. But not sure it will work for PostgreSQL. Any suggestions?
Related
I have a basic Postgres instance in GCP with the following configuration:
https://i.stack.imgur.com/5FFZ6.png
I have a simple python script using pycopg2 that does inserts in a loop to the database connected through the sql auth proxy.
All the inserts are done in the same transaction.
The problem is that it is taking a couple of hours to insert around 200,000 records.
When I run the script on a local database it takes a couple of seconds.
What could be causing this huge difference?
First of all i do not know if is possible what i want to achive, will describe below:
I have access of a remote PostgreSQL that holds the data i need (let's say Remote PostgreSQL 1)
have just credentials to read from database
What i want to achive is to create a localt PostgreSQL on my machine (let's say Local PostgreSQL 2)
Want to copy and check for missing data from Remote PostgreSQL 1 to Local PostgreSQL 2 in real time or at list to copy at the end of the day data
The scenario will work perfect with replication but the issue is that Remote PostgreSQL 1 is not owned by me and can not be used as real time DB, because of this trying to find a solution to get all the data from Remote PostgreSQL 1 to Local PostgreSQL 2.
Could be the following scenarios:
first time setup to downlaod all the database from Remote PostgreSQL 1 to Local PostgreSQL 2
after first time setup to check what data came new inside and add them in Local PostgreSQL 2
Would be great if this could be done on OS level on UBUNTU. My application is written in python 3 i could do scripts to do all this job but i speak of 100 millions of raws per table huge amount of data. Think will be problems to get everything from database and start to check everything what is missing and not.
Any ideas would be great.
If the owner of Remote Database 1 won't cooperate with you other than to give you read only access to the tables, then you don't have any efficient options. If the remote owner does or can be convinced to keep insertion/modification timestamp columns in all the tables (although then deletions would be a problem), or an in-database "audit" log for all the tables, you could use those. I think you have an organizational/political problem rather a programming problem.
Currently I am trying to upgrade my postgres 9.1 to 10 using logical replication. As 9.1 does not support native logical_replication, I tried slony and made a replica successfully.
P.S. The above replica I created is using a sample dump from an year ago which is only of 800mb.
Now I have few questions.
How can I validate whether the replica has all the data replicated successfully. Few suggested to put the master on maintenance mode(few downtime) and do a last N items comparison with both the database on all the tables.
I tried with 800mb. Will there be any problem when I go and try with 100+ GB?
Please share your personal experience here. I have been trying to document what are the things that could go wrong so I can always try to anticipate the next course of action.
You may use the Data Validator that is shipped with trial version of EDB Postgres Replication server for validating the data between old and new PostgreSQL databases.
You may read the details of Data Validator at Data Validator Document
To download the EDB Replication Server please follow this link: EDB Replication Server
Disclosure: I work for EnterpriseDB (EDB)
I want to execute a long-running stored procedure on PostgreSQL 9.3. Our database server is (for the sake of this question) guaranteed to be running stable, but the machine calling the stored procedure can be shut down at any second (Heroku dynos get cycled every 24h).
Is there a way to run the stored procedure 'detached' on PostgreSQL? I do not care about its output. Can I run it asynchronously and then let the database server keep working on it while I close my database connection?
We're using Python and the psycopg2 driver, but I don't care so much about the implementation itself. If the possibility exists, I can figure out how to call it.
I found notes on the asynchronous support and the aiopg library and I'm wondering if there's something in those I could possibly use.
No, you can't run a function that keeps on running after the connection you started it from terminates. When the PostgreSQL server notices that the connection has dropped, it will terminate the function and roll back the open transaction.
With PostgreSQL 9.3 or 9.4 it'd be possible to write a simple background worker to run procedures for you via a queue table, but this requires the ability to compile and install new C extensions into the server - something you can't do on Heroku.
Try to reorganize your function into smaller units of work that can be completed individually. Huge, long-running functions are problematic for other reasons, and should be avoided even if unstable connections aren't a problem.
we are using Heroku Postgres with Ruby on Rails 3.2.
A few days before, we deleted important data by mistake using 'heroku run db:load' with misconfigured data.yml, that is, drop tables and the recreate tables with almost no data.
Backup data is only available 2 weeeks before, so we lost data of 2 weeks.
So We need to recover not by PG Backup/pg_dump but by postgresql's system data files.
I think, the only way to recover data is to restore data from xlog or archive file, but of course we don't have permission to be Super User/Replication Role to copy postgres database on heroku (or Amazon EC2) to local server.
Is there anyone who confronted such a case and resolved the problem?
Your only option is the backups provided by the PgBackups service (if you had that running). If not, Heroku support might have more options available.
At a minimum, you will have some data loss, but you can guarantee you won't do it again ;)