If I run heroku pg:backups --app MY_APP, I see:
=== Backups
No backups. Capture one with heroku pg:backups:capture
=== Restores
No restores found. Use heroku pg:backups:restore to restore a backup
=== Copies
ID Started at Status Size From To
──── ───────────────────────── ─────────────────────────────────── ──────── ───────── ───────────
c001 2021-02-23 15:47:54 +0000 Completed 2021-02-23 15:48:27 +0000 509.16KB From Name Target Name
If I run heroku pg:backups:info c001 --app MY_APP, I see:
=== Backup c001
Database: From Name
Started at: 2021-02-23 15:47:54 +0000
Finished at: 2021-02-23 15:48:27 +0000
Status: Completed
Type: Manual
Original DB Size: 11.53MB
Backup Size: 509.16KB (96% compression)
=== Backup Logs
However, when I try to restore this backup using heroku pg:backups:restore c001 --app MY_APP, it tells me that Backup c001 not found for MY_APP.
Is there any way I can restore this copy? For context, my database was in the free plan and was deleted by Heroku, so as far as I know this is the last backup I have.
Thank you!
Backups start with b00* and are listed in the top section.
I am not sure what "Copies" are, did not find anything in their docs.
❯ heroku pg:backups -r prod
=== Backups
ID Created at Status Size Database
──── ───────────────────────── ─────────────────────────────────── ──────── ────────
b008 2021-05-04 06:57:18 +0000 Completed 2021-05-04 06:57:20 +0000 664.15KB DATABASE
b007 2021-05-04 05:33:50 +0000 Completed 2021-05-04 05:33:52 +0000 663.53KB DATABASE
=== Restores
No restores found. Use heroku pg:backups:restore to restore a backup
=== Copies
ID Started at Status Size From To
──── ───────────────────────── ─────────────────────────────────── ──────── ───────── ───────────
c009 2022-06-23 03:28:14 +0000 Completed 2022-06-23 03:28:46 +0000 232.80KB From Name Target Name
c002 2020-05-27 15:29:37 +0000 Completed 2020-05-27 15:29:40 +0000 112.59KB From Name Target Name
c001 2020-05-26 19:43:26 +0000 Completed 2020-05-26 19:43:37 +0000 112.97KB From Name Target Name
❯ heroku pg:backups:restore c009 --app <app>
▸ Backup c009 not found for ⬢ <app>
❯ heroku pg:backups:restore b008 --app <app>
▸ WARNING: Destructive Action
▸ This command will affect the app <app>
▸ To proceed, type <app> or re-run this command with --confirm
▸ <app>
...
Related
I installed postgres 14 beta and i want to upgrade to 14 new release. I have just installed the new release pg 14 and when i try to start the pg 14 beta i get the error below. May i know the correct procedure of upgrading from beta to new release
-bash-4.2$ /usr/pgsql-14/bin/pg_ctl -D /var/lib/pgsql/14/data -l logfile start
waiting for server to start.... stopped waiting
pg_ctl: could not start server
Examine the log output.
-bash-4.2$ cat logfile
2021-10-27 13:19:29.507 UTC [5112] FATAL: database files are incompatible with server
2021-10-27 13:19:29.507 UTC [5112] DETAIL: The database cluster was initialized with CATALOG_VERSION_NO 202106151, but the server was compiled with CATALOG_VERSION_NO 202107181.
2021-10-27 13:19:29.507 UTC [5112] HINT: It looks like you need to initdb.
2021-10-27 13:19:29.507 UTC [5112] LOG: database system is shut down
There has been a change in the catalog version since v14 beta 1, so you have to use fump/restore of pg_upgrade to upgrade.
Install 14.0, create a new cluster and use the method of your choice to upgrade.
I am trying to implement Master/Slave streaming replication on Postgres 11.5. I ran the following steps -
On Master
select pg_start_backup('replication-setup',true);
On Slave
Stopped the postgres 11 database and ran
rsync -aHAXxv --numeric-ids --progress -e "ssh -T -o Compression=no -x" --exclude pg_wal --exclude postgresql.pid --exclude pg_log MASTER:/var/lib/postgresql/11/main/* /var/lib/postgresql/11/main
On Master
select pg_stop_backup();
On Slave
rsync -aHAXxv --numeric-ids --progress -e "ssh -T -o Compression=no -x" MASTER:/var/lib/postgresql/11/main/pg_wal/* /var/lib/postgresql/11/main/pg_wal
I created the recovery.conf file on slave ~/11/main folder
standby_mode = 'on'
primary_conninfo = 'user=postgres host=MASTER port=5432 sslmode=prefer sslcompression=1 krbsrvname=postgres'
primary_slot_name='my_repl_slot'
When I start Postgres on Slave, I get the error on both MASTER and SLAVE logs -
019-11-08 09:03:51.205 CST [27633] LOG: 00000: database system was interrupted; last known up at 2019-11-08 02:53:04 CST
2019-11-08 09:03:51.205 CST [27633] LOCATION: StartupXLOG, xlog.c:6388
2019-11-08 09:03:51.252 CST [27633] LOG: 00000: entering standby mode
2019-11-08 09:03:51.252 CST [27633] LOCATION: StartupXLOG, xlog.c:6443
2019-11-08 09:03:51.384 CST [27634] LOG: 00000: started streaming WAL from primary at 12DB/C000000 on timeline 1
2019-11-08 09:03:51.384 CST [27634] LOCATION: WalReceiverMain, walreceiver.c:383
2019-11-08 09:03:51.384 CST [27634] FATAL: XX000: could not receive data from WAL stream: ERROR: requested WAL segment 00000001000012DB0000000C has already been removed
2019-11-08 09:03:51.384 CST [27634] LOCATION: libpqrcv_receive, libpqwalreceiver.c:772
2019-11-08 09:03:51.408 CST [27635] LOG: 00000: started streaming WAL from primary at 12DB/C000000 on timeline 1
2019-11-08 09:03:51.408 CST [27635] LOCATION: WalReceiverMain, walreceiver.c:383
The problem is the START WAL - 00000001000012DB0000000C is available right until I run the pg_stop_backup() and is getting archived and no longer available, once the pg_stop_backup() is executed. So this is not an issue of the WAL being archived out due to low WAL_KEEP_SEGMENTS.
postgres#SLAVE:~/11/main/pg_wal$ cat 00000001000012DB0000000C.00000718.backup
START WAL LOCATION: 12DB/C000718 (file 00000001000012DB0000000C)
STOP WAL LOCATION: 12DB/F4C30720 (file 00000001000012DB000000F4)
CHECKPOINT LOCATION: 12DB/C000750
BACKUP METHOD: pg_start_backup
BACKUP FROM: master
START TIME: 2019-11-07 15:47:26 CST
LABEL: replication-setup-mdurbha
START TIMELINE: 1
STOP TIME: 2019-11-08 08:48:35 CST
STOP TIMELINE: 1
My MASTER has archive_command set, and I have the missing WALs available. I copied them into a restore directory on the SLAVE and tried the recovery.conf below, but it still fails with the MASTER reporting the same WAL segment has already been moved error.
Any idea how I can address this issue? I have used rsync to setup replication without any issues in the past on Postgres 9.6, but have been experiencing this issue on Postgres 11.
standby_mode = 'on'
primary_conninfo = 'user=postgres host=MASTER port=5432 sslmode=prefer sslcompression=1 krbsrvname=postgres'
restore_command='cp /var/lib/postgresql/restore/%f %p'
Put a restore_command into recovery.conf that can restore archived WAL files and you are fine.
I am in the process of experimenting/tinkering/learning/breaking with Docker. I am currently writing Docker code to create a snapshotted testing environment for my application.
By snapshotted I mean that my database is reset on purpose on every restart, so that I can work with old data at a certain time. What is peculiar in my case is that I want to populate a Postgresql database at build time, not at start time. Postgresql image is ready for populating the db with sql scripts at container start, but it takes hours.
My application is made by a Tomcat 8.5 server running my WAR and a Postgresql database, which is the focus of my question now. I am creating a Gist while I write for full code.
The code I have done
Full code on Gist
I have followed a tutorial on how to build a Docker image of Postgres with a full database, rather than have Postgres populate itself on boot. This because I have a million record database and only a .sql.gz dump that sysop gave me.
So the relevant parts of the Dockerfile are
WORKDIR /opt/setup/
COPY db-setup.sh /opt/setup/
COPY db-pack.sh /opt/setup/
COPY db-run.sh /opt/setup/
RUN ./db-setup.sh
RUN ./db-pack.sh
#VOLUME $PGDATA (Note it is commented out, now)
EXPOSE 5432
The db-setup.sh is run on image build, and picks files from data-scripts.d. Of course I am not allowed to share the contents of the dump, but it's a plain .sql.gz with plenties of OIDs that take a huge amount of time to restore. The db-setup.sh shown in Gist is derived from both the tutorial and the original Postgres image so that it handles correctly the compression (the tutorial only uses plain SQL)
Build succeeds, startup fails
When I build the image, it takes considerable amount of time to load the data, which is what I want
2019-08-07 07:57:04.149 UTC [49] LOG: database system was shut down at 2019-08-07 07:57:03 UTC
2019-08-07 07:57:04.231 UTC [48] LOG: database system is ready to accept connections
done
server started
./db-setup.sh: running methodinv_pcp3.sql.gz
2019-08-07 08:49:52.052 UTC [117] ERROR: canceling autovacuum task
2019-08-07 08:49:52.052 UTC [117] CONTEXT: automatic analyze of table "postgres.public.ftt_interactive_data_492"
2019-08-07 08:49:59.086 UTC [118] ERROR: canceling autovacuum task
2019-08-07 08:49:59.086 UTC [118] CONTEXT: automatic analyze of table "postgres.public.ftt_oper_492"
2019-08-07 08:50:34.086 UTC [118] ERROR: canceling autovacuum task
2019-08-07 08:50:34.086 UTC [118] CONTEXT: automatic analyze of table "postgres.public.ftt_validation_492"
2019-08-07 08:51:11.889 UTC [119] ERROR: canceling autovacuum task
2019-08-07 08:51:11.889 UTC [119] CONTEXT: automatic analyze of table "postgres.public.ftt_oper_492"
2019-08-07 08:54:21.131 UTC [123] ERROR: canceling autovacuum task
2019-08-07 08:54:21.131 UTC [123] CONTEXT: automatic analyze of table "postgres.public.ftt_oper_492"
waiting for server to shut down...2019-08-07 08:54:28.652 UTC [48] LOG: received fast shutdown request
.2019-08-07 08:54:28.797 UTC [48] LOG: aborting any active transactions
2019-08-07 08:54:28.799 UTC [48] LOG: worker process: logical replication launcher (PID 55) exited with exit code 1
2019-08-07 08:54:28.800 UTC [50] LOG: shutting down
..2019-08-07 08:54:31.407 UTC [48] LOG: database system is shut down
done
When I run the image with docker run, startup fails because it can't find Postgres configuration
D:\IdeaProjects\pcp\ftt-containers\ftt-db-method>docker run -p 5432:5432 -l ftt-db-method ftt-db-method:latest
Restoring /var/lib/postgresql/data ...
Done.
Launching command: postgres ...
postgres: could not access the server configuration file "/var/lib/postgresql/data/postgresql.conf": No such file or directory
Originally, my Dockerfile exposed a VOLUME which is now commented out. The above output occurs both when I declare a volume (which is not exactly what I want, I am new to Docker and copied&pasted on first chance) and when I comment the volume out.
Question
What is wrong with the Docker image of Postgres fully loaded with s**tloads of data I am experimenting?
How can I effectively start Postgres with an already full database that will not (necessarily) survive container restarts?
Edit 1
By bash-ing into the container I have found that the data dump created during build time is 10K, so basically empty.
This doesn't solve my problem yet, but answers why Postgres is unable to find its beloved data dir
Edit 2
I was able to bash into a temporary container, in particular between the moment the database is restored and the data lib is packed.
Basically the Dockerfile does
RUN ./db-setup.sh
Which executes the restore of the sql
echo "$0: running $f"; gunzip -c "$f" | "${psql[#]}" > /dev/null 2>&1 ; echo ;;
The output is saved to a temporary container.
Now Dockerfile does
RUN ./db-pack.sh
Which tars /var/lib/postgresql/data into /zdata. I have
2019-08-07 16:43:51.532 UTC [42] LOG: received fast shutdown request
waiting for server to shut down....2019-08-07 16:43:51.676 UTC [42] LOG: aborting any active transactions
2019-08-07 16:43:51.679 UTC [42] LOG: worker process: logical replication launcher (PID 49) exited with exit code 1
2019-08-07 16:43:51.681 UTC [44] LOG: shutting down
...2019-08-07 16:43:54.952 UTC [42] LOG: database system is shut down
done
server stopped
Removing intermediate container 8dbe2a4e776a
---> 263896b905ce
Step 15/19 : RUN ./db-pack.sh
---> Running in 56132ecb90cc
Packing data folder: /var/lib/postgresql/data
Pack & clean finished successfully.
Removing intermediate container 56132ecb90cc
---> 1a7f8d68e8df
Step 16/19 : VOLUME $PGDATA
---> Running in 10d222beed81
Removing intermediate container 10d222beed81
---> e1a9355882d1
So I tagged 263896b905ce (YHMV if you replicate on your pc) into a new image, then executed bash on it. The data dir was empty, the script would have packed nothing
docker tag 263896b905ce examine
docker run -it --entrypoint /bin/bash examine
root#ab963ace16a1:/opt/setup# ls
data-scripts.d db-pack.sh db-run.sh db-setup.sh
root#ab963ace16a1:/opt/setup# cd /zdata/
root#ab963ace16a1:/zdata# ls
root#ab963ace16a1:/zdata# cd /var/lib/postgresql/
root#ab963ace16a1:/var/lib/postgresql# ls
data
root#ab963ace16a1:/var/lib/postgresql# cd data/
root#ab963ace16a1:/var/lib/postgresql/data# ls
root#ab963ace16a1:/var/lib/postgresql/data# ls -lah
total 8.0K
drwxrwxrwx 2 postgres postgres 4.0K Jul 17 23:55 .
drwxr-xr-x 1 postgres postgres 4.0K Jul 17 23:55 ..
root#ab963ace16a1:/var/lib/postgresql/data#
root#ab963ace16a1:/var/lib/postgresql/data# ls^C
root#ab963ace16a1:/var/lib/postgresql/data# exit
exit
Fixed
According to https://stackoverflow.com/a/52762779/471213
"why doesn't VOLUME work?" When you define a VOLUME in the Dockerfile, you can only define the target, not the source of the volume. During the build, you will only get an anonymous volume from this. That anonymous volume will be mounted at every RUN command, prepopulated with the contents of the image, and then discarded at the end of the RUN command. Only changes to the container are saved, not changes to the volume.
So I had basically to run both RUNs at the same time
RUN ./db-setup.sh && ./db-pack.sh
#RUN ./db-pack.sh
C:\Users\krishnava\Downloads\git>heroku pg:backups:restore "https://s3.amazonaws.com/backup_xxx"
DATABASE_URL
! WARNING: Destructive Action
! This command will affect the app gcesalem
! To proceed, type gcesalem or re-run this command with --confirm appname
> appname
Starting restore of https://s3.amazonaws.com/backup_xxx
to postgresql-round-xxx... done
Use Ctrl-C at any time to stop monitoring progress; the backup will continue restoring.
Use heroku pg:backups to check progress.
Stop a running restore with heroku pg:backups:cancel.
Restoring... !
! An error occurred and the backup did not finish.
!
! waiting for restore to complete
! pg_restore finished with errors
! waiting for download to complete
! download finished with errors
! please check the source URL and ensure it is publicly accessible
!
! Run heroku pg:backups:info r006 for more details.
Info
C:\Users\krishnava\Downloads\git>heroku pg:backups:info r006
=== Backup r006
Database: BACKUP
Started at: 2019-07-16 15:34:40 +0000
Finished at: 2019-07-16 15:34:40 +0000
Status: Failed
Type: Manual
Backup Size: 0.00B (0% compression)
=== Backup Logs
2019-07-16 15:34:40 +0000 pg_restore: [archiver] did not find magic string in file header
2019-07-16 15:34:40 +0000 waiting for restore to complete
2019-07-16 15:34:40 +0000 pg_restore finished with errors
2019-07-16 15:34:40 +0000 waiting for download to complete
2019-07-16 15:34:40 +0000 download finished with errors
2019-07-16 15:34:40 +0000 please check the source URL and ensure it is publicly accessible- -
Instead of doing like this, You can export a copy of the local database and import it to Heroku.
For export from local database,
pg_dump <DATABASE_NAME> > <FILENAME>.sql
This will ask you to enter your database password. But in Windows, this will ask the User password, because the default user name is the system user name. For this you have to specify your username
pg_dump -U <USER_NAME> <DATABASE_NAME> > <FILENAME>.sql
For your case the command will be like this:
pg_dump -U postgres gce > gce.sql
After exporting the local database, you can upload this directly to heruko.
heroku pg:psql --app <APP_NAME> < gce.sql
I am trying to create a hot_standby server, and I receive the following error after pg_basebackup completes. Notice I use a shell script, replicator.sh, to start the replication. Can anyone give me some insight?
My specs:
Debian Wheezy 7.6
Postgresql 9.3
Database size: ~115GB
Error:
postgres#database-master:/etc/postgresql/9.3/main$ sh replicator.sh
Stopping PostgreSQL
[ ok ] Stopping PostgreSQL 9.3 database server: main.
Cleaning up old cluster directory
Starting base backup as replicator
Password:
113720266/113720266 kB (100%), 1/1 tablespace
NOTICE: WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup
pg_basebackup: base backup completed
Starting Postgresql
[....] Starting PostgreSQL 9.3 database server: main[....] The PostgreSQL server failed to start.
Please check the log output: 2014-09-11 17:56:33 UTC LOG: database system was interrupted; last
known up at 2014-09-11 16:54:29 UTC 2014-09-11 17:56:33 UTC LOG: creating missing WAL directory
"pg_xlog/archive_status" 2014-09-11 17:56:33 UTC LOG: incomplete startup packet 2014-09-11 17:56:33
UTC LOG: invalid checkpoint record 2014-09-11 17:56:33 UTC FATAL: could not locate required
checkpoint record 2014-09-11 17:56:33 UTC HINT: If you are not restoring from a backup, try
removing the file "/var/lib/p[FAILesql/9.3/main/backup_label". 2014-09-11 17:56:33 UTC LOG: startup
process (PID 21972) exited with exit code 1 2014-09-11 17:56:33 UTC LOG: aborting startup due to
startup process failure ... failed! failed!
Contents of replicator.sh:
#!/bin/bash
echo Stopping PostgreSQL
/etc/init.d/postgresql stop
echo Cleaning up old cluster directory
rm -rf /var/lib/postgresql/9.3/main
echo Starting base backup as replicator
pg_basebackup -h 123.456.789.123 -D /var/lib/postgresql/9.3/main -U replicator -v -P
echo Writing recovery.conf file
sudo -u postgres bash -c "cat > /var/lib/postgresql/9.3/main/recovery.conf <<- _EOF1_
standby_mode = 'on'
primary_conninfo = 'host=123.456.789.123 port=5432 user=replicator password=XXXXX sslmode=require'
trigger_file = '/tmp/postgresql.trigger'
_EOF1_
"
echo Starting Postgresql
/etc/init.d/postgresql start
Thank you,
Jake
My best guess from the above is that the pg_basebackup failed and your shell script doesn't check for error return codes or use set -e to automatically abort after errors, so it just carried on regardless.
It's also possible that you don't have WAL archiving configured, or don't have a restore_command set in the replica. In that case, the transaction logs required to start the base backup will not be available and startup will fail.
I strongly recommend that you:
Use pg_basebackup -X stream so that the required transaction logs get copied along with the backup; and
Use set -e in your shell script, or test for errors with a suitable if ! pg_basebackup .... ; then block.