I've installed pgAgent in our PostgreSQL database for scheduling our jobs, which really works like a charm!
However, after I restored a backup from our database in a test server, pgAgent simply won't launch. Interestingly enough, it seems that pgAgent ignores the current state of the data in this log tables and tries to populate them from zero.
See error message at the log file:
postgres#postgres ERROR: duplicate key value violates unique constraint "pga_jobsteplog_pkey"
postgres#postgres DETAIL: Key (jslid)=(1) already exists.
postgres#postgres STATEMENT: INSERT INTO pgagent.pga_jobsteplog(jslid, jsljlgid, jsljstid, jslstatus) SELECT 1, 25, 3, 'r' FROM pgagent.pga_jobstep WHERE jstid=3
In case you're wondering how the backup is performed:
pg_dumpall --file "/media/jones/Daten/fulldump.sql" --host "address-to-my-server.de" --port "5432" --username "myuser" --no-password --database "mydb" --clean --if-exists --verbose
Environment:
Ubuntu 16.04
PostgreSQL 9.5
pgAgent 3.4.1-2
Any ideas how to make pgAgent come back to life?
In case anybody is still looking, I've found what works in this situation..
Log onto the database, install the pgagent extension
Do the pg_restore of the pgagent schema
Then restore the pgagent.pga_jobsteplog after the rest of the schema has restored.
Then reset the value of the sequences for each table to be 1 above the max value of the column the sequence is based on
Doing this, means you pgagent working, and you also have you job and jobstep logs.
I'm definitely not happy with the solution, but so far I couldn't find anything better than the following options:
Truncating the log table does the trick, but deletes all job history you had (no big deal in most use cases):
TRUNCATE TABLE pgagent.pga_jobsteplog;
Alternatively, updating the sequences manually also works, e.g.:
SELECT SETVAL('pgagent.pga_exception_jexid_seq', max(jexid)) FROM pgagent.pga_exception;
SELECT SETVAL('pgagent.pga_job_jobid_seq', max(jobid)) FROM pgagent.pga_job;
SELECT SETVAL('pgagent.pga_jobclass_jclid_seq', max(jclid)) FROM pgagent.pga_jobclass;
SELECT SETVAL('pgagent.pga_joblog_jlgid_seq', max(jlgid)) FROM pgagent.pga_joblog;
SELECT SETVAL('pgagent.pga_jobstep_jstid_seq', max(jstid)) FROM pgagent.pga_jobstep;
SELECT SETVAL('pgagent.pga_jobsteplog_jslid_seq', max(jslid)) FROM pgagent.pga_jobsteplog;
SELECT SETVAL('pgagent.pga_schedule_jscid_seq', max(jscid)) FROM pgagent.pga_schedule;
If anyone has a more elegant solution, please let me know in the comments.
Related
I try dumping tables from a production environment to a dev one. However, when dumping and restoring this table, using the following command:
pg_restore --no-owner --no-acl --clean --if-exists -d database dump_file.dump
I get an error stating that I can't drop that table unless I use something like CASCADE (i.e. dropping all other tables that depend on that one). Is there a way to determine the tables to be dropped? is there a way of maybe state in the pg_dump command to dump the table I'm looking to dump and all related tables ?
Here's the error raised:
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 4066; 2606 30526 CONSTRAINT table1 pkey user
pg_restore: error: could not execute query: ERROR: cannot drop constraint pkey on table public.table1 because other objects depend on it
DETAIL: constraint id_fkey on table public.dag depends on index public.pkey
constraint id_fkey on table public.dag depends on index public.pkey
HINT: Use DROP ... CASCADE to drop the dependent objects too...
You have a table on the dev database that has a pkey that is dependent and therefore can not be dropped before the restore. This is proper behavior.
I am not seeing dumping/restoring a particular table. You are dumping/restoring the entire database.
If you want recreate the production database as a dev one then do:
pg_restore -C --no-owner --no-acl --clean --if-exists -d postgres dump_file.dump
The -C with --clean will DROP DATABASE db_name and then rebuild it from scratch by connecting to the database postgres to do the DROP/CREATE db_name and then connect to db_name to load the rest of the objects.
This is the best way to clean out cruft and start at a consistent state.
UPDATE
Update your question with the pg_dump command so it is evident what you are doing.
If you want to see whether a particular table has dependencies, in the original database use psql and do \d the_table to see what the dependencies to and from the table are. If you tell pg_dump to dump a single table it will dump just that table. It will not follow dependencies and dump those also. That is up to you to do.
Look into using a schema management tool to do your changes/migrations. I use Sqitch for this.
I was provided with a .bk file that I need to restore on a postgreSQL DB.
Reference for Restoring DB
I used the following command-line from the link above : psql -d EneFrame -U username -f backupfile.bk
After asking for the password, it seems to do some copying of rows, ALTER TABLE, CREATE INDEX and at the end REVOKE and GRANT - However, in the beginning I get the following errors, but still seems to run and finish... :
Then when I open pgAdmin - I dont see any change in the DB I am trying to restore to ?
I just started with postgreSQL today, so dont know much. Am I trying to restore incorrectly ?
Thanks for any help!
I have a postgres instance with a postgres user.
root#f23ada822ac8:/# su - postgres
No directory, logging in with HOME=/
$ bash
postgres#f23ada822ac8:$ psql
psql (9.6.1)
Type "help" for help.
postgres=# \echo :AUTOCOMMIT
on
In / which is postgres home directory I have .psqlrc which looks like
postgres#f23ada822ac8:/$ cat ~/.psqlrc
\set AUTOCOMMIT off
Ideally this should have set autocommit off but it doesn't switch it off. I am new to postgres. Any pointers here would help. I know this is not permanent solution. I also read that even if auto commit is on if you have transactions then it would only get committed when you call commit no matter what the auto commit setting is. Is this true?
It depends on how you have interfaced with DB's in the past. Applications like TOAD have an auto-commit feature that can be turned off, and it has saved my bacon many times. That said.
psql has a start up file psqlrc. You can add the autocommit off option to this file. This will only work on psql however.
I'm trying to restore a database from backup but I can't connect to postgresql.
namespace :db do
task import: :environment do
import_path = "~/backups"
sql_file = "PostgreSQL.sql"
database_config = Rails.configuration.database_configuration[Rails.env]
system "psql --username=#{database_config['username']} -no-password # {database_config['database']} < #{import_path}/#{sql_file}"
end
end
I tried changing the pg_hba.conf file (peer to md5).
In the console I tried the same thing with the super user postgres, but it still fails.
BTW, does anyone know a better way to restore a database? I used the backup gem.
EDIT:
I restarted the postgresql server and the passed the authentication. But, didn't restored the db. I reverted the changes in the file and just added -h localhost to the psql command. The database restores now. The only errors I get now are:
must be owner of extension plpgsql //and
no privileges could be revoked for "public"
after change pg_hba.conf, you shold reload or send a SIGHUP signal to postmaster pid. so that change applyed.
why not use psql -f to execute the backup sql file?
or you can use pg_dump backup and pg_restore restore. or copy command backup and restore.
LIKE :
digoal=# copy tbl_join_1 to '/home/pg93/tbl_join_1.dmp';
COPY 10
digoal=# delete from tbl_join_1;
DELETE 10
digoal=# copy tbl_join_1 from '/home/pg93/tbl_join_1.dmp';
COPY 10
OR
pg93#db-172-16-3-150-> pg_dump -f ./tbl_join_1.dmp -t tbl_join_1
pg93#db-172-16-3-150-> psql
psql (9.3.3)
Type "help" for help.
digoal=# drop table tbl_join_1;
DROP TABLE
digoal=# \q
pg93#db-172-16-3-150-> psql -f ./tbl_join_1.dmp
SET
SET
SET
SET
SET
SET
SET
SET
SET
CREATE TABLE
ALTER TABLE
ALTER TABLE
I tried pg_dump and then on a separate machine I tried to import the sql and populate the database, I see
CREATE TABLE
ERROR: role "prod" does not exist
CREATE TABLE
ERROR: role "prod" does not exist
CREATE TABLE
ERROR: role "prod" does not exist
CREATE TABLE
ERROR: role "prod" does not exist
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
WARNING: no privileges could be revoked for "public"
REVOKE
ERROR: role "postgres" does not exist
ERROR: role "postgres" does not exist
WARNING: no privileges were granted for "public"
GRANT
which means my user and roles and grant information is not in pg_dump
On the other hand we have pg_dumpall, I read conversation, and this does not lead me anywhere?
Question
- Which one should I be using for database backups? pg_dump or pg_dumpall?
- the requirement is that I can take the backup and should be able to import to any machine and it should work just fine.
The usual process is:
pg_dumpall --globals-only to get users/roles/etc
pg_dump -Fc for each database to get a nice compressed dump suitable for use with pg_restore.
Yes, this kind of sucks. I'd really like to teach pg_dump to embed pg_dumpall output into -Fc dumps, but right now unfortunately it doesn't know how so you have to do it yourself.
Up until PostgreSQL 11 there was also a nasty caveat with this approach: Neither pg_dump, nor pg_dumpall in --globals-only mode would dump user access GRANTs on DATABASEs. So you pretty much had to extract them from the catalogs or filter a pg_dumpall. This is fixed in PostgreSQL 11; see the release notes.
Make pg_dump dump the properties of a database, not just its contents (Haribabu Kommi)
Previously, attributes of the database itself, such as database-level GRANT/REVOKE permissions and ALTER DATABASE SET variable settings, were only dumped by pg_dumpall. Now pg_dump --create and pg_restore --create will restore these database properties in addition to the objects within the database. pg_dumpall -g now only dumps role- and tablespace-related attributes. pg_dumpall's complete output (without -g) is unchanged.
You should also know about physical backups - pg_basebackup, PgBarman and WAL archiving, PITR, etc. These offer much "finer grained" recovery, down to the minute or individual transaction. The downside is that they take up more space, are only restoreable to the same PostgreSQL version on the same platform, and back up all tables in all databases with no ability to exclude anything.