I try to do a table data dump using pg_dump, something like this:
pg96\bin\pg_dump ... --format plain --section data --column-inserts --file obj.account.backup --table obj.account database_xyz
Instead of getting
INSERT INTO obj.account(name, email, password) VALUES ('user1','email1','password1');
INSERT INTO obj.account(name, email, password) VALUES ('user2','email2','password2');
I would like to get
INSERT INTO obj.account (name, email, password) VALUES
('user1','email1','password1'),
('user2','email2','password2');
Is there a way for this without any Non-PostgreSQL postprocessing?
There is no way to get INSERT statements like that with pg_dump.
Since PostgreSQL 12 you can use pg_dump with --rows-per-insert=nrows, see https://www.postgresql.org/docs/12/app-pgdump.html
I'm aware that this is an old question but I wanted to mention it in case somebody else (like me) finds this while searching for a solution. There are cases where COPY can't be used and for bigger data sets using a single INSERT statement is much faster when importing.
Related
I am building a api that is developed by multiple developers and we require to have the same db for testing purposes. We are using UUID and so it would be ideal if we all use the same UUIDs. I can't seem to find a way to export the db contents as plain executable SQL, preferably with insert statements. NO drop tables, NO recreation of database.
I would like the end result to look something like:
INSERT INTO public.bla_bla(
id, bla_bla, bla_bla1, bla_bla2, bla_bla3, bla_bla4)
VALUES (?, ?, ?, ?, ?, ?);
INSERT INTO public.bla_bla(
id, bla_bla, bla_bla1, bla_bla2, bla_bla3, bla_bla4)
VALUES (?, ?, ?, ?, ?, ?);
...
I am using pgAdmin4 as ui. But also have Dbeaver.
I have tried using the Backup wizard to export the data.
On the database, - Does not produce a result if only data is selected. If instead the "sections" category sliders are used the result includes drop statements, which is not wanted and no readable insert statements.
on the schema - same as above
on the table - produces a CSV file, which is not ideal.
I tried following the steps here, but they do not yield the produce I need the result I need.
How to export Postgres schema/data to plain SQL in PgAdmin 4
At this point I am considering just doing it by hand.
Use pg_dump
pg_dump -Fp -d your_database -U your_db_user --column-inserts --data-only -f db_dump.sql
--data-only will skip the creation of the CREATE TABLE statements (or any other DDL).
If you want, you can add --rows-per-insert <nnnn> to only create a single INSERT statement for multiple rows.
You can try pg_dump with the option format plaintext and --column-inserts .
For more details please read here
I'd like to get a hash for data in an entire table. I need to compare two databases after migration to validate that the data migration was successful. Is it possible to reliably and reproducibly generate a hash for an entire table in a database?
You can do this from the command line (replacing of course my_database and my_table):
psql my_database -c 'copy my_table to stdout' |sha1sum
If you want to use a query to limit columns, add ordering, etc., just modify the query:
psql my_database -c 'copy (select * from my_table order by my_id_column) to stdout' |sha1sum
Note that this does not hash anything except the column data. No schema information, constraints, indexes, metadata, permissions, etc.
Note also that sha1sum is an arbitrary hashing program; you can pipe this to any program that generates a hash. Some cuspy options are sha256sum and md5sum.
I am trying to drop the schema masterdata from a postgres database, but it does not work. I am using PostgreSQL 9.5. I have 2 databses, one of them has the masterdata schema, which I want to drop. Here is the structure (in DBeaver):
My first attempt was just to execute the SQL statement DROP SCHEMA masterdata in DBeaver, but it tells me, that such a schema does not exist (but it does show it, as we can see in the picture!). Maybe, it does not know, in which of the 2 databses to look for this schema? If so, then how to specify it? I was looking for a way to specify the database, but did not find anything.
However, my second attempt was to use psql and to type the command
psql -U postgres -d bobd -h localhost
So here I concretely specify, which databse to use! psql does not answer anything, it just asks for the next command. And when I type \dn to view the current schemas, then the schema masterdata is still there! Also, the data in the tables is still there. I tried the same with other users instead of the user postgres (also with the owner of the schema to remove), but the result is the same.
Any ideas, what I am doing wrong?
Your DBeaver session was probably connected to the wrong database (postgres?).
Do it from the psql session, that is easiest.
Right after \dn showed you that there is indeed such a schema, enter
DROP SCHEMA masterdata;
It may complain that there are objects in that schema. Then you can use
DROP SCHEMA masterdata CASCADE;
Maybe your schema name contains characters that do not display or display differently (maybe capital letters) in DBeaver. I suggest you check names by running following query:
SELECT '~~' || nspname || '~~' FROM pg_catalog.pg_namespace;
Try adding quotes around schema name if you have added capital letters when creating schema.
I followed the manual on: https://docs.timescale.com/v1.0/using-timescaledb/backup
When I dump it into a binary file everything work out as expected (can restore it easily).
However, when I dump it into plain text SQL, insertions to hyper tables will be created. Is that possible to create INSERTION to the table itself?
Say I have an 'Auto' table with columns of id,brand,speed
and with only one row: 1,Opel,170
dumping into SQL will result like this:
INSERT INTO _timescaledb_catalog.hypertable VALUES ...
INSERT INTO _timescaledb_internal._hyper_382_8930_chunk VALUES (1, 'Opel',170);
What I need is this (and let TS do the work in the background):
INSERT INTO Auto VALUES (1,'Opel',170);
Is that possible somehow? (I know I can exclude tables from pg_dump but that wouldn't create the needed insertion)
Beatrice. Unfortunately, pg_dump will dump commands that mirror the underlying implementation of Timescale. For example, _hyper_382_8930_chunk is a chunk underlying the auto hypertable that you have.
Might I ask why you don't want pg_dump to behave this way? The SQL file that Postgres creates on a dump is intended to be used by pg_restore. So as long as you dump and restore and see correct state, there is no problem with dump/restore.
Perhaps you are asking a different question?
pg_dump -U postgres mydb > mydb.bak.sql
From the docs, it doesn't seem like I need to pass any flag to include table data in a dump. Yet, the dump resulting from the above includes data for a strange, tiny subset of tables--aside from their create statements, most tables are only listed as
COPY <tablename> (vals) FROM stdin;
\.
Is there some circumstance where you have to explicitly tell pg_dump to include all table data?