pg_dump --table=export_view --data-only --column-inserts mydb > export_view.sql
pg_dump (PostgreSQL) 10.7 (Ubuntu 10.7-1.pgdg18.04+1)
Export specific rows from a PostgreSQL table as INSERT SQL script and the postgresql documentation (https://www.postgresql.org/docs/10/app-pgdump.html) suggest it is possible to pg_dump from a view with the --table flag. If I export from the table directly I get the expected result (ie, data is exported). If I select from the view in psql I get the expected result. However whether I create a view or a materialized view and then try and pg_dump, I get only the normal pg_dump headers and no data. A commenter (https://stackoverflow.com/users/2036135/poshest) also appears to have faced the same issue in the above SO question, with no solution given.
If I CREATE TABLE blah AS SELECT x, y, z FROM MYTABLE then I can export fine. If I CREATE VIEW blah AS SELECT x, y, z FROM MYTABLE then the export is empty.
What am I doing wrong?
As #bugmenot points out, version 13 (and above?) - the current at the time this answer is written - indeed has clarification on what gets dumped:
As well as tables, this option can be used to dump the definition
of matching views, materialized views, foreign tables, and sequences.
It will not dump the contents of views or materialized views, and the contents of foreign tables will only be dumped if the
corresponding foreign server is specified with --include-foreign-data.
(emphasis added).
So the answer (to myself) is: "You are not doing anything wrong, except that you incorrectly interpreted the documentation for Postgres <=12. What you want to do is not possible."
Views do not store data, they provide a dynamic view onto it. When you include views in your dump, you will only get the view definition.
Related
This question already has an answer here:
will pg_restore overwrite the existing tables?
(1 answer)
Closed 9 months ago.
I have an database on aws' rds and I use a pg_dump from a local version of the database, then psql dbname > pg_dump_file with proper arguments for remote upload to populate the database.
I'd like to know what is expected to happen if that rds db already contains data. More specifically:
Data present in the local dump, but absent in rds
Data present on rds, but absent in the local data
Data present in both but that have been modified
My current understanding:
New data will be added and be present in both after upload
Data in rds should be unaffected?
The data from the pg_dump will be present in both (assuming the same pk, but different fields otherwise)
Is that about correct? I've been reading this, but it's a little thin on how the restore is actually performed, so I'm having a harder time figuring that out. Thanks.
EDIT: following #wildplasser comment, by looking at the pg_dump file it appears that the following happens:
CREATE TABLE [....]
ALTER TABLE [setting table owner]
ALTER SEQUENCE [....]
For each table in the db. Then, again one table at a time:
COPY [tablename] (list of cols) FROM stdin;
[data to be copied]
Finally, more ALTER statements to set contraints, foreign keys etc.
So I guess the ultimate answer is "it depends". One could I suppose remove the CREATE TABLE [...], ALTER TABLE, ALTER SEQUENCE statements if those are already created as they should. I am not positive yet what happens if one tries CREATE TABLE with an existing table (error thrown perhaps?).
Then I guess the COPY statements would overwrite whatever already exists. Or perhaps throw an error. I'll have to test that. I'll write up an answer once I figure it out.
So the answer is a bit dull. Turns out that even if one removes the initial statements before the copy, if the table as an primary key (thus uniqueness constrains) then it won't work:
ERROR: duplicate key value violates unique constraint
So one gets shutdown pretty quickly there. One would have I guess to rewrite the dump as a list of UPDATE statements instead, but then I guess might as well write a script to do so. Unsure if pg_dump is all that useful in that case.
I followed the manual on: https://docs.timescale.com/v1.0/using-timescaledb/backup
When I dump it into a binary file everything work out as expected (can restore it easily).
However, when I dump it into plain text SQL, insertions to hyper tables will be created. Is that possible to create INSERTION to the table itself?
Say I have an 'Auto' table with columns of id,brand,speed
and with only one row: 1,Opel,170
dumping into SQL will result like this:
INSERT INTO _timescaledb_catalog.hypertable VALUES ...
INSERT INTO _timescaledb_internal._hyper_382_8930_chunk VALUES (1, 'Opel',170);
What I need is this (and let TS do the work in the background):
INSERT INTO Auto VALUES (1,'Opel',170);
Is that possible somehow? (I know I can exclude tables from pg_dump but that wouldn't create the needed insertion)
Beatrice. Unfortunately, pg_dump will dump commands that mirror the underlying implementation of Timescale. For example, _hyper_382_8930_chunk is a chunk underlying the auto hypertable that you have.
Might I ask why you don't want pg_dump to behave this way? The SQL file that Postgres creates on a dump is intended to be used by pg_restore. So as long as you dump and restore and see correct state, there is no problem with dump/restore.
Perhaps you are asking a different question?
I am using a tool called giswater to model our water sewer and storm networks. The tools builds a database from a few simple inputs in a gui one of which is the SRID. the geometry columns are all xy and I want to make them all ZM aware. When I run the alter table command to update the geometry I get an error "Cannot alter column participating in a view or rule"
Anyway to force this change and ignore the error? I tried changing the view to not reference the column and I tried adding a new geometry column to switch the view to temporarily while I make the change. Apparently I cannot drop a column in a view or change it to another column. I also tried writing the schema to SQL then edited the sql lines for linestring and point to linestringzm and pointzm and using psql to run the file to update the schema; all I get is access denied using "psql -U postgres -d utility -1 -f \i Z:......\xyz_test.sql"
Also tried pg_restore.
Anyway to just force the change using pg_admin4? or other suggestions?
I am not familiar with the tool you are using but I would suggest checking all the objects (views and rules) referencing your table by using the below command.
SELECT DISTINCT so.name
FROM syscomments sc
INNER JOIN sysobjects so ON sc.id=so.id
WHERE sc.TEXT LIKE '%tablename%'
Then backing up these object and dropping them completely before attempting to alter the table.
I have two databases on the same server and need to copy data from a table in the first db to a table in the second. A few caveats:
Both tables already exist (ie: I must not drop the 'copy-to' table first. I need to just add the data to the existing table)
The column names differ. So I need to specify exactly which columns to copy, and what their names are in the new table
After some digging I have only been able to find this:
pg_dump -t tablename dbname | psql otherdbname
But the above command doesn't take into account the two caveats I listed.
For a table t, with columns a and b in the source database, and x and y in the target:
psql -d sourcedb -c "copy t(a,b) to stdout" | psql -d targetdb -c "copy t(x,y) from stdin"
I'd use an ETL tool for this. There are free tools available, they can help you change column names and they are widely used and tested. Most tools allow external schedulers like the windows task scheduler or cron to run transformations based on whatever time schedule you need.
I personally have used Pentaho PDI for similar tasks in the past and it has always worked well for me. For your requirement I'd create a single transformation that first loads the table data from the source database, modify the column names in a "Select Values"-step and then insert the values into the target table using the "truncate" option to remove the existing rows from the target table. If your table is too big to be re-filled each time, you'd need to figure out a delta load procedure.
I am trying to
create a snapshot of a PostgreSQL database (using pg_dump),
do some random tests, and
restore to the exact same state as the snapshot, and do some other random tests.
These can happen over many/different days. Also I am in a multi-user environment where I am not DB admin. In particular, I cannot create new DB.
However, when I restore db using
gunzip -c dump_file.gz | psql my_db
changes in step 2 above remain.
For example, if I make a copy of a table:
create table foo1 as (select * from foo);
and then restore, the copied table foo1 remains there.
Could some explain how can I restore to the exact same state as if step 2 never happened?
-- Update --
Following the comments #a_horse_with_no_name, I tried to to use
DROP OWNED BY my_db_user
to drop all my objects before restore, but I got an error associated with an extension that I cannot control, and my tables remain intact.
ERROR: cannot drop sequence bg_gid_seq because extension postgis_tiger_geocoder requires it
HINT: You can drop extension postgis_tiger_geocoder instead.
Any suggestions?
You have to remove everything that's there by dropping and recreating the database or something like that. pg_dump basically just makes an SQL script that, when applied, will ensure all the tables, stored procs, etc. exist and have their data. It doesn't remove anything.
You can use PostgreSQL Schemas.