Can someone help me, please? I need to perform pg_dump from DB and that must be read by Athena
pg_dump --> s3 < ---- aws athena
How can I do this?
The PostgreSQL pg_dump format is pure SQL that can be run on a PostgreSQL database to create tables and load data. Open it in a text editor and take a look -- you'll see what I mean.
As a result, pg_dump files are not in a format that can be used with Amazon Athena.
Related
Hello experts I am using postgres 9.5 . When I take a backup from pgadmin it has 950 MB size but when i take the same db backup from pgdump.exe command the backup size is with 7.5 GB. I am confused which backup file will be secured for me that I can use to restore? the restoring process is also slow in postgresql. Please help me.
When you backup something in pgadmin it just calls pg_dump with appropriate options, so both your backups are made by the same pg_dump utility.
I guess you're comparing dumps in two different formats.
Default format for pg_dump is plain, which is basically an enormous uncompressed SQL file.
As for pgadmin, it uses custom format by default, which is a highly compressed binary file.
Also note that pgadmin always displays the actual pg_dump command used to create your dump in the log window, along with its full output.
You should be able to call this command in your command prompt to generate an identical backup file.
You can read more about different output formats and other pg_dump options in PostgreSQL docs.
I need to create reports/summary tables on Redshift using SAS. My client data is on Amazon Redshift and he provided me all credentials to access the database. I have SAS 9.2 (32bit) and downloaded PostgresSQL 32bit driver to my system (as Redshift is based on PostgresSQL). I setup ODBC data source successfully and now I am connecting SAS using below command:
LIBNAME RdSft ODBC DSN='Redshift server' user='xxxxxxx' pw='xxxxxx';
data Rdsft.new_table;
set Rdsft.old_table(obs=10);
run;
I am able to connect and can see contents of tables on Redshift but not able to make any table there. Sometimes I could but its taking hours to create a table just with 10 observations. Someone suggested me to use DbVisulizer to do this task but I am comfortable with SAS only.
Please suggest.
If you have SAS/ACCESS try using the postgres engine for the library instead of going via ODBC eg:
libname RdSft postgres server="<server-address>" database=<db-name> port=5432 user='xxxxxxx' pw='xxxxxx';
Also, try adding conopts="UseServerSidePrepare=1" to the libname as suggested by this article: http://support.sas.com/kb/52/585.html
The simple fact of the matter, is that when you're connecting to Redshift via ODBC, even your simple data step query:
"data Rdsft.new_table;
set Rdsft.old_table(obs=10);
run;"
Is essentially translating to "select * from rdsft.old_table" before the obs subset is getting applied.
The SAS/ACCESS postgres solution is solid, you may also want to use proc sql, select only the columns you want, and subset as much as possible. Proc Sql will translate a bit easier into Redshift query language through an ODBC than the data step will.
SAS will hopefully be issuing a SAS/ACCESS for REDSHIFT option sometime soon! :)
Is there any way to dump a postgres db using psql only ( without pg_dump )?
Thanks.
Theoretically you have access to all the data needed. In practice you're more likely to be able to dump/save some data using COPY command, but not the database schema, etc.
Note, that you do not have to have pg_dump on the same machine where your database server is, if it listens to the network. But well, I don't know why you even ask :)
In theory you could run queries to extract the schema and then use those results to extract the data. But it wouldn't be easy to manipulate all of that into something usable for a restore using just psql.
I want sql script of postgres 9 database schema which is not on local server. I tried pg_dump command on sql editor of pgAdmin and its not working there. I m not sure where to run that command. Please assist me with the same....
Thanks..
pg_dump is a command line utility; it isn't SQL, so it won't work in pgAdmin or anywhere else that executes SQL.
pgAdmin however does have a facility to do what you want:
Right-click on the database you want to export
Select Backup from the pop-up menu
Chose "format" Plain
Chose "plain option" Only schema
You can also use pgAdmin tool to generate or take plain text Database Backup.
You can generate plain text backup for data or schema only.
I have shared few screen shots, please visit this blog for more details.
Is it possible to have the postgres database dump(pg_dump) using SQLAlchemy? i can get the dump using pg_dump but I am doing all other db operations using SQLALchemy and thus want to know if this dump operation is also opssible using SQLAlchemy. Any suggestion, link would be of great help.
Thanks,
Tara Singh
pg_dump is a system command.so I do not think you could have postgres database dump using SQLAlchemy.
SqlAlchemy do not manage sort of pg_dump. You probably can mimic it with a buch of queries but it will be painfull.
The more easy way is to use pg_dump itself inside a python script with os.system or subprocess.call
If it's for regular saves also have a look to safekeep project who speak for you to your databases