Currently I can almost get a CSV from psql by simply running
psql -A -t -F'\t' -c "SELECT ...;" > myfile.csv
However it returns the number of rows at the end of the file. I can fix his with head -n -1
psql -A -t -F'\t' | head -n -1 | -c "SELECT ...;"
But with very large files seems like overkill. Is here a flag in psql where I can turn off number of records returned?
There is a number of common ways to get a CSV from PostrgeSQL (see e.g. this question). However not all of these ways are appropriate when working with Redshift, partly because Amazon Redshift is based on Postgres 8.0.2.
One can try to use --pset="footer=off" option to prevent psql from outputting the number of rows. Also, please consult 8.0.26 psql documentation.
Related
Among the hundreds of tables in the postgres DB, only the structure dump of tables starting with a specific name is found.
Example
the name of the table
abc
abc_hello
abc_hi
def
def_hello
def_hi
If you think there is
I want to dump only tables starting with abc*.
pg_dump abc.txt -Fp -t abc* -s DBName
However, it was not recognized because the amount of tables was too large.
it answered,
pg_dump: error: too many command-line arguments (first is "DBName")
but this command works fine
pg_dump abc_hello.txt -Fp -t abc_hello -s DBName
How can I pick them out?
Your main mistake is that you didn't consider that * also is a special character for the shell. If you run
pg_dump -t abc* mydatabase
the shell will replace abc* with the names of all files in the current directory that start with abc, so that your command will actually be something like
pg_dump -t abcfile1 abcfile2 abcfile3 mydatabase
which is syntactically not correct and will make pg_dump complain.
You have to protect the asterisk from being expanded by the shell with single quotes:
pg_dump -t 'abc*' mydatabase
The other error in your command line is that you forgot the -f in -f abc.txt.
I have a dump file (size around 5 GB) which is taken via this command:
pg_dump -U postgres -p 5440 MYPRODDB > MYPRODDB_2022.dmp
The database consists multiple schemas (let's say Schema A,B,C and D) but i need to restore only one schema (schema A).
How can i achieve that? The command below didn't work and gave error:
pg_restore -U postgres -d MYPRODDB -n A -p 5440 < MYPRODDB_2022.dmp
pgrestore: error: input file appears to be a text format dump. please
use psql.
You cannot do that with a plain format dump. That's one of the reasons why you always use a different format unless you need an SQL script.
If you want to stick with a plain text dump:
pg_dump -U postgres -p 5440 -n A MYPRODDB > MYPRODDB_2022.dmp
psql -U postgres -d MYPRODDB -p 5440 -f MYPRODDB_2022.dmp
Though dumping back over the same database as above will throw errors unless you use --clean or its short form -c to create commands to drop existing objects before restoring them:
-c
--clean
Output commands to clean (drop) database objects prior to outputting the commands for creating them. (Unless --if-exists is also specified, restore might generate some harmless error messages, if any objects were not present in the destination database.)
This option is ignored when emitting an archive (non-text) output file. For the archive formats, you can specify the option when you call pg_restore.
Probably also a good idea to throw in --if-exists:
--if-exists
Use conditional commands (i.e., add an IF EXISTS clause) when cleaning database objects. This option is not valid unless --clean is also specified.
Assuming I have the following table, functions and data
create table people (
id bigserial primary key,
age int,
height int,
weight int
);
create or replace function add_person (p_age int, p_height int, p_weight int);
create or replace function get_person_by_age (p_age int);
create or replace function get_person_by_height (p_height int);
create or replace function get_person_by_weight (p_weight int);
add_person (20,180,100);
add_person (20,181,101);
add_person (20,182,102);
add_person (20,183,103);
I am currently testing my database purely in bash so I would have these very long files like so (pseudocode only)
#!/bin/bash
# Insert data
sudo -u postgres psql -d $db_name -c "add_person(20,180,100)"
sudo -u postgres psql -d $db_name -c "add_person(20,181,101)"
sudo -u postgres psql -d $db_name -c "add_person(20,182,102)"
sudo -u postgres psql -d $db_name -c "add_person(20,183,103)"
# Retrieve data
persons=$(sudo -u postgres psql -d $db_name -c "get_person_by_age (20)")
# Count number of rows and make sure it matches the expected outcome
# (You have to manually strip header and footer lines but ignore for now)
if [ $(echo $persons | wc -l) -ne 4]
then
echo "Fail"
exit 1
fi
My test scripts have grown too large and there are so many things I am trying to catch (actions which should throw errors but which do not ie. false positives, actions which should not throw errors but which do ie false negatives, actions which throw errors other than that which they are supposed to, etc.). More importantly, the tests are incredibly slow as bash keeps trying to establish a connection to Postgre.
The reason I am not doing this in PGSQL is because the logic of queries can grow very complex as my db queries have many filters.
Is there a better existing solution to solve my problem? I looked at pgTAP but the documentation for that is horrendous
First of all, I think you could gain a lot of speed by running multiple commands in one client initialization. The "-c" flag only runs one command, and there is some small overhead in starting a connection that really adds up if you are running many commands, as you said.
For example, you could do something like this for your insert commands:
sudo -u postgres psql -d $db_name << EOF
add_person(20,180,100);
add_person(20,181,101);
add_person(20,182,102);
add_person(20,183,103);
Alternatively, you could list all the commands you want to run in a file and run them with "--file"/"-f":
sudo -u postgres psql -d $db_name -f "my_add_commands.psql"
I can't speak to why you are getting false errors, but generally I find that writing complicated test scripts in bash is a pain. If possible, I would consider using any language you are familiar with that has proper try/catch logic and a robust testing library with assertions. This is personally how I write tests for SQL functions. However, you would probably also need to use library that lets you make and process psql queries in the respective language.
There is a light psql testing library for python, but I haven't used it myself. Maybe worth a look though.
I need to know how to migrate from Postgres to MonetDB. Postgres is getting slow and we are trying to change to Monet. Someone now if already exists a script or another thing to migrate to Monet?
exist something equivalent to plpgsql on MonetDB?
exist materialized view on MonetDB?
The following booklet may be relevant to quickly identify some syntactic feature differences. https://en.wikibooks.org/wiki/SQL_Dialects_Reference
And the citrus performance is covered in a blogpost
https://www.monetdb.org/content/citusdb-postgresql-column-store-vs-monetdb-tpc-h-shootout
firstly: You can export data from postgres like:
psql -h xxxxx -U xx -p xx -d postgres -c "copy (select * from db40.xxx) to '/tmp/xxx.csv' delimiter ';'"
secondly: You must replace the NULL like:
sed 's/\\N/NULL/g' xxx.csv >newxxx.csv
last: You can use this to copy data into monetdb like:
mclient -u monetdb -d voc -h 192.168.205.8 -p 50000 -s "COPY INTO newxxx from '/tmp/newxxx.csv' using delimiters ';';"
I have a database with hundreds of tables, what I need to do is export specified tables and insert statements for the data to one sql file.
The only statement I know can achieve this is
pg_dump -D -a -t zones_seq interway > /tmp/zones_seq.sql
Should I run this statement for each and every table or is there a way to run a similar statement to export all selected tables into one big sql big. The pg_dump above does not export the table schema only inserts, I need both
Any help will be appreciated.
Right from the manual: "Multiple tables can be selected by writing multiple -t switches"
So you need to list all of your tables
pg_dump --column-inserts -a -t zones_seq -t interway -t table_3 ... > /tmp/zones_seq.sql
Note that if you have several table with the same prefix (or suffix) you can also use wildcards to select them with the -t parameter:
"Also, the table parameter is interpreted as a pattern according to the same rules used by psql's \d commands"
If those specific tables match a particular pattern, you can use that with the -t option in pg_dump.
pg_dump -D -a -t zones_seq -t interway -t "<pattern>" -f /tmp/zones_seq.sql <DBNAME>
For example to dump tables which start with "test", you can use
pg_dump -D -a -t zones_seq -t interway -t "^test*" -f /tmp/zones_seq.sql <DBNAME>