We are using Postgres 9.4. Following is the requirement.
Can Anyone share me the shell script for importing CSV files into a table.
Which ever CSV file import is done then it should be moved to different directory.
Thanks
I have written like below. And its working fine.
For all CSV to 1 table
cd /tmp--Files are located in /tmp folder
for x in $(ls /tmp/*.csv);
do PGPASSWORD=psqlpassword psql -U pega -c "copy table_name from '/tmp/$x' csv" -h <dbhost>; done
mv /tmp/*.csv /backup_folder/
Related
I have a requirement of taking the table backup into S3 in csv format . I already tried taking the pg_dump of the tables, but those are in .sql format. Also I tried using gz format, but they all contains the dump of the data. I have tried this command:
pg_dump -v -h <hostname> -d <dbname> -t <tablename> -U <username>| gzip | aws s3 cp - s3://xxx/xxx/tablename.csv.gz
Is it possible to get the tables exactly in csv file format so that it could be easy to create QuickSight reports out of that because we cant use pg_dump directly to create the QuickSight reports.
Any help would be much appreciated.
Note: the table size is between 50 to 100 GB
You can use COPY command to take data dump in CSV.
Server Side Export:
copy table_name to 'Absolute/path/to/filename.csv' csv header;
Client Side Export:
\copy table_name to 'Relative/path/to/filename.csv' csv header;
According to a website I can download their sample file dvdrental.zip, but
The database file is in zipformat ( dvdrental.zip) so you need to extract > it to dvdrental.tar
First of all, what is a tar? I thought it had to be tar.gz to be compressed? I don't even know how to create a "tar" by itself. I tried:
tar -zcvf dvdrental.tar.gz dvdrental
and
tar -cf dvdrental.tar dvdrental
I try to import with pgAdmin 4 and I get either:
pg_restore: [archiver] input file does not appear to be a valid archive
or
pg_restore: [tar archiver] could not find header for file "toc.dat" in tar archive
respectively. Now, don't ask me why a popular tutorial site created a file in the wrong format. But, can you tell me how to repackage this file so I can use it as a sample DB?
Using Mac OS 10.12.4. Postgres 9.6. And PgAdmin 4 (not sure if it's in beta? It crashed and does all kinds of nonsensical window movement and highlighting)
I have extracted .zip archive first. Then opened pgAdmin and followed the guide "Load the DVD Rental database using the pgAdmin"
https://www.postgresqltutorial.com/load-postgresql-sample-database/
Pay attention to changing 'Format' field from 'Custom or Tar' to 'Directory'. Then you should be able to restore DB.
If you look into the .tar archive you will find the restore.sql where at the top:
-- File paths need to be edited. Search for $$PATH$$ and
-- replace it with the path to the directory containing
-- the extracted data files.
So to create sample DB you could to extract .tar content somewhere and use single command:
sed -e 's/\$\$PATH\$\$/\/path\/to\/extracted\/files/g' restore.sql | psql
Or
sed -e 's/\$\$PATH\$\$/\/path\/to\/extracted\/files/g' restore.sql > r.sql
and try to execute the r.sql content using PgAdmin.
get sample dataset from the link you cited and save somewhere.
Assuming postgres is installed and running do the following:
Run createdb dvdrental
Run pg_restore -d dvdrental ./dvdrental where "./dvdrental" is the path to the downloaded and unzipped file.
For create sample DB in postgres you following this steps:
1.- Create directory and enter it:
mkdir -p /tmp/dvdrental && cd /tmp/dvdrental
2.- Download zip file dvdrental.zip:
wget https://www.postgresqltutorial.com/wp-content/uploads/2019/05/dvdrental.zip
3.- Uncompress file .zip and later .tar:
unzip dvdrental.zip
tar -xvf dvdrental.tar
4.- Replace in execution time $$PATH variable and review it with grep command:
sed -e 's/\$\$PATH\$\$/\/tmp\/dvdrental/g' restore.sql | grep --color dvdrental
5.- Import DB sample for specific host (localhost), port (5433), user (db) and database name (postgres):
sed -e 's/\$\$PATH\$\$/\/tmp\/dvdrental/g' restore.sql | psql -h localhost -p 5433 -U db -d postgres
Finally, I show import successful with program pgAdmin III
I just started learning Postgresql, and cannot figure out how to import a csv or text file into it. I do not have sql background, and I am not sure if I need to create a table in the database with exactly the same column names as the csv file I want to import. I cannot open my csv file because it is too big (10G), so I do not know what its column names are. In this case, how can I import it to Postgresql. Another thing is that I am not sure if I need to put the csv file in the certain document.
To import a csv file into PostreSQL you will need to create the table that will match the columns of the csv file values in the necessary database.
When doing the import from Linux you will need to give the PostgreSQL user access to the csv file that is going to be imported.
sudo chown postgres:postgres /path/to/csvfile
If you still don't have enough permissions then you can give full access to the file
sudo chmod 777 /path/to/csvfile
Then if you like you can execute from the Linux command line
/path/to/psql -d databasename -U postgres -c "copy tablename from '/path/to/csvfile' delimiter ',' csv"
If you want to execute it from the psql command line then you'd use
copy tablename from '/path/to/csvfile' delimiter ',' csv
first off im relatively new to pgadmin and postgresql
so i have a backup done with pgadmin in plain text which dumps the data i need from one table into a file called lodging. it came from a plans_to_lodge table from our live server. now to test a new web application front end i want to pg_restore it but this is what i get
> pg_restore -h localhost -d TestingDatabase2 -t lodging_of_plans -n plan_authentication "C:\lodging"
pg_restore: [archiver] could not open input file "C:\lodging": No such file or directory
the documentation doesnt give an example for a table restore
and yes the file is at that location in the C drive
The issue was it being the root folder, so when i put the file named lodging into another folder like so
C:\folder\lodging
it worked now onto more errors
I have a bunch of SQL scripts that create tables in the database. Each table is located in a separate file so that editing them is much easier.
I wanted to prepare a single SQL script that will create the full schema, create tables, insert test data and generate sequences for the tables.
I was able to do such thing for oracle database but I am having problems with postgres.
The thing is - I do not know how to run the table creating script from another script.
In oracle I do it using the following syntax:
##'path of the script related to the path of the currently running sql file'
And everything works like a charm.
In postgres I was trying to search for something alike and found this:
\ir 'relative path to the file'
Unfortunately when I run my main script I get the message:
No such file or directory.
The example call is here:
\ir './tables/map_user_groups.sql'
I use Postgres 9.3. I tried to run the script using psql:
psql -U postgres -h localhost -d postgres < "path to my main sql file"
The file executes fine except for the calling of those other scripts.
Does anybody know how to solve the problem ?
If something in the question is unclear - just let me know :)
Based on the answer It is possible to reference another SQL file from SQL script, on PostgreSQL, you can include another SQL's files just using the \i syntax. I just tested and is working good on PostgreSQL 9.6:
\i other_script.sql
SELECT * FROM table_1;
SELECT * FROM table_2;
By the #wildplasser comment:
psql -U postgres -h localhost -d postgres < "path to my main sql file"
From psql's perspective the main_sql file is just stdin, and stdin has no "filename". Use -f filename to submit a file with a name:
psql -U postgres -h localhost -d postgres -f filename.sql
How to run postgres sql script from another script?
Seems the same question as: How to import external sql scripts from a sql script in PostgreSQL?