I have the text file to import with the following format:
columA | columnB | columnC
-----------------------------------------
1 | A | XYZ
2 | B | XZ
3 | C | YZ
I can skip first line by using:
WITH CSV HEADER;
in copy command, but got stuck while skipping second line.
If you're using COPY FROM 'filename', you could instead use COPY FROM PROGRAM to invoke some shell command which removes the header from the file and returns the rest.
In Windows:
COPY t FROM PROGRAM 'more +2 "C:\Path\To\File.txt"'
In Linux:
COPY t FROM PROGRAM 'tail -n +3 /path/to/file.txt'
If you're trying to send a local file to a remote server, you can do something similar through psql, e.g.:
tail -n +3 file.txt | psql -c 'COPY t FROM STDIN'
The COPY command can only skip the first line. The easiest solution would be to manually remove the second line before importing but if that is not possible, then you have to use a "dirty" trick.
You create a table that has a single column of varchar type and import the text file into that table. After import you run a PL/pgSQL function to read all the rows in the table (except the header rows, obviously) and extract the information you want to insert in the destination table with, for instance, the regexp_matches() or regexp_split_to_array() function. You can also automate the whole process by using an after insert trigger on the import table, if you have to import many files with the same issue.
Related
I have a file in my TEMP directory on a windows server
echo %TEMP%
C:\Users\BOB\AppData\Local\Temp\2
Below command to insert file to table:
psql -d BOBDB01 -c "\COPY "BOBTB01" FROM 'C:\Users\BOB\AppData\Local\Temp\2\file.csv' with csv";
This results in an error:
Password for user postgres: <pw_given>
ERROR: relation "bobtb01" does not exist
It does exist though:
\d
List of relations
Schema | Name | Type | Owner
--------+-----------------------------------------------------------------+----------+--------
public | BOBTB01 | table | BOB
Can anyone please help me kick around some ideas as to why the copy command fails given that the table is there?
Thanks.
"but it IS in double quotes...? no?"
No. The windows shell eats the double quotes, so PostgreSQL never sees them. You can get them though to PostgreSQL by either backwacking them, or doubling them. (backwacking them works in bash as well, doubling them is just a Windows-shell thing. If you are using MS PowerHell not cmd shell, then only doubling them works, backwacking does something different)
What currently happens is that that the unescaped " before B is regarded by the cmd shell as closing the very first " on the line, then you have the bare word BOBTB01, then the " after the 1 re-opens the quote, which extends to the rest of the line. The effect is that internal " are not passed on to the psql program.
DB2 command to Postgres command.
db2 IMPORT FROM test.csv OF DEL MODIFIED BY USEDEFAULTS COMMITCOUNT 100000 "INSERT_UPDATE INTO TEST.person (name,old,sex)" > ${TEMPTXT}
How can i use postgres command to do the same thing like this db2 command to import from file to insert and update the table ?
Postgres has COPY, but it doesn't perform update.So, first run COPY into a TEMP table and then merge into main table.
For a comma delimiter,
CREATE TEMP TABLE TEST.tmp_person (name text,old text,sex text)
COPY TEST.tmp_person FROM test.csv WITH DELIMITER ',' CSV
-- ^temp table
There are various techniques in Postgres to do INSERT_UPDATE or merge. Refer this post. Use proper casting to appropriate target data types while inserting/updating.
I was trying to import a CSV file to my Postgresql with the frist 8 lines skipped and start from the 9th line. My codes below works to read from the second line and treat the first line as header:
create table report(
id integer,
name character(3),
orders integer,
shipments float
);
COPY report
FROM 'C:\Users\sample.csv' DELIMITER ',' CSV HEADER;
Now how to improve this code to read from the 9th line.
Thank you!
CSV details
With PostgreSQL 9.3 or newer, COPY can refer to a program to preprocess the data, for instance Unix tail.
To start an import at line 9:
COPY report FROM PROGRAM 'tail -n +9 /path/to/file.csv' delimiter ',' csv;
Apparently you're using Windows, so tail might not be immediately available. Personally I would install it from MSYS, otherwise there are alternatives mentioned in
Looking for a windows equivalent of the unix tail command
or Windows equivalent of the 'tail' command.
I have my database hosted on heroku, and I want to download specific parts of the database (e.g. all the rows with id > x from table 1, all the rows with name = x from table 2, etc.) in a single file.
From some research and asking a question here it seems that some kind of modified pg_dump would solve my problem. However, I won't be able to use pg_dump because I won't have access to the command line (basically I want to be able to click a button in my web app and it will generate + download the database file).
So my new strategy is to use the postgres copy command. I'll go through the various tables in my server database, run COPY (Select * FROM ... WHERE ...) TO filename , where filename is just a temporary file that I will download when complete.
The issue is that this filename file will just have the rows, so I can't just turn around and import it into pgadmin. Assuming I have an 'empty' database set up (the schema, indices, and stuff are all already set up), is there a way I can format my filename file so that it can be easily imported into a postgres db?
Building on my comment about to/from stdout/stdin, and answering the actual question about including multiple tables in one file; you can construct the output file to interleave copy ... from stdin with actual data and load it via psql. For example, psql will support input files that look like this:
copy my_table (col1, col2, col3) from stdin;
foo bar baz
fizz buzz bizz
\.
(Note the trailing \. and that the separators should be tabs; you could also specify the delimiter option in the copy command).
psql will treat everything between the ';' and '.' as stdin. This essentially emulates what pg_dump does when you export table data and no schema (e.g., pg_dump -a -t my_table).
The resulting load could be as simple as psql mydb < output.dump.
Is there any utility available to Import the table data from CSV file.
Since I have huge no of tables and rows.Writing COPY FROM sql will take a time,so in need of any utility or another approaches
You really should use copy:
#!/bin/bash
(
echo 'copy table_name from stdin (format csv);'
cat table_name.csv || exit 1
echo '\.'
) | psql database_name
You don't need to change your big files in any way. You may need to tweak some copy options (delimiter, quote etc.).
This might be helpful:
http://csvloader.sourceforge.net/