I'm trying to use the copy command to copy the content of a file into a database.
One of the lines have this:
CCc1ccc(cc1)C(=O)/N=c\1/n(ccs1)C
and when i insert this normally into database there is no errors.
But when i'm trying to use the following command, this line is not insert correctly.
cat smile_test.txt | psql -c "copy testzincsmile(smile) from stdout" teste
This i what i get (it is wrong):
CCc1ccc(cc1)C(=O)/N=c/n(ccs1)C
What's wrong here?
Thank you :)
copy expects a specific input format and cannot just be used to read random text from a file into a field.
See the manual.
The specific issue you're hitting is probably a backslash being interpreted as an escape by the default copy in/out format.
I figure out how to do this:
This is my answer:
cat smile_test.txt | sed '1d; s/\\/\\\\/g' | psql -c "copy testzincsmile(smile) from stdout" teste
Related
I'm using PostgreSQL on Windows 7 through the command line. I want to import the content of different CSV files into a newly created table.
After executing the command the database name appeared like:
database=#
Now appears like
database*# after executing:
type directory/*.csv | psql -c 'COPY sch.trips(value1, value2) from stdin CSV HEADER';
What does *# mean?
Thanks
This answer is for Linux and as such doesn't answer OP's question for Windows. I'll leave it up anyway for anyone that comes across this in the future.
You accidentally started a block comment with your type directory/*.csv. type doesn't do what you think it does. From the bash built-ins:
With no options, indicate how each name would be interpreted if used as a command name.
Try doing cat instead:
cat directory/*.csv | psql -c 'COPY sch.trips(value1, value2) from stdin CSV HEADER';
If this gives you issues because each CSV has its own header, you can also do:
for file in directory/*.csv; do cat "$file" | psql -c 'COPY sch.trips(value1, value2) from stdin CSV HEADER'; done
Type Command
The type built-in command in Bash is a way of viewing command interpreter results. For example, using it with ssh:
$ type ssh
ssh is /usr/bin/ssh
This indicates how ssh would be interpreted when you run ssh as a command in the current Bash environment. This is useful for things like aliases. As an example for this, ll is usually an alias to ls -l. Here's what my Bash environment had for ll:
$ type ll
ll is aliased to `ls -l --color=auto'
For you, when you pipe the result of this command to psql, it encounters the /* in the input and assumes it's a block comment, which is what the database*# prompt means (the * indicates it's waiting for the comment close pattern, */).
Cat Command
cat is for concatenating multiple files together. By default, it writes to standard out, so cat directory/*.csv will write each CSV file to standard out one after another. However, piping this means that each CSV's header will also be piped mid-stream of the copy. This may not be desirable, so:
For Loop
We can use for to loop over each file and individually import it. The version I have above, for file in directory/*.csv, will properly handle files with spaces. Properly formatted:
for file in directory/*; do
cat "$file" | psql -c 'COPY sch.trips(value1, value2) from stdin CSV HEADER'
done
References
PostgreSQL 10 Comments Documentation (postgresql.org)
type built-in Manual page (mankier.com)
cat Manual page (mankier.com)
Bash looping tutorial (tldp.org)
I've looked around to find a solution to my problem in other posts listed bellow, but it looks my regex is quit different and need special care:
How to output only captured groups with sed
Replace one capture group with another with GNU sed (macOS) 4.4
sed replace line with capture groups
I'm trying to replace a regex match group in big JSON file,
My file has mongoDB exported objects, and I'm trying to replace the objectId with the string:
{"_id":{"$oid":"56cad2ce0481320c111d2313"},"recordId":{"$oid":"56cad2ce0481320c111d2313"}}
So the output in the original file should look like this:
{"_id":"56cad2ce0481320c111d2313","recordId":"56cad2ce0481320c111d2313"}
That's the command I run in the shell:
sed -i 's/(?:{"\$oid":)("\w+")}/\$1/g' data.json
I get no error, but the file remains the same.
What exactly am I doing wrong?
Finally I've managed to make it work, the way regex works in bash is different then in regexr.com tester tool.
echo '{"$oid":"56cad2ce0481320c111d2313"}' | sed 's/{"$oid":\("\w*"\)}/\1/g'
gives the correct output:
"56cad2ce0481320c111d2313"
I found it even better to read from stdin and output to file, instead of writing first to JSON file, then read, replace and write again.
Since I use mongoexport to export collection, replace the objectId and write the output to JSON file, my final solution looks like this:
mongoexport --host localhost --db myDB --collection my_collection | sed 's/{"$oid":\\("\\w*"\\)}/\\1/g' >> data.json
Is there any way to use copy command for batch data import and read data from a url. For example, copy command has a syntax like :
COPY sample_table
FROM 'C:\tmp\sample_data.csv' DELIMITER ',' CSV HEADER;
What I want is not to give a local path but a url. Is there any way?
It's pretty straightforward, provided you have an appropriate command-line tool available:
COPY sample_table FROM PROGRAM 'curl "http://www.example.com/file.csv"'
Since you appear to be on Windows, I think you'll need to install curl or wget yourself. There is an example using wget on Windows here which may be useful.
My solution is
cat $file |
tail -$numberLine |
sed 's/ / ,/g' |
psql -q -d $dataBaseName -c "COPY tableName FROM STDIN DELIMITER ','"
You can insert a awk between sed and psql to add missing column.
Interesting if already you know what to put in the missing column.
awk '{print $0" , "'info_about_missing_column'"\n"}'
I have done that and it works and faster than INSERT.
I want to use to copy command to copy data into postgres; while there other processes are simultaneously writing into the CSV file.
Is something like this possible? Take the stdout from tail and pipe into the stdin of postgres.
COPY targetTable ( column1, column2 )
FROM `tail -f 'path/to/data.csv'`
WITH CSV
Assuming PostgreSQL 9.3 or better, there's the possibility of copying from a program output with:
COPY FROM PROGRAM 'command'
From the doc:
PROGRAM
A command to execute. In COPY FROM, the input is read from standard
output of the command, and in COPY TO, the output is written to the
standard input of the command.
This may be what you need except for the fact that tail -f being a never-ending command by design, it's not obvious how you plan for the COPY to ever finish. Presumably you'd need to replace tail -f by a more elaborate script with some exit condition.
you can also do COPY FROM STDIN;
example:
tail -f datafile.csv | psql -tc "COPY table from STDIN" database
I'm trying to export a PostgreSQL table with headings to a CSV file via command line, however I get it to export to CSV file, but without headings.
My code looks as follows:
COPY products_273 to '/tmp/products_199.csv' delimiters',';
COPY products_273 TO '/tmp/products_199.csv' WITH (FORMAT CSV, HEADER);
as described in the manual.
From psql command line:
\COPY my_table TO 'filename' CSV HEADER
no semi-colon at the end.
instead of just table name, you can also write a query for getting only selected column data.
COPY (select id,name from tablename) TO 'filepath/aa.csv' DELIMITER ',' CSV HEADER;
with admin privilege
\COPY (select id,name from tablename) TO 'filepath/aa.csv' DELIMITER ',' CSV HEADER;
When I don't have permission to write a file out from Postgres I find that I can run the query from the command line.
psql -U user -d db_name -c "Copy (Select * From foo_table LIMIT 10) To STDOUT With CSV HEADER DELIMITER ',';" > foo_data.csv
This works
psql dbname -F , --no-align -c "SELECT * FROM TABLE"
The simplest way (using psql) seems to be by using --csv flag:
psql --csv -c "SELECT * FROM products_273" > '/tmp/products_199.csv'
For version 9.5 I use, it would be like this:
COPY products_273 TO '/tmp/products_199.csv' WITH (FORMAT CSV, HEADER);
This solution worked for me using \copy.
psql -h <host> -U <user> -d <dbname> -c "\copy <table_name> FROM '<path to csvfile/file.csv>' with (format csv,header true, delimiter ',');"
Heres how I got it working power shell using pgsl connnect to a Heroku PG database:
I had to first change the client encoding to utf8 like this: \encoding UTF8
Then dumped the data to a CSV file this:
\copy (SELECT * FROM my_table) TO C://wamp64/www/spider/chebi2/dump.csv CSV DELIMITER '~'
I used ~ as the delimiter because I don't like CSV files, I usually use TSV files, but it won't let me add '\t' as the delimiter, so I used ~ because its a rarely used characeter.
The COPY command isn't what is restricted. What is restricted is directing the output from the TO to anywhere except to STDOUT. However, there is no restriction on specifying the output file via the \o command.
\o '/tmp/products_199.csv';
COPY products_273 TO STDOUT WITH (FORMAT CSV, HEADER);
copy (anysql query datawanttoexport) to 'fileablsoutepathwihname' delimiter ',' csv header;
Using this u can export data also.
I am posting this answer because none of the other answers given here actually worked for me. I could not use COPY from within Postgres, because I did not have the correct permissions. So I chose "Export grid rows" and saved the output as UTF-8.
The psql version given by #Brian also did not work for me, for a different reason. The reason it did not work is that apparently the Windows command prompt (I was using Windows) was meddling around with the encoding on its own. I kept getting this error:
ERROR: character with byte sequence 0x81 in encoding "WIN1252" has no equivalent in encoding "UTF8"
The solution I ended up using was to write a short JDBC script (Java) which read the CSV file and issued insert statements directly into my Postgres table. This worked, but the command prompt also would have worked had it not been altering the encoding.
Try this:
"COPY products_273 FROM '\tmp\products_199.csv' DELIMITER ',' CSV HEADER"
In pgAdmin, highlight your query statement just like when you use F5 to execute and press F9 - this will open the file browser so you can pick where you save your CSV.
If you are using Azure Data Studio, the instruction are here: Azure Data Studio: Save As CSV.
I know this isn't a universal solution, but most of the time you just want to grab the file by hand.