I successfully put a binary file (a jpg) into a bytea field in postgres, with the following code.
CREATE TABLE file_locker_test
(
ID integer PRIMARY KEY,
THE_FILE_ITSELF bytea
);
INSERT INTO file_locker_test (ID, THE_FILE_ITSELF)
VALUES (1, bytea('\\Users\\My Name\\Pictures\\picture.jpg'));
Now, I'm trying to download the file back to make sure that it has uploaded correctly.
I tried this:
\copy (SELECT encode(file_locker_test(the_file_itself), 'hex') FROM file_locker_test LIMIT 1) TO '\\Users\\My Name\\Desktop\\picture.hex';
And got this error:
//Users/My Name/Desktop/picture.hex: No such file or directory
Does anyone have any insights?
You are confused.
The INSERT didn't insert the binary file, but the (binary) string \\Users\\My Name\\Pictures\\picture.jpg.
The file /Users/My Name/Desktop/picture.hex cannot be created on the database server, probably because one of the directories on the path does not exist.
If you want to insert the content of the binary file into the database, you'll have to write a program that opens and reads the file into memory and then inserts that.
Related
I have a table to store file information in postgresql.
select id,filestream,name from Table_file_info
Here filestream is bytea datatype. How to get bytea data into actual text (content of my file) in postgresql.
I tried with below query:
select encode(filestream, 'escape')::text as name from Table_file_info
but i am getting as below
ICAgICAgICAgc2FkZnNhZGZhZCBzZGRkZGRkZGRkIFRlc3R0dA==
actual content of my file is: sadfsadfad sddddddddd Testtt
It looks like base64. Meaning your file was first converted to base64, then converted to bytea (which is kind of pointless since base64 is already text)
select encode(decode(encode(filestream,'escape'),'base64'),'escape') from Table_file_info;
Is there a way to COPY the CSV file data directly into a JSON or JSONb array?
Example:
CREATE TABLE mytable (
id serial PRIMARY KEY,
info jSONb -- or JSON
);
COPY mytable(info) FROM '/tmp/myfile.csv' HEADER csv;
NOTE: each CSV line is mapped to a JSON array. It is a normal CSV.
Normal CSV (no JSON-embeded)... /tmp/myfile.csv =
a,b,c
100,Mum,Dad
200,Hello,Bye
The correct COPY command must be equivalent to the usual copy bellow.
Usual COPY (ugly but works fine)
CREATE TEMPORARY TABLE temp1 (
a int, b text, c text
);
COPY temp1(a,b,c) FROM '/tmp/myfile.csv' HEADER csv;
INSERT INTO mytable(info) SELECT json_build_array(a,b,c) FROM temp1;
It is ugly because:
need the a priory knowledge about fields, and a previous CREATE TABLE with it.
for "big data" need a big temporary table, so lost CPU, disk and my time — the table mytable have CHECKs and UNIQUEs constraints for each line.
... Needs more than 1 SQL command.
Perfect solution!
Not need to know all the CSV columns, only extract what you know.
Use at SQL CREATE EXTENSION PLpythonU;: if the command produce an error like "could not open extension control file ... No such file" you need to install pg-py extra-packages. In standard UBUNTU (16 LTS) is simple, apt install postgresql-contrib postgresql-plpython.
CREATE FUNCTION get_csvfile(
file text,
delim_char char(1) = ',',
quote_char char(1) = '"')
returns setof text[] stable language plpythonu as $$
import csv
return csv.reader(
open(file, 'rb'),
quotechar=quote_char,
delimiter=delim_char,
skipinitialspace=True,
escapechar='\\'
)
$$;
INSERT INTO mytable(info)
SELECT jsonb_build_array(c[1],c[2],c[3])
FROM get_csvfile('/tmp/myfile1.csv') c;
The split_csv() function was defined here. The csv.reader is very reliable (!).
Not tested for big-big CSV... But expected Python do job.
PostgreSQL workaround
It is not a perfect solution, but it solves the main problem, that is the
... big temporary table, so lost CPU, disk and my time"...
This is the way we do it, a workaround with file_fdw!
Adopt your conventions to avoid file-copy and file-permission confusions... The standard file path for a CSV. Example: /tmp/pg_myPrj_file.csv
Initialise your database or SQL script with the magic extension,
CREATE EXTENSION file_fdw;
CREATE SERVER files FOREIGN DATA WRAPPER file_fdw;
For each CSV file, myNewData.csv,
3.1. make a symbolic link (or scp remote copy) for your new file ln -sf $PWD/myNewData.csv /tmp/pg_socKer_file.csv
3.2. configure the file_fdw for your new table (suppose mytable).
CREATE FOREIGN TABLE temp1 (a int, b text, c text)
SERVER files OPTIONS (
filename '/tmp/pg_socKer_file.csv',
format 'csv',
header 'true'
);
PS: after running SQL script with psql, when having some permission problem, change owner of the link by sudo chown -h postgres:postgres /tmp/pg_socKer_file.csv.
3.3. use the file_fdw table as source (suppose populating mytable).
INSERT INTO mytable(info)
SELECT json_build_array(a,b,c) FROM temp1;
Thanks to #JosMac (and his tutorial)!
NOTE: if there is a STDIN way to do it (exists??), will be easy, avoiding permission problems and use of absolute paths. See this answer/discussion.
I have an issue trying to copy a CSV file in a table.
Here is my SQL statement:
DROP TABLE IF EXISTS nom_graph;
CREATE TABLE nom_graph
(
DATE VARCHAR(50),
EDP_REC FLOAT,
EDP_EC FLOAT,
NB_KO FLOAT
);
\copy nom_graph FROM '/home/giutools/EDP/out/SYNTHESE_RESYNC.csv' (DELIMITER('|'))
;
and this is the error I get:
psql:nom_graph.sql:179: ERROR: missing data for column "edp_rec"
CONTEXT: COPY nom_graph, line 1: "DATE;EDP_REC;EDP_EC;NB_KO"
The CSV file is composed by a : date ; and all the other values are FLOAT.
I really can't understand what's the issue, been trying to solve it for two days now.
The Problem is with your CSV file,
Step1: Convert the excel file to CSV using http://www.zamzar.com .
Step2: Create table in postgresql with the same column that you see in your excel file.
Step3: Copy the CSV file to the already created table using below command,
copy table_name (column1,column2,..) from 'C:\Users\Public\lifile_name.csv' delimiter ',' csv header;
Done, hope you find this helpful!
I'm migrating data into Postgresql. I can generate my data into CSV or tab-delimited files, and I'm trying to import these files using pgAdmin.
An example CSV file looks exactly like this:
86,72,1,test
72,64,1,another test
The table I'm importing into looks like this:
CREATE TABLE common.category
(
id integer NOT NULL,
parent integer,
user_id integer,
name character varying(128),
CONSTRAINT category_pkey PRIMARY KEY (id),
CONSTRAINT category_parent_fkey FOREIGN KEY (parent)
REFERENCES common.category (id) MATCH SIMPLE
ON UPDATE CASCADE ON DELETE CASCADE
)
However, upon importing this example, pgAdmin complains about an Invalid Input Syntax for Integer: "86" on the first line.
What am I missing here? I've tried performing the same import using a tab-delimited file, I've tried converting to both Windows and Unix EOLs.
Your sample have dependencies in the order of data imported. There is a foreign key 'parent' referencing 'id'. Having id 64 already in table, changing the order of your sample lines it imports just fine with:
COPY common.category
FROM 'd:\temp\importme.txt'
WITH CSV
I came across the same problem. After 2 hours of google, this did solve it. I just re-added the first line of the csv file, and every thing goes well now.
I had the same error after creating a new text file in Windows Explorer and changing the file extension to .csv.
I copied columns from an existing CSV file in Excel to the new one, also in Excel. After reading #Litty's comment about it not being tab-delimited it made me wonder if that was my problem.
Sure enough, opening the files in Excel hid the tab delimiting. When I opened it in Notepad++ it was obvious. I had to Export->Change File Type->CSV (Comma delimited) before I could import the file using pgAdmin as a default CSV file.
Database: Postgres 9.1
I have a table called logos defined like this:
create type image_type as enum ('png');
create table logos (
id UUID primary key,
bytes bytea not null,
type image_type not null,
created timestamp with time zone default current_timestamp not null
);
create index logo_id_idx on logos(id);
I want to be able to insert records into this table in 2 ways.
The first (and most common) way rows will be inserted in the table will be that a user will provide a PNG image file via an html file upload form. The code processing the request on the server will receive a byte array containing the data in the PNG image file and insert a record in the table using something very similar to what is explained here. There are plenty of example of how to insert byte arrays into a postgresql field of type bytea on the internet. This is an easy exercise. An example of the insert code would look like this:
insert into logos (id, bytes, type, created) values (?, ?, ?, now())
And the bytes would be set with something like:
...
byte[] bytes = ... // read PNG file into a byte array.
...
ps.setBytes(2, bytes);
...
The second way rows will be inserted in the table will be from a plain text file script. The reason this is needed is only to populate test data into the table for automated tests, or to initialize the database with a few records for a remote development environment.
Regardless of how the data is entered in the table, the application will obviously need to be able to select the bytea data from the table and convert it back into a PNG image.
Question
How does one properly encode a byte array, to be able to insert the data from within a script, in such a way that only the original bytes contained in the file are stored in the database?
I can write code to read the file and spit out insert statements to populate the script. But I don't know how to encode the byte array for the plain text script such that when running the script from psql the image data will be the same as if the file was inserted using the setBytes jdbc code.
I would like to run the script with something like this:
psql -U username -d dataBase -a -f test_data.sql
The easiest way, IMO, to represent bytea data in an SQL file is to use the hex format:
8.4.1. bytea Hex Format
The "hex" format encodes binary data as 2 hexadecimal digits per byte, most significant nibble first. The entire string is preceded by the sequence \x (to distinguish it from the escape format). In some contexts, the initial backslash may need to be escaped by doubling it, in the same cases in which backslashes have to be doubled in escape format; details appear below. The hexadecimal digits can be either upper or lower case, and whitespace is permitted between digit pairs (but not within a digit pair nor in the starting \x sequence). The hex format is compatible with a wide range of external applications and protocols, and it tends to be faster to convert than the escape format, so its use is preferred.
Example:
SELECT E'\\xDEADBEEF';
Converting an array of bytes to hex should be trivial in any language that a sane person (such a yourself) would use to write the SQL file generator.