We have decided to move from OIDs in our PostgreSQL 9.0 database and use bytea columns instead. I'm trying to copy the data from one column to the other, but I can't figure out the right query. This is the closest I've gotten to:
update user as thistable set pkcs_as_bytea = (select array_agg(mylargeobject.data) from
(select * from pg_largeobject where loid = thistable.pkcs12_as_oid order by pageno) as mylargeobject) where thistable.pkcs12 is not null
And that gives me the following error message:
ERROR: column "pkcs_as_bytea" is of type bytea but expression is of type bytea[]
What would be the right query then?
Another way which doesn't require a custom function is to use the loread(lo_open(...)) combination, like:
UPDATE user SET pkcs_as_bytea = loread(lo_open(pkcs12_as_oid, 262144), 1000000) WHERE thistable.pkcs12 IS NOT NULL
There is a problem with this code, the loread function requires as the second parameter the maximum number of bytes to read (the 1000000 parameter I used above), so you should use a really big number here if your data is big. Otherwise, the content will be trimmed after this many bytes, and you won't get all the data back into the bytea field.
If you want to convert from OID to a text field, you should also use a conversion function, as in:
UPDATE user SET pkcs_as_text = convert_from(loread(lo_open(pkcs12_as_oid, 262144), 1000000), 'UTF8')
(262144 is a flag for the open mode, 40000 in hexa, which means "open read-only")
Here is a stored procedure that does the magic:
CREATE OR REPLACE FUNCTION merge_oid(val oid)
returns bytea as $$
declare merged bytea;
declare arr bytea;
BEGIN
FOR arr IN SELECT data from pg_largeobject WHERE loid = val ORDER BY pageno LOOP
IF merged IS NULL THEN
merged := arr;
ELSE
merged := merged || arr;
END IF;
END LOOP;
RETURN merged;
END
$$ LANGUAGE plpgsql;
well, i did something like this. I have attachment table and content column with data in oid type. I migrated with four actions:
ALTER TABLE attachment add column content_bytea bytea
UPDATE attachment SET content_bytea = lo_get(content)
ALTER TABLE attachment drop column content
ALTER TABLE attachment rename column content_bytea to content
You need something like array_to_string(anyarray, text) for text arrays, but in this case an array_to_bytea(largeobjectarray) to concat all sections. You have to create this function yourself, or handle this in application logic.
This is what you can do.
--table thistable --
ALTER TABLE thistable add column se_signed_bytea bytea;
UPDATE thistable SET se_signed_bytea = lo_get(pkcs_as_bytea);
ALTER TABLE thistable drop column pkc`enter code here`s_as_bytea;
ALTER TABLE thistable rename column se_signed_bytea to pkcs_as_bytea;
Related
I need to create a function that will generate a table.
This table will have columns brought with several left joins.
At some point i need to filter the data based on the value of a dynamic column (i need to have: WHERE table_name.dynamic_column_name = 1)(that column has 1s and 0s, i need to keep only the 1s)
So when I 'call' the function, user should type like this: SELECT * FROM function_name(dynamic_column_name)
What i did:
CREATE OR REPLACE FUNCTION check_gap (_col_name varchar)
RETRUNS TABLE ( ... here i have several columns with their type ...)
LANGUAGE plpsql
AS $function$
BEGIN
RETURN QUERY
SELECT ... a bunch of columns ...
FROM ... a bunch of tables left joined ...
WHERE _col_name = 1;
END;
$function$
;
I even tried with table_name._col_name .. though it wasn't necessary, the query (select from) works just as fine without
** I found some solutions for dynamic value but not for a dynamic column
*** I am using PostgreSQL (from DBeaver)
You need dynamic SQL for that. Pls. note that the code below is SQL injection prone.
CREATE OR REPLACE FUNCTION check_gap (col_name varchar)
RETURNS TABLE (... several columns with their type ...) LANGUAGE plpgsql AS
$$
BEGIN
RETURN QUERY EXECUTE format
(
'SELECT ... a bunch of columns ...
FROM ... a bunch of tables left joined ...
WHERE %I = 1;', col_name
);
END;
$$;
I would like to access a column by using variable instead of a static column name.
Example:
variable := 'customer';
SELECT table.variable (this is what I would prefer) instead of table.customer
I need this functionality as records in my table vary in terms of data length (eg. some have data in 10 columns, some in 14 or 16 columns etc.) so I need to address columns dynamically. As I understand, I can't address columns by their index (eg. select 8-th column of the table) right?
I can loop and put the desired column name in a variable for the given iteration. However, I get errors when I try to access a column using that variable (e.g. table_name.variable is not working).
For the sake of simplicity, I paste just some dummy code to illustrate the issue:
CREATE OR REPLACE FUNCTION dynamic_column_name() returns text
LANGUAGE PLPGSQL
AS $$
DECLARE
col_name text;
return_value text;
BEGIN
create table customer (
id bigint,
name varchar
);
INSERT INTO customer VALUES(1, 'Adam');
col_name := 'name';
-- SELECT customer.name INTO return_value FROM customer WHERE id = 1; -- WORKING, returns 'Adam' but it is not DYNAMIC.
-- SELECT customer.col_name INTO return_value FROM customer WHERE id = 1; -- ERROR: column customer.col_name does not exist
-- SELECT 'customer.'||col_name INTO return_value FROM customer WHERE id = 1; -- NOT working, returns 'customer.name'
-- SELECT customer||'.'||col_name INTO return_value FROM customer WHERE id = 1; -- NOT working, returns whole record + .name, i.e.: (1,Adam).name
DROP TABLE customer;
RETURN return_value;
END;
$$;
SELECT dynamic_column_name();
So how to obtain 'Adam' string with SQL query using col_name variable when addressing column of customer table?
SQL does not allow to parameterize identifiers (incl. column names) or syntax elements. Only values can be parameters.
You need dynamic SQL for that. (Basically, build the SQL string and execute.) Use EXECUTE in a plpgsql function. There are multiple syntax variants. For your simple example:
CREATE OR REPLACE FUNCTION dynamic_column_name(_col_name text, OUT return_value text)
RETURNS text
LANGUAGE plpgsql AS
$func$
BEGIN
EXECUTE format('SELECT %I FROM customer WHERE id = 1', _col_name)
INTO return_value;
END
$func$;
Call:
SELECT dynamic_column_name('name');
db<>fiddle here
Data types have to be compatible, of course.
More examples:
How to use text input as column name(s) in a Postgres function?
https://stackoverflow.com/search?q=%5Bpostgres%5D+%5Bdynamic-sql%5D+parameter+column+code%3AEXECUTE
I have 2 tables in Postgresql with the same schema, the only difference is that in one of the table id field is of type bigint. Schema of the table I need to fill with data looks like this:
create table test_int_table(
id int,
description text,
hash_code int
);
I need to copy the data from test_table with bigint id to public.test_int_table. And some of the values which are bigger than id range should be filtered out. How can I track those values without hardcoding the range?
I can do something like this, but I would like to build more generic solution:
insert into test_int_table
select * from test_table as test
where test.id not between 2147483647 and 9223372036854775808
By generic I mean without constraints on the columns names and their number. So that in case, I have multiple columns of bigint type in other tables how can I filter all of their columns values generically (without specifying a column name)?
There is no generic solution, as far as I can tell.
But I would write it as
INSERT INTO test_int_table
SELECT *
FROM test_table AS t
WHERE t.id BETWEEN -2147483647 AND 2147483647;
You can do something like this if you want to track :
Create a function like this :
CREATE OR REPLACE FUNCTION convert_to_integer(v_input bigint)
RETURNS INTEGER AS $$
DECLARE v_int_value INTEGER DEFAULT NULL;
BEGIN
BEGIN
v_int_value := v_input::INTEGER;
EXCEPTION WHEN OTHERS THEN
RAISE NOTICE 'Invalid integer value: "%". Returning NULL.', v_input;
RETURN NULL;
END;
RETURN v_int_value;
END;
$$ LANGUAGE plpgsql;
and write a query like this :
INSERT INTO test_int_table SELECT * FROM test_table AS t WHERE convert_to_integer(t.id) is not null;
Or you can modify a function to return 0.
Postgres has a great RETURNING clause for INSERT, DELETE and UPDATE...and it's made me a bit greedy. In a few cases, what I'd like to get is not only the current value, but the previous value:
UPDATE analytic_productivity
SET points = 1000
WHERE points > 1000
RETURNING id,
points,
OLD.points;
I don't believe there's any way to access previous values outside of the lifespan and context of a trigger. So, I'll guess what I'd like isn't possible as such. If that's right, can anyone suggest an alternative? I'm overwriting outliers with some set values, and would like to record the modified values in another table. This is why I don't know the current value in advance. This is a rare (and clearly suspect) operation, and I don't want to record the change on normal inserts and updates.
As an alternative, I'm thinking that I can select the outliers, revise them, and then write back the modifications. So, do most of the work on the client side with a couple of requests to Postgres. If so, can someone suggest the right locking level to apply between my initial SELECT and my following UPDATE? I believe that the FOR UPDATE lock is right.
Any suggestions on a smart way to capture previous values, during an update, without a trigger would be great to hear about.
Follow-up
Thanks to comments here, I experimented a bit and came up with a solution that works in my case. To make my objectives clearer:
I've got a table named outlier_rule that defines values that are too high for a specific column.
The goal is to loop over the table, and apply the rules to set outliers to a fixed value.
Stomping on outliers like this is...questionable. There must be leaks in the app's UI that allow for unreasonable values. To help track these down, I'm recording the large values in a table named outlier_change.
I'd like to push this behavior into server-side function so that any of our servers, regardless of their codebase version, can invoke the current logic.
The client servers compose and send an email with a result summary, when outliers are found and corrected.
So, a server-side function to do everything, log some data, and return a result. I've got that working, but it's got the smell of You Don't Know What You're Doing So Just Keep Adding Code Until it Works. I've at least got a better handle on using FORMAT and think I understand now that a single function can do many things, and that you can choose what to return with the RETURN clause. For reference, the various bits of code:
CREATE TABLE IF NOT EXISTS data.outlier_rule (
id uuid NOT NULL DEFAULT extensions.gen_random_uuid(),
schema_name text NOT NULL DEFAULT NULL,
table_name text NOT NULL DEFAULT NULL,
column_name text NOT NULL DEFAULT NULL,
threshold integer,
set_to integer,
CONSTRAINT outlier_rule_id_pkey
PRIMARY KEY (schema_name,table_name,column_name)
);
For tracking the modifications, I've got a second table named outlier_change:
------------------------------
-- Table
------------------------------
DROP TABLE IF EXISTS data.outlier_change CASCADE;
CREATE TABLE IF NOT EXISTS data.outlier_change (
id uuid NOT NULL DEFAULT NULL,
outlier_rule_id uuid NOT NULL DEFAULT NULL,
value_was integer NOT NULL DEFAULT NULL,
set_to integer NOT NULL DEFAULT NULL,
change_count integer NOT NULL DEFAULT 0,
last_changed_dts timestamptz NOT NULL DEFAULT NOW(),
CONSTRAINT outlier_change_id_pkey
PRIMARY KEY (id,outlier_rule_id)
);
ALTER TABLE data.outlier_change OWNER TO user_change_structure;
------------------------------
-- Trigger Function
------------------------------
CREATE OR REPLACE FUNCTION data.on_outlier_change_upsert()
RETURNS pg_catalog.trigger AS $BODY$
BEGIN
NEW.last_changed_dts := NOW();
NEW.change_count := OLD.change_count + 1;
RETURN NEW; -- important!
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
------------------------------
-- Trigger
------------------------------
CREATE TRIGGER outlier_change_upsert BEFORE INSERT OR UPDATE ON data.outlier_change
FOR EACH ROW
EXECUTE PROCEDURE data.on_outlier_change_upsert();
DROP FUNCTION IF EXISTS data.outlier_fix ();
CREATE OR REPLACE FUNCTION data.outlier_fix ()
RETURNS TABLE (
schema_name text,
table_name text,
column_name text,
id uuid,
value_was integer,
set_to integer,
change_count integer
)
AS $$
DECLARE
rule record;
now_ timestamptz = NOW();
BEGIN
FOR rule IN SELECT * FROM data.outlier_rule LOOP
EXECUTE FORMAT (
'INSERT INTO outlier_change (
outlier_rule_id,
set_to,
id,
value_was)
SELECT %6$L,
%5$s,
%2$I.id,
%2$I.%3$I
FROM %1$I.%2$I
WHERE %3$I > %4$s
ON CONFLICT(id,outlier_rule_id) DO UPDATE SET
value_was = EXCLUDED.value_was,
set_to = EXCLUDED.set_to
RETURNING outlier_rule_id,
id,
value_was,
set_to
change_count;
UPDATE %1$I.%2$I
SET %3$I = %5$s
WHERE %3$I > %4$s;',
rule.schema_name,
rule.table_name,
rule.column_name,
rule.threshold,
rule.set_to,
rule.id);
END LOOP;
RETURN QUERY EXECUTE ('
SELECT outlier_rule.schema_name,
outlier_rule.table_name,
outlier_rule.column_name,
outlier_change.id,
outlier_change.value_was,
outlier_change.set_to,
outlier_change.change_count
FROM outlier_change
JOIN outlier_rule ON (outlier_rule.id = outlier_change.outlier_rule_id)
WHERE last_changed_dts = $1')
USING now_;
END;
$$ LANGUAGE plpgsql;
ALTER FUNCTION data.outlier_fix() OWNER TO user_bender;
You could achieve that with a bit of a hack. You can self join the table in your update query like this:
UPDATE analytic_productivity NEW
SET points = 1000
FROM analytic_productivity OLD
WHERE NEW.points > 1000
and NEW.id = OLD.id
RETURNING NEW.id,
NEW.points,
OLD.points as old_points;
I would like to know How can I insert an image "bytea" into a table of my postgreSql database? I've been searching forums for hours and have seen the same question posted dozens of times, but yet to find a single answer. All I see is how to insert .jpeg's into an old column which isn't what I need.
Here's the database table:
create table category (
"id_category" SERIAL,
"category_name" TEXT,
"category_image" bytea,
constraint id_cat_pkey primary key ("id_category"))without oids;
and when I add a new line, it doesn't work :
insert into category(category_name,category_image) values('tablette', lo_import('D:\image.jpg'));
If the column type is bytea then you can simply use the 'pg_read_binary_file'.
Example: pg_read_binary_file('/path-to-image/')
check postgresql documentation of pg_read_binary_file
insert into category(category_name,category_image) values('tablette', bytea('D:\image.jpg'));
The above solution works if column type is bytea
insert into category(category_name,category_image) values('tablette', lo_import('D:\image.jpg'));
The above solution works if column type is oid i.e., Blob
insert into category(category_name,category_image) values('tablette',decode('HexStringOfImage',hex));
The above decode function take two parameters. First parameter is HexString of Image.The second parameter is hex by default.Decode function coverts the hexString to bytes and store in bytea datatype column in postgres.
None of the above example worked well for me and on top of that I needed to add many images at once.
Full working example (python 3) with explanations:
With get_binary_array we get the value of the image (or file) as a binary array, using its path and file name as parameter (ex: '/home/Pictures/blue.png').
With send_files_to_postgresql we send all the images at once.
I previously created the database with one sequential 'id' that will automatically be incremented (but you can use your own homemade id) and one bytea 'image' field
import psycopg2
def get_binary_array(path):
with open(path, "rb") as image:
f = image.read()
b = bytes(f).hex()
return b
def send_files_to_postgresql(connection, cursor, file_names):
query = "INSERT INTO table(image) VALUES (decode(%s, 'hex'))"
mylist = []
for file_name in file_names:
mylist.append(get_binary_array(file_name))
try:
cursor.executemany(query, mylist)
connection.commit() # commit the changes to the database is advised for big files, see documentation
count = cursor.rowcount # check that the images were all successfully added
print (count, "Records inserted successfully into table")
except (Exception, psycopg2.DatabaseError) as error:
print(error)
def get_connection_cursor_tuple():
connection = None
try:
params = config()
print('Connecting to the PostgreSQL database...')
connection = psycopg2.connect(**params)
cursor = connection.cursor()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
return connection, cursor
connection, cursor = connect_db.get_connection_cursor_tuple()
img_names = ['./blue.png', './landscape.jpg']
send_files_to_postgresql(connection, cursor, img_names)
Something like this function (slightly adapted from here) could work out.
create or replace function img_import(filename text)
returns void
volatile
as $$
declare
content_ bytea;
loid oid;
lfd integer;
lsize integer;
begin
loid := lo_import(filename);
lfd := lo_open(loid,131072);
lsize := lo_lseek(lfd,0,2);
perform lo_lseek(lfd,0,0);
content_ := loread(lfd,lsize);
perform lo_close(lfd);
perform lo_unlink(loid);
insert into category values
('tablette',
content_);
end;
$$ language plpgsql
Use it like select * from img_import('D:\image.jpg');
or rewrite to procedure if feeling like it.
create below function:
create or replace function bytea_import(p_path text, p_result out bytea)
language plpgsql as $$
declare
l_oid oid;
begin
select lo_import(p_path) into l_oid;
select lo_get(l_oid) INTO p_result;
perform lo_unlink(l_oid);
end;$$;
and use like this:
insert into table values(bytea_import('C:\1.png'));
For Linux users this is how to add the path to the image
insert into blog(img) values(bytea('/home/samkb420/Pictures/Sam Pics/sam.png'));
create table images (imgname text, img bytea);
insert into images(imgname,img) values ('MANGO', pg_read_binary_file('path_of_image')::bytea);
Use SQL workbench - Database explorer - insert a row and follow the dialogue...
enter image description here