I updated a bunch of fields in my db using this Feistel Cipher. According to the documentation, the cipher can be undone to get the original value. How can I undo the values if need be?
Here is the original cipher function:
CREATE OR REPLACE FUNCTION pseudo_encrypt(VALUE int) returns bigint AS $$
DECLARE
l1 int;
l2 int;
r1 int;
r2 int;
i int:=0;
BEGIN
l1:= (VALUE >> 16) & 65535;
r1:= VALUE & 65535;
WHILE i < 3 LOOP
l2 := r1;
r2 := l1 # ((((1366.0 * r1 + 150889) % 714025) / 714025.0) * 32767)::int;
l1 := l2;
r1 := r2;
i := i + 1;
END LOOP;
RETURN ((l1::bigint << 16) + r1);
END;
$$ LANGUAGE plpgsql strict immutable;
You could use this self-reversible variant in the first place:
CREATE FUNCTION rev_pseudo_encrypt(VALUE bigint) returns bigint AS $$
DECLARE
l1 int;
l2 int;
r1 int;
r2 int;
i int:=0;
BEGIN
l1:= (VALUE >> 16) & 65535;
r1:= VALUE & 65535;
WHILE i < 3 LOOP
l2 := r1;
r2 := l1 # ((((1366.0 * r1 + 150889) % 714025) / 714025.0) * 32767)::int;
l1 := l2;
r1 := r2;
i := i + 1;
END LOOP;
RETURN ((r1::bigint<<16) + l1);
END;
$$ LANGUAGE plpgsql strict immutable;
It differs from the original version by taking bigint instead of int as input (but the input should still less than 2^32), and by the fact that the two 16 bits blocks are swapped in the 32 bits final result.
It has the property that rev_pseudo_encrypt(rev_pseudo_encrypt(x)) = x in addition to the uniquess of results.
Also there's the advantage that the input type is the same as the output type.
On the other hand, to reverse the values generated by the original version, their 16 bits blocks need to be swapped before being fed to the algorithm, and the result again swapped:
create function swap16(bigint) returns bigint as
'select (($1&65535)<<16)+(($1)>>16)'
language sql stable;
select pseudo_encrypt(1234);
pseudo_encrypt
----------------
223549288
select swap16(pseudo_encrypt(swap16(223549288)::int));
swap16
--------
1234
Related
WORKING CODE AT THE END
I'm trying to get along with plpgsql but it's giving me a hard time. I'm trying to make a function on the database that will be called by my server to expand or create my terrain. I can't make it compile, no matter what I try, it blocks on the first usage of one of the two parameters the function has.
I havec tried sevral manners of declaring the parameters (refering to them as ALIAS FOR $1 or declaring them with a name as the following code shows) I also tried to change the parameter type to INTEGER or NUMERIC.
CREATE FUNCTION public.generate_terrain (
inner NUMERIC,
outer NUMERIC
)
RETURNS void AS
$body$
DECLARE
q NUMERIC;
r NUMERIC;
BEGIN
q := -outer;
r := -outer;
WHILE q < outer DO
WHILE r < outer DO
IF(r > -inner AND r < inner AND q > -inner AND q > inner) THEN
r := inner;
END IF;;
--insert into public.t_cell (coo_q, coo_r) values (q,r);
RAISE NOTICE 'Cell %,%', q,r;
r := r + 1;
END WHILE;
q := q + 1;
END WHILE;
END;
$body$
LANGUAGE 'plpgsql'
VOLATILE
CALLED ON NULL INPUT
SECURITY INVOKER
PARALLEL UNSAFE;
I get this error message when I try to compile it :
ERROR: syntax error at end of input
LINE 8: q := -outer;
^
I cannot have my client do the loop and then push the results onto the database it would generate to much traffic I want to have my database do this on it's own thus the need to be able to compile this. Please Help !
EDIT 1
I forgot to explain what i wanted this function to do : I need a function that populates my database with coherent "cells" of a carthesian grid map (q = x axis, r = y axis because in reality it's hexagonal map). This function must be able to be called to expand them map so if my initial call is generate_terrain(0,2) it must produce the followin terrain :
#####
#####
##0##
#####
#####
(0 is the center of the grid (0,0))
where the coordinates range from (-2,-2) as bottom left up to (2,2) on the top right corner. Later, when i need to expand the map I must be able to call generate_terrain(3,4) to generate the following cells of my terrain :
#########
#########
## ##
## ##
## 0 ##
## ##
## ##
#########
#########
(0 is the center of the grid (0,0))
Where the coordinates range from (-4,-4) as bottom left up to (4,4) on the top right corner but the inner "square" is already present in the database
The function I ended up using and that seems to work is the following :
CREATE OR REPLACE FUNCTION public.generate_terrain (
_inner integer,
_outer integer
)
RETURNS integer AS
$body$
DECLARE
q integer = 0;
r integer = 0;
BEGIN
q := q - _outer;
r := r - _outer;
WHILE q <= _outer
LOOP
WHILE r <= _outer
LOOP
-- This condition is to skip the inner square that is already
-- present in the database.
IF r > -_inner
AND r < _inner
AND q > -_inner
AND q < _inner THEN
r := _inner;
END IF;
--insert into public.t_cell (coo_q, coo_r) values (q, r);
RAISE NOTICE 'Cell %,%', q,r;
r := r + 1;
END LOOP;
q := q + 1;
r := - _outer;
END LOOP;
RETURN 1;
END;
$body$
LANGUAGE 'plpgsql'
Besides using reserved words like a_horse pointed out, you have several syntax violations. This would work:
CREATE OR REPLACE FUNCTION public.generate_terrain (_inner NUMERIC, _outer NUMERIC)
RETURNS void AS
$func$
DECLARE
q NUMERIC := -_outer;
r NUMERIC := -_outer;
BEGIN
WHILE q < _outer
LOOP -- !
WHILE r < _outer
LOOP -- !
IF r > -_inner
AND r < _inner
AND q > -_inner
AND q > _inner THEN -- ??
r := _inner;
END IF; -- !
--insert into public.t_cell (coo_q, coo_r) values (q,r);
RAISE NOTICE 'Cell %,%', q,r;
r := r + 1;
END LOOP; -- !
q := q + 1;
END LOOP; -- !
END
$func$ LANGUAGE plpgsql;
But this seems needlessly twisted. _inner is never used at all. Did you by any chance mean to write q < _inner? (Still odd.)
Else you can just use this instead:
CREATE OR REPLACE FUNCTION public.generate_terrain_simple (_outer int)
RETURNS void AS
$func$
INSERT INTO public.t_cell (coo_q, coo_r)
SELECT -_outer, g FROM generate_series (-_outer, _outer -1) g
$func LANGUAGE sql;
Minor optimization for Erwin Last Query.
create or replace function
public.generate_terrain_simple_1(_outer int)
returns void as $$
declare _x int; _y int;
begin
<<test>>
for _x, _y in
select -_outer, g from generate_series(-_outer,_outer - 1) g
loop
raise info 'test % %', _x,_y;
end loop test;
end
$$ language plpgsql;
select * from generate_terrain_simple_1(4);
_x will be the same as -4, _y will from -4 to 3
How can the PBKDF2 function be done in PostgreSQL? There does not appear to be a native implementation.
Moved Answer out of Question to adhere to Stack Overflow guidelines. See original revision of post.
Original post (Revision link)
Not able to find it natively, and based on PHP code found on the 'net, I came up with this PBKDF2 function for PostgreSQL. Enjoy.
create or replace function PBKDF2
(salt bytea, pw text, count integer, desired_length integer, algorithm text)
returns bytea
immutable
language plpgsql
as $$
declare
hash_length integer;
block_count integer;
output bytea;
the_last bytea;
xorsum bytea;
i_as_int32 bytea;
i integer;
j integer;
k integer;
begin
algorithm := lower(algorithm);
case algorithm
when 'md5' then
hash_length := 16;
when 'sha1' then
hash_length = 20;
when 'sha256' then
hash_length = 32;
when 'sha512' then
hash_length = 64;
else
raise exception 'Unknown algorithm "%"', algorithm;
end case;
block_count := ceil(desired_length::real / hash_length::real);
for i in 1 .. block_count loop
i_as_int32 := E'\\000\\000\\000'::bytea || chr(i)::bytea;
i_as_int32 := substring(i_as_int32, length(i_as_int32) - 3);
the_last := salt::bytea || i_as_int32;
xorsum := HMAC(the_last, pw::bytea, algorithm);
the_last := xorsum;
for j in 2 .. count loop
the_last := HMAC(the_last, pw::bytea, algorithm);
--
-- xor the two
--
for k in 1 .. length(xorsum) loop
xorsum := set_byte(xorsum, k - 1, get_byte(xorsum, k - 1) # get_byte(the_last, k - 1));
end loop;
end loop;
if output is null then
output := xorsum;
else
output := output || xorsum;
end if;
end loop;
return substring(output from 1 for desired_length);
end $$;
I've tested against other implementations without deviation, but be sure to test it yourself.
Because I can't test this easily with billions of table insertions, I want to get help on figuring out how to use the pseudo_encrypt (https://wiki.postgresql.org/wiki/Pseudo_encrypt) function for my table ids that already have sequential ids in them. For example, our users table has approx 10,000 users. Ids go from 1..10,000.
Now I want to use the pseudo_encrypt function to get the 10,001 ID which would look something like this: 1064621387932509969
The problem is that there is a chance that the "random" pseudo encrypt return value may collide at one point with my early 1-10,000 user IDs.
I do not want to change the first 10,000 user IDs as that would cause some pain for the current users (have to re-login again, urls broken, etc.).
My idea was to use some sort of recursive function to handle this... would something like this work, or am I mission something?
CREATE OR REPLACE FUNCTION "pseudo_encrypt"("VALUE" int) RETURNS int IMMUTABLE STRICT AS $function_pseudo_encrypt$
DECLARE
l1 int;
l2 int;
r1 int;
r2 int;
return_value int;
i int:=0;
BEGIN
l1:= ("VALUE" >> 16) & 65535;
r1:= "VALUE" & 65535;
WHILE i < 3 LOOP
l2 := r1;
r2 := l1 # ((((1366.0 * r1 + 150889) % 714025) / 714025.0) * 32767)::int;
r1 := l2;
l1 := r2;
i := i + 1;
END LOOP;
return_value = ((l1::int << 16) + r1); // NEW CODE
// NEW CODE - RECURSIVELY LOOP UNTIL VALUE OVER 10,000
WHILE return_value <= 10000
return_value = pseudo_encrypt(nextval('SEQUENCE_NAME'))
END LOOP
RETURN return_value;
END;
$function_pseudo_encrypt$ LANGUAGE plpgsql;
Great comments! This seems to do the job:
CREATE OR REPLACE FUNCTION get_next_user_id() returns int AS $$
DECLARE
return_value int:=0;
BEGIN
WHILE return_value < 10000 LOOP
return_value := pseudo_encrypt(nextval('test_id_seq')::int);
END LOOP;
RETURN return_value ;
END;
$$ LANGUAGE plpgsql strict immutable;
In PostgreSQL, how to generate random unique integer number for column,
return which not exits in table column?
See the pseudo_encrypt function, which implements a permutation based on the Feistel network technique. Combined with a postgres sequence, this guarantees unicity of the result, as well as randomness to the human eye.
Example:
CREATE OR REPLACE FUNCTION pseudo_encrypt(VALUE int) returns int AS $$
DECLARE
l1 int;
l2 int;
r1 int;
r2 int;
i int:=0;
BEGIN
l1:= (VALUE >> 16) & 65535;
r1:= VALUE & 65535;
WHILE i < 3 LOOP
l2 := r1;
r2 := l1 # ((((1366 * r1 + 150889) % 714025) / 714025.0) * 32767)::int;
l1 := l2;
r1 := r2;
i := i + 1;
END LOOP;
RETURN ((r1 << 16) + l1);
END;
$$ LANGUAGE plpgsql strict immutable;
create sequence seq maxvalue 2147483647;
create table tablename(
id int default pseudo_encrypt(nextval('seq')::int),
[other columns]
);
A variant with a 64-bit output space can be found at: pseudo_encrypt() function in plpgsql that takes bigint.
EDIT: pseudo_encrypt implements only one permutation, and it does not accept a user-supplied key. If you prefer having your own permutations, depending on secret keys, you may consider skip32 (a 32-bit block cipher based on Skipjack, with 10 bytes wide keys).
A plpgsql function (ported from Perl/C) is available at:
https://wiki.postgresql.org/wiki/Skip32
I am trying to understand how to work with binary data in postgresql (v 8.3).
Let's say I have a following table
Table "public.message"
Column | Type | Modifiers
---------+---------+-----------
id | integer |
message | bytea |
I would like to store a packet in the message field in this format:
version (1 byte), identifier (1 byte), epoch (4 bytes)
I would like to pack this data into the message field. Lets say I have version=1, identifier=8 and epoch=123456. How would I pack this data into the message field? How would I convert my integer values to hex.. or octal?
I also need to get the message back and parse it. I was looking at the get_byte function, unless there is another way to parse the data out..
Thanks!
Here is some sample code showing how to do it with server-side Perl. Annoyingly, pack/unpack are considered untrusted operations by PG so this has to be created with plperlu by a superuser and then access granted with GRANT EXECUTE to non superusers.
On the other hand, this choice of language makes it easy to deal with more complex packed structures, which is a significant advantage over code that would be based on the SQL get_bytes()/set_bytes() functions. See Perl's pack() features.
1) first step: define a SQL composite type representing an non-packed record.
create type comp as (a smallint, b smallint, c int);
2) make a function to pack the record value into bytea:
create function pack_comp(comp) returns bytea
as $body$
my $arg=shift;
my $retval = pack("CCL", $arg->{a},$arg->{b},$arg->{c});
# encode bytea according to PG doc. For PG>=9.0, use encode_bytea() instead
$retval =~ s!(\\|[^ -~])!sprintf("\\%03o",ord($1))!ge; # from PG doc
return $retval;
$body$ language plperlu;
3) make a function to unpack bytea into the composite type:
create or replace function unpack_comp(bytea) returns comp
as $body$
my $arg=shift;
# decode bytea according to PG doc. For PG>=9.0, use decode_bytea() instead
$arg =~ s!\\(?:\\|(\d{3}))!$1 ? chr(oct($1)) : "\\"!ge;
my ($v,$i,$e)= unpack("CCL", $arg);
return {"a"=>$v, "b"=>$i, "c"=>$e};
$body$ language plperlu;
4) usage:
# select encode(pack_comp((254,14,1000000)::comp), 'hex');
encode
--------------
fe0e40420f00
# select unpack_comp(decode('fe0e40420f00','hex'));
unpack_comp
------------------
(254,14,1000000)
# select * from unpack_comp(decode('fe0e40420f00','hex'));
a | b | c
-----+----+---------
254 | 14 | 1000000
So I was able to figure out how to do it in plpg
Here's the code to pack
CREATE FUNCTION pack_numeric_bytes(i_values NUMERIC[], i_byte_sizes NUMERIC[], i_big_endian BOOLEAN)
RETURNS BYTEA
DECLARE
v_bytes BYTEA := NULL;
v_start INTEGER := 1;
v_byte BYTEA;
v_byte_size INTEGER;
v_value NUMERIC;
v_binary_value TEXT;
v_num NUMERIC;
i INTEGER;
x INTEGER;
v_sql TEXT;
BEGIN
IF array_upper(i_values, 1) != array_upper(i_byte_sizes, 1) THEN
RETURN v_bytes;
END IF;
FOR x IN array_lower(i_values, 1) .. array_upper(i_values, 1) LOOP
/* Given value and size at x position */
v_byte_size := i_byte_sizes[x]::INTEGER;
v_value := i_values[x];
/* Convert number to binary form */
v_sql := $$SELECT $$|| v_value ||$$::bit($$|| v_byte_size*8 ||$$);$$;
EXECUTE v_sql INTO v_binary_value;
IF i_big_endian IS TRUE THEN
/* Convert each byte at a time */
FOR i IN 1 .. v_byte_size LOOP
/* Extract byte from our binary value.
Big endian starts at 1 and then increments of 8 */
v_byte := substring(v_binary_value, v_start, 8);
/* Convert binary 8 bits to an integer */
v_sql := $$SELECT B$$||quote_literal(v_byte)||$$::int8$$;
EXECUTE v_sql INTO v_num;
/* Build bytea of bytes */
v_bytes := COALESCE(v_bytes, '') || set_byte(E' '::BYTEA, 0, v_num::INTEGER);
v_start := v_start + 8;
END LOOP;
ELSE
/* Small endian is extracted starting from last byte */
v_start := (v_byte_size * 8) + 1;
/* Convert each byte at a time */
FOR i IN 1 .. v_byte_size LOOP
v_start := v_start - 8;
v_byte := substring(v_binary_value, v_start, 8);
/* Convert binary 8 bits to an integer */
v_sql := $$SELECT B$$||quote_literal(v_byte)||$$::int8$$;
EXECUTE v_sql INTO v_num;
/* Build bytea of bytes */
v_bytes := COALESCE(v_bytes, '') || set_byte(E' '::BYTEA, 0, v_num::INTEGER);
END LOOP;
END IF; /* END endian check */
v_start := 1;
END LOOP;
RETURN v_bytes;
END;
And here's the code to unpack:
CREATE OR REPLACE FUNCTION public.unpack_numeric_bytes(i_bytes bytea, i_byte_sizes INTEGER[], i_big_endian BOOLEAN)
RETURNS NUMERIC[]
SECURITY DEFINER AS
DECLARE
v_bytes BYTEA;
v_start INTEGER := 1;
v_byte_index INTEGER := 0;
v_bit_shift INTEGER := 0;
v_length INTEGER;
v_size INTEGER;
v_sum_byte_sizes INTEGER;
v_vals NUMERIC[] := '{}';
v_val BIGINT := 0;
i INTEGER;
x INTEGER;
v_sql TEXT;
BEGIN
v_sql := $$SELECT $$|| array_to_string(i_byte_sizes, '+')||$$;$$;
EXECUTE v_sql INTO v_sum_byte_sizes;
IF length(i_bytes) != v_sum_byte_sizes::INTEGER THEN
RETURN v_vals;
END IF;
/* Loop through values of bytea (split by their sizes) */
FOR x IN array_lower(i_byte_sizes, 1) .. array_upper(i_byte_sizes, 1) LOOP
v_size := i_byte_sizes[x];
v_bytes := substring(i_bytes, v_start, v_size);
v_length := length(v_bytes);
IF i_big_endian IS TRUE THEN
v_byte_index := v_length - 1;
FOR i IN 1..v_length LOOP
v_val := v_val + (get_byte(v_bytes, v_byte_index) << v_bit_shift);
v_bit_shift := v_bit_shift + 8;
v_byte_index := v_byte_index - 1;
END LOOP;
ELSE
FOR i IN 1..v_length LOOP
v_val := v_val + (get_byte(v_bytes, v_byte_index) << v_bit_shift);
v_bit_shift := v_bit_shift + 8;
v_byte_index := v_byte_index + 1;
END LOOP;
END IF;
v_vals := array_append(v_vals, v_val::NUMERIC);
/* Calculate next value start index */
v_start := v_start + v_size;
v_byte_index := 0;
v_bit_shift := 0;
v_val := 0;
END LOOP;
RETURN v_vals;
END;
I hope this will help someone.