Related
My algorithm was working fine but with a new big database, my integers variable exceed the max limit size. (I use the powerset algorithm : https://www.postgresql.org/message-id/20060924054759.GA71934%40winnie.fuhr.org)
So I've decided to change all of my integer to bigint, but now I have a problem on the comparison operator... I don't know how to manage it :
CREATE OR REPLACE FUNCTION powerset(a anyarray)
RETURNS SETOF anyarray AS
$BODY$
DECLARE
retval a%TYPE;
alower bigint := array_lower(a, 1);
aupper bigint := array_upper(a, 1);
j bigint;
k bigint;
BEGIN
FOR i IN 1 .. COALESCE((CAST(1 AS BIGINT) << (aupper - alower + 1)) - 1, 0) LOOP
retval := '{}';
j := alower;
k := i;
WHILE k > CAST(0 AS BIGINT) LOOP
IF k & CAST(1 AS BIGINT) = CAST(1 AS BIGINT) THEN
retval := array_append(retval, a[j]);
END IF;
j := j + CAST(1 AS BIGINT);
k := k >> CAST(1 AS BIGINT);
END LOOP;
RETURN NEXT retval;
END LOOP;
RETURN;
END;
$BODY$
LANGUAGE plpgsql IMMUTABLE STRICT
COST 100
ROWS 1000;
ALTER FUNCTION powerset(anyarray)
OWNER TO postgres;
I've got the error on line :
FOR i IN 1 .. COALESCE((CAST(1 AS BIGINT) << (aupper - alower + 1)) - 1, 0) LOOP
Error 42883
Postgresql ERROR: operator does not exist: bigint << bigint
The type of the right operand of bitwise shift operators is integer. Unfortunately, this is not mentioned in the documentation.
You should cast the right operand of shift operators to integer:
-- instead of
-- COALESCE((CAST(1 AS BIGINT) << (aupper - alower + 1)) - 1, 0)
-- use
select COALESCE(1 << (aupper - alower + 1)::int- 1, 0)::bigint
-- instead of
-- k := k >> CAST(1 AS BIGINT);
--- use
k := k >> 1;
-- etc
You can check possible types of operands by querying the system catalog pg_operator, e.g.:
select oprname, oprleft::regtype, oprright::regtype
from pg_operator
where oprname = '<<'
and oprcode::text like '%shl%' -- shift left functions
oprname | oprleft | oprright
---------+----------+----------
<< | smallint | integer
<< | integer | integer
<< | bigint | integer
(3 rows)
The above result shows that left operand of the operator << (bitwise shift left) can be smallint, integer or bigint and right operand must be integer.
I'm trying to extract the timestamp from a Version 1 UUID, naively wished this worked:
SELECT '3efe0a20-f1b3-11e3-bb44-14109fec739e'::uuid::timestamp;
Here is a quick example showing how to extract the time in Go, but I'm hoping PostgreSQL has something built-in rather than creating a one-off PL/pgSql function:
http://play.golang.org/p/XRCooLgfaG
I've tested this with uuid's from my database and it seems to work very well, even without the unsigned bigints
CREATE FUNCTION uuid_timestamp(id uuid) RETURNS timestamptz AS $$
select TIMESTAMP WITH TIME ZONE 'epoch' +
(((('x' || lpad(split_part(id::text, '-', 1), 16, '0'))::bit(64)::bigint) +
(('x' || lpad(split_part(id::text, '-', 2), 16, '0'))::bit(64)::bigint << 32) +
((('x' || lpad(split_part(id::text, '-', 3), 16, '0'))::bit(64)::bigint&4095) << 48) - 122192928000000000) / 10000000 ) * INTERVAL '1 second';
$$ LANGUAGE SQL
IMMUTABLE
RETURNS NULL ON NULL INPUT;
a V1 uuid I created in the 2099 future!
select uuid_timestamp('6d248400-65b7-1243-a57a-14109fec739e');
uuid_timestamp
------------------------
2099-08-01 11:30:00-07
(1 row)
An alternative to #Krut's implementation that is quite a bit faster based in our testing:
CREATE OR REPLACE FUNCTION uuid_timestamp(uuid UUID) RETURNS TIMESTAMPTZ AS $$
DECLARE
bytes bytea;
BEGIN
bytes := uuid_send(uuid);
RETURN to_timestamp(
(
(
(get_byte(bytes, 0)::bigint << 24) |
(get_byte(bytes, 1)::bigint << 16) |
(get_byte(bytes, 2)::bigint << 8) |
(get_byte(bytes, 3)::bigint << 0)
) + (
((get_byte(bytes, 4)::bigint << 8 |
get_byte(bytes, 5)::bigint)) << 32
) + (
(((get_byte(bytes, 6)::bigint & 15) << 8 | get_byte(bytes, 7)::bigint) & 4095) << 48
) - 122192928000000000
) / 10000 / 1000::double precision
);
END
$$ LANGUAGE plpgsql
IMMUTABLE PARALLEL SAFE
RETURNS NULL ON NULL INPUT;
Note, it will only do millisecond precision so you may want to tweak the function's "/ 10000 / 1000::double precision" bit to just "/ 10000000::double precision" instead.
Example:
=> select uuid_timestamp(uuid_generate_v1()), now();
uuid_timestamp | now
----------------------------+-------------------------------
2020-04-29 17:40:54.519+00 | 2020-04-29 17:40:54.518204+00
(1 row)
Also, it is assuming the input is a v1. If you attempt to give it something like a v4, expect to get weird answers or alter the function to RAISE if it isn't v1.
=> select uuid_timestamp(uuid_generate_v4());
uuid_timestamp
----------------------------
4251-12-19 17:38:34.866+00
(1 row)
Here is a rough pl/pgsql implementation which cast (timestamp, clock_seq, macaddr) into a version 1 uuid.
-- Build UUIDv1 via RFC 4122.
-- clock_seq is a random 14bit unsigned int with range [0,16384)
CREATE OR REPLACE FUNCTION form_uuid_v1(ts TIMESTAMPTZ, clock_seq INTEGER, mac MACADDR)
RETURNS UUID AS $$
DECLARE
t BIT(60) := (extract(EPOCH FROM ts) * 10000000 + 122192928000000000) :: BIGINT :: BIT(60);
uuid_hi BIT(64) := substring(t FROM 29 FOR 32) || substring(t FROM 13 FOR 16) || b'0001' ||
substring(t FROM 1 FOR 12);
BEGIN
RETURN lpad(to_hex(uuid_hi :: BIGINT) :: TEXT, 16, '0') ||
(to_hex((b'10' || clock_seq :: BIT(14)) :: BIT(16) :: INTEGER)) :: TEXT ||
replace(mac :: TEXT, ':', '');
END
$$ LANGUAGE plpgsql;
-- Usage
select form_uuid_v1(now(), 666, '44:88:AA:DD:BB:88');
Is there a query that would be able to accomplish this?
For example given an entry '216.55.82.34' ..I would want to split the string by the '.'s, and apply the equation:
IP Number = 16777216*w + 65536*x + 256*y + z
where IP Address = w.x.y.z
Would this be possible from just a Query?
You can simply convert inet data-type to bigint: (inet_column - '0.0.0.0'::inet)
For example:
SELECT ('127.0.0.1'::inet - '0.0.0.0'::inet) as ip_integer
will output 2130706433, which is the integer representation of IP address 127.0.0.1
You can use split_part(). For example:
CREATE FUNCTION ip2int(text) RETURNS bigint AS $$
SELECT split_part($1,'.',1)::bigint*16777216 + split_part($1,'.',2)::bigint*65536 +
split_part($1,'.',3)::bigint*256 + split_part($1,'.',4)::bigint;
$$ LANGUAGE SQL IMMUTABLE RETURNS NULL ON NULL INPUT;
SELECT ip2int('200.233.1.2');
>> 3370713346
Or, if don't want to define a function, simply :
SELECT split_part(ip,'.',1)::bigint*16777216 + split_part(ip,'.',2)::bigint*65536 +
split_part(ip,'.',3)::bigint*256 + split_part(ip,'.',4)::bigint;
The drawback of the later is that, if the value is given by some computation instead of being just a table field, it can be inefficient to compute, or ugly to write.
PG 9.4
create or replace function ip2numeric(ip varchar) returns numeric AS
$$
DECLARE
ip_numeric numeric;
BEGIN
EXECUTE format('SELECT inet %L - %L', ip, '0.0.0.0') into ip_numeric;
return ip_numeric;
END;
$$ LANGUAGE plpgsql;
Usage
select ip2numeric('192.168.1.2');
$ 3232235778
create function dbo.fn_ipv4_to_int( p_ip text)
returns int
as $func$
select cast(case when cast( split_part(p_ip, '.', 1 ) as int ) >= 128
then
(
( 256 - cast(split_part(p_ip, '.', 1 ) as int ))
*
-power ( 2, 24 )
)
+ (cast( split_part(p_ip, '.', 2 ) as int ) * 65536 )
+ (cast( split_part(p_ip, '.', 3 ) as int ) * 256 )
+ (cast( split_part(p_ip, '.', 4 ) as int ) )
else (cast(split_part(p_ip, '.', 1 ) as int) * 16777216)
+ (cast(split_part(p_ip, '.', 2 ) as int) * 65536)
+ (cast(split_part(p_ip, '.', 3 ) as int) * 256)
+ (cast(split_part(p_ip, '.', 4 ) as int))
end as int )
$func$ LANGUAGE SQL IMMUTABLE RETURNS NULL ON NULL INPUT;
in case you need to get a 32 bit int. it'll return negative numbers for ips over 128.0.0.0. I'd use bigint if you can, but i had a case when i had the numbers stored as 32 bit numbers from another database.
Consider changing the column data type to inet, maybe is more efficient.
ALTER TABLE iptable ALTER COLUMN ip_from TYPE inet
USING '0.0.0.0'::inet + ip_from::bigint;
create index on iptable using gist (ip_from inet_ops);
Then to query
SELECT ip_from
FROM iptable
WHERE ip_from = '177.99.194.234'::inet
I have 7 integer values (with 3,1,3,4,4,5,4 digits respectively) and I have to concatenate them to a single integer (i.e a 24 digit no.) . I tried to do it like this
create or replace function gen_id(int,int,int,int,int,int,int) returns bigint as $$
declare
id bigint;
begin
id = $1 * 1000000000000000000000 + $2 * 100000000000000000000 + $3 * 100000000000000000 + $4 * 10000000000000 + $5 * 1000000000 + $6 * 10000 + $7;
return id;
end;
$$ language plpgsql;
select * from gen_id(100,1,101,1000,1001,10001,1000);
But when I execute it I get error: bigint out of range . Is there any other better way to do it ?
thanks
What about:
SELECT CAST(CAST(num1 AS text)||CAST(num2 AS text)||... AS numeric(24,0))
If you happen to have your IDs in some table, then you can do:
SELECT CAST(string_agg(CAST(num AS text), '') AS numeric(24,0)) FROM srctab;
As I can concatenate a string to integer
SELECT REPLACE(STR(ISNULL(MAX(usuarioid) + 1, 1), 6), ' ', '0') FROM usuarios
usuarioid is string + 1
I am trying to convert hex to decimal using PostgreSQL 9.1
with this query:
SELECT to_number('DEADBEEF', 'FMXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX');
I get the following error:
ERROR: invalid input syntax for type numeric: " "
What am I doing wrong?
Ways without dynamic SQL
There is no cast from hex numbers in text representation to a numeric type, but we can use bit(n) as waypoint. There are undocumented casts from bit strings (bit(n)) to integer types (int2, int4, int8) - the internal representation is binary compatible. Quoting Tom Lane:
This is relying on some undocumented behavior of the bit-type input
converter, but I see no reason to expect that would break. A possibly
bigger issue is that it requires PG >= 8.3 since there wasn't a text
to bit cast before that.
integer for max. 8 hex digits
Up to 8 hex digits can be converted to bit(32) and then coerced to integer (standard 4-byte integer):
SELECT ('x' || lpad(hex, 8, '0'))::bit(32)::int AS int_val
FROM (
VALUES
('1'::text)
, ('f')
, ('100')
, ('7fffffff')
, ('80000000') -- overflow into negative number
, ('deadbeef')
, ('ffffffff')
, ('ffffffff123') -- too long
) AS t(hex);
int_val
------------
1
15
256
2147483647
-2147483648
-559038737
-1
Postgres uses a signed integer type, so hex numbers above '7fffffff' overflow into negative integer numbers. This is still a valid, unique representation but the meaning is different. If that matters, switch to bigint; see below.
For more than 8 hex digits the least significant characters (excess to the right) get truncated.
4 bits in a bit string encode 1 hex digit. Hex numbers of known length can be cast to the respective bit(n) directly. Alternatively, pad hex numbers of unknown length with leading zeros (0) as demonstrated and cast to bit(32). Example with 7 hex digits and int or 8 digits and bigint:
SELECT ('x'|| 'deafbee')::bit(28)::int
, ('x'|| 'deadbeef')::bit(32)::bigint;
int4 | int8
-----------+------------
233503726 | 3735928559
bigint for max. 16 hex digits
Up to 16 hex digits can be converted to bit(64) and then coerced to bigint (int8, 8-byte integer) - overflowing into negative numbers in the upper half again:
SELECT ('x' || lpad(hex, 16, '0'))::bit(64)::bigint AS int8_val
FROM (
VALUES
('ff'::text)
, ('7fffffff')
, ('80000000')
, ('deadbeef')
, ('7fffffffffffffff')
, ('8000000000000000') -- overflow into negative number
, ('ffffffffffffffff')
, ('ffffffffffffffff123') -- too long
) t(hex);
int8_val
---------------------
255
2147483647
2147483648
3735928559
9223372036854775807
-9223372036854775808
-1
-1
uuid for max. 32 hex digits
The Postgres uuid data type is not a numeric type. But it's the most efficient type in standard Postgres to store up to 32 hex digits, only occupying 16 bytes of storage. There is a direct cast from text to uuid (no need for bit(n) as waypoint), but exactly 32 hex digits are required.
SELECT lpad(hex, 32, '0')::uuid AS uuid_val
FROM (
VALUES ('ff'::text)
, ('deadbeef')
, ('ffffffffffffffff')
, ('ffffffffffffffffffffffffffffffff')
, ('ffffffffffffffffffffffffffffffff123') -- too long
) t(hex);
uuid_val
--------------------------------------
00000000-0000-0000-0000-0000000000ff
00000000-0000-0000-0000-0000deadbeef
00000000-0000-0000-ffff-ffffffffffff
ffffffff-ffff-ffff-ffff-ffffffffffff
ffffffff-ffff-ffff-ffff-ffffffffffff
As you can see, standard output is a string of hex digits with typical separators for UUID.
md5 hash
This is particularly useful to store md5 hashes:
SELECT md5('Store hash for long string, maybe for index?')::uuid AS md5_hash;
md5_hash
--------------------------------------
02e10e94-e895-616e-8e23-bb7f8025da42
See:
What is the optimal data type for an MD5 field?
You have two immediate problems:
to_number doesn't understand hexadecimal.
X doesn't have any meaning in a to_number format string and anything without a meaning apparently means "skip a character".
I don't have an authoritative justification for (2), just empirical evidence:
=> SELECT to_number('123', 'X999');
to_number
-----------
23
(1 row)
=> SELECT to_number('123', 'XX999');
to_number
-----------
3
The documentation mentions how double quoted patterns are supposed to behave:
In to_date, to_number, and to_timestamp, double-quoted strings skip the number of input characters contained in the string, e.g. "XX" skips two input characters.
but the behavior of non-quoted characters that are not formatting characters appears to be unspecified.
In any case, to_number isn't the right tool for converting hex to numbers, you want to say something like this:
select x'deadbeef'::int;
so perhaps this function will work better for you:
CREATE OR REPLACE FUNCTION hex_to_int(hexval varchar) RETURNS integer AS $$
DECLARE
result int;
BEGIN
EXECUTE 'SELECT x' || quote_literal(hexval) || '::int' INTO result;
RETURN result;
END;
$$ LANGUAGE plpgsql IMMUTABLE STRICT;
Then:
=> select hex_to_int('DEADBEEF');
hex_to_int
------------
-559038737 **
(1 row)
** To avoid negative numbers like this from integer overflow error, use bigint instead of int to accommodate larger hex numbers (like IP addresses).
pg-bignum
Internally, pg-bignum uses the SSL library for big numbers. This method has none of the drawbacks mentioned in the other answers with numeric. Nor is it slowed down by plpgsql. It's fast and it works with a number of any size. Test case taken from Erwin's answer for comparison,
CREATE EXTENSION bignum;
SELECT hex, bn_in_hex(hex::cstring)
FROM (
VALUES ('ff'::text)
, ('7fffffff')
, ('80000000')
, ('deadbeef')
, ('7fffffffffffffff')
, ('8000000000000000')
, ('ffffffffffffffff')
, ('ffffffffffffffff123')
) t(hex);
hex | bn_in_hex
---------------------+-------------------------
ff | 255
7fffffff | 2147483647
80000000 | 2147483648
deadbeef | 3735928559
7fffffffffffffff | 9223372036854775807
8000000000000000 | 9223372036854775808
ffffffffffffffff | 18446744073709551615
ffffffffffffffff123 | 75557863725914323415331
(8 rows)
You can get the type to numeric using bn_in_hex('deadbeef')::text::numeric.
If anybody else is stuck with PG8.2, here is another way to do it.
bigint version:
create or replace function hex_to_bigint(hexval text) returns bigint as $$
select
(get_byte(x,0)::int8<<(7*8)) |
(get_byte(x,1)::int8<<(6*8)) |
(get_byte(x,2)::int8<<(5*8)) |
(get_byte(x,3)::int8<<(4*8)) |
(get_byte(x,4)::int8<<(3*8)) |
(get_byte(x,5)::int8<<(2*8)) |
(get_byte(x,6)::int8<<(1*8)) |
(get_byte(x,7)::int8)
from (
select decode(lpad($1, 16, '0'), 'hex') as x
) as a;
$$
language sql strict immutable;
int version:
create or replace function hex_to_int(hexval text) returns int as $$
select
(get_byte(x,0)::int<<(3*8)) |
(get_byte(x,1)::int<<(2*8)) |
(get_byte(x,2)::int<<(1*8)) |
(get_byte(x,3)::int)
from (
select decode(lpad($1, 8, '0'), 'hex') as x
) as a;
$$
language sql strict immutable;
Here is a version which uses numeric, so it can handle arbitrarily large hex strings:
create function hex_to_decimal(hex_string text)
returns text
language plpgsql immutable as $pgsql$
declare
bits bit varying;
result numeric := 0;
exponent numeric := 0;
chunk_size integer := 31;
start integer;
begin
execute 'SELECT x' || quote_literal(hex_string) INTO bits;
while length(bits) > 0 loop
start := greatest(1, length(bits) - chunk_size);
result := result + (substring(bits from start for chunk_size)::bigint)::numeric * pow(2::numeric, exponent);
exponent := exponent + chunk_size;
bits := substring(bits from 1 for greatest(0, length(bits) - chunk_size));
end loop;
return trunc(result, 0);
end
$pgsql$;
For example:
=# select hex_to_decimal('ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff');
32592575621351777380295131014550050576823494298654980010178247189670100796213387298934358015
Here is a poper way to convert hex to string... then you can check whether it's a numric type or not
SELECT convert_from('\x7468697320697320612076657279206C6F6E672068657820737472696E67','utf8')
returns
this is a very long hex string
Here is a other version which uses numeric, so it can handle arbitrarily large hex strings:
create OR REPLACE function hex_to_decimal2(hex_string text)
returns text
language plpgsql immutable as $pgsql$
declare
bits bit varying;
result numeric := 0;
begin
execute 'SELECT x' || quote_literal(hex_string) INTO bits;
while length(bits) > 0 loop
result := result + (substring(bits from 1 for 1)::bigint)::numeric * pow(2::numeric, length(bits) - 1);
bits := substring(bits from 2 for length(bits) - 1);
end loop;
return trunc(result, 0);
end
$pgsql$;
For example:
=# select hex_to_decimal('ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff');
32592575621351777380295131014550050576823494298654980010178247189670100796213387298934358015
For example:
=# select hex_to_decimal('5f68e8131ecf80000');
110000000000000000000
Here is another implementation:
CREATE OR REPLACE FUNCTION hex_to_decimal3(hex_string text)
RETURNS numeric
LANGUAGE plpgsql
IMMUTABLE
AS $function$
declare
hex_string_lower text := lower(hex_string);
i int;
digit int;
s numeric := 0;
begin
for i in 1 .. length(hex_string) loop
digit := position(substr(hex_string_lower, i, 1) in '0123456789abcdef') - 1;
if digit < 0 then
raise '"%" is not a valid hexadecimal digit', substr(hex_string_lower, i, 1) using errcode = '22P02';
end if;
s := s * 16 + digit;
end loop;
return s;
end
$function$;
It is a straightforward one that works digit by digit, using the position() function to compute the numeric value of each character in the input string. Its benefit over hex_to_decimal2() is that it seems to be much faster (4x or so for md5()-generated hex strings).
CREATE OR REPLACE FUNCTION numeric_from_bytes(bytea)
RETURNS numeric
LANGUAGE plpgsql
AS $$
declare
bits bit varying;
result numeric := 0;
exponent numeric := 0;
bit_pos integer;
begin
execute 'SELECT x' || quote_literal(substr($1::text,3)) into bits;
bit_pos := length(bits) + 1;
exponent := 0;
while bit_pos >= 56 loop
bit_pos := bit_pos - 56;
result := result + substring(bits from bit_pos for 56)::bigint::numeric * pow(2::numeric, exponent);
exponent := exponent + 56;
end loop;
while bit_pos >= 8 loop
bit_pos := bit_pos - 8;
result := result + substring(bits from bit_pos for 8)::bigint::numeric * pow(2::numeric, exponent);
exponent := exponent + 8;
end loop;
return trunc(result);
end;
$$;
In a future PostgreSQL version, when/if Dean Rasheed's patch 0001-Add-non-decimal-integer-support-to-type-numeric.patch gets committed, this can be simplified:
CREATE OR REPLACE FUNCTION numeric_from_bytes(bytea)
RETURNS numeric
LANGUAGE sql
AS $$
SELECT ('0'||right($1::text,-1))::numeric
$$;