I have the following issue:
Right now I have a table with the subnet/mask information (example 192.168.1.0 / 255.255.255.0 ) .. but I need to obtain the MAX and MIN IP from this subnet:
192.168.1.0 / 192.168.1.255
I've found this answer:
how to query for min or max inet/cidr with postgres
But it seems that:
network_smaller(inet, inet) and network_larger(inet, inet)
Doesn't exists. Even googling that I can't find any answer for those functions.
Thanks!
Edit:
Version info:
PostgreSQL 9.2.15 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4), 64-bit
I don't think that question is relavent to your needs anyway. The min and max defined there are similar to the SQL min() and max() functions for finding the smallest / largest in a table, not the smallest / largest in a subnet.
I'm not generally a fan of relying on undocumented features. They may be safe but may isn't a word I generally like.
There's a page of documented network functions here:
https://www.postgresql.org/docs/current/static/functions-net.html
The two you would need would be:
Min would be network(inet)
Max would be broadcast(inet)
That's because the network name is always the "first" ip in the range and the broadcast address is always the "last" ip in the range.
Don't google, just try:
select network_smaller('192.168.0.9'::inet, '192.168.0.11'::inet);
network_smaller
-----------------
192.168.0.9
(1 row)
Postgres has more than 2,600 internal functions. Most of them are useful for creating operator classes of various types. Not all of them are described in the documentation, but they are all generally available.
You can find them using pgAdmin III in pg_catalog. You only need to set the option: File -> Options -> UI Miscellaneous -> Show System Objects in treeview.
The aggregate functions min(inet) and max(inet) has been introduced in Postgres 9.5:
with test(ip) as (
values
('192.168.0.123'::inet),
('192.168.0.12'),
('192.168.0.1'),
('192.168.0.125')
)
select max(ip), min(ip)
from test;
max | min
---------------+-------------
192.168.0.125 | 192.168.0.1
(1 row)
See how the aggregate min(inet) is defined (it can be found in pg_catalog):
CREATE AGGREGATE min(inet) (
SFUNC=network_smaller,
STYPE=inet,
SORTOP="<"
);
The question How to query for min or max inet/cidr with postgres concerned Postgres 9.4. In my answer I suggested to use the functions network_smaller(inet, inet) and network_larger(inet, inet). I'm sure they were added for creating aggregate functions min(inet) and max(inet) but for some reasons (maybe oversight) the aggregates appeared only in Postgres 9.5.
In Postgres 9.2 you can create your own functions as substitutes, e.g.
create or replace function inet_larger(inet, inet)
returns inet language sql as $$
select case when network_gt($1, $2) then $1 else $2 end
$$;
create or replace function inet_smaller(inet, inet)
returns inet language sql as $$
select case when network_lt($1, $2) then $1 else $2 end
$$;
Related
We are investigating using PostGIS to perform some spacial filtering of data that has been gathered from a roving GPS engine. We have defined some start and end points that we use in our processing with the following table structure:
CREATE TABLE IF NOT EXISTS tracksegments
(
idtracksegments bigserial NOT NULL,
name text,
approxstartpoint geometry,
approxendpoint geometry,
maxpoints integer
);
If the data in this table is queried:
SELECT ST_AsText(approxstartpoint) FROM tracksegments
we get ...
POINT(-3.4525845 58.5133318)
Note that the Long/Lat points are given to 7 decimal places.
To get just the longitude element, we tried:
SELECT ST_X(approxstartpoint) AS long FROM tracksegments
we get ...
-3.45
We need much more precision than the 2 decimal places that are returned. We've searched the documentation and there does not appear to be a way to set the level of precision. Any help would be appreciated.
Vance
Your problem is definitely client related. Your client is most likely truncating double precision values for some reason. As ST_AsText returns a text value, it does not get affected by this behaviour.
ST_X does not truncate the coordinate's precision like that, e.g.
SELECT ST_X('POINT(-3.4525845 58.5133318)');
st_x
------------
-3.4525845
(1 Zeile)
Tested with psql in PostgreSQL 9.5 + PostGIS 2.2 and PostgreSQL 12.3 + PostGIS 3.0 and with pgAdmin III
Note: PostgreSQL 9.5 is a pretty old release! Besides the fact that it will reach EOL next January, you're missing really kickass features in the newer releases. I sincerely recommend you to plan a system upgrade as soon as possible.
We are in the process of migrating a MySQL 5.7 database to PostgreSQL 9.6.
A real issue is the lack of bit_count function in PostgreSQL. This function is also not available in the upcoming version 10.
Current MySQL code snippet (simplified):
-- mysql specific, tested with 5.7.19
select code,phash,bit_count(phash ^ -9187530158960050433) as hd
from documents
where phash is not null and bit_count(phash ^ -9187530158960050433) < 7
order by hd;
We tried a naive solution (converting the BIGINT to a String and counting the "1"'s), but it performs terribly compared to MySQL.
Java uses a trick from Hacker's Delight, but AFAIK this is not possible with PostgreSQL, because the >>> operator is (also) not available.
Question: Is a there solution/workaround available comparable with MySQL performance wise?
UPDATE 1
Best solution i could find is based on this SO answer:
First create bit_count function:
CREATE OR REPLACE FUNCTION bit_count(value bigint)
RETURNS numeric
AS $$ SELECT SUM((value >> bit) & 1) FROM generate_series(0, 63) bit $$
LANGUAGE SQL IMMUTABLE STRICT;
Now we can use almost the same SQL as with MySQL:
-- postgresql specific, tested with 9.6.5
select code,phash,bit_count(phash # -9187530158960050433) as hd
from documents
where phash is not null and bit_count(phash # -9187530158960050433) < 7
order by hd;
UPDATE 2
Based on #a_horse_with_no_name comment, i tried this function:
-- fastest implementation so far. 10 - 11 x faster than the naive solution (see UPDATE 1)
CREATE OR REPLACE FUNCTION bit_count(value bigint)
RETURNS integer
AS $$ SELECT length(replace(value::bit(64)::text,'0','')); $$
LANGUAGE SQL IMMUTABLE STRICT;
However, this is still 5 - 6 times slower than MySQL (tested wit exact the same data set of 200k phash values on the same hardware).
Function bit_count is available since PostgreSQL version 14, see
Bit String Functions and Operators.
Example:
select bit_count(B'1101');
Result is 3.
Note that the function is defined for types bit and bit varying. So if you want to use it with integer values, you need to cast.
Example:
select cast (cast (1101 as text) as bit varying);
Result is B'1101'.
Combining both examples:
select bit_count(cast (cast (1101 as text) as bit varying));
Question: Is a there solution/workaround available comparable with
MySQL performance wise?
To get a comparable speed, a compiled C function should be used.
If you can compile C code, see for instance
https://github.com/dverite/postgresql-functions/tree/master/hamming_weight
The code itself is very simple.
The result seems 10 times faster than the bit_count function based on counting the 0 characters in the bit(64) string as text.
Example:
plpgsql function:
test=> select sum(bit_count(x)) from generate_series(1,1000000) x;
sum
---------
9884999
(1 row)
Time: 2442,340 ms
C function:
test=> select sum(hamming_weight(x::int8)) from generate_series(1,1000000) x;
sum
---------
9884999
(1 row)
Time: 239,749 ms
If you are trying to compute the hamming distance of perceptual hashes or similar LSH bit strings, then this question may be a closely related to this answer
If you are looking specifically for a pre-built way to do hamming distance queries on a PostgreSQL database, then this may be the cure: an extension for hamming distance search
I'm looking for a DB2 function to calculate hashes on large CLOB values in order to quickly track changes. Other engines have functions such as CHECKSUM,CRC32 or MD5. The function in LUW is GET_HASH_VALUE but is not available in zOS.
Constraints: No access to UDFs or Stored Procedures.
Here is a quick and dirty code fragment that computes a CRC32, it only works to about 100 characters.
WITH crc(t,c,j) AS (
SELECT 'Hello World!',4294967295,0 FROM SYSIBM.SYSDUMMY1
UNION ALL
SELECT SUBSTR(t,2),bitxor(c,ASCII(t)),8 FROM crc WHERE t>'' AND j=0
UNION ALL
SELECT t,BITXOR(c/2,BITAND(3988292384,-BITAND(c,1))),j-1 FROM crc WHERE j>0
)
SELECT RIGHT(HEX(BITNOT(c)),8) FROM CRC WHERE t='' AND j=0
Result checked against http://www.lammertbies.nl/comm/info/crc-calculation.html :
1
--------
1C291CA3
Source: http://www.hackersdelight.org/hdcodetxt/crc.c.txt
The answer depends on which version of DB2 you have. If you are on DB2 9.7 or higher, have a look here: https://www.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.sql.rtn.doc/doc/r0055167.html
In postgresql if I want percentages I just write:
select x / sum(x) over() ...
Inside a function it doesn't work since aggregate functions don't behave well.
I tried to find a solution but with no success.
This is a simple version of what I really need, but I believe the solution to this problem would surely point me in the right direction.
Some more details...
If I create this simple table:
create table ttt(v1 numeric, v2 numeric);
insert into ttt values (2,1),(5,2),(10,4);
If I run:
select v1/sum(v1) over() from ttt; --returns relative frequencies
I get:
select v1/sum(v1) over() from ttt;
?column?
------------------------
0.11764705882352941176
0.29411764705882352941
0.58823529411764705882
(3 rows)
Now, if I want to create a function which does the same thing, I would write:
create or replace function rfreq (double precision)
returns double precision
AS
'
select
$1 / sum($1) over()
'
LANGUAGE 'sql';
I get:
select rfreq(v1) from bruto;
rfreq
-------
1
1
1
(3 rows)
Postgresql is not summing up inside a function.
Any suggestions?
Thank you,
Ali.
To debug your function, write the query with arbitrary parameters in a text file, and then use psql to run it:
\i ./myfunc.sql
Content of myfunc.sql would be:
select x / sum(y) over (...) ...
This will allow you to debug the function before wrapping it in a function.
When you're done and happy with the results for a few samples, copy/paste it into your function, and replace the hard-coded test values with parameters where applicable.
As to optimizing it when it has parameters, I'm not aware of any means to run explain analyze within the Postgres function, but you can get a plan which -- best I'm aware -- is the same as the function will use by preparing a statement with the same parameters. So you can explain analyze the latter instead.
Seeing the new details, note that if you prepare the query that you're running in function, you should always get 1 -- bar with zero.
You've an error in there, in the sense that you'd need to keep state from a call to the next first to return the expected result. Per Pavel's suggestion, you actually need a custom aggregate or a custom window function here. See the link he suggested in a comment, as well as:
http://www.postgresql.org/docs/current/static/xaggr.html
I found the solution browsing through the pl/r mailing list.
Percentages (or relative frequencies) can be calculated in postgres using the following code:
CREATE OR REPLACE
FUNCTION rel_freq(float8)
RETURNS float8 AS
$BODY$
var <- as.vector(farg1)
return((var/sum(var))[prownum]
$BODY$
LANGUAGE plr WINDOW;
I'm currently writing some installer script that fires SQL files against different database types depending on the system's configuration (the webapplication supports multiple database server like MySQL, MSSQL and PostgreSQL).
One of those types is PostgreSQL. I'm not fluent with it and I would like to know if it's possible to make a statement into a define/populate SQL file that makes an SQL query conditional to a specific PostgreSQL server version.
How to make an SQL statement conditionally in plain PGSQL so that it is only executed in version 9? The command is:
ALTER DATABASE dbname SET bytea_output='escape';
The version check is to compare the version with 9.
Postgres does have version() function, however there is no major_vesion(). Assuming that output string always includes version number as number(s).number(s).number(s) you could write your own wrapper as:
CREATE OR REPLACE FUNCTION major_version() RETURNS smallint
AS $BODY$
SELECT substring(version() from $$(\d+)\.\d+\.\d+$$)::smallint;
$BODY$ LANGUAGE SQL;
Example:
=> Select major_version();
major_version
---------------
9
(1 row)
However real issue here is that AFAIK you can't execute your commands conditionally in "pure" SQL and best what you can do is to write some stored function like this:
CREATE OR REPLACE FUNCTION conditionalInvoke() RETURNS void
AS $BODY$
BEGIN
IF major_version() = 9 THEN
ALTER DATABASE postgres SET bytea_output='escape';
END IF;
RETURN;
END;
$BODY$ LANGUAGE plpgsql;
I think that you should rather use some scripting language and generate appropriate SQL with it.
Or you could just use
select setting from pg_settings where name = 'server_version'
Or
select setting from pg_settings where name = 'server_version_num'
If you need major version only
select Substr(setting, 1, 1) from pg_settings where name = 'server_version_num'
or
select Substr(setting, 1, strpos(setting, '.')-1) from pg_settings where name = 'server_version'
if you want it to be compatible with two digit versions.
Maybe you could make things dependent on the output of
select version();
(probably you'll have to trim and substring that a bit)
BTW (some) DDL statements may not be issued from within functions; maybe you'll have to escape to shell-programming and here-documents.