Background
A service running same tasks against a number of similar PostgreSQL instances. Most environments are on version 10, but some are on 9. Upgrading them is not an option in short term at least.
Problem
To improve performance, we used PostgreSQL 10 feature CREATE STATISTICS. It works just fine on on environments on v10, but is not supported on v9.
One way to deal with it could be to duplicate each script that uses CREATE STATISTICS, maintain a copy of it without those statement and choose which script to run at application level. I'd like to avoid it as it's a lot of duplicated code to maintain.
I've tried to cheat it by only creating the statistics if the script finds appropriate version (code below), but on v9 it still gets picked up as a syntax error.
DO $$
-- server_version_num is a version number melted to an integer:
-- 9.6.6 -> 09.06.06 -> 90606
-- 10.0.1 -> 10.00.01 -> 100001
DECLARE pg_version int := current_setting('server_version_num');
BEGIN
IF pg_version >= 100000 THEN
CREATE STATISTICS table_1_related_col_group_a (NDISTINCT)
ON col_a1, col_a2
FROM schema_1.table_1;
CREATE STATISTICS table_2_related_col_group_b (NDISTINCT)
ON col_b1, col_b2, col_b3
FROM schema_1.table_2;
END IF;
END $$ LANGUAGE plpgsql;
Question
Is there a way to run a script that has an unsupported statement like CREATE STATISTICS in it without tipping postgres 9 off?
Use dynamic SQL. It won't be evaluated unless executed.
DO $$
-- server_version_num is a version number melted to an integer:
-- 9.6.6 -> 09.06.06 -> 90606
-- 10.0.1 -> 10.00.01 -> 100001
DECLARE pg_version int := current_setting('server_version_num');
BEGIN
IF pg_version >= 100000 THEN
EXECUTE 'CREATE STATISTICS table_1_related_col_group_a (NDISTINCT)
ON col_a1, col_a2
FROM schema_1.table_1;
CREATE STATISTICS table_2_related_col_group_b (NDISTINCT)
ON col_b1, col_b2, col_b3
FROM schema_1.table_2;';
END IF;
END $$ LANGUAGE plpgsql;
Related
I have simple function, if I run it it takes around 40 second to finish.
select * from f_cyklus1(100000000)
but if I run this function 8 times in 8 separated instances, meaning all 8 function are running in parallel it takes around 210 to 260 seconds to finish for each of it's instances. Which is a massive drop in performance. I tried to compile it as 8 individual functions and run it again but it had no change in performance.
select * from f_cyklus1(100000000);
select * from f_cyklus2(100000000);
select * from f_cyklus3(100000000);
select * from f_cyklus4(100000000);
select * from f_cyklus5(100000000);
select * from f_cyklus6(100000000);
select * from f_cyklus7(100000000);
select * from f_cyklus8(100000000);
So why it takes 40s compare to 210-260s to finish? Our virtual machine has 16 CPUs and physical hardware was at low usage. I was also the only one using the Postgre database at the time of testing.
create or replace function f_cyklus1 (p_rozsah int) returns bigint as -- drop function f_cyklus(int)
$body$
declare
declare
v_exc_context TEXT;
v_result INTEGER;
p_soucet bigint :=0;
begin
for i in 0..p_rozsah
loop
p_soucet = p_soucet + i;
end loop;
return p_soucet;
EXCEPTION
WHEN OTHERS THEN
GET STACKED DIAGNOSTICS v_exc_context = PG_EXCEPTION_CONTEXT;
PERFORM main.ut_log('ERR', SQLERRM || ' [SQL State: ' || SQLSTATE || '] Context: ' || v_exc_context );
RAISE;
END;
$body$ LANGUAGE plpgsql
PostgreSQL 11.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5
20150623 (Red Hat 4.8.5-39), 64-bit
Virtual machine: Centos 7 + KVM
HW: 2x AMD EPYC 7351 + 256 GB RAM
Note: I already asked similar question where I thought it was due to asynchronous processing, but this shows the problem is actually in raw Postgres performance, therefore I deleted my former question and asked this new one.
p_soucet = p_soucet + i;
Each time you do this, it has to acquire a "snapshot" in which to run the statement, as it uses the regular SQL engine behind the scenes and that always needs to run in snapshot. Acquiring a snapshot requires a system-wide lock. The more processes you have running simultaneously, the more time they spend fighting to acquire the snapshot, rather than doing useful work.
If you run the function in a transaction which is set to "repeatable read", you will find they scale better because they keep the same snapshot for the duration and keep re-using it. Of course that might interfere with your real use case.
plpgsql is not really well suited for this kind of work, scaling aside. You can use one of the other pl languages, like plperl or plpythonu.
How expressions are evaluated by main SQL engine is described at https://www.postgresql.org/docs/current/plpgsql-expressions.html
Snapshots are discussed in general at the docs starting at https://www.postgresql.org/docs/current/mvcc.html
I am not aware that the interaction between the two are documented anywhere for end users.
I am attempting to get a better understanding of the DO command in postgreSQL 9.1
I have following code block,
DO
$do$
BEGIN
IF 1=1 THEN
SELECT 'foo';
ELSE
SELECT 'bar';
END IF;
END
$do$
However it returns the following error:
[42601] ERROR: query has no destination for result data Hint: If you want to discard the results of a SELECT, use PERFORM instead. Where: PL/pgSQL function "inline_code_block" line 4 at SQL statement
PostgreSQL DO command creates and executes some specific short life function. This function has not any interface, and then it cannot to return any output other then changes data in tables and some debug output.
Your example has more than one issues:
Only PostgreSQL table functions can returns some tabular data. But the mechanism is significantly different than MSSQL. PostgreSQL user's should to use RETURN NEXT or RETURN QUERY commands.
CREATE OR REPLACE FUNCTION foo(a int)
RETURNS TABLE(b int, c int) AS $$
BEGIN
RETURN QUERY SELECT i, i + 1 FROM generate_series(1,a) g(i);
END;
$$ LANGUAGE plpgsql;
SELECT * FROM foo(10);
DO anonymous functions are not table functions - so no output is allowed.
PostgreSQL 9.1 is not supported version, please upgrade
If you have some experience only from MSSQL, then forget it. A concept of stored procedures of PostgreSQL is very similar to Oracle or DB2, and it is significantly different to MSSQL does. T-SQL integrates procedural and SQL constructs to one set. Oracle, PostgreSQL, ... procedural functionality can embedded SQL, but procedural functionality is not integrated to SQL.
Please, read PostgreSQL PLpgSQL documentation for better imagine about this technology.
I'm trying to create a trigger and am getting the error "[Error] PLS-00357: PLS-00357: Table, View Or Sequence reference 'table_data_seq.nextval' not allowed in this context"
I have read a lot of information on the error and cannot find the difference between the PL/SQL that people say works and mine. Below is my code for creating the trigger ( keeping it as basic as possible to get it working ):
create or replace trigger tr_tabData
before insert on table_data
for each row
DECLARE
seq_value int;
begin
select table_data_sq.nextval into seq_value from dual;
end;
Oracle version is 10.2.0.5
As requested here it the script for the sequence:
DROP SEQUENCE DATA_ADMIN.TABLE_DATA_SQ;
CREATE SEQUENCE DATA_ADMIN.TABLE_DATA_SQ
START WITH 1000
MAXVALUE 999999999999999999999999999
MINVALUE 1
NOCYCLE
CACHE 20
NOORDER;
This is not possible before 11g. You can use sequence_name.NEXTVAL in regular assignments from 11g not before that, and that by the following:
select TABLE_DATA_SQ.NEXTVAL into :NEW.YourID from dual;
It turned out that this was a bug with the TOAD version and my Oracle database version. The same code in SQL*Plus and SQL Developer worked as expected.
This is a small thing, but it's a bit annoying to me and seems like there is probably a way to configure it. Let's say I have the following:
CREATE OR REPLACE FUNCTION baz()
RETURNS void AS
$BODY$
DECLARE
BEGIN
RAISE NOTICE 'I also did some work!';
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
CREATE OR REPLACE FUNCTION bar()
RETURNS void AS
$BODY$
DECLARE
BEGIN
RAISE NOTICE 'I did a bunch of work and want you to know about it';
PERFORM baz();
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
CREATE OR REPLACE FUNCTION foo()
RETURNS void AS
$BODY$
DECLARE
BEGIN
PERFORM bar();
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
select foo();
I get the following output messages:
NOTICE: I did a bunch of work and want you to know about it
CONTEXT: SQL statement "SELECT bar()"
PL/pgSQL function "foo" line 4 at PERFORM
NOTICE: I also did some work!
CONTEXT: SQL statement "SELECT baz()"
PL/pgSQL function "bar" line 5 at PERFORM
SQL statement "SELECT bar()"
PL/pgSQL function "foo" line 4 at PERFORM
Total query runtime: 31 ms.
1 row retrieved.
What I want (usually) is to just see something like:
NOTICE: I did a bunch of work and want you to know about it
NOTICE: I also did some work!
Total query runtime: 31 ms.
1 row retrieved.
Is there a way to control/alter this? Again, it's a small thing and hardly worth a question on Stackoverflow, but if you have a lot of stuff going on it starts to introduce a lot of "noise" into the output and it makes my already overloaded brain hurt trying to sift through it. :)
I'm using PostgreSQL 9.1.5 with pgAdminIII 1.16.0
Try connecting with -q option. The q stands for Quiet and may be what you need.
psql -q -d foo_db
you may also try:
\set verbosity terse
\set quiet on
If you want more control over messaging and interface, it might be worth moving to a real scripting language. Writing database scripts in tools like Python with psycopg2 or Perl with DBD::Pg and DBI is pretty simple.
Not only does using a real scripting language give you total control over messaging, but it also gives you control over error handling, gives you loops and other control structures, and generally offers a much nicer model than raw SQL for scripting tasks.
I'm currently writing some installer script that fires SQL files against different database types depending on the system's configuration (the webapplication supports multiple database server like MySQL, MSSQL and PostgreSQL).
One of those types is PostgreSQL. I'm not fluent with it and I would like to know if it's possible to make a statement into a define/populate SQL file that makes an SQL query conditional to a specific PostgreSQL server version.
How to make an SQL statement conditionally in plain PGSQL so that it is only executed in version 9? The command is:
ALTER DATABASE dbname SET bytea_output='escape';
The version check is to compare the version with 9.
Postgres does have version() function, however there is no major_vesion(). Assuming that output string always includes version number as number(s).number(s).number(s) you could write your own wrapper as:
CREATE OR REPLACE FUNCTION major_version() RETURNS smallint
AS $BODY$
SELECT substring(version() from $$(\d+)\.\d+\.\d+$$)::smallint;
$BODY$ LANGUAGE SQL;
Example:
=> Select major_version();
major_version
---------------
9
(1 row)
However real issue here is that AFAIK you can't execute your commands conditionally in "pure" SQL and best what you can do is to write some stored function like this:
CREATE OR REPLACE FUNCTION conditionalInvoke() RETURNS void
AS $BODY$
BEGIN
IF major_version() = 9 THEN
ALTER DATABASE postgres SET bytea_output='escape';
END IF;
RETURN;
END;
$BODY$ LANGUAGE plpgsql;
I think that you should rather use some scripting language and generate appropriate SQL with it.
Or you could just use
select setting from pg_settings where name = 'server_version'
Or
select setting from pg_settings where name = 'server_version_num'
If you need major version only
select Substr(setting, 1, 1) from pg_settings where name = 'server_version_num'
or
select Substr(setting, 1, strpos(setting, '.')-1) from pg_settings where name = 'server_version'
if you want it to be compatible with two digit versions.
Maybe you could make things dependent on the output of
select version();
(probably you'll have to trim and substring that a bit)
BTW (some) DDL statements may not be issued from within functions; maybe you'll have to escape to shell-programming and here-documents.