I have a function that creates a temporary table to store and process data. Problem is I want to run this function on the order of 1M times within a single transaction, without having:
NOTICE: relation "foo" already exists, skipping
output ~1M times. Is there an efficient way to do so?
What is not efficient:
Dropping the table instead
DROP TABLE IF EXISTS
Leads to running out of shared memory
Catching the duplicate_table exception (less efficient than using IF NOT EXISTS?)
BEGIN
CREATE TEMPORARY TABLE foo () ON COMMIT DROP;
EXCEPTION
WHEN duplicate_table THEN --do nothing
END;
As others have pointed out, the client_min_messages setting is what you want. There are a number of ways to configure this.
SET client_min_messages = warning or SELECT set_config('client_min_messages', 'warning', false) will persist for the rest of the current session/connection.
SET LOCAL client_min_messages = warning or SELECT set_config('client_min_messages', 'warning', true) resets at the end of the current transaction.
The CREATE FUNCTION statement's SET clause will scope the setting only to this function; this sounds like the most suitable option in your case. For example:
CREATE FUNCTION f()
RETURNS void
SET client_min_messages = warning
LANGUAGE plpgsql
AS ...
Simply
SET client_min_messages = error;
before running the statement(s).
You can also set it on the psql command line for batch executions
PGOPTIONS="-c client_min_messages=error" psql -f somefile.sql dbname
Related
Trigger Function below would trigger if there's a new record inserted into auth users table in supabase
declare
api_url text;
begin
SET search_path = extensions;
api_url := concat('https://cccccxzxxt?nme=',new.id);
SELECT * FROM http_get(api_url);
SET search_path = none;
return new;
end;
The http_get does get triggered all ok on this but I'm getting {"code":500,"msg":"Database error saving new user","error_id":"xxx"} when trying to create a new user and there's no user created in the auth users table.
Most likely, the problem is that you are setting the search_path to none at the end of your trigger. This means that any SQL statements that run AFTER your trigger will fail to execute UNLESS they use ONLY fully-qualified names (e.g. auth.users vs users).
Indeed, when I reproduce your trigger, I see the following error in the PostgreSQL logs:
{
"query": "UPDATE \"users\" SET \"role\" = $1, \"updated_at\" = $2 WHERE users.id = $3",
"event_message": relation "users" does not exist
}
This is most likely the update that Supabase's GoTrue server executes right after inserting the new user (i.e. right after your trigger). See: https://github.com/supabase/gotrue/blob/bfaa68ec2412abb44b76838dcfb817e68eb49aed/api/signup.go#L311).
To solve the issue, you can either:
Avoid manipulating the search_path altogether and instead use the fully qualified name of function you are calling, i.e. extensions.http_get
Use PostgreSQL's special syntax for changing the values of configuration parameters (e.g. search_path) ONLY for the duration of a function and then reverting them back. See: https://www.postgresql.org/docs/current/sql-createfunction.html, SET configuration_parameter { TO value | = value | FROM CURRENT })
How to prevent or avoid running update or delete statements without where clauses in PostgreSQL?
Same as SQL_SAFE_UPDATES statement in MySQL is needed for PostgreSQL.
For example:
UPDATE table_name SET active=1; -- Prevent this statement or throw error message.
UPDATE table_name SET active=1 WHERE id=1; -- This is allowed
My company database has many users with insert and update privilege any one of the users do that unsafe update.
In this secoario how to handle this.
Any idea can write trigger or any extension to handle the unsafe update in PostgreSQL.
I have switched off autocommits to avoid these errors. So I always have a transaction that I can roll back. All you have to do is modify .psqlrc:
\set AUTOCOMMIT off
\echo AUTOCOMMIT = :AUTOCOMMIT
\set PROMPT1 '%[%033[32m%]%/%[%033[0m%]%R%[%033[1;32;40m%]%x%[%033[0m%]%# '
\set PROMPT2 '%[%033[32m%]%/%[%033[0m%]%R%[%033[1;32;40m%]%x%[%033[0m%]%# '
\set PROMPT3 '>> '
You don't have to insert the PROMPT statements. But they are helpful because they change the psql prompt to show the transaction status.
Another advantage of this approach is that it gives you a chance to prevent any erroneous changes.
Example (psql):
database=# SELECT * FROM my_table; -- implicit start transaction; see prompt
-- output result
database*# UPDATE my_table SET my_column = 1; -- missed where clause
UPDATE 525125 -- Oh, no!
database*# ROLLBACK; -- Puh! revert wrong changes
ROLLBACK
database=# -- I'm completely operational and all of my circuits working perfectly
There actually was a discussion on the hackers list about this very feature. It had a mixed reception, but might have been accepted if the author had persisted.
As it is, the best you can do is a statement level trigger that bleats if you modify too many rows:
CREATE TABLE deleteme
AS SELECT i FROM generate_series(1, 1000) AS i;
CREATE FUNCTION stop_mass_deletes() RETURNS trigger
LANGUAGE plpgsql AS
$$BEGIN
IF (SELECT count(*) FROM OLD) > TG_ARGV[0]::bigint THEN
RAISE EXCEPTION 'must not modify more than % rows', TG_ARGV[0];
END IF;
RETURN NULL;
END;$$;
CREATE TRIGGER stop_mass_deletes AFTER DELETE ON deleteme
REFERENCING OLD TABLE AS old FOR EACH STATEMENT
EXECUTE FUNCTION stop_mass_deletes(10);
DELETE FROM deleteme WHERE i < 100;
ERROR: must not modify more than 10 rows
CONTEXT: PL/pgSQL function stop_mass_deletes() line 1 at RAISE
DELETE FROM deleteme WHERE i < 10;
DELETE 9
This will have a certain performance impact on deletes.
This works from v10 on, when transition tables were introduced.
If you can afford making it a little less convinient for your users, you might try revoking UPDATE privilege for all "standard" users and creating a stored procedure like this:
CREATE FUNCTION update(table_name, col_name, new_value, condition) RETURNS void
/*
Check if condition is acceptable, create and run UPDATE statement
*/
LANGUAGE plpgsql SECURITY DEFINER
Because of SECURITY DEFINER this way your users will be able to UPDATE despite not having UPDATE privilege.
I'm not sure if this is a good approach, but this way you can force as strict UPDATE (or anything else) requirements as you wish.
Of course the more complicated UPDATES are required, the more complicated has to be your procedure, but if this is mostly just about updating single row by ID (as in your example) this might be worth a try.
I have a function which uses temporary table, that must be dropped if exists.
drop table if exists t_xy;
create temp table t_xy on commit drop as select ...;
Subsequently I use this function in a view. The function is called many times while select is in progress. I like to use "raise notice" command because it is almost the only reliable way to report any variables in functions for debug purposes. The problem is I must search for them in huge amount of unwanted lines like:
NOTICE: table "t_xy" does not exist, skipping
CONTEXT: SQL statement "drop table if exists t_xy"
PL/pgSQL function f_pending_operations(uuid) line5 in SQL command
Is there a way to suppress such notices that haven't been generated by raise notice command, but by drop table if exists or dropping other objects? Setting 'client_min_messages' option to 'debug' makes the problem worse.
You can silence notices to the client from any command with a local setting for client_min_messages:
SET LOCAL client_min_messages = warning; -- "debug" would have opposite effect
DROP TABLE if exists t_xy;
-- RESET client_min_messages;
If you don't issue RESET you effectively silence notices for the rest of the transaction. The manual:
The effects of SET LOCAL last only till the end of the current transaction
Alternatively, you can set client_min_messages in the call from the command line (for the duration of the session):
How to suppress INFO messages when running psql scripts
You can also reduce the verbosity of the messages using the GUC parameter:
set log_error_verbosity='terse';
which can of course be set at the function level.
Is there a way to save, temporarily change, and then restore the value of the psql ON_ERROR_STOP variable?
Basically, I'd like to have the "moral equivalent" of the following in a psql script:
save_on_error_stop=ON_ERROR_STOP
\unset ON_ERROR_STOP
ALTER TABLE foo DROP COLUMN bar; -- (for example)
\set ON_ERROR_STOP save_on_error_stop
ALTER TABLE foo ADD COLUMN bar;
The point being that the '\set' command at the end won't actually set ON_ERROR_STOP unless it was set before.
I don't think it is possible to do this in a single psql session. According to the manual, one can use \set command on it's own to list all psql variables.
One can memorize the setting in shell, but this makes the whole thing useless, as it is much more simple just to execute the desired set of queries enforcing the desired ON_ERROR_STOP value.
Another alternative is to write an anonymous code block and DO some extra logic to detect, whether column needs to be dropped before adding it.
BTW, What is the purpose of dropping and adding column straight after?
If it is only to make sure no such column exists, how bout this DO block:
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_schema='public' AND table_name='foo'
AND column_name='bar')
THEN
EXECUTE format('ALTER TABLE %I.%I ADD %I text',
'public','foo','bar');
END IF;
END; $$;
You can also create a function if you tend to do such check quite often.
I have this Trigger in Postgresql that I can't just get to work (does nothing). For understanding, there's how I defined it:
CREATE TABLE documents (
...
modification_time timestamp with time zone DEFAULT now()
);
CREATE FUNCTION documents_update_mod_time() RETURNS trigger
AS $$
begin
new.modification_time := now();
return new;
end
$$
LANGUAGE plpgsql;
CREATE TRIGGER documents_modification_time
BEFORE INSERT OR UPDATE ON documents
FOR EACH ROW
EXECUTE PROCEDURE documents_update_mod_time();
Now to make it a bit more interesting.. How do you debug triggers?
Use the following code within a trigger function, then watch the 'messages' tab in pgAdmin3 or the output in psql:
RAISE NOTICE 'myplpgsqlval is currently %', myplpgsqlval; -- either this
RAISE EXCEPTION 'failed'; -- or that
To see which triggers actually get called, how many times etc, the following statement is the life-saver of choice:
EXPLAIN ANALYZE UPDATE table SET foo='bar'; -- shows the called triggers
Note that if your trigger is not getting called and you use inheritance, it may be that you've only defined a trigger on the parent table, whereas triggers are not inherited by child tables automatically.
To step through the function, you can use the debugger built into pgAdmin3, which on Windows is enabled by default; all you have to do is execute the code found in ...\8.3\share\contrib\pldbgapi.sql against the database you're debugging, restart pgAdmin3, right-click your trigger function, hit 'Set Breakpoint', and then execute a statement that would cause the trigger to fire, such as the UPDATE statement above.
Turns out I was using inheritance in the above problem and forgot to mention it. Now for everybody who might run into this as well, here's some debugging hints:
Use the following code to debug what a trigger is doing:
RAISE NOTICE 'test'; -- either this
RAISE EXCEPTION 'failed'; -- or that
To see which triggers actually get called, how many times etc, the following statement is the life-saver of choice:
EXPLAIN ANALYZE UPDATE table SET foo='bar'; -- shows the called triggers
Then there's the one thing I didn't know before: triggers only fire when updating the exact table they're defined on. If you use inheritance, you MUST define them on the child tables as well!
You can use 'raise notice' statements inside your trigger function to debug it. To debug the trigger not being called at all is another story.
If you add a 'raise exception' inside your trigger function, can you still do inserts/updates?
Also, if your update test occurs in the same transaction as your insert test, now() will be the same (since it's only calculated once per transaction) and therefore the update won't seem to do anything. If that's the case, either do them in separate transactions, or if this is a unit test and you can't do that, use clock_timestamp().
I have a unit test that depends on some time going by between transactions, so at the beginning of the unit test I have something like:
ALTER TABLE documents
ALTER COLUMN modification_time SET DEFAULT clock_timestamp();
Then in the trigger, use "set modification_time = default".
So normally it doesn't do the extra calculation, but during a unit test this allows me to do inserts with pg_sleep in between to simulate time passing and actually have that be reflected in the data.