Asynchronous data load by parallel sessions - postgresql

Question

Looking for help with a data load function designed to support asynchronous execution by parallel sessions.
Process_Log table contains the list of data load functions, with current status and a list of upstream dependencies.
Each session first looks for a function that is ready for execution, calls it, and updates status.
For further details please see comments in the code.
In Oracle PL/SQL I would design it as a nested block within the loop, and autonomous transaction for status updates.
Not sure how to achieve that in Postgres. Running 9.2.
CREATE OR REPLACE FUNCTION dm_operations.dm_load()
RETURNS void AS
$BODY$
declare
_run_cnt integer;
_ready_cnt integer;
_process_id dm_operations.process_log.process_id%type;
_exec_name dm_operations.process_log.exec_name%type;
_rowcnt dm_operations.process_log.rows_affected%type;
_error text;
_error_text text;
_error_detail text;
_error_hint text;
_error_context text;
begin
loop
--(1) Find one function ready to run
select sum(case when process_status = 'RUNNING' then 1 else 0 end) run_cnt,
sum(case when process_status = 'READY' then 1 else 0 end) ready_cnt,
min(case when process_status = 'READY' then process_id end) process_id
into _run_cnt, _ready_cnt, _process_id
from dm_operations.process_log; --One row per each executable data load function
--(2) Exit loop if nothing is ready
if _ready_cnt = 0 then exit;
else
--(3) Lock the row until the status is updated
select exec_name
into _exec_name
from dm_operations.process_log
where process_id = _process_id
for update;
--(4) Set status of the function to 'RUNNING'
--New status must be visible to other sessions
update dm_operations.process_log
set process_status = 'RUNNING',
start_ts = now()
where process_id = _process_id;
--(5) Release lock. (How?)
--(6) Execute data load function. See example below.
-- Is this correct syntax for dynamic call to a function that returns void?
execute 'perform dm_operations.'||_exec_name;
--(7) Get number of rows processed by the data load function
GET DIAGNOSTICS _rowcnt := ROW_COUNT;
--(8) Upon successful function execution set status to 'SUCCESS'
update dm_operations.process_log
set process_status = 'SUCCESS',
end_ts = now(),
rows_affected = _rowcnt
where process_id = _process_id;
--(9) Check dependencies and update status
--These changes must be visible to the next loop iteration, and to other sessions
update dm_operations.process_log pl1
set process_status = 'READY'
where process_status is null
and not exists (select null from dm_operations.process_log pl2
where pl2.process_id in (select unnest(pl1.depends_on))
and (coalesce(pl2.process_status,'NULL') <> 'SUCCESS'));
end if;
--(10) Log error and allow the loop to continue
EXCEPTION
when others then
GET STACKED DIAGNOSTICS _error_text = MESSAGE_TEXT,
_error_detail = PG_EXCEPTION_DETAIL,
_error_hint = PG_EXCEPTION_HINT,
_error_context = PG_EXCEPTION_CONTEXT;
_error := _error_text||
_error_detail||
_error_hint||
_error_context;
update dm_operations.process_log
set process_status = 'ERROR',
start_ts = now(),
rows_affected = _rowcnt,
error_text = _error
where process_id = _process_id;
end;
end loop;
end;
$BODY$
LANGUAGE plpgsql;
Data load function example (6):
CREATE OR REPLACE FUNCTION load_target()
RETURNS void AS
$BODY$
begin
execute 'truncate table target_table';
insert into target_table
select ...
from source_table;
end;
$BODY$
LANGUAGE plpgsql;

Answer 1

You cannot start asynchronous operations in PL/pgSQL.
There are two options I can think of:
The hard way: Upgrade to a more recent PostgreSQL version and write a background worker in C that executes load_target. You'd have to use
Don't write your function in the database, but on the client side. Then you can simply open several database sessions and run functions in parallel that way.