Why can 'create table' in Postgres take several seconds? - postgresql

Question

In my project we have to sometimes copy all the data from one schema into another. I automated this by simple truncate / insert into select * script, but sooner realized that this way is not tolerant to changes in the source schema (adding/deleteing tables required modifying the script). So today I decided to change it to PL/PGSQL script which creates tables and copies data using dynamic queries. My first implementation was something like this:
do
$$
declare
source_schema text := 'source_schema';
dest_schema text := 'dest_schema';
obj_name text;
source_table text;
dest_table text;
alter_columns text;
begin
for dest_table in
select table_schema || '.' || table_name
from information_schema.tables
where table_schema = dest_schema
order by table_name
loop
execute 'drop table ' || dest_table;
end loop;
raise notice 'Data cleared';
for obj_name in
select table_name
from information_schema.tables
where table_schema = source_schema
order by table_name
loop
source_table := source_schema || '.' || obj_name;
dest_table := dest_schema || '.' || obj_name;
execute 'create unlogged table ' || dest_table
|| ' (like ' || source_table || ' including comments)';
alter_columns := (
select string_agg('alter column ' || column_name || ' drop not null', ', ')
from information_schema.columns
where table_schema = dest_schema and table_name = obj_name
and is_nullable = 'NO');
if alter_columns is not null then
execute 'alter table ' || dest_table || ' ' || alter_columns;
end if;
execute 'insert into ' || dest_table || ' select * from ' || source_table;
raise notice '% done', obj_name;
end loop;
end;
$$
language plpgsql;
As destination schema is read only, I create it without constrants to reach maximum performance. I don't think that NOT NULL constraints is a big deal, but I decided to leave everything here as it was.
This solution worked perfectly but I noticed that it was taking longer time to copy data in comparison to static script. Not dramatically, but steadily it took 20-30 seconds longer than static script.
I decided to investigate it. My first step was to comment insert into select * statement to find out what time takes everything else. It shown that it takes only half a second to clear and recreate all tables. My clue was that INSERT statements somehow work longer in procedureal context.
Then I added measuring of the execution time:
ts := clock_timestamp();
execute 'insert into ...';
raise notice 'obj_name: %', clock_timestamp() - ts;
Also I performed the old static script with \timing in psql. But this shown that my assumption was wrong. All insert statements took more or less the same time, predominantly even faster in dynamic script (I suppose it was due to autocommit and network roundtrips after each statement in psql). However the overal time of dynamic script was again longer than time of static script.
Mysticism?
Then I added very verbose logging with timestamps like this:
raise notice '%: %', clock_timestamp()::timestamp(3), 'label';
I discovered that sometimes create table executes immediately, but sometimes it takes several seconds to finish. OK, but how come all these statements for all tables took just milliseconds to complete in my first experiment?
Then I basically split one loop into two: first one creates all the tables (and we now know it takes just milliseconds) and the second one only inserts data:
do
$$
declare
source_schema text := 'onto_oper';
dest_schema text := 'onto';
obj_name text;
source_table text;
dest_table text;
alter_columns text;
begin
raise notice 'Clearing data...';
for dest_table in
select table_schema || '.' || table_name
from information_schema.tables
where table_schema = dest_schema
order by table_name
loop
execute 'drop table ' || dest_table;
end loop;
raise notice 'Data cleared';
for obj_name in
select table_name
from information_schema.tables
where table_schema = source_schema
order by table_name
loop
source_table := source_schema || '.' || obj_name;
dest_table := dest_schema || '.' || obj_name;
execute 'create unlogged table ' || dest_table
|| ' (like ' || source_table || ' including comments)';
alter_columns := (
select string_agg('alter column ' || column_name || ' drop not null', ', ')
from information_schema.columns
where table_schema = dest_schema and table_name = obj_name
and is_nullable = 'NO');
if alter_columns is not null then
execute 'alter table ' || dest_table || ' ' || alter_columns;
end if;
end loop;
raise notice 'All tables created';
for obj_name in
select table_name
from information_schema.tables
where table_schema = source_schema
order by table_name
loop
source_table := source_schema || '.' || obj_name;
dest_table := dest_schema || '.' || obj_name;
execute 'insert into ' || dest_table || ' select * from ' || source_table;
raise notice '% done', obj_name;
end loop;
end;
$$
language plpgsql;
Surprisingly it fixed everything! This version works faster than the old static script!
We are coming to very weird conclusion: create table after inserts sometime may take long time. This is very frustrating. Despite the fact I solved my problem I don't understand why it happens. Does anybody have any idea?