How do I reduce the cost of set_bit in Postgres? - postgresql

Question

I am running PostgreSQL 9.6 and am running an experiment on the following table structure:
CREATE TABLE my_bit_varying_test (
id SERIAL PRIMARY KEY,
mr_bit_varying BIT VARYING
);
Just to understand how much performance I could expect if I was resetting bits on 100,000-bit data concurrently, I wrote a small PL/pgSQL block like this:
DO $$
DECLARE
t BIT VARYING(100000) := B'0';
idd INT;
BEGIN
FOR I IN 1..100000
LOOP
IF I % 2 = 0 THEN
t := t || B'1';
ELSE
t := t || B'0';
end if;
END LOOP ;
INSERT INTO my_bit_varying_test (mr_bit_varying) VALUES (t) RETURNING id INTO idd;
UPDATE my_bit_varying_test SET mr_bit_varying = set_bit(mr_bit_varying, 100, 1) WHERE id = idd;
UPDATE my_bit_varying_test SET mr_bit_varying = set_bit(mr_bit_varying, 99, 1) WHERE id = idd;
UPDATE my_bit_varying_test SET mr_bit_varying = set_bit(mr_bit_varying, 34587, 1) WHERE id = idd;
UPDATE my_bit_varying_test SET mr_bit_varying = set_bit(mr_bit_varying, 1, 1) WHERE id = idd;
FOR I IN 1..100000
LOOP
IF I % 2 = 0 THEN
UPDATE my_bit_varying_test
SET mr_bit_varying = set_bit(mr_bit_varying, I, 1)
WHERE id = idd;
ELSE
UPDATE my_bit_varying_test
SET mr_bit_varying = set_bit(mr_bit_varying, I, 0)
WHERE id = idd;
end if;
END LOOP ;
END
$$;
When I run the PL/pgSQL though, it takes several minutes to complete, and I've narrowed it down to the for loop that is updating the table. Is it running slowly because of the compression on the BIT VARYING column? Is there any way to improve the performance?
Edit This is a simulated, simplified example. What this is actually for is that I have tens of thousands of jobs running that each need to report back their status, which updates every few seconds.
Now, I could normalize it and have a "run status" table that held all the workers and their statuses, but that would involve storing tens of thousands of rows. So, my thought is that I could use a bitmap to store the client and status, and the mask would tell me in order which ones had run and which ones had completed. The front bit would be used as an "error bit" since I don't need to know exactly which client failed, only that a failure exists.
So for example, you might have 5 workers for one job. If they all completed, then the status would be "01111", indicating that all jobs were complete, and none of them failed. If worker number 2 fails, then the status is "111110", indicating that there was an error, and all workers completed except the last one.
So, you can see this as a contrived way of handling large numbers of job statuses. Of course I'm up for other ideas, but even if I go that route, for the future, I'd still like to know how to update a variable bit quickly, because well, I'm curious.

Answer 1

If it is really the TOAST compression that is your problem, you can simply disable it for that table:
ALTER TABLE my_bit_varying_test SET STORAGE EXTERNAL;

Answer 2

You can try a set based approach to replace the second loop. A set based approach is usually fatser than looping. Use generate_series() to get the indexes.
UPDATE my_bit_varying_test
SET mr_bit_varying = set_bit(mr_bit_varying, gs.i, abs(gs.i % 2 - 1))
FROM generate_series(1, 100000) gs(i)
WHERE id = idd;
Also consider creating an index on my_bit_varying_test (id), if you don't already have one.