I use macOS and I am having issue with full-refresh on a large table. During the run it appears as if it hangs and there is no query running in redshift. It does not happen with smaller tables and it does not happen if I run an incremental. This table used to be smaller and I was able to run a full refresh as long as I specified the table. Now that it is bigger I seem to be running into this issue. There are 6 tables that this model is dependent on. Almost like the command isn’t being sent. Any suggestions?
There is no error because it just doesn't run. Other team members running this on windows and macos expect it to finish in 10 min. Currently it is 30 min but I have let it sit a lot longer than that.
My command is
dbt run --models +fct_mymodel --full-refresh --vars "run_date_start: 2020-06-01"
Thank you
Redhift UI usually shows only the long running queries. I ran into similar problems and they were caused by lock on some tables - in our case caused by uncommitted explicit transactions (BEGIN without COMMIT or ROLLBACK).
run this query to see current transactions and their locks:
select a.txn_owner, a.txn_db, a.xid, a.pid, a.txn_start, a.lock_mode, a.relation as table_id,nvl(trim(c."name"),d.relname) as tablename, a.granted,b.pid as blocking_pid ,datediff(s,a.txn_start,getdate())/86400||' days '||datediff(s,a.txn_start,getdate())%86400/3600||' hrs '||datediff(s,a.txn_start,getdate())%3600/60||' mins '||datediff(s,a.txn_start,getdate())%60||' secs' as txn_duration
from svv_transactions a
left join (select pid,relation,granted from pg_locks group by 1,2,3) b
on a.relation=b.relation and a.granted='f' and b.granted='t'
left join (select * from stv_tbl_perm where slice=0) c
on a.relation=c.id
left join pg_class d on a.relation=d.oid
where a.relation is not null;
read AWS knowledge base entry for more details https://aws.amazon.com/premiumsupport/knowledge-center/prevent-locks-blocking-queries-redshift/
Related
I have a process which is creating thousands of temporary tables a day to import data into a system.
It is using the form of:
create temp table if not exists test_table_temp as
select * from test_table where 1=0;
This very quickly creates a lot of dead rows in pg_attribute as it is constantly making lots of new columns and deleting them shortly afterwards for these tables. I have seen solutions elsewhere that suggest using on commit delete rows. However, this does not appear to have the desired effect either.
To test the above, you can create two separate sessions on a test database. In one of them, check:
select count(*)
from pg_catalog.pg_attribute;
and also note down the value for n_dead_tup from:
select n_dead_tup
from pg_stat_sys_tables
where relname = 'pg_attribute';
On the other one, create a temp table (will need another table to select from):
create temp table if not exists test_table_temp on commit delete rows as
select * from test_table where 1=0;
The count query for pg_attribute immediately goes up, even before we reach the commit. Upon closing the temp table creation session, the pg_attribute value goes down, but the n_dead_tup goes up, suggesting that vacuuming is still required.
I guess my real question is have I missed something above, or is the only way of dealing with this issue vacuuming aggressively and taking the performance hit that comes with it?
Thanks for any responses in advance.
No, you have understood the situation correctly.
You either need to make autovacuum more aggressive, or you need to use fewer temporary tables.
Unfortunately you cannot change the storage parameters on a catalog table – at least not in a supported fashion that will survive an upgrade – so you will have to do so for the whole cluster.
For almost a year, I have been using this sql to report on the rank of game profiles, based on the number of days that a game has been above the rank of 10:
SELECT P.id, P.name, P.rank, COUNT(P.id)
FROM application_classes_models_gameprofile P
LEFT JOIN application_classes_models_gamedeveloper D ON D.id = P."developerId"
LEFT JOIN application_classes_models_gameprofileposition PP ON
PP."gameProfileId" = P.id AND
PP.position <= 10 AND
PP.position > 0
WHERE
P.inactive = false AND
D."excludeFromRanking" = false AND
P.rank <= 10 AND
P.rank > 0
GROUP BY P.id
ORDER BY COUNT(P.id) DESC
Grouping is always a big of a pain in postgresql, but the above sql has been working fine for almost a year, returning the expected results.
Yesterday, I had an issue with the game profile table which forced me to have to restore a backup, for that table. I did so using pg_restore -v --clean -t application_classes_models_gameprofile < backup.bak.
This morning, when we ran our reports, postgresql came back with the error:
column "p.name" must appear in the GROUP BY clause or be used in an aggregate function
Just to clarify, this sql has been running for almost a year, and the above error has never appeared for this specific sql query, however, it seems that after we have cleaned and restored the game profile table we're getting the above error...
I know that I can solve the problem by fixing the sql query to remove the name/rank, but I worry if there is a deeper issue here... so does anyone know why the above might happen?
Postgresal version is 9.6, running on debian 9
I have a PostgreSQL server running on 10.6 with an openSUSE distribution of Linux, and I just set up pgpool-II, to allow me to cache queries. It works mostly fine, but for unknown reasons, sometimes I get this warning message :
WARNING: memcache: adding table oid maps, failed to create directory:"/var/log/pgpool/oiddir". error:"No such file or directory"
I already created the directory, changed the owner to the user that runs the pgpool server, and allowed read, write, and execution on this directory to the same user.
This message appears when a query is not yet cached, and it doesn't seem to have any impact, i.e. the query is cached as it should and if I do it again, the result is directly pulled from the cache.
But I also have another problem, and I don't know if it's related to the first one :
When I write big queries (using JOINs, subqueries, lots of conditions, etc), pgpool-II will not cache their results (or cache it but not use it, I have no idea), even though the results are not big (less than 500 rows). Also, in this case, I don't get the "oid" warning message. I tried raising the different limits of shared memory pgpool-II is allowed to use (see documentation) but it changed nothing as I expected, because when pgpool-II fails to cache a query due to the lack of available shared memory, it's supposed to return a message like this one :
LOG: pid 13756: pool_add_temp_query_cache: data size exceeds memqcache_maxcache. current:4095 requested:111 memq_maxcache:4096
But in my case, I don't get any message. Examples of both cases below.
1st problem
Simple query, result is cached, "oid" error :
SELECT *
FROM some_table
WARNING: memcache: adding table oid maps, failed to create directory:"/var/log/pgpool/oiddir". error:"No such file or directory"
-- Doing it a second time will just give me the cached result without any warning, as expected
2nd problem
Complex query, result is not cached (or cached and not used), no warning/error message :
SELECT A.geom, A.id, to_char(C.timestamp, 'DD/MM/YY') as date, C.timestamp::time as time, ROUND(C.value) as value
FROM segments A, lines B, ( SELECT DISTINCT ON (B.id) B.id, A.timestamp, ROUND(A.nb1+A.nb2+A.nb3) as value
FROM records A
CROSS JOIN LATERAL (SELECT *
FROM points B
WHERE A.id = B.id
AND A.direction = B.direction
ORDER BY A.position <-> B.geom
LIMIT 1) AS B
ORDER BY B.id, A.timestamp DESC) AS C
WHERE A.id = B.id
AND B.id = C.id
AND A.direction = B.direction
AND B.direction = C.direction
-- Doing it a second time will just directly request the PostgreSQL server again, instead of pulling the result from the cache, as it should if the result was cached.
Your first issue looks like a permissions issue on the oiddir. Basically pgpool can't create the necessary directories for caching. Depending on how you start pgpool (through pgpoolAdmin or command line), you need to
Create the oiddir folder, mkdir /var/log/pgpool/oiddir
Give permissions of the folder to pgpool
Assuming pgpoolAdmin starts pgpool:
chown -R _www /var/log/pgpool/oiddir (I use pgpool on mac os, so the apache user is "_www", change accordingly)
Assuming you start pgpool from command line:
chown -R postgres /var/log/pgpool/oiddir (or you can use your current user instead of postgres)
Your second issue looks based on the first issue. Basically you can't cache because the oiddir isn't created. Fixing the first should allow you to cache.
At my work, I needed to build a new join table in a postgresql database that involved doing a lot of computations on two existing tables. The process was supposed to take a long time so I set it up to run over the weekend before I left on Friday. Now, I want to check to see if the query finished or not.
How can I check if an INSERT command has finished yet while not being at the computer I ran it on? (No, I don't know how many rows it was suppose to add.)
Select * from pg_stat_activity where state not ilike 'idle%' and query ilike 'insert%'
This will return all non-idle sessions where the query begins with insert, if your query does not show in this list then it is no longer running.
pg_stat_activity doc
You can have a look at the table pg_stat_activity which contains all database connections including active query, owner etc.
At https://gist.github.com/rgreenjr/3637525 there is a copy-able example how such a query could look like.
What I did is on the website How to select efficiently in a very large table
I used RPostgreSQL package under R because I want to manipulate other files at the same time. As you can see from the above website, the table is very large. It will cost more than two hours to complete the selection using RPostgreSQL. But today I use the same SQL code under psql instead of using RpostgreSQL. It only took several minutes.
Why?
The code of R is:
sql='SELECT * into new_table FROM table_1 WHERE EXISTS (SELECT 1 FROM table_2 WHERE column=table_1.column_1) AND EXISTS (SELECT 1 FROM table_2 WHERE column=table_1.column_2)'
rs=dbSendQuery(con,sql)
Updated:
I used the psycopg2 under python. It also runs pretty fast.
If no data are being sent to R, then it would be surprising if there were any difference at all.
If data are being sent back to R, that can be relatively slow (I have found psycopg2 and pandas to be faster for data transfer in many cases).
Depending on what is being done with the data in R, it can be more efficient to use PL/R.