Issue:
PostgreSQL 12.2 (Ubuntu 12.2-2.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0, 64-bit
Client is Data Grip and same behavior from my report server which uses the driver that comes with Jaspersoft
I am running the same function for reporting with different parameters multiple times under the same session. Also does the same thing using the same parameters.
These are being one run one after the other and not at the same time.
The result only has a few rows but does read from quite a few tables, no writes.
It is just table joins and selects no inputs or updates to the tables themselves (would like to be able to post query but can't for security reasons).
After I run the function a few times it starts to slow down and gets to an unacceptable level. For example one of the functions goes from 1 second to over 90+ seconds (that is where I stopped testing).
Troubleshooting:
I have gone to the server and terminated the session and after that it starts to run normally for a few runs.
The standard report does use a temp table but I have removed that for testing.
I have run the following after it starts having the issue to try and fix the issue.
VACUUM all touched tables; -- I know this should not be required as there are no major changes to these tables but was trying pretty much anything.
Language plpgsql.
SET SESSION AUTHORIZATION DEFAULT;
RESET ALL;
DEALLOCATE ALL;
CLOSE ALL;
UNLISTEN *;
SELECT pg_advisory_unlock_all();
DISCARD PLANS;
DISCARD SEQUENCES;
DISCARD TEMP;
There does not seem to be a good way to terminate idle connections without a separate application or script run through a job.
Turns out there is a known issue but seldom talked about issue with parameters in Execution Plans where starting with the 6th run in the same session it will take a lot longer to run. The solution is to just set the execution plan to force_custom_plan and it works fine or did in my case.
ALTER FUNCTION xy SET plan_cache_mode = force_custom_plan;
Related
I have a pretty basic SQL Query I'm running (SQL Server 2016). I'm pulling about 15 columns from some simple inner and left joins on 8 tables that all have WITH (NOLOCK) on the join. My where clause is checking for a few string, uniqueidentifier, and bit values. The stored procedure has no calculations, no loops/cursors/case statements, etc.. It is very straightforward.
For some reason our application keeps intermittently freezing up because the SQL call is timing out. When I grab the call in profiler and run it in SSMS it runs in sub-second time, but from the application it just wont finish.
However, if I script out an ALTER command on the query and just add a blank line and execute the change, the problem goes away and the calls run instantaneously. The change I make is nothing substantive, just the act of changing the query in some way seems to unlock it.
Does anyone have any idea what could be causing this? I don't see any locks when it is timing out. I was thinking maybe a bad execution plan being cached?
The only other oddity is that this is running from an old legacy application, our last one in old classic asp. But it doesn't seem related to the web server or architecture, just the database.
I'm writing a program to run mass calculation and output results into PostgreSQL.
My platform is Windows Sever 2008, PostgreSQL 10. My program is written in C.
The results would be produced group by group, finishing of each group will create an extra thread to write the output.
Now since the output threads are created one by one, it is possible that two or more SQL input commands will be created simultaneously, or the previous one is under process when new ones call the function.
So my questions are:
(1) What would happen if one thread is in SQL processing and another thread called PQexec(PGconn *conn, const char *query), would they effect each other?
(2) What if I apply different PGconn? Would it speed up?
If you try to call PQexec on a connection that is in the process of executing an SQL statement, you would cause a protocol violation. That just doesn't work.
Processing could certainly be made faster if you use several database connections in parallel — concurrent transactions is something that PostgreSQL is designed for.
Is there any way/tool to trace and debug a query in PostgreSQL 9.3.18?
I'm a SQL programmer and sometimes I need to trace and debug my queries and see the values of different fields at execution time. I've Googled this but didn't get any relevant result.
Any idea would be appreciated
PG Admin (database interaction GUI) that is sometimes bundled with PostgreSQL includes a step through debugger for query/calls to Postgres database functions (as opposed to every query that goes to the server).
https://www.pgadmin.org/docs/pgadmin4/4.29/debugger.html
Before using it you have to enable it as a plugin/library in PG Admin.
The debugger will step to statements so sometimes a complex single statement will execute without letting you step through it's details. Still, if you need to see a basic step through of a longer multi statement function or variable values at some points it can be useful. Note, this debug applies to database functions and not general queries.
I have a cronjob that is kicked off every night that involves building up a temporary table, dropping the current table on Redshift, and swapping in the temporary table for the old one. More than half of the time, this specific job gets stuck when dropping the existing table and behaving as if there is some pending transaction that is stopping the drop from going through.
This is just one of dozens of jobs that uses the exact same script to run overnight, none of which have ever had this issue; however, there are a few minor differences:
The box that this particular job is running on is a different box from all of the other production jobs, as this one is currently in a testing state.
The S3 key used on this box is different from the other box.
In addition to the fact that I have never seen this on any other job, this issue has been extremely difficult to troubleshoot for the following reasons:
I have not been able to replicate this issue by running the script manually on the same box it is currently being run on; the script executes as expected, with the table drop occurring in mere seconds. The only difference I can think of here is that I'm executing the script as ubuntu whereas the cronjob is executed from root.
I have not had any success identifying or terminating the sessions that are causing the drop to stall; I've looked high and low on Stack Overflow (this is the most applicable question with answers - redshift drop or truncate table very very slow), the Redshift docs, and otherwise, but nothing I've found has been the answer. When I see that the job is stalled, I've checked the following tables on Redshift and usually find that things are in the following state:
The temporary table has been created, but the old version of the destination table is still there.
The stv_locks table shows that that there are three processes running, with the lock_status of "Holding write lock," "Holding delete lock," and "Holding insert lock" respectively. The process ID associated with these is NOT the ID related to the current job.
The stv_tr_conflict table shows nothing.
The stv_recents table shows the drop with a status of Running.
The query that's supposedly creating the lock described above shows up in the svl_qlog as finished, so that seems to contradict the stv_locks table.
Using pg_terminate_backend to stop the associated process does not actually remove the session when querying stv_sessions, but DOES free up something that allows the job to finish.
Any help in figuring out what exactly is going on here would be greatly appreciated!
I faced the same problem, I just rebot RS then it works again normally.
I'm encountering some major performance problems with simple SQL queries generated by the Entity Framework (4.2) running against SQL Server 2008 R2. In some situations (but not all), EF uses the following syntax:
exec sp_executesql 'DYNAMIC-SQL-QUERY-HERE', #param1...
In other situations is simply executes the raw SQL with the provided parameters baked into the query. The problem I'm encountering is that queries executed with the sp_executesql are ignoring all indexes on my target tables, resulting in an extremely poor performing query (confirmed by examining the execution plan in SSMS).
After a bit of research, it sounds like the issue might be caused by 'parameter sniffing'. If I append the OPTION(RECOMPILE) query hint like so:
exec sp_executesql 'DYNAMIC-SQL-QUERY-HERE OPTION(RECOMPILE)', #param1...
The indexes on the target tables are used and the query executes extremely quickly. I've also tried toggling on the trace flag used to disable parameter sniffing (4136) on the database instance (http://support.microsoft.com/kb/980653), however this didn't appear to have any effect whatsoever.
This leaves me with a few questions:
Is there anyway to append the OPTION(RECOMPILE) query hint to the SQL generated by Entity Framework?
Is there anyway to prevent Entity Framework from using exec sp_executesql, and instead simply run the raw SQL?
Is anyone else running into this problem? Any other hints/tips?
Additional Information:
I did restart the database instance through SSMS, however, I will try restarting the service from the service management console.
Parameterization is set to SIMPLE (is_parameterization_forced: 0)
Optimize for adhoc workloads has the following settings
value: 0
minimum: 0
maximum: 1
value_in_use: 0
is_dynamic: 1
is_advanced: 1
I should also mention that if I restart the SQL Server Service via the service management console AFTER enabling trace flag 4136 with the below script, appears to actually clear the trace flag...perhaps I should be doing this a different way...
DBCC TRACEON(4136,-1)
tl;dr
update statistics
We had a delete query with one parameter (the primary key) that took ~7 seconds to complete when called through EF and sp_executesql. Running the query manually, with the parameter embedded in the first argument to sp_executesql made the query run quickly (~0.2 seconds). Adding option (recompile) also worked. Of course, those two workarounds aren't available to us since were using EF.
Probably due to cascading foreign key constraints, the execution plan for the long running query was, uhmm..., huge. When I looked at the execution plan in SSMS I noticed that the arrows between the different steps in some cases were wider than others, possibly indicating that SQL Server had trouble making the right decisions. That led me to thinking about statistics. I looked at the steps in the execution plan to see what table was involved in the suspect steps. Then I ran update statistics Table for that table. Then I re-ran the bad query. And I re-ran it again. And again just to make sure. It worked. Our perf was back to normal. (Still somewhat worse than non-sp_executesql performance, but hey!)
It turned out that this was only a problem in our development environment. (And it was a big problem because it made our integration tests take forever.) In our production environment, we had a job running that updated all statistics on a regular basis.
At this point I would recommend:
Set the optimize for ad hoc workloads setting to true.
EXEC sp_configure 'show advanced', 1;
GO
RECONFIGURE WITH OVERRIDE;
GO
EXEC sp_configure 'optimize for ad hoc', 1;
GO
RECONFIGURE WITH OVERRIDE
GO
EXEC sp_configure 'show advanced', 0;
GO
RECONFIGURE WITH OVERRIDE;
GO
If after some time this setting doesn't seem to have helped, only then would I try the additional support of the trace flag. These are usually reserved as a last resort. Set the trace flag using the command line via SQL Server Configuration Manager, as opposed to in a query window and using the global flag. See http://msdn.microsoft.com/en-us/library/ms187329.aspx