We are using Materialised Views in Redshift to house queries used in our Looker BI tool.
We regularly refresh our base data and so these views are required to be refreshed every hour, and so we have set these views to auto refresh with the following command.
ALTER MATERIALIZED VIEW view_name AUTO REFRESH YES
This works for a period of time, and then just ceases with no failure messages showing in SVL_MV_REFRESH_STATUS, the refreshes simply cease across all mat. views.
At this point, we have tried to manually refresh the mat. views using REFRESH MATERIALIZED VIEW but this query hangs for long periods of time and does not get killed until we manually cancel the PID some time later.
After failing to refresh the views manually, we then attempted to DROP the mat. views but this query also hangs and needs to be manually killed using the PID.
The only solution we have found thus far is rebooting the Redshift cluster, which obviously is not a robust solution and doesn't get us to the root of the problem.
Has anybody encountered this issue and could shed some light on it?
Thanks!
This behavior is indicative of a lock issue. I suspect you have a lock blocking the update of the view. Since this has been going on for a while look at the connected session and see if any are very old. This can happen when people open a session on their client and never end it. Their locks can stay around for ever.
Related
I am using a Postgres table which gets 2000-3000 updates per second.
I am using for update this table queries generated with the update helper of pg-promise library.
Each update triggers a notify with pg_notify() function. Some nodejs scripts are handling these notifications. For some reason in Postgres logs keep appearing 'too many notifications in the NOTIFY queue' messages and also indication about the notify queue size which keep increasing up to 100%.
I read some posts like: https://postgrespro.com/list/thread-id/1557124
or https://github.com/hasura/graphql-engine/issues/6263
but I cannot find a way to debug this issue.
Which would be a good way to approach this situation?
Your listener doesn't seem to be consuming the notices fast enough, or possibly not at all. So the first step would be something like logging the processing of each notice from your app code, to figure out what is actually going on.
This might be because there is a long-running transaction that is blocking the release of older messages from the buffer. The process is explained in the manuals and is somewhat analoguous to vacuuming - old transactions need to finish in order to clean up old data.
A gotcha here is that any long-running query can hold up the cleanup; for me it was the process that was running the Listen - it was designed to just keep running forever. PG server log has a backend PID that might be the culprit, so you can look it up in pg_stat_activity and proceed from there.
I have a little issue with a form in a delphy XE2 application:
It's an old issue on this application and i have begin to work on it just since a little time.
When the user choose to launch the process with a button's event, my application launch a connexion with an OPCServer , an SQLServer and construct the form for a good following of data take on the tow servers.
The construction of my form involves a blockage of the interface (for approximately 15 sec) because of lot's of data which are necessary for make it.
When it freeze, if the user want drag the form, she go far away, and usually with the TMainMenu which go out of the screen. After that, it's impossible to use the application because we can't drag and we need to close and re-open.
In the old version, the form be already construct before the connexion. So the modification for a dynamic form isn't in link with this issue.
Life of my event :
-Open connexion with OPC Server
-Open SQL Connexion
-Send SQL Command Text
-FieldByName('') for update my UI (Button.Caption// TPage.TStaticText.Caption // TPage.Label1.Caption)
-FieldByName('') for update an array of record
-Close SQL Connexion
-Open SQL Connexion
-Send SQL CommandText
-FieldByName('') for update an other array of record
-Panel.Visible(false)
-TPage.Panel.Show;
-TPage.Panel.BringToFront;
So I haven't MainForm modification can change its position.
I'm a young developer, so I don't know why it moving and what I can make for repair that...
If you want a part of code, ask me what and i edit this, it's very long and i don't want spam answer.
Thank's for read.
The core of your problem is that you have a lengthy process (form construction) which completely blocks the main thread so your application isn't able to process normal Windows messages at the same time. That is why when you move your application it doesn't properly update its interface.
Now based on your description you already have this form construction process split into multiple steps so you could call Application.ProcessMessages between them.
This will force your application to update its UI part.
But beware calling Application.ProcessMessages often could hurt your application performance quite a bit. Why? It is usually a lengthy process because it forces your application to process all the messages that are in its queue.
Normally not all of these messages get processed as soon as they arrive. Windows groups them in the message queue by their priority list, making sure that high priority messages like WM_PAINT are processed as soon as possible while some other low priority messages like demand for application to respond to OS so that OS can see if the application is still working are mostly processed when application is idle or when they are in queue for certain amount of time.
So that is why Application.ProcessMessages can be so slow as it forces your application to process all messages regardless of their priority.
Also bear in mind that using Application.ProcessMessages can in certain scenarios actually become a bit dangerous.
Let me give you an example:
Lets say that clicking on a button starts a lengthy job which can take some time to finish. Now in order to allow your form to be updated you call Application.ProcessMessages in certain intervals. So far it is all good. But what happens if user clicks on that button again?
Since you are calling Application.ProcessMessages which forces your application to process all the messages and since clicking on button creates a MouseClick message which then fires buttons OnClick event which then executes the OnClick method that has been assigned to buttons OnClick event in the end this will cause the same method that was executed on first button click to be executed again.
So now you have this method partially done from first button click and same method executing again for second mouse click. Now the method that was executed from the second click will finish first and then the method that was started from first button click but was interrupted with Application.ProcessMessages handling the second button click will continue its execution to the end.
This all can lead to strange bugs which are hard to track, because you as a programmer normally don't predict that your end user might have clicked the button twice.
So to avoid this I strongly recommend you implement some safeguard mechanisms to prevent such scenarios by temporarily disabling a button for instance.
But the best solution is always to show your user that your application is working which in most cases will dissuade them from clicking the button again, but unfortunately not always.
You should also take a good care when dynamically constructing a form to enable the controls only after all of the controls have been successfully constructed. Failing to do so the user might click on one of your controls and that control could attempt to access some other control which hasn't yet been created at the time. The result hard to track bug which causes Access Violation.
You might also consider showing a splash screen at start instead of half built form. Why?
For once it is much nicer to see and it tells your users to wait a bit. And for second having main form hidden until it is fully created makes sure that user won't be doing any clicks on it prematurely.
I have read a few tutorials and did some search but I must be missing something.
I've had a bunch of different problems with sqlite on my iphone app the last few days.
The most common problem was crashing when I was doing inserts at the same time on different tables, not on the same table.
I can achieve several inserts using threads by the way, so I am sure there is concurrency.
So does the db lock the whole DB or just the table I am inserting information into?
I'm also looking into another possible problems who might be causing the crashes but this little piece of information would help a lot.
After further tests I can say with certainty that you can perform multiple inserts at the same time in the database as long as it is on different tables. Multiple inserts at the same time on the same table will throw an error.
The crashes I was having earlier seem to be related to the fact that I was init + alloc the db on each view I wanted to use it and then releasing it on each view.
After init+alloc only on the app delegate it solved the crashes.
Although you can have multiple instances of the same DB throughout your App it seems to be preferable to just having one in the App Delegate and accessing it as you need it.
This might not be entirely correct but was my conclusion after several hours searching and testing.
I hope this helps other people.
There are 5 types of locks, but all are related to the whole database file and not a table-lock, or not to mention row-lock.
Are you calling sqlite3_finalize at the proper times when you're done? I've noticed that failure to call sqlite3_finalize in sqlite will pretty much bring things to a halt.
So I'm working on this application that requests and retrieves webservice content for iPhone. The problem I am running into is this: When I initially request data, it is spawned off as an independent thread so that the application does not become unresponsive due to the network being slow. What this means is that if the user navigates away from the current page before this data finishes downloading, unexpected things can happen.
I have managed to narrow down the problem cases to one relatively simple one: I have some nested tables, so if a user goes down into the "Messages" table, which can sometimes take a little while to download, then back out immediately, and select a different set of messages to view, the previous set of messages ends up loading, because it was still in the queue.
Here are things I have tried:
1) I tried cancelling the operations, but this is futile, because since I only allow one operation in the queue at the time, it triggers immediately
2) I tried validating that the recipient of the data is the same, but this doesn't work because the actual table object is the between the two selections, it just needs a different data set.
Anyone have any general programming suggestions on how to solve this tricky threading problem?
On an iPhone specific note: if I could just stop the user from being able to back out of the messages table, I wouldn't have this problem, because they would basically be locked into that view until the data has finished loading.
Thanks!
This post has some design advice relating to iOS networking and threading. The basic gist of it is "Don't use explicit threading", and I couldn't agree more. NSURLConnection has great built-in functionality for asynchronously loading data from a URL while managing all of the threading for you. They can also be cancelled easily at will.
If you were to use the NSURLConnection paradigm, you can simply cancel any pending request when you back out of the requesting view controller.
I am new to objective-c/cocoa programming. I am making an application which is to constantly sync with a server and keep its view updated.
Now in a nutshell, heres what I thought of: Initiate an NSTimer to trigger every second or two, contact the server, if there is a change, update the view. Is this a good way of doing it?
I have read elsewhere that you can have a thread running in the background which monitors the changes and updates the view. I never worked with threads before and I know they can be quite troublesome and you need a good amount of experience with memory management to get most out of them.
I have one month to get this application done. What do you guys recommend? Just use an NSTimer and do it the way I though of...or learn multithreading and get it done that way (but keep in mind my time frame).
Thanks!
I think using separate thread in this case would be too much. You need to use threads when there is some task that runs for considerable amount of time and can freeze your app for some time.
In your case do this:
Create timer and call some method (say update) every N seconds.
in update send asynchronous request to server and check for any changes.
download data using NSURLConnection delegate and parse. Note: if there is probability that you can receive a huge amount of data from server and its processing can take much time (for example parsing of 2Mb of XML data) then you do need to perform that is a separate thread.
update all listeners (appropriate view controllers for example) with processed data.
continue polling using timer.
Think about requirements. The most relevant questions, IMO, are :
does your application have to get new data while running in background?
does your application need to be responsive, that is, not sluggish when it's fetching new data?
I guess the answer to the first question is probably no. If you are updating a view depending on the data, it's only required to fetch the data when the view is visible. You cannot guarantee always fetching data in background anyway, because iOS can always just kill your application. Anyway, in your application's perspective, multithreading is not relevant to this question. Because either you are updating only in foreground or also in background, your application need no more than one thread.
Multithreading is relevant rather to the second question. If your application has to remain responsive while fetching data, then you will have to run your fetching code on a detached thread. What's more important here is, the update on the user interface (like views) must happen on the main thread again.
Learning multithreading in general is something indeed, but iOS SDK provides a lot of help. Learning how to use operation queue (I guess that's the easiest to learn, but not necessarily the easiest to use) wouldn't take many days. In a month period, you can definitely finish the job.
Again, however, think clearly why you would need multithreading.