What I want is to update a table every night and cache it so it doesn't have to run each time we run a query based on it. So I figure I need a materialised view (not a view).
Top answer to below question is spot on what I need.
How can I ensure that a materialized view is always up to date?
So, I searched around about materialised views for Postgresql and it seems perfect. All I need is a scheduler.
Pg_cron looks to be popular but from what I understand it is not compatible with Amazon Redshift(See https://github.com/citusdata/pg_cron/)(?)
Is there some other scheduling tool that is useable or some work around to the problem?
Many thanks!
Hannes
Redshift has no inbuilt support for materialised views yet. You will need to have external service do it for you. We are using airflow, where we have written templates DAG which fills materialised views.
Redshift now supports Materialized Views
https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-overview.html
There is however currently no way to schedule the refresh from within RedShift, so you will have to invoke the REFRESH command from some external timer. https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-refresh-sql-command.html
Since 2020/11 it is also possible to allow Redshift to have control of refreshing the materialized view with the AUTO REFRESH YES option. https://docs.amazonaws.cn/en_us/redshift/latest/dg/materialized-view-create-sql-command.html
If you want to keep it serverless, you can use Redshift data API and invoke MV REFRESH from Lambda.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/redshift-data.html
Related
Is it possible to update view table on specific interval? (For example every hour)
How can I do this?
Postgres does not have an internal scheduler, so there is NO Direct method to do what you wanting. There are alternatives however. For Linux there is cron, pgAgent, and a Postgres extension pg_cron. For Windows there is Task Scheduler. Additionally, there are many commercial applications.
I have an application which has read and write inline queries in the code, I am facing a challenge while pointing the read and write queries to respective Databases. Is there any best of doing it for Go application?
My thought is to have two ORMs up with Read and Write databases and select appropriate based on the operation. e.g: ReadDbMap.Select("query"); WriteDbMap.Update("query");
But this change effects entire application, that is the concern I have
I am afraid that there is no simpler way.
Streaming replication is not primarily a load balancing feature. For one, you'll have to be aware that a change you made on the primary server is not immediately visible on the standby, so your application will have to deal with these temporary inconsistencies.
Could anyone suggest best approach to monitor schema change of Snowflake database?
I would to trigger my app when any schema changes, and furthermore, I would trigger my app when any permission changes.
One way is to use the last_altered column in Schemata View in Information_schema and run a task to pull information.
Reference: https://docs.snowflake.net/manuals/sql-reference/info-schema/schemata.html
I'm using airflow with celery Executor. Now I'm planning to develop user interaction for a task to decide to select branch using BranchOperator in a DAG. Its working by running continuous loop to checking value in database. But I feel it is not the good way of approach. Is there any alternative to do this?
And I want to wait for this interaction up-to particular time otherwise I want to stop. Is it possible to do this in airflow? And if is possible then is the any possibility to change this time bound dynamically?
Thank you in advance.
You shouldn't be using a BranchOperator for this. If you want to proceed in your dag based on some value in the db, you should use a Sensor. There are some off the shelf sensors in airflow and you could also look at some of those to create your own. Sensors basically poll for a certain criteria and timeout after a configurable period of time. From your question it seems this is exactly what you need.
I want to be able to set up one or more Jobs/Triggers when my app runs. The list of Jobs and triggers will come from a db table. I DO NOT care about persisting the jobs to the db for restarting or tracking purposes. Basically I just want to use the DB table as an INIt device. Obviously I can do this by writing the code myself but I am wondering if there is some way to use the SQLJobStore to get this functionality without the overhead of keeping the db updated throughout the life of the app using the scheduler.
Thanks for you help!
Eric
The job store's main purpose is to store the scheduler's state, so there is no way built in to do what you want. You can always write directly to the tables if you want and this will give you the results you want, but this isn't really the best way to do it.
The recommended way to do this would be to write some code that reads the data from your table and then connects to the scheduler using remoting to schedule the jobs.