I have few functions that need to run every day at certain time. Is there any way to schedule it from postgres? or do I have to use an external tool?
One option you have is to make use of pgAgent. http://www.pgadmin.org/docs/1.8/pgagent.html
Related
Given some Custom SQL, I want to create a Tableau Extract programmatically.
Is this possible?
Context of the process is:
Generate SQL scripts for each extract (100+)
Create the (100+) extracts from Step 1
Publish the Extracts to Tableau Online
Refresh them there on a schedule
Step 2 can be done manually using Tableau Desktop's Custom SQL.
As seen here in this help doc, https://help.tableau.com/current/pro/desktop/en-us/customsql.htm
I want to do it (Step 2) programmatically, due to the number of extracts needed and the time it will take.
Yes, you can programmatically create extracts with the Hyper API. Then you have the option to use either Tabcmd, Tableau REST API or Tableau Server Client Python library to publish the extract. If you go with Python to create the extract, then in the same script, you could use the Server Client to publish. Instead of tableau server refreshing it, you would schedule your script with some sort task scheduler, like in Windows Task Scheduler.
Tableau can do Step 4, you just need to configure it.
Is the problem with steps 1-3 that you have the SQL and just need to automate that part ?
Is it possible to update view table on specific interval? (For example every hour)
How can I do this?
Postgres does not have an internal scheduler, so there is NO Direct method to do what you wanting. There are alternatives however. For Linux there is cron, pgAgent, and a Postgres extension pg_cron. For Windows there is Task Scheduler. Additionally, there are many commercial applications.
What I want is to update a table every night and cache it so it doesn't have to run each time we run a query based on it. So I figure I need a materialised view (not a view).
Top answer to below question is spot on what I need.
How can I ensure that a materialized view is always up to date?
So, I searched around about materialised views for Postgresql and it seems perfect. All I need is a scheduler.
Pg_cron looks to be popular but from what I understand it is not compatible with Amazon Redshift(See https://github.com/citusdata/pg_cron/)(?)
Is there some other scheduling tool that is useable or some work around to the problem?
Many thanks!
Hannes
Redshift has no inbuilt support for materialised views yet. You will need to have external service do it for you. We are using airflow, where we have written templates DAG which fills materialised views.
Redshift now supports Materialized Views
https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-overview.html
There is however currently no way to schedule the refresh from within RedShift, so you will have to invoke the REFRESH command from some external timer. https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-refresh-sql-command.html
Since 2020/11 it is also possible to allow Redshift to have control of refreshing the materialized view with the AUTO REFRESH YES option. https://docs.amazonaws.cn/en_us/redshift/latest/dg/materialized-view-create-sql-command.html
If you want to keep it serverless, you can use Redshift data API and invoke MV REFRESH from Lambda.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/redshift-data.html
I'm using airflow with celery Executor. Now I'm planning to develop user interaction for a task to decide to select branch using BranchOperator in a DAG. Its working by running continuous loop to checking value in database. But I feel it is not the good way of approach. Is there any alternative to do this?
And I want to wait for this interaction up-to particular time otherwise I want to stop. Is it possible to do this in airflow? And if is possible then is the any possibility to change this time bound dynamically?
Thank you in advance.
You shouldn't be using a BranchOperator for this. If you want to proceed in your dag based on some value in the db, you should use a Sensor. There are some off the shelf sensors in airflow and you could also look at some of those to create your own. Sensors basically poll for a certain criteria and timeout after a configurable period of time. From your question it seems this is exactly what you need.
I want to be able to set up one or more Jobs/Triggers when my app runs. The list of Jobs and triggers will come from a db table. I DO NOT care about persisting the jobs to the db for restarting or tracking purposes. Basically I just want to use the DB table as an INIt device. Obviously I can do this by writing the code myself but I am wondering if there is some way to use the SQLJobStore to get this functionality without the overhead of keeping the db updated throughout the life of the app using the scheduler.
Thanks for you help!
Eric
The job store's main purpose is to store the scheduler's state, so there is no way built in to do what you want. You can always write directly to the tables if you want and this will give you the results you want, but this isn't really the best way to do it.
The recommended way to do this would be to write some code that reads the data from your table and then connects to the scheduler using remoting to schedule the jobs.