Could anyone suggest best approach to monitor schema change of Snowflake database?
I would to trigger my app when any schema changes, and furthermore, I would trigger my app when any permission changes.
One way is to use the last_altered column in Schemata View in Information_schema and run a task to pull information.
Reference: https://docs.snowflake.net/manuals/sql-reference/info-schema/schemata.html
Related
I have a Redshift database in my company (not in my power to change that) and recently some data just desapear. I thinked in do some kind of trigger to identify when any delete happen and try to found the source, but I learn Redshift don't have trigger. There are any opcions for monitoring what user and when delete from database?
Ideally you should be having different users or roles for each and every process or client connecting to redshift. Use grants to solve/debug this problem.
Then you should be using grant to provide DELETE grants to some specific user/users or roles.
Also, there is sql_history table that you could query to see which user has issued delete query.
I hope it will help.
What I want is to update a table every night and cache it so it doesn't have to run each time we run a query based on it. So I figure I need a materialised view (not a view).
Top answer to below question is spot on what I need.
How can I ensure that a materialized view is always up to date?
So, I searched around about materialised views for Postgresql and it seems perfect. All I need is a scheduler.
Pg_cron looks to be popular but from what I understand it is not compatible with Amazon Redshift(See https://github.com/citusdata/pg_cron/)(?)
Is there some other scheduling tool that is useable or some work around to the problem?
Many thanks!
Hannes
Redshift has no inbuilt support for materialised views yet. You will need to have external service do it for you. We are using airflow, where we have written templates DAG which fills materialised views.
Redshift now supports Materialized Views
https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-overview.html
There is however currently no way to schedule the refresh from within RedShift, so you will have to invoke the REFRESH command from some external timer. https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-refresh-sql-command.html
Since 2020/11 it is also possible to allow Redshift to have control of refreshing the materialized view with the AUTO REFRESH YES option. https://docs.amazonaws.cn/en_us/redshift/latest/dg/materialized-view-create-sql-command.html
If you want to keep it serverless, you can use Redshift data API and invoke MV REFRESH from Lambda.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/redshift-data.html
I'm trying to set up an architecture with 2 databases, say preview and live, that have the exact same schemas. The use case is that edits can be made to the preview database and then pushed to the live database after they are vetted and approved. The production application would read from the live database.
What would be the most appropriate way to push all data from the preview database to the live database without bringing the live database down? Ideally the copy from preview to live would be an atomic transaction.
I've worked with this type of setup in MSSQL, but I'm fairly new to Postgres. So I'm open to hearing other ways to architect this (with Schemas perhaps?).
EDIT: The main reason to use separate databases is that I may need more than 1 target database (not just a single "live" database). I also may need to switch target databases on the fly without altering the source database schema.
I think what you're looking for is a "hot standby". This would be a separate instance of Postgresql, possibly on the same server but usually not, which is a near-real-time replica of the primary server.
In broad strokes, this is done by shipping the binary transaction logs from the primary server to the backup server, and then "replaying" them there. The exact mechanism for transmitting the logs may vary depending on your requirements.
Fortunately, the docs on this are excellent:
https://www.postgresql.org/docs/9.3/static/warm-standby.html
https://www.postgresql.org/docs/9.0/static/hot-standby.html
I'm going to develop a multi-tenant application, where each tenant lives in its own database or schema (I've not decided this yet).
In this scenario, if I wanted to use point in time recovery (PITR), I also want to have it per-tenant. If a tenant has a problem, I want to be able to roll back only his database or schema and not the whole server.
While I found information how to do backup/restore in such situations with pg_dump and pg_restore, I haven't found any information for PITR.
Is this even possible? If yes, only per database or even per schema?
I can imagine that postgres maybe stores the log of the whole server in a single file, which may be the reason why it could not be possible. But I may be wrong..
To connect to the database I use this example. But I can't find lessons on how to create a database.
For example:
connect to server
create new database
do something
drop database
close connection
Can anybody show me how to do it?
Thanks!
Follow the manual on how to create a database cluster:
http://www.postgresql.org/docs/9.1/interactive/creating-cluster.html
The database and users are created only once and you can use the client applications for that. Or are you trying to do it automatically as part of a software install package? After that you connect to it as many times as needed.
Since you are creating a new database and then dropping it, why not use the built-in SQLite database? You can do a completely in-memory database that will be lightning fast (unless you fill up available RAM).
I believe you can create databases by issuing standard SQL commands just as you can create tables in a database, as long as you are using a user (e.g. admin or similarly entitled user) that has permissions to create new databases.
So, all you need is to connect to the DB with the right user and then issue SQL commands with db.SQLExecute, such as "create database newDBname".