I am using Hashicorp Vault to generate dynamic creds in multiple database clusters. We have one database cluster that is somewhat ephemeral so on occasion it will be refreshed from another database cluster. This database cluster will be connected to via Vault dynamic creds just like the other database clusters.
I have the process to clean up the database users brought over by the backup from the source system when this cluster is refreshed but I don't know how I should handle the Vault cleanup. The database config will be the same (same host/user) but all the existing database user accounts recently created by Vault will be gone after the refresh so I don't know what I need to do to reset/clean up Vault for that database. The database system I'm using (Redshift) doesn't seem to have DROP USER ... IF EXISTS type of syntax otherwise I would simply use that in the dynamic role's revocation_statements and let it cycle out naturally that way.
So my main question is how do I reset or delete all the dynamic creds that were created for a specific database cluster in Vault if the database cluster is refreshed or no longer exists?
I figured out the answer to this and I wanted to share here in case anyone else encounters this.
The "lease revoke" documentation explains that you can use the -prefix switch to revoke leases based on a partial or full prefix match.
Using this information you can run a command similar to the following in order to force revoke existing leases for a specific role:
vault lease revoke -force -prefix database/creds/ROLE_NAME
Using the -force switch will remove the lease even if the revocation_statements fails to process (case when the database user no longer exists).
As an aside, the following command can be used to list leases and is useful to check before and after that all the leases are, in fact, revoked:
vault list sys/leases/lookup/database/creds/ROLE_NAME
This solves my problem of "how to I remove leases for orphaned Vault dynamic credentials" in cases where the target database is refreshed from a backup which is the case I am using this for.
Related
I am trying to write a synthetic monitoring for my on-prem postgresql service, using airflow. The monitoring should return if a cluster is available for creating tables, writing and reading data, and deleting tables.
The clusters on my service are using SSL certificates for authentication, which means a client is required to provide a suitable client certificate in order to connect to the cluster.
Currently, I have implemented my monitoring by creating a global user which will have a certificate with permissions to all the cluster. The user will have permissions to create, write and read only on one schema, dedicated to this monitoring. Using airflow, I will connect with this user each of my postgresql clusters and try to create a table, write to it, read, and then delete it. If one of the actions fails - the DAG will write a log describing the reason for failure.
My main problem with this solution it not being able to limit such a powerful user with accessibility to all of my clusters. In case an intruder will get the user's client certificate, he would be able to explode the DB storage by writing huge amount of data or overload queries and fail the cluster.
I am looking for some ideas for limiting this user so it will be able to act only for it's purpose- the simple actions required for this monitoring, and could not be exploit by an attacker. Alternatively, I would appreciate any suggestions for different implementation for this monitoring.
I searched for build in postgresql configurations that will allow me to limit the dedicated monitoring schema / limiting the amount of queries performed by the user.
I am migrating from AccountA(source) to AccountB(Target), same region.
I ran templates so AccountB already has a RDS cluster but with no data. The db instance id is exactly same as what I have on the sourcing account.
**My goal is to have the same endpoint as before since we're retiring AccountA completely and I don't want to change codes side for the updated endpoint. **
I took a snapshot of the cluster(writer instance) then copy snapshot with a KMS key, shared it with AccountB. All good. Now, from the AccountB(target), copied snapshot and attempted to restore. I thought I could restore directly into the RDS cluster but I see that's not doable with restore as it always creates a new instance.
Then, I renamed the existing empty RDS cluster to something else to free up the DB instance ID, then renamed the temp cluster to the same name. It worked but this seems not efficient.
What's is the best practice for the RDS data migration ?
Clustered RDS ( writer - reader ) and Cross Account
I didn't create a snapshot for Reader. Will it be synced from the writer automatically once I restore?
Your experience is correct -- RDS Snapshots are restored as a new RDS instance (rather than loading data into an existing instance).
By "endpoint", if you are referring to the DNS Name used to connect to the database, then the new database will not have the same endpoint. If you want to preserve an endpoint, then you can create a DNS Name with a CNAME record that resolves to the DNS Name of the database. That way, you can change the CNAME record to point to a different endpoint without needing to change the code. (However, you probably haven't done this, so you'll need to change the code to point to the new DNS Name anyway, so it's the same amount of work.)
You are correct that you do not need to snapshot/copy the Readers -- you can simply create them from the new database. You will need to 'create' the Readers on the new database after the restore, since the Snapshot simply contains data for the main database.
I have a problem with a shared mongodb cluster: I try to get data via the nodejs implementation of mongodb. A few days ago, it worked just fine. But now, every getmore command I send to the cluster is very, very slow. So I thought: Maybe I just have to turn it off and on again.
So I tried to connect to the cluster with the mongo shell. Everything works fine, my user has the atlasAdmin role (with can be seen via db.getUser("admin")), but when I try to execute commands like db.shutdownServer() or show users, the server tells me that I'm not authorized. Even the command "db.auth("admin", ...pw...)" returns 1.
After some research, I found out I have to shutdown the server to have a chance to fix this problem. But without permission, how should I do it? Is there any other possibility to perform this, like a button on the atlas webapp or something?
Atlas is a hosted service, so the privileges are different vs. a bare metal MongoDB server. From MongoDB Database User Privileges This is the list of privileges of atlasAdmin:
readWriteAnyDatabase
readAnyDatabase
dbAdminAnyDatabase
clusterMonitor
cleanupOrphaned
enableSharding
flushRouterConfig
moveChunk
splitChunk
viewUser
shutdown privilege is part of the hostManager role, which is not included in the list above.
Depending on your type of Atlas deployment, here are the list of restricted commands/privileges:
Unsupported Commands in M0/M2/M5 Clusters
Unsupported Commands in M10+ Clusters
If you need to "turn on and off" your deployment, you might be able to use the Test Failover button if your type of deployment supports it. That button will step down the primary node and elect a new primary, which for most cases is almost equivalent to "turn off and on again".
I have an AWS account with a Postgres RDS database that represents the production environment for an app. We have another team that is building an analytics infrastructure in a different AWS account. They need to be able to pull data from our production database to hydrate their reports.
From my research so far, it seems there are a couple options:
Create a bash script that runs on a CRON schedule that uses pg_dump and pg_restore and stash that on an EC2 instance in one of the accounts.
Automate the process of creating a Snapshot on a schedule and then ship that to the other accounts S3 bucket. Then create a Lambda (or other script) that triggers when the snapshot is placed in the S3 bucket and restore it. Downside to this is we'd have to create a new RDS instance with each restore (since you can't restore a Snapshot to an existing instance), which changes the FQDN of the database (which we can mitigate using Route53 and a CNAME that gets updated, but this is complicated).
Create a read-replica in the origin AWS account and open up security for that instance so they can just access it directly (but then my account is responsible for all the costs associated with hosting and accessing it).
None of these seem like good options. Is there some other way to accomplish this?
I would suggest to use AWS Data Migration Service It can listen to changes on your source database and stream them to a target (https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html)
There is also a third-party blog post explaining how to set this up
https://medium.com/tensult/cross-account-and-cross-region-rds-mysql-db-replication-part-1-55d307c7ae65
Pricing is per hour, depending on the size of the replication EC2 instance. It runs in the target account, so it will not be on your cost center.
Regarding using the COPY command for populating a Redshift table with data from S3; I'm wondering if there is a reason for why you have to specify a role via its ARN which provides the permissions even though the Redshift cluster is already associated with that rule. This seems redundant to me but probably there is a reason for this. Hence my question.
This question arose upon reading the Redshift getting started guide; specifically regarding steps 2, 3 and 6.
It's not mandatory to reference an IAM Role when using the COPY command. This is one of several authorization methods available for the cluster to access external resources (e.g. files stored on S3). The reason for specifying the IAM_ROLE clause is to tell Redshift that this is the authorization method to use, you could alternatively specify ACCESS_KEY_ID/SECRET_ACCESS_KEY or CREDENTIALS.
https://docs.aws.amazon.com/redshift/latest/dg/copy-usage_notes-access-permissions.html
The reason you need to add the ARN for a specific IAM role is that it's possible to add more than one role to a cluster.