Currently, I am using DevOps Pipelines YAML to run a flyway cli to perform migrations to a single Database. When code gets pushed, yaml triggers and runs flyway which updates DB.
We are planning to create multiple instances of this DB. However, the DB instances are created dynamically. We plan to store connection strings in a master DB + Key Vault.
Is it possible to achieve the following?
Code gets commit
YAML queries master DB, and gets X number connection strings
YAML loops X, and runs flyway for each connection string
All DBs gets updated
I do not think there is such a way. An alternative I can think of is create a Console Application that does the querying + calling flyway, and YAML just calls this in the build server.
Related
Is it possible to apply DB Snapshot to existing DB using CloudFormation in one step?
I know it is possible if you run your CloudFormation twice - first time to rename existing resource and second time reverting the name back and set DBSnapshotIdentifier: !Ref DBSnapshot variable.
But it take to much time, is it possible to do it in one step?
The main goal - update database using snapshot from another database (for example apply STG db to DEV to have similar data)
Currently using GCP BigQuery UI for scheduled queries, everything is manually done.
Wondering if there's a way to automatically deploy to GCP using a config JSON that contains the scheduled query's parameters and scheduled times through github actions?
So far, this is one option I've found that makes it more "automated":
- store query in a file on Cloud Storage. When invoking Cloud Function, you read the file content and you perform a bigQuery job on it.
- have to update the file content to update the query
- con: read file from storage, then call BQ: 2 api calls and query file to manage
Currently using DBT in other repos to automate and make this process quicker: https://docs.getdbt.com/docs/introduction
Would prefer the github actions version though, just haven't found a good docu yet :)
I'm exploring the DACPAC feature on the SQL Server database deployments.
I'm using the EXTRACT action to get the DACPAC generated from the source and the PUBLISH action to deploy it to the target.
Extract
sqlpackage.exe /Action:Extract /SourceDatabaseName:%sourcedatabaseName% /SourceServerName:%sourceserverName% /TargetFile:%FilePath%
Publish
sqlpackage.exe /Action:Publish /SourceFile:%SourceFile% /TargetServerName:%serverName% /TargetDatabaseName:%databaseName%
Here, when I have new columns introduced in the source table when I do the DACPAC deployment, it works fine. The new columns are reflected in the target.
But, when I drop columns in the source and do the DACPAC deployment, the changes are not reflected. The column is not getting dropped in the target. Is it because I have data in that column?
In the other scenario, I have some test tables and test stored procedures in the source, when I generate DACPAC and do the deployment the same test tables and stored procedures are getting deployed in the target. Is there a way to restrict this?
So would like to understand what are all the limitations of using DACPAC?
Using SQL Server 2019.
Column removal from non-empty table could lead to data loss.
It could be overriden with: /p:BlockOnPossibleDataLoss=false
DacDeployOptions.BlockOnPossibleDataLoss Property
Get or set boolean that specifies whether deployment should stop if the operation could cause data loss.
True to stop deployment if possible data loss if detected; otherwise, false. Default is true.
We are planning to go for PostgreSQL RDS in AWS environment. There are some files in S3 which we will need to load every week. I don't see any option in AWS documentation where we can load data from S3 to PostgreSQL RDS. I see it is possible for Aurora but cannot find anything for PostgreSQL.
Any help will be appreciated.
One option is to use AWS Data Pipeline. It's essentially a JSON script that allows you to orchestrate the flow of data between sources on AWS.
There's a template offered by AWS that's setup to move data between S3 and MySQL. You can find it here. You can easily follow this and swap out the MySQL parameters with those associated with your Postgres instance. Data Pipeline simply looks for RDS as the type and does not distinguish between MySQL and Postgres instances.
Scheduling is also supported by Data Pipeline, so you can automate your weekly file transfers.
To start this:
Go to the Data Pipeline service in your AWS console
Select "Build from template" under source
Select the "Load S3 to MySQL table" template
Fill in the rest of the fields and create the pipeline
From there, you can monitor the progress of the pipeline in the console!
I am using Entity Framework (version 6.1.3) - Code First - for my application.
The application is hosted on the Azure platform, and uses Azure SQL Databases.
I have a database instance in two different regions, and I am using the Sync Preview to keep the data in sync.
Since the sync takes care of ensuring the data is kept synchronised, when I run a migration, I'd like the schema changes and seed to happen in only one database, and the schema changes only (with no seed) in the other.
Is this possible with the EF tooling, or do I need to move the seeding out to a manual script?
This is possible by spreading out your deployment.
if worker role 1 updates your database and seed
if after the sync worker role 2 connects to your other database it will see that the migration already took place.
One way to trigger this is to disable automatic migrations on all but 1 worker role. The problem is that you potentially have to deal with downtime/issues while part of your application landscape is updated/migrated but your database is still syncing.
(worker role can also be replaced by webjob , website etc )