Couple of days ago we made a mistake. We have a kubernetes cluster with a pipeline that times out in 25 minutes, meaning if the deployment wasn't done in 25 minutes, it will fail. We deployed a flyway migration that involves some queries that run for more than an hour. Stupid, I know. now we ran the queries in the migration manually, We want to manually mark the flyway migration as done, otherwise redeployment won't work. Is there a way this could be done?
So we ended up manually inserting a migration row in the database. flyway keeps a table flyway_schema_history in your schema. If you manually insert a row there it will skip the migration. The only tricky part is calculating the checksum. You can either migrate locally, get the checksum and inject it the live database, or just re-calculate the checksum on your own.
You will find how they calculate the checksum in the AbstractLoadableResource class.
Related
I read a lot about best practices applying DataBase Migrations for Kubernetes Apps. Actually there are three common solutions:
Separate CI/CD stage for DB Migrations (usually we run it before deploy new App version but in some cases we can deploy first, it doesn't matter for current question). Just decouple app and migrations, and run them in different stages one by one.
InitContainer: we can run DB migraitions before main app container starts. It is a good solution, but we have to be careful to tune it to run migrations once for all pods and replicas, do not run on pod restarts and check probes (kubelet can kill container on timeout while migration still running)
Separate Kubernetes Job or Helm Hook. Is close to initContainer, cannot share data with app container (but it is not really necessary for DB Migrations, so it's ok). Must be careful with timeout too - Helm can kill Job before migration is completed.
But the question - how to apply rollbacks for these solutions???
Lets try to form my ideas:
Separate stage in CI/CD: we can save previous migration name, and then rollback to it in another stage. Pipeline: Migrations -> Deploy -> Test -> Rollback DB and ReDeploy
But for InitContainer and HelmHook I have to idea how to realise Rollback! Do we need additional containers for that? Does helm rollback affect DB too (don't think so). What are the best practices for that?
I will be very glad to any suggestions!
I started investigating this issue too, and it looks like all manuals, tutorials and practices are meant to be forward-only. Should you ever need to rollback a database migration, you face with the limitations of Helm rollback. There is an open issue in Helm (https://github.com/helm/helm/issues/5825) that addresses this very problem, but it looks like no solution so far.
So probably Helm rollback is not a suitable mechanism for databases as you end up with creating a batch/v1.Job with container image that knows nothing about the changes you need to rollback. I use Liquibase and it requires that the changesets you need to rollback are present in the changelog.
I think for now this problem can be solved with forward-only approach:
Build the container image that introduces a new changeset into your database.
In release values for Helm specify the container image that should be run to perform a migration, and a command (like liquibase update).
If the migration fails and you see that rollback is required, you do not invoke helm rollback. You deploy a new release, and in its values you specify the container image that should perform the database migration rollback, and the command (like liquibase rollback <tag>). I am myself going this way now and it looks like the best solution I could came up with.
Hope that helps.
I have set up a Fivetran connector to connect to a PostgreSQL database in an EC2 server and snowflake. The connection seems to work (no error), but the data is not really updated.
On the EC2 server, every day a script will pull down the latest dump of our app production database and restore it on the EC2 server, and then the Fivetran connector is expected to sync the database to snowflake. But the data after the first setup date is not synced with the snowflake. Could FiveTran be used in such a setup? If so, do you know what may be the issue of the sync failing?
Could FiveTran be used in such a setup?
Yes, but it's not ideal.
If so, do you know what may be the issue of the sync failing?
It's hard to answer this question without more context, however: Fivetran uses logging to replicate your DB (WAL in the case of PostgreSQL), so if you restore the DB every single day Fivetran will loose track of the changes and will need to re-sync the whole database.
The point made by NickW is completely valid, why not replicate from the DB? I assume the answer is along the lines of the data you need to modify. You can use column blocking and/or hashing to prevent sensible data from being transfered, or to obfuscate it before it's flushed to Snowflake.
Build servers are generally detached from the VPC running the instance. Be it Cloud Build on GCP, or utilising one of the many CI tools out there (CircleCI, Codeship etc), thus running DB schema updates is particularly challenging.
So, it makes me wonder.... When's the best place to run database schema migrations?
From my perspective, there are four opportunities to automatically run schema migrations or seeds within a CD pipeline:
Within the build phase
On instance startup
Via a warm-up script (synchronously or asynchronously)
Via an endpoint, either automatically or manually called post deployment
The primary issue with option 1 is security. With Google Cloud Sql/Google Cloud Build, it's been possible for me to run (with much struggle), schema migrations/seeds via a build step and a SQL proxy. To be honest, it was a total ball-ache to set up...but it works.
My latest project is utilising MongoDb, for which I've connected in migrate-mongo if I ever need to move some data around/seed some data. Unfortunately there is no such SQL proxy to securely connect MongoDb (atlas) to Cloud Build (or any other CI tools) as it doesn't run in the instance's VPC. Thus, it's a dead-end in my eyes.
I'm therefore warming (no pun intended) to the warm-up script concept.
With App Engine, the warm-up script is called prior to traffic being served, and on the host which would already have access via the VPC. The warmup script is meant to be used for opening up database connections to speed up connectivity, but assuming there are no outstanding migrations, it'd be doing exactly that - a very light-weight select statement.
Can anyone think of any issues with this approach?
Option 4 is also suitable (it's essentially the same thing). There may be a bit more protection required on these endpoints though - especially if a "down" migration script exists(!)
It's hard to answer you because it's an opinion based question!
Here my thoughts about your propositions
It's the best solution for me. Of course you have to take care to only add field and not to delete or remove existing schema field. Like this, you can update your schema during the Build phase, then deploy. The new deployment will take the new schema and the obsolete field will no longer be used. On the next schema update, you will be able to delete these obsolete field and clean your schema.
This solution will decrease your cold start performance. It's not a suitable solution
Same remark as before, in addition to be sticky to App Engine infrastructure and way of working.
No real advantage compare to the solution 1.
About security, Cloud Build will be able to work with worker pool soon. Still in alpha but I expect in the next month an alpha release of it.
The Flyway FAQ says I can't make structural changes to the DB outside of Flyway. Does that includes creating new partitions for an existing table?
If so, is there any way to use Flyway to automatically create daily partitions as required? Bear in mind that the process will be running for more than one day so it's not something that can be just triggered on start-up.
We're stuck with Postgres 9.6 at the mo, so the partitions have to be created manually.
When I take a backup of a database (SQL Server), is there any way that I can include a scheduled job in the backup?
I have a database with stored procedures and a maintenance job that runs some of the stored procedures nightly.
I would like to achieve a minimal effort to schedule the job, when the .bak file is restored into a server, back as a database.
I don't have the quick button to click for your problem, but I think (not sitting in front of it right now) You can right click a job and get a script for its creation including scheduling specifics. I don't know how to include a job in a backup and how to restore it, though. I think restoring a job would require a script with the CREATE for the job to be run.
You can also back up the msdb database. The msdb database is where all the jobs live, it is one of the system databases, and then restore your database plus msdb.