According to Alembic's docs, the migration algorithm is trying to calculate the migration "path" to target revision from the version it finds in the alembic_version table. Upon checking that in the db of the service I'm working with, I found out that the current version that Alembic operates from is not in the version history, i.e. there is no migration script in projects migrations folder with that revision ID.
It seems that for this reason, when I added a manually written migration script (with empty template from alembic revision), calling alembic upgrade with this revision's ID failed silently without carrying out the migration. I got an output ending with
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
and exit code 1.
However, when I changed the value in alembic_version table to the penultimate version ID (the one revised by my own script), alembic upgrade head worked.
It is unclear whether or not the database actually was in the state corresponding with the revision ID I set manually. The script I migrated with only creates a table like this:
...
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql as pg
# revision identifiers, used by Alembic.
revision = '09d841dcd114'
down_revision = '450e39fe439d'
branch_labels = None
depends_on = None
def upgrade():
op.create_table('table_name',
sa.Column('id', pg.UUID(), autoincrement=False, nullable=False),
sa.Column('user_id', sa.INTEGER(), autoincrement=False, nullable=False),
...
sa.PrimaryKeyConstraint('id'),
sa.ForeignKeyConstraint(['user_id'], ['other_schema.users.id'], name='schemaname_tablename_user_id_fkey'),
schema='schema_name'
)
...
so it's partly reliant on existing database state, but does not determine the state exacly.
So my question is: is it normally possible that the alembic_version in db is not among existing revisions? Was that the reason why my previous attempts to migrate the db failed silently? What would be the correct solution for this problem? Setting the alembic version manually feels like an improper way to deal with this.
The alembic commands here were mostly given using Flask-Migrate, but it seems to not affect anything in this context.
There was a hotfix revision not merged into the main branch, but used on the service. That's where the missing version was
Related
I have a Spring Boot application that uses Flyway for database migrations in Postgres.
It's about four years old now, so we're talking Flyway 4.0.3, Spring Boot 1.3.x, and Postgres 9.x. Versions could probably be upgraded, but I'd like to fix any existing issues before doing that.
In the meantime, Postgres was upgraded to higher than 9.x. Unfortunately, with that, a few of the existing migrations became outdated as they contain deprecated syntax. So now starting the app with a fresh database (i.e. in a development environment) leads to those migrations failing. In production it is fine as those migrations already have already been applied and won't be again.
I am curious as to what the best practices are to go about this. I can't just go and fix the syntax in the existing migrations, as this will lead to the checksums in the production environment failing. I know repair is a thing, however I am unsure how it works and how to use it with Spring Boot.
Failing SQL:
UPDATE config
SET (description) = 'my description'
WHERE ...
Correct SQL:
UPDATE config
SET description = 'my description'
WHERE ...
Error:
Message : ERROR: source for a multiple-column UPDATE item must be a sub-SELECT or ROW() expression
EDIT 24/04/2020 Spring Boot solution:
After Grant Fitchey posted the correct answer about how to use repair for this below, I am just going to add how I did this with Spring Boot specifically. I just created a bean of FlywayMigrationStrategy that calls
repair before it calls migrate:
#Bean
public FlywayMigrationStrategy cleanMigrateStrategy() {
return flyway -> {
flyway.repair();
flyway.migrate();
};
}
When deploying this to the production environment, during startup the checksums in the schema_versions table in Postgres were fixed. In another version I will remove the flyway.repair(); line again, as otherwise obviously this would create the risk of applying invalid migrations.
You are looking for the repair option. I don't know directly how to call it through spring boot, but the documentation is here. This should take care of exactly what you're looking for.
So the first step in this case would be to fix the migrations so they execute correctly in the development environment. Development should now be fine, and flyway should migrate successfully.
On production you should now get a validation error because the checksums differ. Flyway repair will 'repair' the schema history table so that the checksums it has stored match the new ones on disk, and therefore flyway validation passes again.
Specifically what flyway repair is doing is making the schema history table match what you have on disk. It updates all the checksums for applied migrations to the checksums of the ones you have on disk (and therefore, only use this if you are confident the changes are identical). It also removes all failed migration entries from the table (again, only use this once you have cleaned up the database yourself).
I have created a Postgresql database A using liquibase changesets. Now, I'm creating an application that allows creating a new database B and copies the schema from database A in real-time including the liquibase changesets as the database can still be updated later. Note that at the time of the copied schema in database A could already be updated, making the base changesets outdated.
My main question would be:
How to copy PostgreSQL schema x from database a (dynamically generated at run-time) to b using liquibase? Database b could be on another server.
If it's not possible with liquibase, what other tools or approaches would make this possible?
--
Let me add more context:
We initialize a new database a schema using liquibase changeset.
We can add a new table and field to the database an at run-time. Or during the time when the application is running. For example, we add a new table people to the schema of database a, which is not originally in the changeset. This is done using liquibase classes too. So changeset is added to databasechangelog table.
Now, we create a new database b.
We want to import the schema of the database a to b, with people table.
I hope that is clear.
Thanks.
All schema changes must be run through your schema migration tool
The point of using a database schema migration tool such as Liquibase or Flyway is to have a “single source of truth” regarding the structure of your database tables. Your set of Liquibase changesets (or Flyway scripts) is supposed to be that single source of truth for your database.
If you are altering the structure of you database at runtime, such as adding a table named people, outside the scope of your migration tool, well, then you have violated the rules of the game. You will have defeated the purpose of using a schema migration tool. The intention of using a schema migration tool is that you make all schema changes through that tool.
If you need to add a table while running in production, you should be dropping the physical file for the Liquibase changeset (or Flyway script) into the file system of your database server environment, and then invoking Liquibase (or Flyway) to run a migration.
Perhaps you have been misunderstanding the sequence of events:
If you have built a database on server "A", that means you installed Postgres, created an empty database, then installed the collection of Liquibase changesets you have carefully built, then ran a Liquibase migration operation on that server.
When you go to create a database on server "B", you should be following the same steps: Install Postgres, create an empty database, installing the very same collection of Liquibase changesets, and then running a Liquibase migration operation.
Alternatively, if making a copy of server "A" to create server "B", that copy should include the exact same Liquibase changesets. So at the end of your copy process, the two databases+changesets are identical.
Here's how I solved this problem of mine using the Liquibase Java library:
1.) Export the changelog from the source database into a temporary file (XML).
Liquibase liquibase = new Liquibase(liquibaseOutFile.getAbsolutePath(), new FileSystemResourceAccessor(), sourceDatabase);
liquibase.generateChangeLog(catalogAndSchema, changeLogWriter, new PrintStream(liquibaseOutFile.getAbsolutePath()), null);
2.) Execute the temporary file to the new data source.
Liquibase targetLiquibase = new Liquibase(liquibaseOutFile.getAbsolutePath(), new FileSystemResourceAccessor(), targetDatabase);
Contexts context = new Contexts();
targetLiquibase.update(context);
Here's the complete code: https://czetsuya-tech.blogspot.com/2019/12/generate-postgresql-schema-using-java.html
I work on a team using alembic to manage db migrations. I recently pulled master, and tried to run alembic upgrade heads. I got the following message;
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
ERROR [alembic.util.messaging] Requested revision a04c53fd8c74 overlaps with other requested revisions 453d88f67d34
FAILED: Requested revision a04c53fd8c74 overlaps with other requested revisions 453d88f67d34
I got the same message when I tried to run alembic downgrade -1. Running alembic history prints this;
453d88f67d34 -> a04c53fd8c74 (label_1, label_2) (head), Create such and such tables.
2f15c778e709, 9NZSZX -> 453d88f67d34 (label_1, label_2) (mergepoint), empty message
b1861bb8b23f, b8aa3acdf260 -> 2f15c778e709 (label_1, label_2) (mergepoint), Merge heads b18 and b8a
(...many more old revisions)
which to me looks like a perfectly fine history. alembic heads reports a04c53fd8c74 (label_1, label_2) (head).
The only thing that looks odd to me is that my alembic version db has two values in it;
my_postgres=# SELECT * FROM alembic_version;
version_num
--------------
a04c53fd8c74
453d88f67d34
(2 rows)
The only reference I can find from googling the exception is the source code, which I'd rather not read through.
How could this situation have come about? How should I fix it? What does "overlaps" mean?
I "fixed" it by deleting the older version number in the database;
my_postgres=# DELETE FROM alembic_version WHERE version_num = '453d88f67d34';
DELETE 1
my_postgres=# SELECT * FROM alembic_version;
version_num
--------------
a04c53fd8c74
(1 row)
I can now run upgrades and downgrades. My history and heads look the same. But I still don't know why it happened, or whether there is some db state that is subtly messed up, so if anyone has a better answer please post it!
For those who find this, this happened to me as well because I tried restoring my database to an older version of it without first dropping it. I believe you should only ever have one row in your alembic_version table with the version_num of whatever version your database should currently be at.
So when I restored my database and did not drop it first, rather than replacing the current version number, it added a new row. In order to fix it I had to delete the incorrect version from my alembic table (whatever the version number of the database was that I should have dropped first). ... just to give a little more context to Altair's answer.
The alembic_version table should have only one row for consistency.
If you restore a database data from another version, restore operation WILL ADD another row.
After you restore the data, delete the row which was added from the
restore operation.
This one worked for me:
delete from alembic_version where version_num='e69ad4aec63b';
Find the last applied migration and copy revision name.
Delete every records from alembic_version table.
Insert the copied revision name as the only revision name (RECORD 1).
alembic upgrade head
When I try
alembic upgrade head
Alembic runs the previous migration script, obviously raising an error because my schema has changed.
In my database, I have version_num set to 48957fdfe8d5. After running
alembic revision -m '<my message>'
Alembic created the new script file—the one I want to run—with this at the top
revision = '28cc06993b73'
down_revision = '4d5f9ba76c5e'
In other words, everything looks good. So why is it clearly running the code in 4d5f9ba76c5e rather than 28cc06993b73? I have also tried
alembic upgrade 28cc06993b73
But it still runs the code in 4d5f9ba76c5e. Here is that log:
$ alembic upgrade 28cc06993b73
INFO [alembic.migration] Context impl MySQLImpl.
INFO [alembic.migration] Will assume non-transactional DDL.
Starting in DEBUG mode
INFO [alembic.migration] Running upgrade 48957fdfe8d5 -> 4d5f9ba76c5e, Breaking up metadata into required and optional
Also, if I check Alembic's history, I see that head is on 28cc06993b73:
$ alembic history
Starting in DEBUG mode
4d5f9ba76c5e -> 28cc06993b73 (head), creating soft file table
48957fdfe8d5 -> 4d5f9ba76c5e, Breaking up metadata into required and optional
<base> -> 48957fdfe8d5, Init
Thanks in advance.
Well firstly you said that your version number is 48957fdfe8d5 in the db so an upgrade from 48957fdfe8d5 -> 4d5f9ba76c5e is expected based on your alembic history.
You also say "Alembic runs the previous migration script, obviously raising an error because my schema has changed." What error is it showing? Did you manually change the db so that it reflects what is supposed to happen in 28cc06993b73? If that is the case then you could run
alembic stamp head
to get your db in sync with alembic migrations
I'd like to make a migration for a Flask app. I am using Alembic.
However, I receive the following error.
Target database is not up to date.
Online, I read that it has something to do with this.
http://alembic.zzzcomputing.com/en/latest/cookbook.html#building-an-up-to-date-database-from-scratch
Unfortunately, I don't quite understand how to get the database up to date and where/how I should write the code given in the link.
After creating a migration, either manually or as --autogenerate, you must apply it with alembic upgrade head. If you used db.create_all() from a shell, you can use alembic stamp head to indicate that the current state of the database represents the application of all migrations.
This Worked For me
$ flask db stamp head
$ flask db migrate
$ flask db upgrade
My stuation is like this question, When I execute "./manage.py db migrate -m 'Add relationship'", the error occused like this "
alembic.util.exc.CommandError: Target database is not up to date."
So I checked the status of my migrate:
(venv) ]#./manage.py db heads
d996b44eca57 (head)
(venv) ]#./manage.py db current
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
715f79abbd75
and found that the heads and the current are different!
I fixed it by doing this steps:
(venv)]#./manage.py db stamp heads
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running stamp_revision 715f79abbd75 -> d996b44eca57
And now the current is same to the head
(venv) ]#./manage.py db current
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
d996b44eca57 (head)
And now I can do the migrate again.
This can be solved bby many ways :
1 To fix this error, delete the latest migration file ( a python file) then try to perform a migration afresh.
If issue still persists try these commands :
$ flask db stamp head # To set the revision in the database to the head, without performing any migrations. You can change head to the required change you want.
$ flask db migrate # To detect automatically all the changes.
$ flask db upgrade # To apply all the changes.
$ flask db stamp head # To set the revision in the database to the head, without performing any migrations. You can change head to the required change you want.
$ flask db migrate # To detect automatically all the changes.
$ flask db upgrade # To apply all the changes.
You can find more info at the documentation https://flask-migrate.readthedocs.io/en/latest/
I had to delete some of my migration files for some reason. Not sure why. But that fixed the problem, kind of.
One issue is that the database ends up getting updated properly, with all the new tables, etc, but the migration files themselves don't show any changes when I use automigrate.
If someone has a better solution, please let me know, as right now my solution is kind of hacky.
I did too run into different heads and I wanted to change one of the fields from string to integer, so first run:
$ flask db stamp head # to make the current the same
$ flask db migrate
$ flask db upgrade
It's solved now!
To fix this error, delete the latest migration file ( a python file) then try to perform a migration afresh.
This can also happen if you, like myself, have just started a new project and you are using in-memory SQLite database (sqlite:///:memory:). If you apply a migration on such a database, obviously the next time you want to say auto-generate a revision, the database will still be in its original state (empty), so alembic will be complaining that the target database is not up to date. The solution is to switch to a persisted database.
I also had the same problem input with flask db migrate
I used
flask db stamp head
and then
flask db migrate
Try to drop all tables before execute the db upgrade command.
To solve this, I drop(delete) the tables in migration and run these commands
flask db migrate
and
flask db upgrade