Flyway - Flyway Schema migration failed - postgresql

I have successfully configured spring boot with a new project to work
with flyway
Migrated with the Postgres database from the version 0001.0 to 0008.0
I have made manually alter the script in local but
flyway migration getting failed.
Sample Error message:
org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'flywayInitializer' defined in class path
resource
[org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]:
Invocation of init method failed; nested exception is
org.flywaydb.core.api.FlywayException: Validate failed: Migration
checksum mismatch for migration version 0006.0
How to alter the database tables without affecting flyway script from the flyway_schema_history?
For example, I need to change the table name using alter command but executing the flyway migration script without failed.
Any suggestions, Kindly appreciated.
Note:- I don't want to remove the script entries from the table flyway_schema_history.

There are a few ways to do this:-
1) Create a new script file with incremented version. Put your DDL commands to alter the table in this file. Then run migration.
2) If you don't want to delete the entry from the schema_version table, you can change the checksum value in that table. To calculate checksum, use the following method copied from org.flywaydb.core.internal.resolver.sql.SqlMigrationResolver. You can pass null for resource parameter:-
/**
* Calculates the checksum of this string.
*
* #param str The string to calculate the checksum for.
* #return The crc-32 checksum of the bytes.
*/
/* private -> for testing */
static int calculateChecksum(Resource resource, String str) {
final CRC32 crc32 = new CRC32();
BufferedReader bufferedReader = new BufferedReader(new StringReader(str));
try {
String line;
while ((line = bufferedReader.readLine()) != null) {
crc32.update(line.getBytes("UTF-8"));
}
} catch (IOException e) {
String message = "Unable to calculate checksum";
if (resource != null) {
message += " for " + resource.getLocation() + " (" + resource.getLocationOnDisk() + ")";
}
throw new FlywayException(message, e);
}
return (int) crc32.getValue();
}
3) If you are using Flyway Pro version 5+, you can rollback the migration https://flywaydb.org/getstarted/undo.
The answers here are outdated but can still help you.

It sounds like you might be in one of two situations:
You want to re-run a versioned migration. This isn't really how flyway works, as Kartik has suggested, create a new versioned migration to alter the table.
A migration file has been modified and you want to leave it that way and run new ones (eg 0009.0). In this situation you can try:
Run repair. Which will recalculate the checksums (among other things).
Turn off the validateOnMigrate option which will not fail a migration if there are modified migration files.

To solve this error locally without dropping your whole db:
Fix the migration error which caused the root problem
Disconnect your db server
Open the table "flyway_schema_history" which would be created automatically
Delete the rows with the versions that are causing the mismatch problem
Open the tables that have columns depending on the conflict migrations and drop those columns (if needed)
Run again your db server with the new migrations

Related

PostgreSQL database Versioning using Liquibase

I'm trying to use Liquibase version 3.6 to maintain database version changes. I'm able to execute database version changes when i need to execute single file changes.
I have using below code to execute version changes successfully, but my intention is to dynamically execute the change logs from a folder and not single file. I'm using only Java for all configuration of Liquibase
#Startup
#Singleton
#TransactionManagement(TransactionManagementType.BEAN)
public class InitializerBean {
#Resource(mappedName = "java:/M2M_RESOURCES")
private DataSource ds;
#PostConstruct
protected void bootstrap() {
ResourceAccessor resourceAccessor = new ClassLoaderResourceAccessor(getClass().getClassLoader());
try (Connection connection = ds.getConnection()) {
JdbcConnection jdbcConnection = new JdbcConnection(connection);
Database db = DatabaseFactory.getInstance().findCorrectDatabaseImplementation(jdbcConnection);
Liquibase liquiBase = new Liquibase("data/liquibase/", resourceAccessor, db);
liquiBase.update("Development");
} catch (SQLException | LiquibaseException e) {
}
}
}
When the first parameter of Liquibase class is single file , liquibase is able to execute changes but when I intent to execute all file of single folder is not able to track and execute changes.
I'm using JDK 1.8 and file in data/liquibase/ is dbChangelog.sql and dbChangelog_2.sq. This all code is deployed on Wildfly 10 as part of ear archive

Cannot run tests on h2 in-memory database, rather it runs on PostgreSQL

(I have multiple related questions, so I highlight them as bold)
I have a play app.
play: 2.6.19
scala: 2.12.6
h2: 1.4.197
postgresql: 42.2.5
play-slick/play-slick-evolutions: 3.0.1
slick-pg: 0.16.3
I am adding a test for DAO, and I believe it should run on an h2 in-memory database that is created when tests start, cleared when tests end.
However, my test always runs on PostgreSQL database I configure and use.
# application.conf
slick.dbs.default.profile="slick.jdbc.PostgresProfile$"
slick.dbs.default.db.driver="org.postgresql.Driver"
slick.dbs.default.db.url="jdbc:postgresql://localhost:5432/postgres"
Here is my test test/dao/TodoDAOImplSpec.scala.
package dao
import play.api.inject.guice.GuiceApplicationBuilder
import play.api.test.{Injecting, PlaySpecification, WithApplication}
class TodoDAOImplSpec extends PlaySpecification {
val conf = Map(
"slick.dbs.test.profile" -> "slick.jdbc.H2Profile$",
"slick.dbs.test.db.driver" -> "org.h2.Driver",
"slick.dbs.test.db.url" -> "jdbc:h2:mem:test;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=FALSE"
)
val fakeApp = new GuiceApplicationBuilder().configure(conf).build()
//val fakeApp = new GuiceApplicationBuilder().configure(inMemoryDatabase()).build()
//val fakeApp = new GuiceApplicationBuilder().configure(inMemoryDatabase("test")).build()
"TodoDAO" should {
"returns current state in local pgsql table" in new WithApplication(fakeApp) with Injecting {
val todoDao = inject[TodoDAOImpl]
val result = await(todoDao.index())
result.size should_== 0
}
}
}
For fakeApp, I try all three, but none of them work as expected - my test still runs on my local PostgreSQL table (in which there are 3 todo items), so the test fails.
What I have tried/found:
First, inMemoryDatabase() simply returns a Map("db.<name>.driver"->"org.h2.Driver", "db.<name>.url"->""jdbc:h2:mem:play-test-xxx"), which looks very similar to my own conf map. However, there are 2 main differeneces:
inMemoryDatabase uses db.<name>.xxx while my conf map uses slick.dbs.<name>.db.xxx. Which one should be correct?
Second, rename conf map's keys to "slick.dbs.default.profile", "slick.dbs.default.db.driver" and "slick.dbs.default.db.url" will throw error.
[error] p.a.d.e.DefaultEvolutionsApi - Unknown data type: "status_enum"; SQL statement:
ALTER TABLE todo ADD COLUMN status status_enum NOT NULL [50004-197] [ERROR:50004, SQLSTATE:HY004]
cannot create an instance for class dao.TodoDAOImplSpec
caused by #79bg46315: Database 'default' is in an inconsistent state!
The finding is interesting - is it related to my use of PostgreSQL ENUM type and slick-pg? (See slick-pg issue with h2). Does it mean this is the right configuration for running h2 in-memory tests? If so, the question becomes How to fake PostgreSQL ENUM in h2.
Third, I follow this thread, run sbt '; set javaOptions += "-Dconfig.file=conf/application-test.conf"; test' with a test configuration file conf/application-test.conf:
include "application.conf"
slick.dbs.default.profile="slick.jdbc.H2Profile$"
slick.dbs.default.db.driver="org.h2.Driver"
slick.dbs.default.db.url="jdbc:h2:mem:test;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=FALSE"
Not surprisingly, I get the same error as the 2nd trial.
It seems to me that the 2nd and 3rd trials point to the right direction (Will work on this). But why must we set name to default? Any other better approach?
In play the default database is default. You could however change that to any other database name to want, but then you need to add the database name as well. For example, I want to have a comment database that has the user table:
CREATE TABLE comment.User(
id int(250) NOT NULL AUTO_INCREMENT,
username varchar(255),
comment varchar(255),
PRIMARY KEY (id));
Then I need to have the configuration of it to connect to it (add it to the application.conf file):
db.comment.url="jdbc:mysql://localhost/comment"
db.comment.username=admin-username
db.comment.password="admin-password"
You could have the test database for your testing as mentioned above and use it within your test.
Database Tests Locally: Why not have the database, in local, as you have in production? The data is not there and running the test on local does not touch the production database; why you need an extra database?
Inconsistent State: This is when the MYSQL you wrote, changes the state of the current database within the database, that could be based on creation of a new table or when you want to delete it.
Also status_enum is not recognizable as a MySQL command obviously. Try the commands you want to use in MySQL console if you are not sure about it.

SQL Timeout when running an Entity Framework Migration

I am currently experiencing a SQL Timeout when running a SQL() command inside of an EF Migration.
Situation:
I am replacing a bunch (>50) tables with one table, and need to convert the data from those tables I'm about to drop into the new table. I've organized the Migration the following way:
1.Create the new table.
In the same Migration, use the SQL() function to run a sql script that migrates the data.
3.Drop all the old tables.
Currently, the migration gives the following error:
System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
The statement has been terminated. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
The error is happening in an environment where I give them an installer, and they run it without me involved, so I can't manually run individual migrations, and pause in the middle to run the SQL script.
Is there any way to change the timeout for a connection, or get around this issue?
Environment:
EF 6.0 Code First
SQL Server 2012
See this answer.
Use Configuration.cs file to set custom time out:
internal sealed class Configuration :
DbMigrationsConfiguration<ApplicationDbContext>
{
public Configuration()
{
AutomaticMigrationsEnabled = false;
ContextKey = "YourDbContext";
// New timeout in seconds
this.CommandTimeout = 60 * 5;
}
}
With this method, you can change the timeout only for migration and not everyone using your default connection string.
You should be able to set the connection timeout in the connection string, something like:
Connection Timeout=180;

Getting DB2 SQL Error while firing a select query using Worklight 6.1.0?

I am trying to connect to DB2 in my local LAN using worklight 6.1.0 and firing a Select Query for lookup of data if exist. But i am getting below error:
{
"errors": [
"Runtime: DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704, SQLERRMC=DATABASE_NAME.REGISTRATION, DRIVER=3.58.82.\nPerformed query:\nSELECT * FROM DATABASE_NAME.registration where DATABASE_NAME.registration.Mob_No = ?"
],
"info": [
],
"isSuccessful": false,
"warnings": [
]
}
My SQL adapter configuration looks like below:
<connectionPolicy xsi:type="sql:SQLConnectionPolicy">
<!-- Example for using a JNDI data source, replace with actual data source name -->
<!-- <dataSourceJNDIName>java:/data-source-jndi-name</dataSourceJNDIName> -->
<!-- Example for using MySQL connector, do not forget to put the MySQL connector library in the project's lib folder -->
<dataSourceDefinition>
<driverClass>com.ibm.db2.jcc.DB2Driver</driverClass>
<url>jdbc:db2://172.21.11.129:50000/MOBILEDB</url>
<user>db2admin</user>
<password>Newuser123</password>
</dataSourceDefinition>
</connectionPolicy>
And js file which has procedure looks like:
var selectStatement1 = "SELECT * FROM DATABASE_NAME.registration where DATABASE_NAME.registration.Mob_No = ?";
var procStmt1 = WL.Server.createSQLStatement(selectStatement1);
function registrationLookup(mobile){
WL.Logger.debug("Inside registrationLookup");
return WL.Server.invokeSQLStatement(
{
preparedStatement : procStmt1,
parameters : [mobile]
}
);
}
I did some Research about connecting DB2 with Worklight and came to know that i need to put below data in worklight.properties file.
wl.db.username=db2admin
wl.db.type=DB2
wl.db.password=Newuser123
wl.db.driver=com.ibm.db2.jcc.DB2Driver
But after adding it, i am not able to deploy Adapter and error says 'db2admin' does not exist. So i have skipped this step in the context of current question. But after going through error which i am getting without adding this worklight.properties data it seems to me that 'Object doesn't exist' as per http://www-01.ibm.com/support/docview.wss?uid=swg21613531 or user table does not exist. Any suggestion would be helpful.
NOTE:
My IP address is 172.21.11.125 from where i am invoking Adapter for DB2.
DB2 instance is running on 172.21.11.129 # 50000.
Already Added db2jcc_license_cu_9.5.jar & db2jcc_9.5.jar in server/lib. It had name appended with '_9.5' which i have removed from both jar and kept only db2jcc_license_cu.jar and db2jcc.jar.
The error message is saying that your SQL statement is invalid, and I therefore infer that your connection to the DB is fine.
To diagnose this first run the SQL using DB2 command line or other tools. My guess is you mean
LARSEN.registration
whereas you are saying DATABASE_NAME.registration
Got Answer to my own Question and its really interesting one. Thanks to https://stackoverflow.com/users/2260967/glen-misquith [Glen Misquith]
Problem was with SQL Query framing which is Different what i did with MYSQL.
Adapter's Java Script file:
var selectStatement4 = "UPDATE \"LARSEN\".\"registration\" SET \"LARSEN\".\"registration\".\"Pass\"=?, \"LARSEN\".\"registration\".\"Re_Pass\"=? WHERE \"User_Name\" = ?";
var procStmt5 = WL.Server.createSQLStatement(selectStatement4);
function updatePassword(username,pass,repass){
WL.Logger.debug("Inside updatePassword "+username+" "+pass+" "+repass);
return WL.Server.invokeSQLStatement(
{
preparedStatement : procStmt5,
parameters : [pass,repass,username]
}
);
}
This is quite a Strange thing Slashes need to be used in SQL Statement while Preparing it.
I would really like to understand this behavior of DB2. And also i cant directly write 'Select * from schema.table_name' and precisely write column name from which data needs to be fetched.

How to detach a LocalDB (SQL Server Express) file in code

When using LocalDB .mdf files in deployment you will often want to move, delete or backup the database file.
It is paramount to detach this file first as simply deleting it will cause errors because LocalDB still keeps a registration of it.
So how is a LocalDB .mdf file detached in code?
I had to string together the answer from several places, so I wil post it here:
Mind, manually detaching the .mdf file from Visual Studio is possible after manually deleting it before detachment by going through SQL Server Object Explorer.
''' <summary>
''' Detach a database from LocalDB. This MUST be done prior to deleting it. It must also be done after a inadvertent (or ill advised) manual delete.
''' </summary>
''' <param name="dbName">The NAME of the database, not its filename.</param>
''' <remarks></remarks>
Private Sub DetachDatabase(dbName As String)
Try
'Close the connection to the database.
myViewModel.CloseDatabase()
'Connect to the MASTER database in order to excute the detach command on it.
Dim connectionString = String.Format("Data Source=(LocalDB)\v11.0;Initial Catalog=master;Integrated Security=True")
Using connection As New SqlConnection(connectionString)
connection.Open()
Dim cmd = connection.CreateCommand
'--Before the database file can be detached from code the workaround below has to be applied.
'http://web.archive.org/web/20130429051616/http://gunnalag.wordpress.com/2012/02/27/fix-cannot-detach-the-database-dbname-because-it-is-currently-in-use-microsoft-sql-server-error-3703
cmd.CommandText = String.Format("ALTER DATABASE [{0}] SET OFFLINE WITH ROLLBACK IMMEDIATE", dbName)
cmd.ExecuteNonQuery()
'--
'--Now detach
cmd.CommandText = String.Format("exec sp_detach_db '{0}'", dbName)
cmd.ExecuteNonQuery()
'--
End Using
Catch ex As Exception
'Do something meaningful here.
End Try
End Sub
I had the same issue and was thinking of how to deal with it.
There are 3 approaches.
Detach at the end of (or during) working with database
I didn't find the way to close connection in LinqToSQL, but actually it is not needed. Simply execute the following code:
var db = #"c:\blablabla\database1.mdf";
using (var master = new DataContext(#"Data Source=(LocalDB)\v11.0;Initial Catalog=master;Integrated Security=True"))
{
master.ExecuteCommand(#"ALTER DATABASE [{0}] SET OFFLINE WITH ROLLBACK IMMEDIATE", db);
master.ExecuteCommand(#"exec sp_detach_db '{0}'", db);
}
and make sure nothing will try to query db after (or you get it attached again).
Detach on start
Before you made any connection to db, detaching is as simple as:
var db = #"c:\blablabla\database1.mdf";
using (var master = new DataContext(#"Data Source=(LocalDB)\v11.0;Initial Catalog=master;Integrated Security=True"))
master.ExecuteCommand(#"exec sp_detach_db '{0}'", db);
This suit very well to my needs, because I do not care about delay to start application (because in this case I will have to attach to db always), but it will fix any kind of
System.Data.SqlClient.SqlException (0x80131904): Database 'c:\blablabla\database1.mdf' already exists. Choose a different database name.
which occurs, if database file is delete and you try to create it programmatically
// DataContext
if (!DatabaseExists())
CreateDatabase();
Another way
You can also run command line tool sqllocaldb like this:
var start = new ProcessStartInfo("sqllocaldb", "stop v11.0");
start.WindowStyle = ProcessWindowStyle.Hidden;
using (var stop = Process.Start(start))
stop.WaitForExit();
start.Arguments = "delete v11.0";
using (var delete = Process.Start(start))
delete.WaitForExit();
It will stop the server, detaching all databases. If you have other application using LocalDB, then they will experience attaching delay next time when they try to do query.
Here is my solution for EntityFramework Core 1.0
As you see the database name can be used with its full file path.
var dbf = fileDlg.FileName;
var options = new DbContextOptionsBuilder();
options.UseSqlServer($#"Server=(localdb)\mssqllocaldb;Initial Catalog=master;MultipleActiveResultSets=False;Integrated Security=True");
using (var master = new DbContext(options.Options))
{
master.Database.ExecuteSqlCommand($"ALTER DATABASE [{dbf}] SET OFFLINE WITH ROLLBACK IMMEDIATE");
master.Database.ExecuteSqlCommand($"exec sp_detach_db '{dbf}'");
}