In our project, we handle DB (Postgres) migrations using EFCore migrations (but we write them ourselves, they are not autogenerated). Example script:
// it is making column nullable
migrationBuilder.AlterColumn<string>(
name: "discount_type",
table: some_table_name,
nullable: true,
schema: "some_schema");
// it is making column nullable
migrationBuilder.AlterColumn<decimal>(
name: "discount_value",
table: some_table_name,
nullable: true,
schema: "some_schema");
migrationBuilder.AddForeignKey(
name: "some_key_name_fk",
table: some_table_name,
column: "code_id",
schema: "some_schema",
principalTable: some_table_name_2,
principalColumn: "id",
principalSchema: "some_schema",
onDelete: ReferentialAction.Restrict);
migrationBuilder.DropColumn(
name: "description_english",
table: some_table_name,
schema: Constants.Schema);
migrationBuilder.DropColumn(
name: "description_german",
table: some_table_name,
schema: "some_schema");
We found that during the migration, our service become unresponsive for few minutes. All queries (SELECT queries in majority) where giving timeout. 100% requests failed. Please note that some_table_name table has over 120 million records.
My question is if EFCore migration somehow locks the entire DB schema, even preventing it from reading data? If 'yes' - how to prevent it?
Related
I am using .net5.0 EF
I have a class AppUser and extend it to IdentityUser
public class AppUser : IdentityUser
{
}
I am using command to generate EF migrations
dotnet ef migrations add "myMessage" -p Persistence -s API
I have deleted all previous migration files, so this is a new migration.
I also deleted the DB as well.
I am ablle to successfullly generate new db in sqlLite.
But when I am trying to do the same in production / testing server SQL, it gives me below issue.
fail: Microsoft.EntityFrameworkCore.Database.Command[20102]
Failed executing DbCommand (38ms) [Parameters=[], CommandType='Text', CommandTimeout='60']
CREATE TABLE [AspNetRoles] (
[Id] TEXT NOT NULL,
[Name] TEXT NULL,
[NormalizedName] TEXT NULL,
[ConcurrencyStamp] TEXT NULL,
CONSTRAINT [PK_AspNetRoles] PRIMARY KEY ([Id])
);
fail: API.Program[0]
An error occured during migration
Microsoft.Data.SqlClient.SqlException (0x80131904): Column 'Id' in table 'AspNetRoles' is of a type that is invalid for use as a key column in an index.
Could not create constraint or index. See previous errors.
at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at Microsoft.Data.SqlClient.SqlCommand.InternalEndExecuteNonQuery(IAsyncResult asyncResult, Boolean isInternal, String endMethod)
at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryInternal(IAsyncResult asyncResult)
at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryAsync(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
The error relates to something in AspNetRoles table
But I am not even touching anything in this AspNetRoles table.
Can anyone help me, please?
Thanks
Autogenerated migration code below
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.CreateTable(
name: "AspNetRoles",
columns: table => new
{
Id = table.Column<string>(type: "TEXT", nullable: false),
Name = table.Column<string>(type: "TEXT", maxLength: 256, nullable: true),
NormalizedName = table.Column<string>(type: "TEXT", maxLength: 256, nullable: true),
ConcurrencyStamp = table.Column<string>(type: "TEXT", nullable: true)
},
constraints: table =>
{
table.PrimaryKey("PK_AspNetRoles", x => x.Id);
});
This error is due to the fact that the migration code EF generates for SQLite is different from the migration code generated for SQL Server. Check your builder configuration logic (Program.cs or Startup.cs) and make sure that you are specifying the correct database options when invoking AddDbContext. When using a SQL Server database you'll need to specify that as follows:
builder.Services.AddDbContext<ApplicationDbContext>(options =>
options.UseSqlServer(connection));
For comparison to your SQLite migration code, this is what the SQL Server migration code looks like for the default AspNetRoles table:
migrationBuilder.CreateTable(
name: "AspNetRoles",
columns: table => new
{
Id = table.Column<string>(type: "nvarchar(450)", nullable: false),
Name = table.Column<string>(type: "nvarchar(256)", maxLength: 256, nullable: true),
NormalizedName = table.Column<string>(type: "nvarchar(256)", maxLength: 256, nullable: true),
ConcurrencyStamp = table.Column<string>(type: "nvarchar(max)", nullable: true)
},
constraints: table =>
{
table.PrimaryKey("PK_AspNetRoles", x => x.Id);
});
I think for sql server's text type index is maximum length required. using FluentApi you can set maxLength for Id.
public class AppDbContext
{
....
protected override void OnModelCreating(ModelBuilder builder)
{
builder.Entity<Role>().Property(x => x.Id).HasMaxLength(250);
base.OnModelCreating(builder);
}
}
and re-migration created required.
The migrations created for SQLite is not useable for Sql Server, so you have to create new migrations like this:
step 1: Change "UseSqlite" to "UseSqlServer"
builder.Services.AddDbContext<ApplicationDbContext>(options =>
options.UseSqlite(connectionString));
To
builder.Services.AddDbContext<ApplicationDbContext>(options =>
options.UseSqlServer(connectionString));
step 2: Remove all migrations (by deleting the "Migrations" folder)
step 3: add a new migration, then update the database
dotnet ef migrations add "CreateIdentitySchema"
dotnet ef database update
There is already created table 'A'
'id' 'b_id' 'name'
1 2 someName
Now I want to add unique index to id and b_id columns
How to do it in yml format
based on documentation, something like this:
changeSet:
id: addUniqueConstraint-example
author: liquibase-docs
changes:
- addUniqueConstraint:
catalogName: cat
clustered: false
columnNames: id, b_id
constraintName: const_name
deferrable: true
disabled: false
forIndexName:
initiallyDeferred: true
schemaName: [yourschema]
tableName: [yourtablename]
tablespace:
validate: true
Little Background Story
Here is my scenario, I have a database named Location in my MariaDB, and I have a User database in my Postgres. I want to chain two of them in my Postgres DB. So I have two tables users and userLocations. Oh yeah, a nice thing to note is that I only have access to read (SELECT and CREATE VIEW) from LocationDB, and connecting to UserDB as root. Yes I have successfully authenticate() both of them. I even able to receive data from LocationDB however the only problem right now is creating that relation between User and Location. Error logs available on the bottom.
Here is my models:
models/user.js
// ... Connection to UserDB (uses Postgres)
const UserDB = require('../datasources/user-db')
const User = UserDB.define('user', {
id: { type: Sequelize.INTEGER, primaryKey: true, autoIncrement: true },
name: { type: Sequelize.STRING, allowNull: false }
})
module.exports = User
models/location.js
// Connection to LocationDB (uses MariaDB)
const LocationDB = require('../datasources/location-db')
const Location = LocationDB.define('ms_location', {
id_Location: { type: Sequelize.INTEGER, primaryKey: true },
name_Location: { type: Sequelize.STRING, allowNull: false }
}, { timestamps: false, freezeTableName: true })
module.exports = Location
models/user-location.js
// ... Connection to UserDB (uses Postgres)
const UserDB = require('../datasources/user-db')
const UserLocation = UserDB.define('userLocation', {
id: { type: Sequelize.INTEGER, primaryKey: true, autoIncrement: true }
})
module.exports = UserLocation
app.js
const Location = require('./models/location')
const User = require('./models/user')
const UserLocation = require('./models/user-location')
User.belongsToMany(Location, { through: UserLocation })
This is a minified version of the case, if you need more information please do ask as I am still new to sequelize as well
The error log: (sorry I don't have any idea to beautify it)
{ SequelizeDatabaseError: relation "ms_location" does not exist
at Query.formatError (<path-to-project>/node_modules/.registry.npmjs.org/sequelize/5.8.6/node_modules/sequelize/lib/dialects/postgres/query.js:354:16)
at query.catch.err (<path-to-project>/node_modules/.registry.npmjs.org/sequelize/5.8.6/node_modules/sequelize/lib/dialects/postgres/query.js:71:18)
at tryCatcher (<path-to-project>/node_modules/.registry.npmjs.org/bluebird/3.5.5/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (<path-to-project>/node_modules/.registry.npmjs.org/bluebird/3.5.5/node_modules/bluebird/js/release/promise.js:517:31)
at Promise._settlePromise (<path-to-project>/node_modules/.registry.npmjs.org/bluebird/3.5.5/node_modules/bluebird/js/release/promise.js:574:18)
at Promise._settlePromise0 (<path-to-project>/node_modules/.registry.npmjs.org/bluebird/3.5.5/node_modules/bluebird/js/release/promise.js:619:10)
at Promise._settlePromises (<path-to-project>/node_modules/.registry.npmjs.org/bluebird/3.5.5/node_modules/bluebird/js/release/promise.js:695:18)
at _drainQueueStep (<path-to-project>/node_modules/.registry.npmjs.org/bluebird/3.5.5/node_modules/bluebird/js/release/async.js:138:12)
at _drainQueue (<path-to-project>/node_modules/.registry.npmjs.org/bluebird/3.5.5/node_modules/bluebird/js/release/async.js:131:9)
at Async._drainQueues (<path-to-project>/node_modules/.registry.npmjs.org/bluebird/3.5.5/node_modules/bluebird/js/release/async.js:147:5)
at Immediate.Async.drainQueues [as _onImmediate] (<path-to-project>/node_modules/.registry.npmjs.org/bluebird/3.5.5/node_modules/bluebird/js/release/async.js:17:14)
at runCallback (timers.js:705:18)
at tryOnImmediate (timers.js:676:5)
at processImmediate (timers.js:658:5)
name: 'SequelizeDatabaseError',
parent:
{ error: relation "ms_location" does not exist
at Connection.parseE (<path-to-project>/node_modules/.registry.npmjs.org/pg/7.11.0/node_modules/pg/lib/connection.js:602:11)
at Connection.parseMessage (<path-to-project>/node_modules/.registry.npmjs.org/pg/7.11.0/node_modules/pg/lib/connection.js:399:19)
at Socket.<anonymous> (<path-to-project>/node_modules/.registry.npmjs.org/pg/7.11.0/node_modules/pg/lib/connection.js:121:22)
at Socket.emit (events.js:189:13)
at addChunk (_stream_readable.js:284:12)
at readableAddChunk (_stream_readable.js:265:11)
at Socket.Readable.push (_stream_readable.js:220:10)
at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
name: 'error',
length: 107,
severity: 'ERROR',
code: '42P01',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'namespace.c',
line: '426',
routine: 'RangeVarGetRelidExtended',
sql:
'CREATE TABLE IF NOT EXISTS "userLocations" ("id" SERIAL , "createdAt" TIMESTAMP WITH TIME ZONE NOT NULL, "updatedAt" TIMESTAMP WITH TIME ZONE NOT NULL, "userId" INTEGER REFERENCES "users" ("id") ON DELETE CASCADE ON UPDATE CASCADE, "msLocationIdLocation" INTEGER REFERENCES "ms_location" ("id_Location") ON DELETE CASCADE ON UPDATE CASCADE, UNIQUE ("userId", "msLocationIdLocation"), PRIMARY KEY ("id"));' },
original:
{ error: relation "ms_location" does not exist
at Connection.parseE (<path-to-project>/node_modules/.registry.npmjs.org/pg/7.11.0/node_modules/pg/lib/connection.js:602:11)
at Connection.parseMessage (<path-to-project>/node_modules/.registry.npmjs.org/pg/7.11.0/node_modules/pg/lib/connection.js:399:19)
at Socket.<anonymous> (<path-to-project>/node_modules/.registry.npmjs.org/pg/7.11.0/node_modules/pg/lib/connection.js:121:22)
at Socket.emit (events.js:189:13)
at addChunk (_stream_readable.js:284:12)
at readableAddChunk (_stream_readable.js:265:11)
at Socket.Readable.push (_stream_readable.js:220:10)
at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
name: 'error',
length: 107,
severity: 'ERROR',
code: '42P01',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'namespace.c',
line: '426',
routine: 'RangeVarGetRelidExtended',
sql:
'CREATE TABLE IF NOT EXISTS "userLocations" ("id" SERIAL , "createdAt" TIMESTAMP WITH TIME ZONE NOT NULL, "updatedAt" TIMESTAMP WITH TIME ZONE NOT NULL, "userId" INTEGER REFERENCES "users" ("id") ON DELETE CASCADE ON UPDATE CASCADE, "msLocationIdLocation" INTEGER REFERENCES "ms_location" ("id_Location") ON DELETE CASCADE ON UPDATE CASCADE, UNIQUE ("userId", "msLocationIdLocation"), PRIMARY KEY ("id"));' },
sql:
'CREATE TABLE IF NOT EXISTS "userLocations" ("id" SERIAL , "createdAt" TIMESTAMP WITH TIME ZONE NOT NULL, "updatedAt" TIMESTAMP WITH TIME ZONE NOT NULL, "userId" INTEGER REFERENCES "users" ("id") ON DELETE CASCADE ON UPDATE CASCADE, "msLocationIdLocation" INTEGER REFERENCES "ms_location" ("id_Location") ON DELETE CASCADE ON UPDATE CASCADE, UNIQUE ("userId", "msLocationIdLocation"), PRIMARY KEY ("id"));' }
You've told Sequelize that you have a table called user and a table called msLocation and that the two are related through another table called userLocation. (Or is it userLocations? Your code and error message are inconsistent.)
It looks like Sequelize is trying to create the userLocations table UserDB but it's failing because as part of the table definition, it's trying to create a foreign key reference to ms_location:
"msLocationIdLocation" INTEGER REFERENCES "ms_location" ("id_Location")
ON DELETE CASCADE ON UPDATE CASCADE
which fails because ms_location is not in the UserDB but rather in the other database.
I don't know if Sequelize can actually handle tables spanning multiple databases servers, but I doubt it. Creating a query that spans two tables in the same database is easy, Sequelize can just create a join. It's a totally different story if the tables are on two different servers; now Sequelize would have to run two different queries and do all the join logic in memory. That's a pretty big lift.
Some databases have a way to create a table in one database that replicates or forwards to a table in another database. If PostgreSQL supports that, you could try using that to replicate or shadow the MariaDB table, which would enable Sequelize to see all the tables as being part of UserDB.
I'm having a very weird issue. When I insert five roles into my "repository" table with unique ids, the following error below comes up multiple times (same id being mentioned!). I'm not using autoincrement for PK.
Error saving repo { error: duplicate key value violates unique constraint "repository_pkey"
at Connection.parseE (/Users/macintosh/node-projects/risingstack/node_modules/pg/lib/connection.js:554:11)
at Connection.parseMessage (/Users/macintosh/node-projects/risingstack/node_modules/pg/lib/connection.js:379:19)
at Socket.<anonymous> (/Users/macintosh/node-projects/risingstack/node_modules/pg/lib/connection.js:119:22)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at addChunk (_stream_readable.js:263:12)
at readableAddChunk (_stream_readable.js:250:11)
at Socket.Readable.push (_stream_readable.js:208:10)
at TCP.onread (net.js:601:20)
name: 'error',
length: 202,
severity: 'ERROR',
code: '23505',
detail: 'Key (id)=(80073079) already exists.',
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: 'public',
table: 'repository',
column: undefined,
dataType: undefined,
constraint: 'repository_pkey',
file: 'nbtinsert.c',
line: '434',
routine: '_bt_check_unique' }
Postgres code generated by knex:
insert into "repository" ("description", "full_name", "html_url", "id", "language", "owner_id", "stargazers_count") values ('Node.js JavaScript runtime :sparkles::turtle::rocket::sparkles:', 'nodejs/node', 'https://github.com/nodejs/node', 27193779, 'JavaScript', 9950313, 56009)
insert into "repository" ("description", "full_name", "html_url", "id", "language", "owner_id", "stargazers_count") values (':closed_book:《Node.js 包教不包会》 by alsotang', 'alsotang/node-lessons', 'https://github.com/alsotang/node-lessons', 24812854, 'JavaScript', 1147375, 13989)
insert into "repository" ("description", "full_name", "html_url", "id", "language", "owner_id", "stargazers_count") values ('Node.js based forum software built for the modern web', 'NodeBB/NodeBB', 'https://github.com/NodeBB/NodeBB', 9603889, 'JavaScript', 4449608, 9399)
insert into "repository" ("description", "full_name", "html_url", "id", "language", "owner_id", "stargazers_count") values (':baby_chick:Nodeclub 是使用 Node.js 和 MongoDB 开发的社区系统', 'cnodejs/nodeclub', 'https://github.com/cnodejs/nodeclub', 3447593, 'JavaScript', 1455983, 7907)
insert into "repository" ("description", "full_name", "html_url", "id", "language", "owner_id", "stargazers_count") values ('Mysterium Node - VPN server and client for Mysterium Network', 'mysteriumnetwork/node', 'https://github.com/mysteriumnetwork/node', 80073079, 'Go', 23056638, 478)
Knex schema for repository:
return knex.schema.createTable('repository', (table) => {
table.integer('id').primary();
table.integer('owner_id');
table.foreign('owner_id').references('user.id').onDelete('CASCADE').onUpdate('CASCADE');
table.string('full_name');
table.string('description');
table.string('html_url');
table.string('language');
table.integer('stargazers_count');
})
Code run to insert Repository:
const fn = composeMany(withOwner, removeIrrelevantProperties, defaultLanguageAndDescToString, saveAndPublish);
const tRepos = r.map(fn);
return Promise.all(tRepos);
const saveAndPublish = (r) => {
return User
.insert(r.owner)
.catch(e => console.log('Error saving User', e))
.then(() => {
const { owner, ...repo } = r;
const q = Repository.insert(repo);
console.log(q.toQuery());
return q;
})
.catch(e => {
console.log('Error saving repo', e)}
);
Sounds like your database already had a row inserted with primary key id == 80073079.
To be sure about it try to query DB rows with that key just before inserting. I just wonder how are those ids generated, since you are clearly not using id sequence for it.
It is possible that input data, where IDs were fetched is corrupted and has duplicate ids
I have a spring boot 2 project(jhipster) and I am a little confused on how hibernate tables work with liquibase.
Right now I have a bunch of data classes with hibernate annotations and I would like to insert some static data for testing purposes. I have a bunch of questions in getting started.
Do I have to define liquibase changeSets to create tables when I already have the hibernate annotated data classes?
How do I run liquibase changeSets when configuring the database?
EDIT -----
How are the foreign key relationships named between the changeset and the domain(java) code? For example I have a Person table and it holds a reference to an Address table. How do I represent this relationship within the changeSet table?
The Person table doesn't hold an id of an Address, it holds the reference.
Spring boot has excellent integration with liquibase.
If you want liquibase to handle creation of the tables (DDL) and not hibernate:
you need to disable the hibernate auto-create flag.
set spring.jpa.hibernate.ddl-auto=none
(or) remove this property from application.yml file.
Just include following into pom.xml:
<dependency>
<groupId>org.liquibase</groupId>
<artifactId>liquibase-core</artifactId>
</dependency>
Below is the sample code to create tables and insert data into table.
File location:
src/main/resources/db/changelog/db.changelog-master.yaml
databaseChangeLog:
- changeSet:
id: 1
author: sgollapinni
changes:
- createTable:
tableName: person
columns:
- column:
name: id
type: int
autoIncrement: true
constraints:
primaryKey: true
nullable: false
- column:
name: first_name
type: varchar(255)
constraints:
nullable: false
- column:
name: last_name
type: varchar(255)
constraints:
nullable: false
- createTable:
tableName: address
columns:
- column:
name: id
type: int
autoIncrement: true
constraints:
primaryKey: true
nullable: false
- column:
name: city
type: varchar(255)
constraints:
nullable: false
- column:
name: person_id
type: varchar(255)
constraints:
nullable: false
foreignKeyName: fk_person_address
references: person(id)
- changeSet:
id: 2
author: sgollapinni
changes:
- insert:
tableName: person
columns:
- column:
name: first_name
value: Sunil
- column:
name: last_name
value: Kumar
- insert:
tableName: address
columns:
- column:
name: city
value: Bangalore
- column:
name: user_id
value: (Select id from person where name = 'Sunil')
Otherwise, if you want hibernate to handle the DDL and only you want to insert some static data for testing purposes, you can still do this using liquibase.
You can use changeSets to add the DML statements.
- changeSet:
id: 1
author: sgollapinni
changes:
- insert:
tableName: person
columns:
- column:
name: first_name
value: Sunil
- column:
name: last_name
value: Kumar
- insert:
tableName: address
columns:
- column:
name: city
value: Bangalore
- column:
name: user_id
value: (Select id from person where name = 'Sunil')
Hope it helps!