typeorm:migration create on New Project Does Not Recognize Entities - "No changes in database schema were found - cannot generate a migration." - mongodb

I'm having trouble creating the initial migration for a nestjs-typeorm-mongo project.
I have cloned this sample project from nestjs that uses typeorm with mongodb. The project does work in that when I run it locally after putting a "Photo" document into my local mongo with db named "test" and collection "photos" then I can call to localhost:3000/photo and receive the photo documents.
Now I'm trying to create migrations with the typeorm cli using this command:
./node_modules/.bin/ts-node ./node_modules/typeorm/cli.js migration:generate -n initial
...but it's not working. I am not able to create an initial commit- even after setting "synchronize: false" in my app.module.ts file I always get the error:
No changes in database schema were found - cannot generate a migration. To create a new empty migration use "typeorm migration:create" command
when trying to generate a migration... 🤔
Other than changing synchronize to false, the only other change I made was adding an ormconfig.json file in the project root by running typeorm init --database mongodb:
{
"type": "mongodb",
"database": "test",
"synchronize": true,
"logging": false,
"entities": [
"src/**/*.entity.ts"
],
"migrations": [
"src/migration/**/*.ts"
],
"subscribers": [
"src/subscriber/**/*.ts"
],
"cli": {
"entitiesDir": "src",
"migrationsDir": "src/migration",
"subscribersDir": "src/subscriber"
}
}

Once you are using MongoDB, you don't have tables and have no need to create your collections ahead of time. Essentially, MongoDB schemas are created on the fly!
Under the hood, if the driver is MongoDB, the command typeorm migration:create is bypassed so it is useless in this case. You can check yourself the PR #3304 and Issue #2867.
However, there is an alternative called migrate-mongo which provides a way to archive incremental, reversible, and version-controlled way to apply schema and data changes. It’s well documented and actively developed.
migrate-mongo example
Run npm install -g migrate-mongo to install it.
Run migrate-mongo init to init the migrations tool. This will create a migrate-mongo-config.js configuration file and a migrations folder at the root of our project:
|_ src/
|_ migrations/
|- 20200606204524-migration-1.js
|- 20200608124524-migration-2.js
|- 20200808114324-migration-3.js
|- migrate-mongo.js
|- package.json
|- package-lock.json
Your migrate-mongo-config.js configuration file may look like:
// In this file you can configure migrate-mongo
const env = require('./server/config')
const config = {
mongodb: {
// TODO Change (or review) the url to your MongoDB:
url: env.mongo.url || "mongodb://localhost:27017",
// TODO Change this to your database name:
databaseName: env.mongo.dbname || "YOURDATABASENAME",
options: {
useNewUrlParser: true, // removes a deprecation warning when connecting
useUnifiedTopology: true, // removes a deprecating warning when connecting
// connectTimeoutMS: 3600000, // increase connection timeout up to 1 hour
// socketTimeoutMS: 3600000, // increase socket timeout up to 1 hour
}
},
// The migrations dir can be a relative or absolute path. Only edit this when really necessary.
migrationsDir: "migrations",
// The MongoDB collection where the applied changes are stored. Only edit this when really necessary.
changelogCollectionName: "changelog"
};
module.exports = config;
Run migrate-mongo create name-of-my-script to add a new migration script. A new file will be created with a corresponding timestamp.
/*
|_ migrations/
|- 20210108114324-name-of-my-script.js
*/
module.exports = {
function up(db) {
return db.collection('products').updateMany({}, { $set: { quantity: 10 } })
}
function down(db) {
return db.collection('products').updateMany({}, { $unset: { quantity: null } })
}
}
The database changelog: In order to know the current database version and which migration should apply next, there is a special collection that stores the database changelog with information such as migrations applied, and when where they applied.
To run your migrations, simply run the command: migrate-mongo up
You can find a full example in this article MongoDB Schema Migrations in Node.js

Related

No migrations are pending using Typeorm CLI and NestJS

The issue I'm facing is with running migrations using #nestjs/typeorm ^9.0.1 and typeorm ^0.3.12. Despite being able to build the project and create migrations, when I try to run them, typeorm can't find the migration files (No migrations are pending).
I have noticed that the version 0.3.x of typeorm requires Datasource object and that in the version 9.x of #nestjs/typeorm, TypeOrmModuleOptions does not have the cli : {migrationsDir : string} attribute.
Here's a link to the project that reproduces the problem, and an example directory structure :
src\configs\database\dataSource.config.ts
src\configs\database\postgres.config.ts
src\configs\database\migrations\1676252827402-Chat.ts
postgres.config.ts
import { DataSourceOptions } from "typeorm";
import { join } from "path";
import { config } from "dotenv";
import CustomNamingStrategy from "./customNamingStrategy";
import { registerAs } from "#nestjs/config";
import { TypeOrmModuleOptions } from "#nestjs/typeorm";
config();
export const databaseConfig: DataSourceOptions = {
name: "default",
type: "postgres",
url: process.env.DATABASE_URL,
ssl:
process.env.DATABASE_ENABLE_SSL === "true"
? {
rejectUnauthorized: false,
}
: false,
logging: process.env.DATABASE_ENABLE_LOGGING === "true",
entities: [join(__dirname, "../../models/*/", "*.entity.{ts,js}")],
migrations: [join(__dirname, "migrations/*.{ts,js}")],
synchronize: false,
migrationsRun: false,
namingStrategy: new CustomNamingStrategy(),
};
export default registerAs(
"database",
() =>
({
...databaseConfig,
keepConnectionAlive: true,
} as TypeOrmModuleOptions)
);
dataSource.config.ts
import { DataSource } from "typeorm";
import { databaseConfig } from "./postgres.config";
export const AppDataSource = new DataSource(databaseConfig);
package.json
"typeorm": "ts-node -r tsconfig-paths/register ./node_modules/typeorm/cli"
"migration:create": "yarn run typeorm migration:create"
"migration:run": "yarn run typeorm migration:run -d src/configs/database/dataSource.config.ts"
Although I can use the create command successfully, when I use the run command, it only displays the message "No migrations are pending". I attempted to resolve the issue by changing migrations path to the DataSourceOptions.migrations list, but it did not work. I also tried replacing the cli parameter datasource with the dist datasource, as well as adjusting the migration paths, but these changes did not fix the issue.
I'm facing a similar issue using the same version of typeorm (^0.3.12), so there might be a bug in this version.
It seems that the migrations are not found anymore if the DataSourceOptions.migrations property is set with the path to the migrations folder.
The ^0.3.12 version seems to be working fine only if the migration classes are directly referenced in DataSourceOptions.migrations, similar to the following:
import { Chat1676252827402 } from './migrations/1676252827402-Chat.ts';
export const databaseConfig: DataSourceOptions = {
...
migrations: [Chat1676252827402],
...
};

Wasm-rust and postgres

is there a way to get data from a database in postgres using wasm?. I'd tried to get it using a library in rust but I got some errore when I build the package using "wasm-pack building--target web". The idea is to build a function in lib.rs file that return data from a db. I have the below code inside lib.rs:
use postgres::{Client, Error, NoTls};
use wasm_bindgen::prelude::*;
...
struct Author {
_id: i32,
name: String,
}
#[wasm_bindgen]
pub fn select_name(name: &String) -> Result<(), Error> {
let mut client = Client::connect("postgresql://user:1234#localhost:5432/db", NoTls)?;
for row in client.query(
"SELECT id, name FROM author WHERE name = $1",
&[&name],
)? {
let author = Author {
_id: row.get(0),
name: row.get(1),
};
println!(
"Select_Name => Author {} :",
author.name
);
}
Ok(())
}
but I get some errors:
error[E0432]: unresolved import `crate::sys::IoSourceState`
error[E0432]: unresolved import `crate::sys`
...
It is not possible directly (as Njuguna Mureithi rightly wrote) but it can be circumvented.
We can use the project: https://github.com/PostgREST/postgrest
and expose the API to our sql server.
We download the latest version of postgrest
https://github.com/PostgREST/postgrest/releases/tag/v9.0.0
in case of Linux we unpack
tar -xf postgrest-v9.0.0-linux-static-x64.tar.xz
then run help
./postgrest -h
create a configuration file for ./postgrest
postgrest -e > cfg.psqlrest
change the user and password for the database in the configuration file.
e.g. with
db-uri = "postgres://user:pasword#localhost:5432/dbname"
to your dbname database access user:pasword configuration
db-uri = "postgres://postgres:zaqwsxc#localhost:5432/dbname"
and run the server which will issue the api to our postgres
./postgrest cfg.psqlrest
The address http://localhost:3000 will start accessing the database dbname, which must be created in the database server beforehand.
Here is a description of the libraries needed to call the query using the API.
https://rustwasm.github.io/wasm-bindgen/examples/fetch.html
examples of API
https://postgrest.org/en/stable/api.html

I want to insert with mikro-orm, but it dont find my table :c (TableNotFoundException)

So
Console:
yarn dev
yarn run v1.22.10
$ nodemon dist/index.js
[nodemon] 2.0.7
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,json
[nodemon] starting `node dist/index.js`
[discovery] ORM entity discovery started, using ReflectMetadataProvider
[discovery] - processing entity Post
[discovery] - entity discovery finished, found 1 entities, took 21 ms
[info] MikroORM successfully connected to database postgres on postgresql://postgres:*****#127.0.0.1:5432
[query] begin
[query] insert into "post" ("created_at", "title", "updated_at") values ('2021-04-05T21:04:23.126Z', 'my first post', '2021-04-05T21:04:23.126Z') returning "_id" [took 12 ms]
[query] rollback
TableNotFoundException: insert into "post" ("created_at", "title", "updated_at") values ('2021-04-05T21:04:23.126Z', 'my first post', '2021-04-05T21:04:23.126Z') returning "_id" - relation "post" does not exist
at PostgreSqlExceptionConverter.convertException (P:\.Projektek\lireddit-server\node_modules\#mikro-orm\postgresql\PostgreSqlExceptionConverter.js:36:24)
at PostgreSqlDriver.convertException (P:\.Projektek\lireddit-server\node_modules\#mikro-orm\core\drivers\DatabaseDriver.js:194:54)
at P:\.Projektek\lireddit-server\node_modules\#mikro-orm\core\drivers\DatabaseDriver.js:198:24
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at async PostgreSqlDriver.nativeInsert (P:\.Projektek\lireddit-server\node_modules\#mikro-orm\knex\AbstractSqlDriver.js:150:21)
at async ChangeSetPersister.persistNewEntity (P:\.Projektek\lireddit-server\node_modules\#mikro-orm\core\unit-of-work\ChangeSetPersister.js:55:21)
at async ChangeSetPersister.executeInserts (P:\.Projektek\lireddit-server\node_modules\#mikro-orm\core\unit-of-work\ChangeSetPersister.js:24:13)
at async UnitOfWork.commitCreateChangeSets (P:\.Projektek\lireddit-server\node_modules\#mikro-orm\core\unit-of-work\UnitOfWork.js:496:9)
at async UnitOfWork.persistToDatabase (P:\.Projektek\lireddit-server\node_modules\#mikro-orm\core\unit-of-work\UnitOfWork.js:458:13)
at async PostgreSqlConnection.transactional (P:\.Projektek\lireddit-server\node_modules\#mikro-orm\knex\AbstractSqlConnection.js:53:25)
at async UnitOfWork.commit (P:\.Projektek\lireddit-server\node_modules\#mikro-orm\core\unit-of-work\UnitOfWork.js:183:17)
at async SqlEntityManager.flush (P:\.Projektek\lireddit-server\node_modules\#mikro-orm\core\EntityManager.js:486:9)
at async SqlEntityManager.persistAndFlush (P:\.Projektek\lireddit-server\node_modules\#mikro-orm\core\EntityManager.js:438:9)
previous error: insert into "post" ("created_at", "title", "updated_at") values ('2021-04-05T21:04:23.126Z', 'my
first post', '2021-04-05T21:04:23.126Z') returning "_id" - relation "post" does not exist
at Parser.parseErrorMessage (P:\.Projektek\lireddit-server\node_modules\pg-protocol\dist\parser.js:278:15)
at Parser.handlePacket (P:\.Projektek\lireddit-server\node_modules\pg-protocol\dist\parser.js:126:29)
at Parser.parse (P:\.Projektek\lireddit-server\node_modules\pg-protocol\dist\parser.js:39:38)
at Socket.<anonymous> (P:\.Projektek\lireddit-server\node_modules\pg-protocol\dist\index.js:10:42)
at Socket.emit (events.js:315:20)
at Socket.EventEmitter.emit (domain.js:467:12)
at addChunk (internal/streams/readable.js:309:12)
at readableAddChunk (internal/streams/readable.js:284:9)
at Socket.Readable.push (internal/streams/readable.js:223:10)
at TCP.onStreamRead (internal/stream_base_commons.js:188:23) {
length: 166,
severity: 'ERROR',
code: '42P01',
detail: undefined,
hint: undefined,
position: '13',
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'd:\\pginstaller_13.auto\\postgres.windows-x64\\src\\backend\\parser\\parse_relation.c',
line: '1376',
routine: 'parserOpenTable'
}
Index.ts:
import { MikroORM } from "#mikro-orm/core";
import { __prod__ } from "./constants";
import { Post } from "./entities/Post";
import mikroConfig from "./mikro-orm.config";
const main = async () => {
const orm = await MikroORM.init(mikroConfig);
await orm.getMigrator().up;
const post = orm.em.create(Post, { title: "my first post" });
await orm.em.persistAndFlush(post);
};
main().catch((err) => {
console.error(err);
});
Post.ts:
import { Entity, PrimaryKey, Property } from "#mikro-orm/core";
#Entity()
export class Post {
#PrimaryKey()
_id!: number;
#Property({ type: "date" })
createdAt = new Date();
#Property({ type: "date", onUpdate: () => new Date() })
updatedAt = new Date();
#Property({ type: "text" })
title!: string;
}
mikro-orm.config.ts:
import { __prod__ } from "./constants";
import { Post } from "./entities/Post";
import { MikroORM } from "#mikro-orm/core";
import path from "path";
export default {
migrations: {
path: path.join(__dirname, "./migrations"),
pattern: /^[\w-]+\d+\.[tj]s$/,
},
entities: [Post],
dbName: "postgres",
debug: !__prod__,
type: "postgresql",
password: "hellothere",
} as Parameters<typeof MikroORM.init>[0];
And the migration I created with npx mikro-orm migration:create:
import { Migration } from '#mikro-orm/migrations';
export class Migration20210405205411 extends Migration {
async up(): Promise<void> {
this.addSql('create table "post" ("_id" serial primary key, "created_at" timestamptz(0) not null, "updated_at" timestamptz(0) not null, "title" text not null);');
}
}
After that im compiling it to js btw, but I guess the problem will be somewhere at my code or idk plz help me, I can give you more info just plz help, I've been trying to fix this bug for 5 hours :/
Btw Im doin Ben Awad's 14 hour fullstack tutorial if its matter.
The TableNotFoundException happens when you try to add data before initializing the table's schema (or structure).
Passing the --initial as in Mosh's Answer did not work for me, possibly because I am passing a username and password in ./mikro-orm.config.ts.
I used Mikro-ORM's SchemaGenerator to initialize the table as seen here in the official docs.
Add the following lines before adding data to post in your main function in index.ts:
const generator = orm.getSchemaGenerator();
await generator.updateSchema();
The main function in index.ts should now look like this:
const main = async () => {
const orm = await MikroORM.init(mikroConfig);
await orm.getMigrator().up;
const generator = orm.getSchemaGenerator();
await generator.updateSchema();
const post = orm.em.create(Post, { title: "my first post" });
await orm.em.persistAndFlush(post);
};
updateSchema creates a table or updates it based on .entities/Post.ts. This could cause issues when the Post file is updated, I haven't run in to any while following Ben's tutorial. Although, I'd still recommend creating ./create-schema.ts and running it when needed as shown in the official docs.
I have had the same issue. This is what I did:
I deleted the migrations folder as well as the dist folder
I ran npx mikro-orm migration:create --initial
After that, I restarted yarn watch and yarn dev and it worked for me.
Notice the --initial flag. I would recommend to check the official documentation. The migrations table is used to keep track of already executed migrations. When you only run npx mikro-orm migration:create, the table will not be created and therefore MikroORM is unable to check if the migration for the Post entity has already been performed (which includes creating the respective table on the database).
Ben does not use the --initial flag in his tutorial, he might have already ran it prior to the tutorial.
I had a similar problem myself (Also doing Ben Awad's tutorial).
I used Mikro-ORM's schema generator to initialize the table like in Fares47's Answer, but the problem still persisted.
It wasn't until I set my user to have Superuser permissions that it started working.
I am using postgresql for my data base which I downloaded using homebrew. If you have a similar set up here is what I did:
Start up psql in your terminal using psql postgres. If you want, you can view your users and check their permissions by typing \du in the shell. Then, to change the permissions for a user use the command ALTER ROLE <username> WITH SUPERUSER;. Make sure you include a semi-colon or else it will not run the command.
Check this article out for more info on psql commands.
I have the same problem i solved by install the ts-node on project
npm i -D ts-node
and set useTsNode on package.json as true.
The problem is the mikro-orm cli only add ts files paths in configPaths if the property useTsNode is true and ts-node is installed.
orther problem that i have is the regex in pattern property in mikro-orm.config.ts was wrong because a typo.
If any of the suggested steps didnt solve it for you, simply...
quit yarn watch and yarn dev
run this command from the command line
npx mikro-orm migration:up
now restart watch and dev and it you should be good.
from https://mikro-orm.io/docs/migrations/#migration-class
I also experienced this. And like Fares47 said it's possibly because I passed the username and password in ./mikro-orm.config.ts.
And my solution is simply execute the sql command that generated in ./src/migrations/Migration<NUMBERS>.ts file in postgresql terminal.
Here is the command that I execute in the database,
create table "post" ("id" serial primary key, "created_at" timestamptz(0) not null, "updated_at" timestamptz(0) not null, "title" text not null);
Just like what they suggested in the doc,
A safe approach would be generating the SQL on development server and
saving it into SQL Migration files that are executed manually on the
production server.

How to integrate PostgreSQL with Corda 3.0 or with Corda 4.0?

I tried to configure PostgreSQL as node's database in Corda 3.0 and in Corda 4.0.
I have added following things in build.gradle file. (Testdb1 is Database name. I have tried with postgres also)
node{
...
// this part i have added
extraConfig = [
jarDirs: ['path'],
'dataSourceProperties': [
'dataSourceClassName': 'org.postgresql.ds.PGSimpleDataSource',
'"dataSource.url"' : 'jdbc:postgresql://127.0.0.1:5432/Testdb1',
'"dataSource.user"' : 'postgres',
'"dataSource.password"': 'admin#123'
],
'database': [
'transactionIsolationLevel': 'READ_COMMITTED'
]
]
// till here
}
this part in reference.conf file
dataSourceProperties = {
dataSourceClassName = org.postgresql.ds.PGSimpleDataSource
dataSource.url = "jdbc:postgresql://127.0.0.1:5432/Testdb1"
dataSource.user = postgres
dataSource.password = "admin#123"
}
database = {
transactionIsolationLevel = "READ_COMMITTED"
}
jarDirs = ["path"]
I got the follwing Error while deploying the nodes:
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':java-source:deployNodes'.
In node-info-gen.log file it showed the CAPSULE EXCEPTION. Then I updated my JDK to 8u191 but still got the same error.
I have go-through the followings to get the things done. One can get reference from here.
https://docs.corda.net/node-database.html ,
https://github.com/corda/corda/issues/4037 ,
How can the Corda node be extended to work with databases other than H2?
You need to add those properties in node.conf in each of your corda nodes. After you do "deployNodes"
After you add these properties in node.conf file , just run the corda jar . It will automatically start . But before that you need to create the tables (The migration to other DBs is already provided in corda documentation)
I have added following things in .conf file of each node and one referene.conf file. I have given all the privileges for the user postgres which are mentioned in corda documentation.
https://docs.corda.r3.com/node-database.html
(previously I used postgresql-42.2.5.jar file but that didn't work so I used one downgrade version of it postgresql-42.1.4.jar.
one can download jar files from https://jdbc.postgresql.org/download.html )
After deployed Nodes successfully add following things:
dataSourceProperties = {
dataSourceClassName = org.postgresql.ds.PGSimpleDataSource
dataSource.url = "jdbc:postgresql://127.0.0.1:5432/Testdb1"
dataSource.user = postgres
dataSource.password = "admin#123"
}
database = {
transactionIsolationLevel = "READ_COMMITTED"
}
jarDirs = ["path"]
(path = jar file's location) after adding this configuration run file called runnodes.bat

Accessing config variables from other config files

I am having problems using in a config file a config var set in another config file. E.g.
// file - config/local.js
module.exports = {
mongo_db : {
username : 'TheUsername',
password : 'ThePassword',
database : 'TheDatabase'
}
}
// file - config/connections.js
module.exports.connections = {
mongo_db: {
adapter: 'sails-mongo',
host: 'localhost',
port: 27017,
user: sails.config.mongo_db.username,
password: sails.config.mongo_db.password,
database: sails.config.mongo_db.database
},
}
When I 'sails lift', I get the following error:
user: sails.config.mongo_db.username,
^
ReferenceError: sails is not defined
I can access the config variables in other places - e.g, this works:
// file - config/bootstrap.js
module.exports.bootstrap = function(cb) {
console.log('Dumping config: ', sails.config);
cb();
}
This dumps all the config settings to the console - I can even see the config settings for mongo_db in there!
I so confuse.
You can't access sails inside of config files, since Sails config is still being loaded when those files are processed! In bootstrap.js, you can access the config inside the bootstrap function, since that function gets called after Sails is loaded, but not above the function.
In any case, config/local.js gets merged on top of all the other config files, so you can get what you want this way:
// file - config/local.js
module.exports = {
connections: {
mongo_db : {
username : 'TheUsername',
password : 'ThePassword',
database : 'TheDatabase'
}
}
}
// file - config/connections.js
module.exports.connections = {
mongo_db: {
adapter: 'sails-mongo',
host: 'localhost',
port: 27017
},
}
If you really need to access one config file from another you can always use require, but it's not recommended. Since Sails merges config files together based on several factors (including the current environment), it's possible you'd be reading some invalid options. Best to do things the intended way: use config/env/* files for environment-specific settings (e.g. config/env/production.js), config/local.js for settings specific to a single system (like your computer) and the rest of the files for shared settings.