Sequelize migration tool can not find schema defined in config.json - postgresql

When start a docker container from postgres image, it only creates a public schema under a database. here is my setting
postgres:
container_name: postgres
image: postgres:latest
ports:
- "${DB_PORT}:5432"
environment:
- POSTGRES_USER=${DB_USER}
- POSTGRES_PASSWORD=${DB_PASSWORD}
- POSTGRES_DB=${DB_NAME}
my solution is
// 111111-create-schema.js
await queryInterface.createSchema('custom');
// 222222-create-user.js
await queryInterface.createTable('Users', {}, {schema: 'custom'}
When I run npx sequelize-cli db:migrate, I do see my user table in custom schema. However, the sequelizeMeta table which stores the migrations is in PUBLIC shema
I also tried to add schema: 'custom' in config.json. however, I got custom schema is not found error when I run db:migrate. I assume at that point my migration script has not run yet before it trys to locate this custom schema.
Then I tried to manually create the schema in the pgadmin. then I see both my user and sequelizeMeta table are in this custom schema.
I am wondering how I can let postgres docker container precreate a schema just like it precreate a database using a POSTGRES_DB environment variable ? something like
postgres:
container_name: postgres
image: postgres:latest
ports:
- "${DB_PORT}:5432"
environment:
- POSTGRES_USER=${DB_USER}
- POSTGRES_PASSWORD=${DB_PASSWORD}
- POSTGRES_DB=${DB_NAME}
- POSTGRES_SCHEMA=${DB_SCHEMA} <---------------
Or there is an easy or proper way in sequelize migration that I missed ?

When you create your sequelize connection say your schema on define object like this:
const sequelize = new Sequelize(database, username, password,
{
host, dialect, port, omitNull: true,
define: {
schema: "your-schema"
timestamps: true,
underscored: true
},
...
})

Related

Failed Authentication when connecting with Flask through PyMongo to MongoDB in Docker Compose

I'm using Docker Compose and trying to make two containers talk to each other. One runs a MongoDB database and the other one is a Flask app that needs to read data from the first one using PyMongo.
The Mongo image is defined with the following Dockerfile:
FROM mongo:6.0
ENV MONGO_INITDB_ROOT_USERNAME admin
ENV MONGO_INITDB_ROOT_PASSWORD admin-pwd
ENV MONGO_INITDB_DATABASE admin
COPY mongo-init.js /docker-entrypoint-initdb.d/
EXPOSE 27017
And my data is loaded through the following mongo-init.js script:
db.auth('admin','admin-pwd')
db = db.getSiblingDB('quiz-db')
db.createUser({
user: 'quiz-admin',
pwd: 'quiz-pwd',
roles: [
{
role: 'readWrite',
db: 'quiz-db'
}
]
});
db.createCollection('questions');
db.questions.insertMany([
{
question: "Do you like sushi?",
answers: {
0:"Yes",
1:"No",
2:"Maybe"
}
}
]);
The Flask app is pretty straightforward. I'll skip the Dockerfile for this one as I don't think it's important to the issue. I try to connect to the database with the following code:
from flask import Flask, render_template
from pymongo import MongoClient
app = Flask(__name__)
MONGO_HOST = "questions-db"
MONGO_PORT = "27017"
MONGO_DB = "quiz-db"
MONGO_USER = "quiz-admin"
MONGO_PASS = "quiz-pwd"
uri = "mongodb://{}:{}#{}:{}/{}?authSource=quiz-db".format(MONGO_USER, MONGO_PASS, MONGO_HOST, MONGO_PORT, MONGO_DB)
client = MongoClient(uri)
db=client["quiz-db"]
questions=list(db["questions"].find())
I'm not an expert when it comes to Mongo, but I've set authSource to 'quiz-db' since that's the database where I've created the user in the 'mongo-init.js' script. I tried to run the database container alone and I did successfully log in using mongosh with the user 'quiz-db'. All the data is there and everything works fine.
The problem is only coming up when trying to connect from the Flask app. Here's my Docker compose file:
version: '3.9'
services:
#Flask App
app:
build: ./app
ports:
- "8000:5000"
depends_on:
- "questions-db"
networks:
- mongo-net
#Mongo Database
questions-db:
build: ./questions_db
hostname: questions-db
container_name: questions-db
ports:
- "27017:27017"
networks:
- mongo-net
networks:
mongo-net:
driver: bridge
When I run 'docker compose up' I get the following error on the Flask container startup:
pymongo.errors.OperationFailure: command find requires authentication
full error: {'ok': 0.0, 'errmsg': 'command find requires authentication', 'code': 13, 'codeName': 'Unauthorized'}
MongoDB stores all user credentials in the admin database, unless you are using a really ancient version.
Use authSource=admin in the URI

Entity framework + postgis Get-Migration fails with Error: 28000: role "my_username" does not exist

I have a postgis database running using the following docker-compose file:
version: '3.9'
volumes:
dbbackups:
postgis-data:
services:
db:
image: kartoza/postgis:14-3.2
volumes:
- postgis-data:/var/lib/postgresql
- dbbackups:/backups
environment:
# If you need to create multiple database you can add coma separated databases eg gis,data
- POSTGRES_DB=my_db
- POSTGRES_USER=my_username
- POSTGRES_PASS=my_password
- ALLOW_IP_RANGE=0.0.0.0/0
# Add extensions you need to be enabled by default in the DB. Default are the five specified below
- POSTGRES_MULTIPLE_EXTENSIONS=postgis,hstore,postgis_topology,postgis_raster,pgrouting
ports:
- "25432:5432"
restart: on-failure
healthcheck:
test: "exit 0"
I have an ASP.Net Core Web API project configured to use this database. In Program.cs:
builder.Services.AddDbContext<PerceptionAnnotationServerContext>(options =>
options.UseNpgsql("Host=localhost;Port=25432;Database=my_db;Username=my_username;Password=my_password"));
If I do Get-Migration or Add-Migration then Update-Database from the VS 2022 pack manager console, I get the error:
An error occurred while accessing the database. Continuing without the information provided by the database. Error: 28000: role "my_username" does not exist
I initially found this answer on how to add the user. However checking with pgAdmin shows that the role "my_username" already exists and has access to "my_db".
Any ideas?
I had forgotten to add to the db context:
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseNpgsql("Host=localhost;Port=25432;Database=my_db;Username=my_username;Password=my_password");
}

ERROR: syntax error at or near "CREATE" in Docker-compose Postgres

I am trying to create two tables using docker-compose and a dockerfile with postgres sql. However, I get the following error.
psql:/docker-entrypoint-initdb.d/tables/users.sql:11: ERROR: syntax error at or near "CREATE" postgres_1 | LINE 2: CREATE TABLE users
I am not sure what I am doing wrong, but I checked my sql query via eversql.com/sql-syntax-check-validator/ and it seems to be valid sql.
could it be the version of the postgres image I am using or something else? My dockerfiles look correct to me but please do let me know if I am missing something.
Here is my Dockerfile
FROM postgres:latest
ADD /tables/ /docker-entrypoint-initdb.d/tables/
ADD /deploy_schemas.sql/ /docker-entrypoint-initdb.d/
Here is my deploy_schemas.sql
-- Deploy login and users tables
\i '/docker-entrypoint-initdb.d/tables/users.sql'
\i '/docker-entrypoint-initdb.d/tables/login.sql'
Here is my users.sql
BEGIN TRANSACTION
CREATE TABLE users (
id serial PRIMARY KEY,
name VARCHAR(100),
email text UNIQUE NOT NULL,
entries BIGINT DEFAULT 0,
joined TIMESTAMP NOT NULL
);
COMMIT;
Here is my login.sql
BEGIN TRANSACTION
CREATE TABLE login (
id serial PRIMARY KEY,
hash VARCHAR(100) NOT NULL,
email text UNIQUE NOT NULL,
);
COMMIT;
and finally here is docker-compose.yml
version: "3.8"
services:
#Backend API
smart-brain-api:
container_name: backend
build: ./
command: npm start
working_dir: /usr/src/test-api
environment:
POSTGRES_URI: postgres://admin:password#postgres:5432/test-api
links:
- postgres
ports:
- "3000:3000"
volumes:
- ./:/usr/src/test-api
#Postgres
postgres:
environment:
POSTGRES_USER: admin
POSTGRES_DB: docker-test-api
POSTGRES_HOST: postgres
POSTGRES_PASSWORD: password
build: ./postgres
ports:
- "5432:5432"
any ideas?
you miss semicolon after BEGIN TRANSACTION

mongo db docker image authentication failed

I'm using https://hub.docker.com/_/mongo mongo image in my local docker environment, but I'm getting Authentication failed error. In docker-compose I add it like:
my-mongo:
image: mongo
restart: always
container_name: my-mongo
environment:
MONGO_INITDB_ROOT_USERNAME: mongo
MONGO_INITDB_ROOT_PASSWORD: asdfasdf
networks:
- mynet
I also tried to run mongo CLI from inside the container but still getting the same error:
root#76e6db78228b:/# mongo
MongoDB shell version v4.2.3
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("c87c0f0e-fe83-41a6-96e9-4aa4ede8fa25") }
MongoDB server version: 4.2.3
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
> use translations
switched to db translations
> db.auth("mongo", "asdfasdf")
Error: Authentication failed.
0
Also, I'm trying to create a separate user:
> use admin
switched to db admin
db.auth("mongo", "asdfasdf")
1
> db.createUser({
user: "user",
pwd: "asdfasdf",
roles: [ {role: "readWrite", db: "translations" } ]
})
Successfully added user: {
"user" : "user",
"roles" : [
{
"role" : "readWrite",
"db" : "translations"
}
]
}
> use translations
switched to db translations
> db.auth("user", "asdfasdf")
Error: Authentication failed.
0
and the same, what I'm doing wrong???
Updated:
root#8bf81ef1fc4f:/# mongo -u mongo -p asdfasdf --authenticationDatabase admin
MongoDB shell version v4.2.3
connecting to: mongodb://127.0.0.1:27017/?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("02231489-eaf4-40be-a108-248cec88257e") }
MongoDB server version: 4.2.3
Server has startup warnings:
2020-02-26T16:24:12.942+0000 I STORAGE [initandlisten]
2020-02-26T16:24:12.943+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-02-26T16:24:12.943+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
> db.createUser({user: "someuser", pwd: "asdfasdf", roles: [{role: "readWrite", db: "translations"}]})
Successfully added user: {
"user" : "someuser",
"roles" : [
{
"role" : "readWrite",
"db" : "translations"
}
]
}
> use translations
switched to db translations
> db.auth("someuser", "asdfasdf")
Error: Authentication failed.
0
>
After some time, I figured out.
On the same folder, create docker-compose.yml and init-mongo.js
docker-compose.yml
version: '3.7'
services:
database:
image: mongo
container_name : your-cont-name
command: mongod --auth
environment:
- MONGO_INITDB_DATABASE=my_db
- MONGO_INITDB_ROOT_USERNAME=admin
- MONGO_INITDB_ROOT_PASSWORD=root
ports:
- '27017-27019:27017-27019'
volumes:
- mongodbdata:/data/db
- ./init-mongo.js:/docker-entrypoint-initdb.d/mongo-init.js:ro
volumes:
mongodbdata:
driver: local
init-mongo.js
db.createUser(
{
user: "your_user",
pwd: "your_password",
roles: [
{
role: "readWrite",
db: "my_db"
}
]
}
);
db.createCollection("test"); //MongoDB creates the database when you first store data in that database
Auth
First, execute the bash inside the container
docker exec -it your-cont-name bash
Now we can login.
For the admin
mongo -u admin -p root
For the your_user you have to specify the db (with the --authenticationDatabase) otherwise you'll have an auth error
mongo -u your_user -p your_password --authenticationDatabase my_db
After that, you should switch to the right db with
use my_db
If you don't execute this command, you'll be on test db
Note
For being sure of having the right config, i prefer to
docker-compose stop
docker-compose rm
docker volume rm <your-volume>
docker-compose up --build -d
as stated in the Docs
These variables, used in conjunction, create a new user and set that
user's password. This user is created in the admin authentication
database and given the role of root, which is a "superuser" role.
so you need to add --authenticationDatabase admin to your command since the mongod will be started with mongod --auth
example:
mongo -u mongo -p asdfasdf --authenticationDatabase admin
i have the same issue, after google two hours finally sovled;
solution:find out the host machine direcory mounted into mongodb container, delete it,then re-create the mongodb container.
mongo db container create by docker-compose.yaml mount a diretory from host mechine to the container for save the mongo datbases. when you remove the container the mouted direcotry do not deleted, so the default username and password pass by env var could be long time ago you set, now you change the user name and password. just do not work,cause recreate the container will not recreate the "admin" database .
I've fallen in this trap and wasted a day while everything was correct.
I'm writing this for future me(s) because it wasn't mentioned anywhere else and also to avoid my mistake while setting up user/pass combination to connect to their database from other services.
Assuming everything is right:👇
If you are mounting some local folder of yours as storage for your database like below:
services:
your_mongo_db:
// ...some config
volumes:
- ./__TEST_DB_DATA__:/data/db
- ./init-mongo.js:/docker-entrypoint-initdb.d/mongo-init.js:ro
environment:
- "MONGO_INITDB_ROOT_USERNAME=root"
- "MONGO_INITDB_ROOT_PASSWORD=pass"
//...more config
Please remember to remove this folder before re-running your compose file. I think when you run the docker-compose command for the first time, Mongo will create and store the user data there (like any other collections) and then reuse it for the next time (since you mounted that volume).
I had the same problem myself,
Please first remove the username and password from credentials.
environment:
MONGO_INITDB_ROOT_USERNAME: mongo
MONGO_INITDB_ROOT_PASSWORD: asdfasdf
after you remove the credentials you may check dbs or users on your mongodb.
show dbs
show users
Those commands also needs auth, so if you can see them, can be null, then you fix your issue.
Than,
then create a admin user,
use admin
db.createUser({user: "root", pwd: "root", roles:["root"]})
then you can logout and try to connect with credentials to the shell as an admin.
In addition if you are still having some issues about creating new user,
In my case I changed mechanisms to SCRAM-SHA-1 than it worked like a charm.
{
user: "<name>",
pwd: passwordPrompt(), // Or "<cleartext password>"
customData: { <any information> },
roles: [
{ role: "<role>", db: "<database>" } | "<role>",
...
],
authenticationRestrictions: [
{
clientSource: ["<IP>" | "<CIDR range>", ...],
serverAddress: ["<IP>" | "<CIDR range>", ...]
},
...
],
mechanisms: [ "<SCRAM-SHA-1|SCRAM-SHA-256>", ... ],
passwordDigestor: "<server|client>"
}
I had the same problem myself, follows this steps:
Steps 1 and 2 are to delete de old configuration, and set and apply the new configuration, its so important:
Delete to the containers mongo:
docker rm mongo -f
If you have created volumes, delete them:
docker volume rm $(docker volume ls -q) -f
In ports field of docker-compose.yml set:
- 27018:27017 ->
Its so important that ports is not 27017:27017, in my case it was generating conflict.
Up docker compose:
docker-compose up
Try now the connection with authentication!
Example of docker-compose.yml:
mongo:
container_name: mongo
image: mongo:4.4
restart: always
environment:
TZ: "Europe/Madrid"
MONGO_INITDB_ROOT_USERNAME: "user"
MONGO_INITDB_ROOT_PASSWORD: "admin1"
volumes:
- ./mongoDataBase:/data/db
ports:
- 27018:27017
Best regards!

Docker failed to run entry-point

So I have created a node/mongo app and I am trying to run everything on docker.
I can get everything to run just fine until I try and add the init file for the mongo instance into the entry-point.
Here is my docker file for mongo: (Called mongo.dockerfile in /MongoDB)
FROM mongo:4.2
WORKDIR /usr/src/mongo
VOLUME /docker/volumes/mongo /user/data/mongo
ADD ./db-init /docker-entrypoint-initdb.d
CMD ["mongod", "--auth"]
The db-init folder contains an init.js file that looks like so (removed the names of stuff):
db.createUser({
user: '',
pwd: '',
roles: [ { role: 'readWrite', db: '' } ]
})
Here is my docker-compose file:
version: "3.7"
services:
web:
container_name: web
env_file:
- API/web.env
build:
context: ./API
target: prod
dockerfile: web.dockerfile
ports:
- "127.0.0.1:3000:3000"
depends_on:
- mongo
links:
- mongo
restart: always
mongo:
container_name: mongo
env_file:
- MongoDB/mongo.env
build:
context: ./MongoDB
dockerfile: mongo.dockerfile
restart: always
The exact error I get when running a docker-compose up is:
ERROR: for mongo Cannot start service mongo: OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \"docker-entrypoint-initdb.d\": executable file not found in $PATH": unknown
I had this working at one point with another project but cannot seem to get this on to work at all.
Any thoughts on what I am doing wrong?
Also note I have seen other issues like this saying to chmod +x the path (tried that didnt work)
Also tried to chmod 777 also didnt work. (Maybe I did this wrong and I dont know exactly what to run this on?)
Your entrypoint has been modified from the upstream image, and it's not clear how from the input you've provided. You may have modified the mongo image itself and need to pull a fresh copy with docker-compose build --pull. Otherwise, you can force the entrypoint back to the upstream value:
ENTRYPOINT ["docker-entrypoint.sh"]