I'm going to conduct CI / CD in gitlab in AWS. My website contains a Postgresql database.
Question 1: But I don't know how to find the db host. Some say the host name is localhost, but I doubt it because I've deployed my website to AWS. Should it be elastic IP?
My .gitlab-ci.yml file is as follows:
image: node:latest
stages:
- testing
variables:
POSTGRES_DB: firstdb
POSTGRES_USER: johndoe
POSTGRES_PASSWORD: 1234
POSTGRES_HOST: //I don't know
testing:
services:
- postgres:latest
before_script:
- npm install -g yarn
- yarn install
- yarn knex migrate:latest --env testing
stage: testing
script:
- yarn jest
Question 2: Also, should I change the database config of development, testing and production of knex.ts accordingly, so that it aligns with .gitlab-ci.yml?
My knex file is as follows:
import * as dotenv from 'dotenv';
dotenv.config();
module.exports = {
development: {
client: 'postgresql',
connection: {
database: process.env.DB_NAME , //should I type actual data?
user: process.env.DB_USERNAME ,
password: process.env.DB_PASSWORD
},
pool: {
min: 2,
max: 10
},
migrations: {
tableName: 'knex_migrations'
}
},
testing:{
client: 'postgresql',
connection: {
host: process.env.POSTGRES_HOST ,//should I type actual data?
database: process.env.POSTGRES_DB ,
user: process.env.POSTGRES_USER ,
password: process.env.POSTGRES_PASSWORD
},
pool: {
min: 2,
max: 10
},
migrations: {
tableName: 'knex_migrations'
}
},
production: {
client: "postgresql",
connection: {
database: process.env.DB_NAME ,//should I type actual data?
user: process.env.DB_USERNAME ,
password: process.env.DB_PASSWORD
},
pool: {
min: 2,
max: 10
},
migrations: {
tableName: "knex_migrations"
}
}
};
Many thanks in advance. :)
The hostname is postgres. the hostname is derived from the image name.
explained here:
https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#how-services-are-linked-to-the-job
Related
I'm having trouble creating a mongo database using the docker-compose command. Docker desktop tells me that everything is up and running including the db, but all I get is the standard 'admin, config, local' not the db I want to create. Here's my docker-compose.yaml
version: '3'
services:
app:
build: ./
entrypoint: ./.docker/entrypoint.sh
ports:
- 3000:3000
volumes:
- .:/home/node/app
depends_on:
- db
db:
image: mongo:4.4.4
restart: always
volumes:
- ./.docker/dbdata:/data/db
- ./.docker/mongo:/docker-entrypoint-initdb.d
environment:
- MONGO_INITDB_ROOT_USERNAME=root
- MONGO_INITDB_ROOT_PASSWORD=root
- MONGO_INITDB_DATABASE=nest
mongo-express:
image: mongo-express
restart: always
ports:
- 8081:8081
environment:
- ME_CONFIG_MONGODB_SERVER=db
- ME_CONFIG_MONGODB_AUTH_USERNAME=root
- ME_CONFIG_MONGODB_AUTH_PASSWORD=root
- ME_CONFIG_MONGODB_ADMINUSERNAME=root
- ME_CONFIG_MONGODB_ADMINPASSWORD=root
depends_on:
- db
my init.js inside .docker/mongo
db.routes.insertMany([
{
_id: "1",
title: "Primeiro",
startPosition: {lat: -15.82594, lng: -47.92923},
endPosition: {lat: -15.82942, lng: -47.92765},
},
{
_id: "2",
title: "Segundo",
startPosition: {lat: -15.82449, lng: -47.92756},
endPosition: {lat: -15.82776, lng: -47.92621},
},
{
_id: "3",
title: "Terceiro",
startPosition: {lat: -15.82331, lng: -47.92588},
endPosition: {lat: -15.82758, lng: -47.92532},
}
]);
and my dockerfile
FROM node:14.18.1-alpine
RUN apk add --no-cache bash
RUN npm install -g #nestjs/cli
USER node
WORKDIR /home/node/app
and this is the 'error' log I get from docker when I run the nest container with mongodb, nest app and mongo express(there is actually a lot more but SO keeps thinking that it is spam for some reason.
about to fork child process, waiting until server is ready for connections.
Successfully added user: {
"user" : "root",
"roles" : [
{
"role" : "root",
"db" : "admin"
}
]
}
Error saving history file: FileOpenFailed Unable to open() file /home/mongodb/.dbshell: No such file or directory
{"t":{"$date":"2022-06-01T19:39:15.542+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2","msg":"Connection ended","attr":{"remote":"127.0.0.1:39304","connectionId":2,"connectionCount":0}}
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.js
{"t":{"$date":"2022-06-01T19:39:15.683+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:39310","connectionId":3,"connectionCount":1}}
{"t":{"$date":"2022-06-01T19:39:15.684+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn3","msg":"client metadata","attr":{"remote":"127.0.0.1:39310","client":"conn3","doc":{"application":{"name":"MongoDB Shell"},"driver":{"name":"MongoDB Internal Client","version":"4.4.4"},"os":{"type":"Linux","name":"Ubuntu","architecture":"x86_64","version":"18.04"}}}}
{"t":{"$date":"2022-06-01T19:39:15.701+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"conn3","msg":"createCollection","attr":{"namespace":"nest.routes","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"f689868e-af6d-4ec6-b555-dcf520f24788"}},"options":{}}}
{"t":{"$date":"2022-06-01T19:39:15.761+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"conn3","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"nest.routes","index":"_id_","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}
uncaught exception: ReferenceError: colection is not defined :
#/docker-entrypoint-initdb.d/init.js:23:1
failed to load: /docker-entrypoint-initdb.d/init.js
exiting with code -3
this is what running docker-compose ps shows
NAME COMMAND SERVICE STATUS PORTS
nest-api-app-1 "./.docker/entrypoin…" app running 0.0.0.0:3000->3000/tcp
nest-api-db-1 "docker-entrypoint.s…" db running 27017/tcp
nest-api-mongo-express-1 "tini -- /docker-ent…" mongo-express running 0.0.0.0:8081->8081/tcp
this what my docker desktop shows
The MongoDB container only creates a database if no database already exists. You probably already have one, which is why a new database isn't created and your initialization script isn't run.
Delete the contents of ./.docker/dbdata on the host. Then start the containers with docker-compose and Mongo should create your database for you.
I used Typeorm with NestJS
I want to config connection Pool more than 10.
But always max connection 10....
// My Typeorm config
import { TypeOrmModuleOptions } from '#nestjs/typeorm'
import * as dotenv from 'dotenv'
import { SnakeNamingStrategy } from 'typeorm-naming-strategies'
const config: TypeOrmModuleOptions = {
type: 'postgres',
host: process.env.POSTGRES_HOST,
port: parseInt(process.env.POSTGRES_PORT || '5432'),
username: process.env.POSTGRES_USER,
password: process.env.POSTGRES_PASS,
database: process.env.POSTGRES_DATABASE,
entities: ['src/**/*.entity{.ts,.js}'],
namingStrategy: new SnakeNamingStrategy(),
migrationsTableName: 'migration',
migrations: ['src/db/migrations/*.ts'],
cli: {
migrationsDir: 'src/db/migrations',
},
extra: {
max: 30
}
}
export default config
What should i do for postgres connection more than 10?
I'm trying to connect to mongodb running in docker from the app running on host using mongoose but it failed.
I can't use the port 27017 for the new mongodb container because it is used by other container. So I followed the guide here for setting it up using the compose.
Below are the snippets:
docker-compose.yml
version: '3'
services:
db:
image: mongo:latest
restart: always
ports:
- '8081:8081'
environment:
MONGO_INITDB_ROOT_USERNAME: root1
MONGO_INITDB_ROOT_PASSWORD: password1
But when I do docker-ps, port 27017 still there but I'm not sure if that causes an issue.
PORTS
0.0.0.0:8081->8081/tcp, 27017/tcp
Then I created a new user in admin database.
use admin
db.createUser(
{
user: "admin1",
pwd: "password2",
roles: [ { role: "userAdminAnyDatabase", db: "admin" }, "readWriteAnyDatabase" ]
}
)
server.js
const connectOption = {
useNewUrlParser: true,
user: 'admin1',
pass: 'password2',
authSource: 'admin',
}
const mongoURL = 'mongodb://localhost:8081/app1';
mongoose.connect(mongoURL, connectOption)
.then(() => console.log('MongoDB Connected'))
.catch(error => console.log(error));
And the error I received is
{
MongoNetworkError: failed to connect to server [localhost:8081] on first connect [MongoNetworkError: write EPIPE]
...
...
...
name: 'MongoNetworkError',
errorLabels: [ 'TransientTransactionError' ],
[Symbol(mongoErrorContextSymbol)]: {}
}
Assuming you're running nodejs application as a docker-compose service, in db service remove ports section (including - '8081:8081'
line). In server.js, change const mongoURL = 'mongodb://localhost:8081/app1'; to const mongoURL = 'mongodb://db:27017/app1';.
If you want to access the db from host machine, change ports 8081:8081 to <give-a-port-number>:27017.
In my project I am using https://www.npmjs.com/package/dotenv-safe in order to declare environment variables needed for configuration. For example:
NODE_ENV=development
JWT_SECRET=xxxxxxx
JWT_EXPIRATION_MINUTES=15
MONGO_URI=mongodb://mongodb:27017/proddb
BASE_URI=http://localhost:3000/
MONGO_URI_TESTS=mongodb://mongodb:27017/testdb
PORT=3000
Then I use those files in a config file :
module.exports = {
env: process.env.NODE_ENV,
port: process.env.PORT,
jwtSecret: process.env.JWT_SECRET,
jwtExpirationInterval: process.env.JWT_EXPIRATION_MINUTES,
mongo: {
uri: process.env.NODE_ENV === 'test'
? process.env.MONGO_URI_TESTS
: process.env.MONGO_URI,
},
logs: process.env.NODE_ENV === 'production' ? 'combined' : 'dev',
};
and in my package.json file, I've got:
"scripts": {
"start": "NODE_ENV=production node ./src/index.js",
"dev": "LOG_LEVEL=debug nodemon --inspect=0.0.0.0 ./src/index.js",
"test": "NODE_ENV=test nyc --reporter=html --reporter=text mocha --timeout 20000 --recursive src/tests"
}
The problem? Everything works fine but when tests are run on Heroku (prod) , they run on the main database and not on the testdb...
I am struggling to get PostgreSQL 9.2 installed with the Opscode postgresql cookbook on a Vagrant VM (Ubuntu 12.04) .
This is the my recipe:
node.set['postgresql']['version'] = "9.2"
node.set['postgresql']['enable_pgdg_apt'] = true
node.set['postgresql']['password'] = {postgres: "pwd"}
node.set['postgresql']['server']['packages'] = ["postgresql-9.2"]
include_recipe "postgresql::apt_pgdg_postgresql"
include_recipe "postgresql::server"
include_recipe "database"
The run results in the following:
[2013-05-23T11:00:52+00:00] FATAL: Chef::Exceptions::EnclosingDirectoryDoesNotExist:
template[/etc/postgresql/9.2/main/postgresql.conf] (postgresql::server line 60) had an error:
Chef::Exceptions::EnclosingDirectoryDoesNotExist: Parent directory /etc/postgresql/9.2/main does not exist.
I am using the latest 3.0.0 version of the poostgresql cookbook.
Rerunning everything from scratch with a clean VM (vagrant destroy, up, etc) gives this error:
[2013-05-23T11:16:37+00:00] FATAL: Chef::Exceptions::EnclosingDirectoryDoesNotExist:
template[/etc/postgresql/9.1/main/postgresql.conf] (postgresql::server line 60) had an error:
Chef::Exceptions::EnclosingDirectoryDoesNotExist: Parent directory /etc/postgresql/9.1/main does not exist.
Suddenly we don't even have the right version.
Here's what finally fixed it for me:
run update-alternatives --remove postmaster.1.gz /usr/share/postgresql/9.1/man/man1/postmaster.1.gz
Use the conf below.
Snip:
postgresql: {
enable_pgdg_apt: true,
dir: "/etc/postgresql/9.2/main",
config: {
data_directory: "/var/lib/postgresql/9.2/main",
hba_file: "/etc/postgresql/9.2/main/pg_hba.conf",
ident_file: "/etc/postgresql/9.2/main/pg_ident.conf",
external_pid_file: "/var/run/postgresql/9.2-main.pid",
ssl_key_file: "/etc/ssl/private/ssl-cert-snakeoil.key",
ssl_cert_file: "/etc/ssl/certs/ssl-cert-snakeoil.pem",
},
client: {
packages: ["postgresql-client-9.2",],
},
server: {
packages: ["postgresql-9.2", "postgresql-server-dev-9.2"],
},
contrib: {
packages: ["postgresql-contrib-9.2"],
},
password: {
postgres: 'postgres'
},
pg_hba: [
{type: 'local', db: 'all', user: 'all', addr: nil, method: 'trust'},
{type: 'host', db: 'all', user: 'all', addr: '127.0.0.1/32', method: 'trust'},
{type: 'host', db: 'all', user: 'all', addr: '::1/128', method: 'trust'}
],
version: "9.2",
},
I was able to fix this by setting the directory:
node.set['postgresql']['dir'] = "/var/lib/postgresql/9.2/main"
Looks like this is the same issue: http://tickets.opscode.com/browse/COOK-2113