I try to connect two docker containers, one is localstack with lambda function I am invoking, the other one is mongo replica set.I tried connecting them with various network configurations, but I fail to reach the mongo from lambda.
Lambda VPC config is empty: "VpcConfig": {},, I read that it should be if I want to access external network.
Key points:
I can connect to mongo from my rest API app on my host.
I can ping the mongo container from within localstack container.
root#e42146a357e1:/opt/code/localstack# ping db
PING db (172.19.0.2) 56(84) bytes of data.
64 bytes from db.localstack_backend (172.19.0.2): icmp_seq=1 ttl=64 time=0.040 ms
64 bytes from db.localstack_backend (172.19.0.2): icmp_seq=2 ttl=64 time=0.038 ms
...
The error I get from Lambda:
MongoServerSelectionError: connect ECONNREFUSED 127.0.0.1:27017
at Timeout._onTimeout (/tmp/function.zipfile.d9dce9f0/node_modules/mongodb/lib/sdam/topology.js:277:38)
at listOnTimeout (node:internal/timers:564:17)
at process.processTimers (node:internal/timers:507:7) {
reason: TopologyDescription {
type: 'ReplicaSetNoPrimary',
servers: Map(3) {
'127.0.0.1:27017' => [ServerDescription],
'127.0.0.1:27018' => [ServerDescription],
'127.0.0.1:27019' => [ServerDescription]
},
stale: false,
compatible: true,
heartbeatFrequencyMS: 10000,
localThresholdMS: 15,
setName: 'rs0',
maxElectionId: new ObjectId("7fffffff0000000000000001"),
maxSetVersion: 1,
commonWireVersion: 0,
logicalSessionTimeoutMinutes: null
},
code: undefined,
[Symbol(errorLabels)]: Set(0) {}
}
What I don't know here is if the error comes from the server or from the client. I also don't know why in the error servers are set to 127.0.0.1:port.
Connection string which lambda uses: mongodb://db:27017,db:27018,db:27019/?replicaSet=rs0
I tried also with IPs from docker network.
Client connection:
const client = new MongoClient('mongodb://db:27017,db:27018,db:27019/?replicaSet=rs0', {
maxPoolSize: 20,
minPoolSize: 0,
retryReads: true,
retryWrites: true,
});
Docker compose file:
services:
db:
image: candis/mongo-replica-set
ports:
- "27017:27017"
- "27018:27018"
- "27019:27019"
networks:
- backend
container_name: db
localstack:
image: localstack/localstack
ports:
- "4510-4559:4510-4559"
- "4566:4566"
networks:
- backend
container_name: localstack
networks:
backend:
driver: bridge
I was thinking that maybe lambda can't connect with outside world due to VPC configuration.
Related
I have a custom made docker image for the backend of my app. I have a yaml file that runs my app image and a mongo image. However, when I use docker-compose on the yml file, I get the following error (about 20 seconds and the containers start running):
(node:33) [MONGOOSE] DeprecationWarning: Mongoose: the `strictQuery` option will be switched back to `false` by default in Mongoose 7. Use `mongoose.set('strictQuery', false);` if you want to prepare for this change. Or use `mongoose.set('strictQuery', true);` to suppress this warning.
(Use `node --trace-deprecation ...` to show where the warning was created)
Server listening on port 3000
/cloudband/node_modules/mongoose/lib/connection.js:825
const serverSelectionError = new ServerSelectionError();
^
MongooseServerSelectionError: connect ECONNREFUSED 127.0.0.1:27017
at Connection.openUri (/cloudband/node_modules/mongoose/lib/connection.js:825:32)
at /cloudband/node_modules/mongoose/lib/index.js:409:10
at /cloudband/node_modules/mongoose/lib/helpers/promiseOrCallback.js:41:5
at new Promise (<anonymous>)
at promiseOrCallback (/cloudband/node_modules/mongoose/lib/helpers/promiseOrCallback.js:40:10)
at Mongoose._promiseOrCallback (/cloudband/node_modules/mongoose/lib/index.js:1262:10)
at Mongoose.connect (/cloudband/node_modules/mongoose/lib/index.js:408:20)
at Object.<anonymous> (/cloudband/server/server.js:15:4)
at Module._compile (node:internal/modules/cjs/loader:1239:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1293:10) {
reason: TopologyDescription {
type: 'Unknown',
servers: Map(1) {
'localhost:27017' => ServerDescription {
address: 'localhost:27017',
type: 'Unknown',
hosts: [],
passives: [],
arbiters: [],
tags: {},
minWireVersion: 0,
maxWireVersion: 0,
roundTripTime: -1,
lastUpdateTime: 28094812,
lastWriteDate: 0,
error: MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017
at connectionFailureError (/cloudband/node_modules/mongodb/lib/cmap/connect.js:387:20)
at Socket.<anonymous> (/cloudband/node_modules/mongodb/lib/cmap/connect.js:310:22)
at Object.onceWrapper (node:events:628:26)
at Socket.emit (node:events:513:28)
at emitErrorNT (node:internal/streams/destroy:151:8)
at emitErrorCloseNT (node:internal/streams/destroy:116:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
cause: Error: connect ECONNREFUSED 127.0.0.1:27017
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1495:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 27017
},
[Symbol(errorLabels)]: Set(1) { 'ResetPool' }
},
topologyVersion: null,
setName: null,
setVersion: null,
electionId: null,
logicalSessionTimeoutMinutes: null,
primary: null,
me: null,
'$clusterTime': null
}
},
stale: false,
compatible: true,
heartbeatFrequencyMS: 10000,
localThresholdMS: 15,
setName: null,
maxElectionId: null,
maxSetVersion: null,
commonWireVersion: 0,
logicalSessionTimeoutMinutes: null
},
code: undefined
}
Here are my files:
Dockerfile:
FROM node:19.4.0
WORKDIR /cloudband
COPY package.json /cloudband/
COPY package-lock.json /cloudband/
RUN npm ci
COPY .env /cloudband/
COPY server /cloudband/server/
EXPOSE 3000
CMD ["npm", "run", "dev:server"]
YAML file:
version: '3'
services:
mongo:
image: mongo
container_name: mongo
ports:
- 27017:27017
environment:
- MONGO_INITDB_ROOT_USERNAME=admin
- MONGO_INITDB_ROOT_PASSWORD=password
cloudband:
image: cloudband
container_name: cloudband
ports:
- 3000:3000
command: npm run dev:server
networks:
app:
I expected my application and mongo db to start running in their respective containers and for them to be able to communicate (i.e. create documents / find documents / etc.).
What I have already tried:
-making sure they are in the same network (they are)
-making sure they can ping each other (they can)
-adding links to my app in the yaml file
-checked configurations and i think they are ok (port, host, ip)
-switching my uri to the following things:
# MONGO_URI_=mongodb://admin:password#localhost:27017/dbname
MONGO_URI_=mongodb://localhost:27017/dbname
# MONGO_URI_=mongodb://127.0.0.1:27017/dbname
Things to consider:
node v18.12.0 is installed on my computer
In a container, localhost means the container itself.
Docker-compose creates a docker network where the containers can talk to each other using their service name or container names as host names.
So, instead of
MONGO_URI_=mongodb://localhost:27017/dbname
you need to use
MONGO_URI_=mongodb://mongo:27017/dbname
I'm having trouble creating a mongo database using the docker-compose command. Docker desktop tells me that everything is up and running including the db, but all I get is the standard 'admin, config, local' not the db I want to create. Here's my docker-compose.yaml
version: '3'
services:
app:
build: ./
entrypoint: ./.docker/entrypoint.sh
ports:
- 3000:3000
volumes:
- .:/home/node/app
depends_on:
- db
db:
image: mongo:4.4.4
restart: always
volumes:
- ./.docker/dbdata:/data/db
- ./.docker/mongo:/docker-entrypoint-initdb.d
environment:
- MONGO_INITDB_ROOT_USERNAME=root
- MONGO_INITDB_ROOT_PASSWORD=root
- MONGO_INITDB_DATABASE=nest
mongo-express:
image: mongo-express
restart: always
ports:
- 8081:8081
environment:
- ME_CONFIG_MONGODB_SERVER=db
- ME_CONFIG_MONGODB_AUTH_USERNAME=root
- ME_CONFIG_MONGODB_AUTH_PASSWORD=root
- ME_CONFIG_MONGODB_ADMINUSERNAME=root
- ME_CONFIG_MONGODB_ADMINPASSWORD=root
depends_on:
- db
my init.js inside .docker/mongo
db.routes.insertMany([
{
_id: "1",
title: "Primeiro",
startPosition: {lat: -15.82594, lng: -47.92923},
endPosition: {lat: -15.82942, lng: -47.92765},
},
{
_id: "2",
title: "Segundo",
startPosition: {lat: -15.82449, lng: -47.92756},
endPosition: {lat: -15.82776, lng: -47.92621},
},
{
_id: "3",
title: "Terceiro",
startPosition: {lat: -15.82331, lng: -47.92588},
endPosition: {lat: -15.82758, lng: -47.92532},
}
]);
and my dockerfile
FROM node:14.18.1-alpine
RUN apk add --no-cache bash
RUN npm install -g #nestjs/cli
USER node
WORKDIR /home/node/app
and this is the 'error' log I get from docker when I run the nest container with mongodb, nest app and mongo express(there is actually a lot more but SO keeps thinking that it is spam for some reason.
about to fork child process, waiting until server is ready for connections.
Successfully added user: {
"user" : "root",
"roles" : [
{
"role" : "root",
"db" : "admin"
}
]
}
Error saving history file: FileOpenFailed Unable to open() file /home/mongodb/.dbshell: No such file or directory
{"t":{"$date":"2022-06-01T19:39:15.542+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn2","msg":"Connection ended","attr":{"remote":"127.0.0.1:39304","connectionId":2,"connectionCount":0}}
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.js
{"t":{"$date":"2022-06-01T19:39:15.683+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:39310","connectionId":3,"connectionCount":1}}
{"t":{"$date":"2022-06-01T19:39:15.684+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn3","msg":"client metadata","attr":{"remote":"127.0.0.1:39310","client":"conn3","doc":{"application":{"name":"MongoDB Shell"},"driver":{"name":"MongoDB Internal Client","version":"4.4.4"},"os":{"type":"Linux","name":"Ubuntu","architecture":"x86_64","version":"18.04"}}}}
{"t":{"$date":"2022-06-01T19:39:15.701+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"conn3","msg":"createCollection","attr":{"namespace":"nest.routes","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"f689868e-af6d-4ec6-b555-dcf520f24788"}},"options":{}}}
{"t":{"$date":"2022-06-01T19:39:15.761+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"conn3","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"nest.routes","index":"_id_","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}
uncaught exception: ReferenceError: colection is not defined :
#/docker-entrypoint-initdb.d/init.js:23:1
failed to load: /docker-entrypoint-initdb.d/init.js
exiting with code -3
this is what running docker-compose ps shows
NAME COMMAND SERVICE STATUS PORTS
nest-api-app-1 "./.docker/entrypoin…" app running 0.0.0.0:3000->3000/tcp
nest-api-db-1 "docker-entrypoint.s…" db running 27017/tcp
nest-api-mongo-express-1 "tini -- /docker-ent…" mongo-express running 0.0.0.0:8081->8081/tcp
this what my docker desktop shows
The MongoDB container only creates a database if no database already exists. You probably already have one, which is why a new database isn't created and your initialization script isn't run.
Delete the contents of ./.docker/dbdata on the host. Then start the containers with docker-compose and Mongo should create your database for you.
I have a mongo container, started with the requireTLS TLS mode, and a mongo-express container. Mongo-express does not seem to manage to connect to mongo using TLS.
My docker-compose.yml:
version: '3.1'
services:
mongodb1:
image : "mongo:4.2"
container_name : "mongodb-001"
ports:
- '27017:27017'
environment:
MONGO_INITDB_ROOT_USERNAME : "admin"
MONGO_INITDB_ROOT_PASSWORD : "adminpasswd"
volumes:
- "./mongo-data:/data/db"
- "./etc_mongod.conf:/etc/mongod.conf"
- "./certificates:/etc/certificates:ro"
command:
- "--tlsMode"
- "preferTLS"
- "--tlsDisabledProtocols"
- "none"
- "--tlsCertificateKeyFile"
- "/etc/certificates/certificateKey.pem"
- "--tlsCAFile"
- "/etc/certificates/CA.crt"
- "--tlsAllowConnectionsWithoutCertificates"
mongo-express:
image : "mongo-express:latest"
container_name : "mongo-express-001"
ports:
- '8081:8081'
depends_on:
- mongodb1
volumes:
- "./certificates/CA.crt:/etc/certificates/CA.crt:ro"
environment:
ME_CONFIG_MONGODB_SERVER: "mongodb-001"
ME_CONFIG_MONGODB_PORT: "27017"
ME_CONFIG_MONGODB_ENABLE_ADMIN: "false"
ME_CONFIG_MONGODB_AUTH_DATABASE: "admin"
ME_CONFIG_MONGODB_AUTH_USERNAME: "admin"
ME_CONFIG_MONGODB_AUTH_PASSWORD: "adminpasswd"
ME_CONFIG_MONGODB_ADMINUSERNAME: "admin"
ME_CONFIG_MONGODB_ADMINPASSWORD: "adminpasswd"
ME_CONFIG_SITE_SSL_ENABLED: "true"
ME_CONFIG_MONGODB_CA_FILE: "/etc/certificates/CA.crt"
...and the error message I get:
mongodb-001 | 2020-10-09T14:16:13.299+0000 I NETWORK [listener] connection accepted from 172.31.0.3:44774 #2 (1 connection now open)
mongodb-001 | 2020-10-09T14:16:13.305+0000 I NETWORK [conn2] Error receiving request from client: SSLHandshakeFailed: The server is configured to only allow SSL connections. Ending connection from 172.31.0.3:44774 (connection id: 2)
mongodb-001 | 2020-10-09T14:16:13.305+0000 I NETWORK [conn2] end connection 172.31.0.3:44774 (0 connections now open)
mongo-express-001 |
mongo-express-001 | /node_modules/mongodb/lib/server.js:265
mongo-express-001 | process.nextTick(function() { throw err; })
mongo-express-001 | ^
mongo-express-001 | Error [MongoError]: connection 0 to mongodb-001:27017 closed
mongo-express-001 | at Function.MongoError.create (/node_modules/mongodb-core/lib/error.js:29:11)
mongo-express-001 | at Socket.<anonymous> (/node_modules/mongodb-core/lib/connection/connection.js:200:22)
mongo-express-001 | at Object.onceWrapper (events.js:422:26)
mongo-express-001 | at Socket.emit (events.js:315:20)
mongo-express-001 | at TCP.<anonymous> (net.js:674:12)
mongo-express-001 exited with code 1
Note that:
I can connect to MongoDB using a mongo shell with the same parameters I pass to mongo-express:
mongo "mongodb://admin:adminpasswd#mongodb-001:27017/admin?authSource=admin" --tls --tlsCAFile certificates/CA.crt
If I start MongoDB in preferTLS mode, the mongo-express connection works
tl;dr
Create a new config.js file with the following code
module.exports = {
mongodb: {
connectionOptions: {
ssl: true,
}
}
};
and mount that file in your docker compose file at /node_modules/mongo-express/config.js
Explanation
It appears to be an issue with their config.default.js file. In it, they have this
module.exports = {
mongodb: {
// if a connection string options such as server/port/etc are ignored
connectionString: mongo.connectionString || getConnectionStringFromEnvVariables(),
connectionOptions: {
// ssl: connect to the server using secure SSL
ssl: process.env.ME_CONFIG_MONGODB_SSL || mongo.ssl,
// sslValidate: validate mongod server certificate against CA
sslValidate: process.env.ME_CONFIG_MONGODB_SSLVALIDATE || true,
// sslCA: array of valid CA certificates
sslCA: sslCAFromEnv ? [sslCAFromEnv] : [],
// autoReconnect: automatically reconnect if connection is lost
autoReconnect: true,
// poolSize: size of connection pool (number of connections to use)
poolSize: 4,
}
}
You'll notice the existence of an env var that's not listed (with reason) in their documentation. ME_CONFIG_MONGODB_SSL
This is required to enable tls support. Setting the env var yourself however does nothing but throw an error and break express due to it not being cast to a Boolean. So it just reads as a string 'true' or 'false'.
This code is fixed in their npm package code, but they haven't updated their docker image since late 2021. So the only "fix" I've found for this is to create a new config.js file with the following code
module.exports = {
mongodb: {
connectionOptions: {
ssl: true,
sslValidate: true,
}
}
};
Then mount this file at /node_modules/mongo-express/config.js in your docker-compose file. It'll read these and overwrite the defaults.
Note: I added the sslValidate key:value pair as well due to the fact that it suffers from the same lack of type casting. So if you omit the ME_CONFIG_MONGODB_SSLVALIDATE env var entirely, it'll be set as true, but if you include the env var as either true or false, it'll just simply break (undefined behaviour).
I'm trying to connect to mongodb running in docker from the app running on host using mongoose but it failed.
I can't use the port 27017 for the new mongodb container because it is used by other container. So I followed the guide here for setting it up using the compose.
Below are the snippets:
docker-compose.yml
version: '3'
services:
db:
image: mongo:latest
restart: always
ports:
- '8081:8081'
environment:
MONGO_INITDB_ROOT_USERNAME: root1
MONGO_INITDB_ROOT_PASSWORD: password1
But when I do docker-ps, port 27017 still there but I'm not sure if that causes an issue.
PORTS
0.0.0.0:8081->8081/tcp, 27017/tcp
Then I created a new user in admin database.
use admin
db.createUser(
{
user: "admin1",
pwd: "password2",
roles: [ { role: "userAdminAnyDatabase", db: "admin" }, "readWriteAnyDatabase" ]
}
)
server.js
const connectOption = {
useNewUrlParser: true,
user: 'admin1',
pass: 'password2',
authSource: 'admin',
}
const mongoURL = 'mongodb://localhost:8081/app1';
mongoose.connect(mongoURL, connectOption)
.then(() => console.log('MongoDB Connected'))
.catch(error => console.log(error));
And the error I received is
{
MongoNetworkError: failed to connect to server [localhost:8081] on first connect [MongoNetworkError: write EPIPE]
...
...
...
name: 'MongoNetworkError',
errorLabels: [ 'TransientTransactionError' ],
[Symbol(mongoErrorContextSymbol)]: {}
}
Assuming you're running nodejs application as a docker-compose service, in db service remove ports section (including - '8081:8081'
line). In server.js, change const mongoURL = 'mongodb://localhost:8081/app1'; to const mongoURL = 'mongodb://db:27017/app1';.
If you want to access the db from host machine, change ports 8081:8081 to <give-a-port-number>:27017.
I have some troubles setting up Graylog2 under docker. Everything works until I try using authentication. All I get is the following error repeated forever.
Trying both root and graylog user (in both graylog and admin db) gives the same result.
The log from mongodb says both users are created during setup. But graylog says it does not find any graylog user in database graylog. Same with user root.
I'm new to MongoDB and have no idea how authentication works. But from what I understand authentication (similar to --auth parameter) is activated when providing user/pw for root account (https://github.com/docker-library/mongo/pull/145).
Is it possible that Graylog ses a different authentication mechanism than MongoDB is excpecting? See line #158 in the pasted log
Error message as root user
mongodb_1 | 2017-04-16T13:27:52.486+0000 I NETWORK [thread1] connection accepted from 172.18.0.4:46566 #12 (1 connection now open)
mongodb_1 | 2017-04-16T13:27:52.495+0000 I NETWORK [conn12] received client metadata from 172.18.0.4:46566 conn12: { driver: { name: "mongo-java-driver", version: "unknown" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "4.4.0-72-generic" }, platform: "Java/Oracle Corporation/1.8.0_72-internal-b15" }
mongodb_1 | 2017-04-16T13:27:52.525+0000 I ACCESS [conn12] SCRAM-SHA-1 authentication failed for root on graylog from client 172.18.0.4:46566 ; UserNotFound: Could not find user root#graylog
mongodb_1 | 2017-04-16T13:27:52.543+0000 I - [conn12] end connection 172.18.0.4:46566 (1 connection now open)
Error message as graylog user (Full log on pastebin)
mongodb_1 | 2017-04-16T15:47:48.404+0000 I NETWORK [thread1] connection accepted from 172.18.0.4:41602 #7 (1 connection now open)
mongodb_1 | 2017-04-16T15:47:48.410+0000 I NETWORK [conn7] received client metadata from 172.18.0.4:41602 conn7: { driver: { name: "mongo-java-driver", version: "unknown" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "4.4.0-72-generic" }, platform: "Java/Oracle Corporation/1.8.0_72-internal-b15" }
mongodb_1 | 2017-04-16T15:47:48.418+0000 I ACCESS [conn7] SCRAM-SHA-1 authentication failed for graylog on graylog from client 172.18.0.4:41602 ; UserNotFound: Could not find user graylog#graylog
mongodb_1 | 2017-04-16T15:47:48.423+0000 I - [conn7] end connection 172.18.0.4:41602 (1 connection now open)
This is my ./docker-composer.yml
version: '2'
services:
mongodb:
build: ./mongodb
volumes:
- /docker/mongodb/data:/data/db
elasticsearch:
image: "elasticsearch:2"
command: "elasticsearch -Des.cluster.name='graylog'"
volumes:
- /docker/elasticsearch/data:/usr/share/elasticsearch/data
graylog:
image: graylog2/server
volumes:
- /docker/graylog/journal:/usr/share/graylog/data/journal
- /docker/graylog/config:/usr/share/graylog/data/config
environment:
#GRAYLOG_MONGODB_URI: mongodb://root:drUqGGCMh#mongodb:27017/graylog
GRAYLOG_MONGODB_URI: mongodb://graylog:vWGzncmBe9#mongodb:27017/graylog
depends_on:
- mongodb
- elasticsearch
ports:
- "9000:9000"
./mongodb/Dockerfile
FROM mongo:3
ENV MONGO_INITDB_ROOT_USERNAME: root
ENV MONGO_INITDB_ROOT_PASSWORD: drUqGGCMh
ADD grayloguser.js /docker-entrypoint-initdb.d/grayloguser.js
./mogodb/grayloguser.js
db.getSiblingDB('graylog');
db.createUser(
{
user: "graylog",
pwd: "vWGzncmBe9",
roles: [
{ role: "dbOwner", db: "graylog" }
]
}
);
Your MongoDB script is incorrect.
Either assign the return value of db.getSiblingDB('graylog') to a variable and use that for createUser(), or keep using use graylog instead:
graylog = db.getSiblingDB('graylog');
graylog.createUser(
{
user: "graylog",
pwd: "vWGzncmBe9",
roles: [
{ role: "dbOwner", db: "graylog" }
]
}
);
In other words, just stick to the MongoDB documentation: https://docs.mongodb.com/manual/tutorial/create-users/#username-password-authentication