I am using a simple go CRUD api that uses MongoDB and moving it to docker containers. I cannot connect to the MongoDB for some reason. After researching I cannot find a solution.
I have tried:
Exposing/publishing ports, so I can use the container name instead of 'mongodb:localhost:27017/' when trying to connect to the client.
I have also removed any network config in my compose.yml file so that there is no network confusion.
This is my compose.yml file:
version: '3.4'
services:
mongodb:
image: mongo:4.0.4
restart: always
ports:
- 27017:27017
mongo_todo:
build: ./mongo_todo
ports:
- 3000:3000
depends_on:
- mongodb
go_todo:
build: ./go_todo
ports:
- 80:80
depends_on:
- mongo_todo
This is my mongo_todo Dockerfile:
FROM golang:1.14
WORKDIR /go/src/app
COPY . .
RUN go get -d -v ./...
RUN go install -v ./...
EXPOSE 80
And this is how I am trying to connect to the client:
// Set client options
clientOptions := options.Client().ApplyURI("mongodb://mongodb:27017")
// Connect to MongoDB
Client, _ = mongo.Connect(context.TODO(), clientOptions)
// Check the connection
err := Client.Ping(context.TODO(), nil)
if err != nil {
log.Fatal(err)
}
The program is logging a Fatal error when trying to ping the client to check the connection.
Log output:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x2e0 pc=0x9a798c]
goroutine 1 [running]:
go.mongodb.org/mongo-driver/mongo.(*Client).Ping(0x0, 0xd61e20, 0xc000028030, 0x0, 0x1, 0x0)
/go/src/go.mongodb.org/mongo-driver/mongo/client.go:229 +0x21c
main.main()
/go/src/app/main.go:30 +0x16f
I understand that it is some kind of networking issue but I have no idea why it is not working and I have a feeling that there's something simple that I am missing.
Please ask me for more information if needed and thanks in advance for any help.
Edit: I can ping the hostname mongodb from my go_todo container so not sure if it is a network issue.
Yeah, this is confusing a bit (read: annoying) part of using docker-compose.
The service called mongodb is not what the container name is that you set your ApplyURI to. docker-compose prefixes a "project name" to the container name.
Run a docker ps and look for the column 'NAMES' (should be the last one). Use the name of your mongodb instance for the URI host, and it should connect.
I am so stupid and clearly a docker noob.
I thought that docker-compose up rebuilt my container images but it doesn't and so my changes were not being used.
Thanks you for your comments though I appreciate it!
You should debug the code and check which object is Nil. Maybe you are calling method on Nil object.
I will suggest you to explicitly create a Docker network and attach your MongoDB docker instance and your application to it. Then access your MongoDB using container name. I also faced the connection issue and I resolved that using Docker network.
As #CenterOrbit suggested, name your containers explicitly otherwise docker will give it a random name, and you won't be able to connect to MongoDB.
Related
I've followed the instructions in https://tsmx.net/docker-local-mongodb/ but I still get the following error:
**panic: unable to connect to MongoDB (local): no reachable servers
**
I even tried the following but still get the same error:
_ = pflag.String("mongodb-addr", "127.0.0.1:27017", "MongoDB connection address")
My connection code is as follows:
dbAddr := d.cfg.GetString("mongodb-addr")
session, err := mgo.Dial(dbAddr)
And my docker run command is as follows:
docker run image_name
I'm using macOS Monterey. Any help would be greatly appreciated. Thanks.
If the application and the MongoDB are on the same docker network, then use the docker name to connect to the MongoDB container.
If the MongoDB is running in the server where the application is running in docker container, then use the IP of the server to communicate to the MongoDB. 127.0.0.1 from within the container will try to find the MongoDB within the same Docker as the application.
if you run mongo like this :
mongo:
image: mongo
restart: always
volumes:
- ./mongo-data:/data/db
env_file: .env
ports:
- 27017:27017
environment:
MONGO_INITDB_ROOT_USERNAME: ${MONGO_USERNAME}
MONGO_INITDB_ROOT_PASSWORD: ${MONGO_PASSWORD}
then you can connect from Go like this :
var cred options.Credential
cred.Username = MongoUsername
cred.Password = MongoPassword
clientOption := options.Client().ApplyURI(mongodb://mongodb:27017).SetAuth(cred)
I was facing the same issue and this command did the trick for me, was mentioned here.
Docker provides a host network which lets containers share your host’s networking stack. This approach means localhost inside a container resolves to the physical host, instead of the container itself.
docker run -d --network=host my-container:latest
hope it help someone.
I currently have two docker containers set up on my PC where one is a PostgreSQL container created with docker compose and the other contains go code to initialize a table within the PostgreSQL db. Docker compose works perfectly in creating the container/db (i am to log into it through psql outside of the container) and the go code itself runs properly outside of a container (creates tables). The problem arises when I try to run the code from inside its own container whether I try to connect to a local database or the containerized one. Both cases lead to the dial error in the title.
I'm just a beginner when it comes to docker so I know its most likely a problem on my end but here are the contents of my dockerfile:
FROM golang:alpine AS builder
#adding needed env variables
ENV GO111MODULE=on \
CGO_ENABLED=0 \
GOOS=linux \
GOARCH=amd64
#move to /build
WORKDIR /build
#copy dependancies
COPY go.mod .
COPY go.sum .
RUN go mod download
#add code to container
COPY . .
#build app
RUN go build -o main .
# Move to /dist directory as the place for resulting binary folder
WORKDIR /dist
# Copy binary from build to main folder
RUN cp /build/main .
# Build a small image
FROM scratch
COPY --from=builder /dist/main /
ENV DATABASE_URL=postgres://short:password#test:8001/shorturl
#change database url variable to match your needs
# Command to run
ENTRYPOINT ["/main"]
and the contents of the docker-compose.yml file:
version: '3'
services:
database:
container_name: test
image: "postgres" # use latest official postgres version
env_file:
- ./db/database.env # configure postgres
volumes:
- database-data:/var/lib/postgresql/data/ # persist data even if container shuts down
ports:
- "8001:5432"
backend:
build: .
volumes:
database-data:
The code itself exists in the following directory structure:
/project/
./db/
db.go
database.env
main.go
go.mod
go.sum
Dockerfile
docker-compose.yml
db.go contains:
package db
import (
"context"
"fmt"
"os"
"github.com/jackc/pgx/v4/pgxpool"
)
//InitDb creates a connection to our postgres db and populates it
func InitDb(dbURL string) *pgxpool.Pool {
connPool, err := pgxpool.Connect(context.Background(), dbURL)
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
os.Exit(1)
}
return connPool
}
//PopulateDb populates our databse with the appropiate tables i.e id, url, base64 encoding, visited, visit count
func PopulateDb(db *pgxpool.Pool) {
_, err := db.Exec(context.Background(), "create table if not exists shortener (id serial primary key not null, url varchar not null, visited boolean default false, count integer default 0);")
if err != nil {
fmt.Fprintf(os.Stderr, "Unable to create users table: %v\n", err)
os.Exit(1)
}
fmt.Printf("Successfully created users table\n")
}
database.env which is used to generate the database with docker-compose contains:
POSTGRES_USER=short
POSTGRES_PASSWORD=password
POSTGRES_DB=shorturl
main.go contains:
package main
import (
"fmt"
"os"
"project/db"
)
func main() {
/*
set databse url in this format from root directory
DATABASE_URL=postgres://user:password#host:port/database
*/
dbURL := os.Getenv("DATABASE_URL")
conn := db.InitDb(dbURL)
db.PopulateDb(conn)
defer conn.Close()
}
As I mentioned before running all of this outside of any containers works and populates the database and docker-compose successfully creates an instance of postgres that I can connect to locally.
What I cant do is connect to the containerized postgres through my code (ran locally or in a container) or connect to a local database from the containerized version of my code. All of those result in some variation of this error:
Unable to connect to database: failed to connect to `host=container_name user=short database=shorturl`: dial error (dial tcp 172.18.0.2:8001: connect: connection refused)
I'm curious about what I might be doing wrong in my code and/or with docker and hoping someone can run it on their end and reproduce my issues/find whats wrong.
I'll try to clarify what Michée Lengronne wrote, I think his answer is only partly right.
You have two containers - test, which has postgres running inside, and an unnamed one with your go app inside, the name for this one is determined randomly every time the container starts.
Both those containers are on the same network. That is important.
When you define ports with value 8001:5432, it means that if you connect from outside the defined network, and you choose port 8001 to connect, it will connect to port 5432 on the inside.
But if you connect only from within the same network, you may omit the ports definition and go ahead and connect directly to test container.
(seriously, choose a better name, test_database at least)
Meaning, using the container name with default port (5432) should be enough.
TL;DR
DATABASE_URL=postgres://short:password#test:5432/shorturl
should work
If you want to connect to a local postgres or a postgres in another docker network, you must provide the IP or domain name of your postgres in that variable: DATABASE_URL=postgres://short:password#test:8001/shorturl
Normally you should replace test by localhost or 127.0.0.1
From the error, your app tries to connect to 172.18.0.2. This IP is the one from the postgres container I suppose. It is not reachable outside the docker network of this specific container. If you want to reach your container by this IP you must add your app container to the postgres docker network.
Docker network and compose: https://docs.docker.com/compose/networking/
I have built a docker image for a flask app I have with some html templates and after running my image I go to localhost:5000which takes me to the start page in my flask app . I press a register button to register a user using a flask endpoint but I get
pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused
Before going to localhost I run my mongodb image with sudo docker start mongodband the connection seems to hit this error whenever I have to search something in my monogdb database for the endpoint . Do I need a docker-compose.yml to connect and I cannot connect without one ?
This is how I connect to mongodb using pymongo
client = MongoClient('mongodb://localhost:27017/')
db = client['MovieFlixDB']
users = db['Users']
movies = db['Movies']
How I run my flask app :
if __name__ == '__main__':
app.run(debug=True, host='0.0.0.0', port=5000)
I would appreciate your help . Thank you in advance
To connect containers to each other you should use networks.
First you create a network
docker network create my-network
Run mongodb specyfing the network.
docker container run -d --name mongodb -p 27017:27017 --network my-network mongodb:latest
Modify your app to connect to mongodb as host instead of localhost. Containers that are connected to a common network can talk to each other by using their names (DNS names) that can be automatically resolved to container IPs.
client = MongoClient('mongodb://mongodb:27017/')
You could also think about providing such deatils (db host, user, password) through environment variables and read them in your app.
Rebuild image with your app and run it
docker container run --name flask-app -d --network my-network my-flaskapp-image
You can read more about container networking in docker docs.
Do I need a docker-compose.yml to connect and I cannot connect without
one ?
If you use docker-compose, it will be easier and don't have to use too many commands to deploy. Look at this example (there are too many however you can refer random service).
Steps -
Build your docker-componse file [I have modified the one in the example of random service, removing rest] e.g.
version: '3.3'
services:
web-random:
build:
context: .
args:
requirements: ./flask-mongodb-example/requirements.txt
image: web-random-image
ports:
- "800:5000"
entrypoint: python ./flask-mongodb-example/random_demo.py
depends_on:
- mongo
mongo:
image: mongo:4.2-bionic
ports:
- "27017:27017"
Refer this example to update your mongo URL in your python code
Now, use the following command to compose and bring up the containers
docker-compose build
docker-compose up
Now, either browse your URL with browser or use the curl command
I'm working on a very basic (I thought) starter program in Go using MongoDB and Docker. Trying to get a handle on these before we start using them at work.
I've got my MongoDB running in a docker container, just using my local host, using the official Docker image. This is running fine, I can connect to it through MongoDB Compass and modify the DB.
My next task was to build a separate Docker container that is able to read and write to the DB. I'm using MongoDB-Go-Driver (https://godoc.org/github.com/mongodb/mongo-go-driver/mongo) for this as mgo is no longer kept up.
This is my code, I'm just following the numerous tutorials online to make a simple connection and then ping the DB to ensure connectivity.
client, err := mongo.Connect("mongodb://localhost:27017")
if err != nil {
log.Fatal("error ", err)
}
// Check the connection
err = client.Ping(context.TODO(), nil)
if err != nil {
log.Fatal("error2 ", err)
}
fmt.Println("Connected to MongoDB!")
It always fails on doing any operation on the DB (Find, FindOne, Ping, etc.) with error2 server selection timeout
This is my docker-compose file I'm running.
version: "3"
services:
datastore:
image: mongo
ports:
- "27017:27017"
networks:
- maccaptionNet
volumes:
- .:/go/src/maccaption_microservice/dbdata
jobservice:
image: jobservicemaccaption:1.0
networks:
- maccaptionNet
depends_on:
- "datastore"
networks:
maccaptionNet:
driver: bridge
I'm brand new to MongoDB and after hours of research haven't made any progress on this.
I've read through https://docs.mongodb.com/manual/core/read-preference-mechanics/
https://docs.mongodb.com/manual/replication/
Can anyone point me in the right direction for this? I haven't been able to find a lot on this specific issue.
Thanks!
When you running the service and mongodb in docker you can't use localhost since the service is in a different container than mongodb, and from docker point of view it's under a different ip address.
You can connect with the service name you specify in docker-compose datastore
mongo.Connect("mongodb://datastore:27017")
Edit:
from: https://docs.docker.com/compose/networking/
By default Compose sets up a single network for your app. Each
container for a service joins the default network and is both
reachable by other containers on that network, and discoverable by
them at a hostname identical to the container name
Meaning that if you run multiple containers via compose, you can access one container from the other by the container name,
Basically when docker-compose starts, it sets up the network, and each container in the compose joins the network under its container name. For a container's point if view, localhost is just the container itself, while he can search for other container's name and get back the container’s IP address.
Assuming that the docker is running on your localhost, you can set the name in etc/hosts file like this:
127.0.0.1 datastore
(if not just replace 127.0.0.1 with the docker ip)
And in the app you will connect with mongodb://datastore:27017
So you will be able to run the service both in the docker and from outside, if you'll decide to run only the db in docker
docker-compose start datastore
If you are connecting to one docker from another (like it is written in your docker-compose file, and using bridge network mode, you have to change your localhost to the hostname, like datastore
client, err := mongo.Connect("mongodb://datastore:27017")
When your go script uses localhost, it expects the database to located in the same docker
I think my answer might be unrelated but still, I was getting the same error and it was because my IP address was not listed in the IP whitelist tab in MongoDB atlas, so make sure you have your IP address there before trying to connect.
I had the same problem but found another way to address this issue. You can just pass network parameter while running docker image and this way docker points to correct localhost.
docker run --network="host" ....
Source for this solution
Somehow i've fix this problem in a different way: by changing ports from "27018:27017" to "27017:27017".
IDK why this helps. Maybe if Mongo sees not default port it thinks there are cluster of Mongo's nodes.
I got this problem when I tired to connect to
mongodb v4.0.10
with
pymongo==4.0.2 not worked
pymongo==3.12.3 worked
Check your packages
mongodb v5.0.2 works with pymongo==4.0.2
I'm new to learning how to use goLang to build microservices. I had a whole project up and running locally, but when I tried deploying it I ran into a problem. The session I was working with (mgo.Dial("localhost")) was no longer working. When I put this into a docker image, it failed to connect to the local host, which makes sense, since the docker image builds it over a new OS (alpine in my case). I was wondering what I should do to get it to connect.
To be clear, when I was researching this, most people wanted to connect to a mongoDB session that is a docker container, I want to connect to a mongoDB session from within a docker container. Also once I'm ready for deployment I'll be using StatefulSet with kubernetes if that changes anything.
For example, this is what I want my program to be like:
sess, err := mgo.Dial("localhost") //or whatever
if err != nil {
fmt.Println("failed to connect")
else {
fmt.Println("connected")
What I tried doing:
Dockerfile:
FROM alpine:3.6
COPY /build/app /bin/
EXPOSE 8080
ENTRYPOINT ["/bin/app"]
In terminal:
docker build -t hell:4 .
docker run -d -p 8080:8080 hell:4
And as you can expect, it says not connected. Also the port mapping is for the rest of the project, not this part.
Thanks for your help!
I think you should not try to connect to the MongoDB server running on your machine. Think about deploying the whole application lateron you want a MongoDB server running together with your service on some cloud or server.
That problem could be solved by setting up an additional container and link it to your Go Web App. Docker compose can handle this. Just place a docker-compose.yml file in the directory you are executing your docker build in.
version: '3'
services:
myapp:
build: .
image: hell:4
ports:
- 8080:8080
links:
- mongodb
depends_on:
- mongodb
mongodb:
image: mongo:latest
ports:
- "27017:27017"
environment:
- MONGODB_USER="user"
- MONGODB_PASS="pass"
Something like this should do it (not tested). You have two services: One for your app that gets build according to your Dockerfile in the directory in which you currently are. Additionally it links to a service called mongodb defined below. The mongodb service is accessible via the service name mongodb.
If your mongoDB server is running in your host machine, replace localhost by you host IP.