the Database field must be set on Operation - mongodb

I have a golang application with mongodb connected using mongo-driver.
Everything is working fine if I run the main file as go run main.go
It is able to connect to database as expected.
However if I dockerize the application it is unable communicate to the database whenever I hit the api that interacts with the database it exits with message
the Database field must be set on Operation
Dockerfile
ARG ROOT="/go/src/bitbucket.org/myteam/myapp"
ARG BIN="/go/bin"
FROM golang:1.13 as build
ARG ROOT
ARG BIN
ENV GOBIN $BIN
ENV GOPATH /go
RUN apt-get update
RUN apt-get install go-dep -y
RUN mkdir -p $ROOT
RUN mkdir -p $BIN
WORKDIR /go
COPY .env .
WORKDIR $ROOT
COPY src/bitbucket.org/myteam/myapp .
RUN ["chmod", "+x", "main.go"]
RUN dep ensure
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo
FROM alpine:3.10
ARG ROOT
RUN apk --no-cache add ca-certificates
RUN mkdir -p /go/bin
WORKDIR /go/bin
RUN mkdir -p /go/bin
WORKDIR /go/bin
COPY --from=build $ROOT/myapp myapp
EXPOSE 3000
CMD ["./myapp"]
Connection File
config := conf.GetInstance()
ctx, _ := context.WithTimeout(context.Background(), 1*time.Second)
var clientOptions *options.ClientOptions
if config.MONGODB_USERNAME != "" && config.MONGODB_PASSWORD != "" {
if config.Env == "dev" {
clientOptions = options.Client().ApplyURI("mongodb://" + config.MONGODB_HOST + ":" + config.MONGODB_PORT).SetAuth(options.Credential{
AuthSource: config.MONGODB_NAME, Username: config.MONGODB_USERNAME, Password: config.MONGODB_PASSWORD,
})
} else {
mongoURI := fmt.Sprintf("mongodb+srv://%s:%s#%s", config.MONGODB_USERNAME, config.MONGODB_PASSWORD, config.MONGODB_HOST)
clientOptions = options.Client().ApplyURI(mongoURI)
}
} else {
clientOptions = options.Client().ApplyURI("mongodb://" + config.MONGODB_HOST + ":" + config.MONGODB_PORT)
}
client, err := mongo.Connect(ctx, clientOptions)
if err != nil {
logrus.Fatal("Error connecting to database", err)
}
db := client.Database(config.MONGODB_NAME)
Migrate(db, ctx)
logrus.Info("Database connection successful")
return db

This error happens when the mongodb database name is not available for the connection.
Make sure your config.MONGODB_* variables are available to set the connection string for mongodb.

Related

Doctrine\ORM\Tools\ToolsException : Schema-Tool failed with tests

I use a Docker, Symfony and Postgres to development my Symfony app:
RUN apk add --no-cache libpq-dev && docker-php-ext-install pdo_pgsql
# Configure non-root user.
ARG PUID=1000
ARG PGID=1000
RUN apk --no-cache add shadow && \
groupmod -o -g ${PGID} www-data && \
usermod -o -u ${PUID} -g www-data www-data
# Source code
RUN chown www-data:www-data /var/www
COPY --chown=www-data:www-data ./ /var/www
WORKDIR /var/www
USER www-data
and i want to use LiipTestFixturesBundle to load fixtures in my database
public function setUp(): void
{
parent::setUp();
self::bootKernel();
$container = static::getContainer();
$this->repository = $container->get(TemplateRepository::class);
$this->databaseTool = $container->get(DatabaseToolCollection::class)->get();
$this->databaseTool->loadFixtures([
TemplateFixture::class]);
}
But when i try to run my tests i get an error
Doctrine\ORM\Tools\ToolsException : Schema-Tool failed with Error 'An exception occurred while executing a query: SQLSTATE[3F000]: Invalid schema name: 7 ERROR: no schema has been selected to create in
LINE 1: CREATE TABLE templates (id UUID NOT NULL, name VARCHAR(255) ...
^' while executing DDL: CREATE TABLE templates (id UUID NOT NULL, name VARCHAR(255) NOT NULL, path VARCHAR(255) NOT NULL, PRIMARY KEY(id))
I have 2 databases - teambox and teambox_test (as expected in doctrine.yaml)
when#test:
doctrine:
dbal:
dbname_suffix: '_test%env(default::TEST_TOKEN)%'
packages/test/liip_test_fixture.yaml
liip_test_fixtures:
keep_database_and_schema: true
cache_metadata: true
cache_db: ~

Patch SSL Certificate handling in mongodump 4.2+ (go lang)

I wish to upgrade my Mongo database from version 4 to 5. I use ssl configuration with Mongo to ensure communication with the database is encrypted. To backup the database I use mongodump.
In Mongo 4.2 mongodump was rewritten in Go lang, causing it to import a common Go bug around ssl certificate handling. Specifically PEM files with intermediate certificates aren't fully loaded. The bug does not impact Mongo server or client itself, any version, or any other apps. Only mongodump is impacted.
The bug is described here: https://jira.mongodb.org/browse/TOOLS-2598
Since tool's SSL/TLS code is copied from Go driver, the current implementation only parses the last certificate inside the pem file
This is a discrepancy with Mongoshell behavior, which only loads the first certificate inside the pem file.
In this related exchange: https://jira.mongodb.org/browse/TOOLS-2996 I fail to see the resolution. I have tried every permutation of keys and certificates in the arguments passed to mongodump.
I've been looking at the source code for mongodump and specifically the SSL loading code.
To aid in go development, I've created this dockerfile to instantly provide a working environment for building this code but I am unfamiliar with go as a language.
FROM centos:8
RUN yum -y update
RUN yum -y install git-core vim-enhanced golang krb5-devel krb5-libs snappy
RUN groupadd -r app -g 1000 && \
useradd -r -g app -u 1000 app -d /app && \
mkdir -p /app && \
chown -R app:app /app
USER 1000
WORKDIR /app
ENTRYPOINT /bin/bash
How plausible is it to fix these PEM loading bugs in this code base? My Mongo estate has many clients so rotating the certificates to solve this issue involves a high degree of planning and downtime. Patching mongodump to accept the existing certs feels like an acceptable medium term trade off.
Is anyone able to help me write the appropriate patch, perhaps there's standard ssl code that go developers use now? Does anyone have any ideas on how I can move this forward please? (Patches would be ideal!)
I apologise in advance that I have no reproducible test case here due to complexities of how the certificates I want to test were created.
This patch I've written (based on a rejected git pull request for GODRIVER-1753) resolves the issues.
https://jira.mongodb.org/browse/GODRIVER-1753
https://github.com/mongodb/mongo-go-driver/pull/521/files
It also fixes an asn1 trailing data issue too.
diff --git a/README.md b/README.md
index 20f3ffe8..4b3bed1a 100644
--- a/README.md
+++ b/README.md
## -1,3 +1,13 ##
+
+** PATCHED MONGO-TOOLS **
+
+In Mongo 4.2, the mongo team rewrote many of their core tools in Go.
+
+This introduced a bug with the way that PEM files are handled.
+Faced with downtime and complexity in a large Mongo estate, this
+patched version of mongo resolves the issues.
+
+
MongoDB Tools
===================================
diff --git a/common/db/db.go b/common/db/db.go
index 3e78abab..9290bb31 100644
--- a/common/db/db.go
+++ b/common/db/db.go
## -186,9 +186,13 ## func addClientCertFromBytes(cfg *tls.Config, data []byte, keyPasswd string) (str
}
if currentBlock.Type == "CERTIFICATE" {
- certBlock = data[start : len(data)-len(remaining)]
- certDecodedBlock = currentBlock.Bytes
- start += len(certBlock)
+ tempCertBlock := data[start : len(data)-len(remaining)]
+ certBlock = append(certBlock, tempCertBlock...) //To handle usecase where multiple certs are present
+
+ tempCertEncodedBlock := currentBlock.Bytes
+ certDecodedBlock = append(certDecodedBlock, tempCertEncodedBlock...)
+
+ start += len(tempCertBlock)
} else if strings.HasSuffix(currentBlock.Type, "PRIVATE KEY") {
isEncrypted := x509.IsEncryptedPEMBlock(currentBlock) || strings.Contains(currentBlock.Type, "ENCRYPTED PRIVATE KEY")
if isEncrypted {
## -244,12 +248,17 ## func addClientCertFromBytes(cfg *tls.Config, data []byte, keyPasswd string) (str
// The documentation for the tls.X509KeyPair indicates that the Leaf certificate is not
// retained.
- crt, err := x509.ParseCertificate(certDecodedBlock)
- if err != nil {
- return "", err
- }
-
- return crt.Subject.String(), nil
+ l := len(certDecodedBlock)
+ for {
+ crt, err := x509.ParseCertificate(certDecodedBlock[0:l])
+ if err == nil {
+ return crt.Subject.String(), nil
+ }
+ l = l - 1
+ if l == 0 {
+ return "", err
+ }
+ }
}
// create a username for x509 authentication from an x509 certificate subject.
diff --git a/vendor/go.mongodb.org/mongo-driver/mongo/options/clientoptions.go b/vendor/go.mongodb.org/mongo-driver/mongo/options/clientoptions.go
index d66114fa..2f3c6554 100644
--- a/vendor/go.mongodb.org/mongo-driver/mongo/options/clientoptions.go
+++ b/vendor/go.mongodb.org/mongo-driver/mongo/options/clientoptions.go
## -941,9 +941,13 ## func addClientCertFromBytes(cfg *tls.Config, data []byte, keyPasswd string) (str
}
if currentBlock.Type == "CERTIFICATE" {
- certBlock = data[start : len(data)-len(remaining)]
- certDecodedBlock = currentBlock.Bytes
- start += len(certBlock)
+ tempCertBlock := data[start : len(data)-len(remaining)]
+ certBlock = append(certBlock, tempCertBlock...) //To handle usecase where multiple certs are present
+
+ tempCertEncodedBlock := currentBlock.Bytes
+ certDecodedBlock = append(certDecodedBlock, tempCertEncodedBlock...)
+
+ start += len(tempCertBlock)
} else if strings.HasSuffix(currentBlock.Type, "PRIVATE KEY") {
isEncrypted := x509.IsEncryptedPEMBlock(currentBlock) || strings.Contains(currentBlock.Type, "ENCRYPTED PRIVATE KEY")
if isEncrypted {
## -997,12 +1001,18 ## func addClientCertFromBytes(cfg *tls.Config, data []byte, keyPasswd string) (str
// The documentation for the tls.X509KeyPair indicates that the Leaf certificate is not
// retained.
- crt, err := x509.ParseCertificate(certDecodedBlock)
- if err != nil {
- return "", err
- }
+ l := len(certDecodedBlock)
+ for {
+ crt, err := x509.ParseCertificate(certDecodedBlock[0:l])
+ if err == nil {
+ return crt.Subject.String(), nil
+ }
+ l = l - 1
+ if l == 0 {
+ return "", err
+ }
+ }
- return x509CertSubject(crt), nil
}
func stringSliceContains(source []string, target string) bool {

Connecting Golang and Postgres docker containers

I'm trying to run a golang server at localhost:8080 that uses a postgres database. I've tried to containerize both the db and the server but can't seem to get them connected.
main.go
func (a *App) Initialize() {
var db *gorm.DB
var err error
envErr := godotenv.Load(".env")
if envErr != nil {
log.Fatalf("Error loading .env file")
}
var dbString = fmt.Sprintf("port=5432 user=sample dbname=sampledb sslmode=disable password=password host=db")
db, err = gorm.Open("postgres", dbString)
if err != nil {
fmt.Printf("failed to connect to databse\n",err)
}
a.DB=model.DBMigrate(db)
a.Router = mux.NewRouter()
a.setRoutes()
}
//Get : get wrapper
func (a *App) Get(path string, f func(w http.ResponseWriter, r *http.Request)) {
a.Router.HandleFunc(path, f).Methods("GET")
}
//Post : post wrapper
func (a *App) Post(path string, f func(w http.ResponseWriter, r *http.Request)) {
a.Router.HandleFunc(path, f).Methods("POST")
}
//Run : run on port
func (a *App) Run(port string) {
handler := cors.Default().Handler(a.Router)
log.Fatal(http.ListenAndServe(port, handler))
}
func (a *App) setRoutes() {
a.Get("/", a.handleRequest(controller.Welcome))
a.Get("/users", a.handleRequest(controller.GetUsers))
a.Get("/user/{id}", a.handleRequest(controller.GetUser))
a.Post("/login", a.handleRequest(controller.HandleLogin))
a.Post("/users/add", a.handleRequest(controller.CreateUser))
a.Post("/validate", a.handleRequest(controller.HandleValidation))
}
func main() {
app := &App{}
app.Initialize()
app.Run(":8080")
}
server Dockerfile
FROM golang:latest
RUN mkdir /app
WORKDIR /app/server
COPY go.mod .
COPY go.sum .
RUN go mod download
COPY . .
docker-compose.yml
version: '3.7'
services:
db:
image: postgres
container_name: ep-db
environment:
- POSTGRES_PORT=${DB_PORT}
- POSTGRES_USER=${DB_USERNAME}
- POSTGRES_PASSWORD=${DB_PASSWORD}
- POSTGRES_DB=${DB_NAME}
ports:
- '5432:5432'
volumes:
- ./db:/var/lib/postgresql/data"
networks:
- internal
server:
container_name: ep-server
build:
context: ./server
dockerfile: Dockerfile
command: bash -c "go build && ./server -b 0.0.0.0:8080 --timeout 120"
volumes:
- './server:/app/server'
expose:
- 8080
depends_on:
- db
networks:
- internal
stdin_open: true
volumes:
db:
server:
networks:
internal:
driver: bridge
I have some get and post requests that return the right values when i run it locally on my computer (for ex. localhost:8080/users would return a JSON full of users from the database) but when I use curl inside the server container, I don't get any results. I am new to docker, Is there something wrong with what I'm doing so far?
Each docker container has its own IP address. When you connect to the postgres db from your application, you are using localhost, which is the container for the application and not the db. Based on your docker-compose, you should use the hostname db (the service name) to connect to the database.
As suggested by #DavidMaze you should verify the logs from your server container. Also,
First ensure ep-server is running (check that the output of docker container ls has status running for ep-server)
Run docker logs ep-server to view errors (if any)
If there are no errors in the logs, then run a docker exec -it ep-server bash to login to your container and run a telnet ep-db 5432 to verify that your postgres instance is reacheable from ep-server

run my test in docker mongo instance using jenkins pipeline

I would like to run my tests against a Docker MongoDB instance using Jenkins pipeline. I have got it working kind of. My problem is the tests are running within the Mongo container. I just want it to load up a container and my tests for it to connect to the Monogo container. At the moment it downloads Gradle within the container and takes about 5 min to run. Hope that makes sense. Here is my JenkinsFile
#!/usr/bin/env groovy
pipeline {
environment {
SPRING_PROFILES_ACTIVE = "jenkins"
}
agent {
node {
label "jdk8"
}
}
parameters {
choice(choices: 'None\nBuild\nMinor\nMajor', description: '', name: 'RELEASE_TYPE')
string(defaultValue: "refs/heads/master:refs/remotes/origin/master", description: 'gerrit refspec e.g. refs/changes/45/12345/1', name: 'GERRIT_REFSPEC')
choice(choices: 'master\nFETCH_HEAD', description: 'gerrit branch', name: 'GERRIT_BRANCH')
}
stages {
stage("Test") {
stages {
stage("Initialise") {
steps {
println "Running on ${NODE_NAME}, release type: ${params.RELEASE_TYPE}"
println "gerrit refspec: ${params.GERRIT_REFSPEC}, branch: ${params.GERRIT_BRANCH}, event type: ${params.GERRIT_EVENT_TYPE}"
checkout scm
sh 'git log -n 1'
}
}
stage("Verify") {
agent {
dockerfile {
filename 'backend/Dockerfile'
args '-p 27017:27017'
label 'docker-pipeline'
dir './maintenance-notifications'
}
}
steps {
sh './gradlew :maintenance-notifications:backend:clean'
sh './gradlew :maintenance-notifications:backend:check :maintenance-notifications:backend:test'
}
post {
always {
junit 'maintenance-notifications/backend/build/test-results/**/*.xml'
}
}
}
}
}
stage("Release") {
when {
expression {
return params.RELEASE_TYPE != '' && params.RELEASE_TYPE != 'None';
}
}
steps {
script {
def gradleProps = readProperties file: "gradle.properties"
def isCurrentSnapshot = gradleProps.version.endsWith("-SNAPSHOT")
def newVersion = gradleProps.version.replace("-SNAPSHOT", "")
def cleanVersion = newVersion.tokenize(".").collect{it.toInteger()}
if (params.RELEASE_TYPE == 'Build') {
newVersion = "${cleanVersion[0]}.${cleanVersion[1]}.${isCurrentSnapshot ? cleanVersion[2] : cleanVersion[2] + 1}"
} else if (params.RELEASE_TYPE == 'Minor') {
newVersion = "${cleanVersion[0]}.${cleanVersion[1] + 1}.0"
} else if (params.RELEASE_TYPE == 'Major') {
newVersion = "${cleanVersion[0] + 1}.0.0"
}
def newVersionArray = newVersion.tokenize(".").collect{it.toInteger()}
def newSnapshot = "${newVersionArray[0]}.${newVersionArray[1]}.${newVersionArray[2] + 1}-SNAPSHOT"
println "release version: ${newVersion}, snapshot version: ${newSnapshot}"
sh "./gradlew :maintenance-notifications:backend:release -Prelease.useAutomaticVersion=true -Prelease.releaseVersion=${newVersion} -Prelease.newVersion=${newSnapshot}"
}
}
}
}
}
and here is my Dockerfile
FROM centos:centos7
ENV container=docker
RUN mkdir -p /usr/java; curl http://configuration/yum/thecloud/artifacts/java/jdk-8u151-linux-x64.tar.gz|tar zxC /usr/java && ln -s /usr/java/jdk1.8.0_151/bin/j* /usr/bin
RUN mkdir -p /usr/mongodb; curl http://configuration/yum/thecloud/artifacts/mongodb/mongodb-linux-x86_64-3.4.10.tgz|tar zxC /usr/mongodb && ln -s /usr/mongodb/mongodb-linux-x86_64-3.4.10/bin/* /usr/bin
ENV JAVA_HOME /usr/java/jdk1.8.0_151/
ENV SPRING_PROFILES_ACTIVE jenkins
RUN yum -y install git.x86_64 && yum clean all
# Set up directory requirements
RUN mkdir -p /data/db /var/log/mongodb /var/run/mongodb
VOLUME ["/data/db", "/var/log/mongodb"]
# Expose port 27017 from the container to the host
EXPOSE 27017
CMD ["--port", "27017", "--pidfilepath", "/var/run/mongodb/mongod.pid"]
# Start mongodb
ENTRYPOINT /usr/bin/mongod

Error when creating container with golang docker engine

I am creating a project in Go and I am using both "github.com/docker/docker/client" and "github.com/docker/docker/api/types", but when I try and create a container I get the following error:
ERROR: 2016/10/03 22:39:26 containers.go:84: error during connect: Post https://%2Fvar%2Frun%2Fdocker.sock/v1.23/containers/create: http: server gave HTTP response to HTTPS client
I can't understand why this is happening and it only happened after using the new golang docker engine(the old "github.com/docker/engine-api" is now deprecated).
The code isn't anything complicated, so I wonder if I am missing something:
resp, err := cli.Pcli.ContainerCreate(context.Background(), initConfig(), nil, nil, "")
if err != nil {
return err
}
And the initConfig that is called does the following:
func initConfig() (config *container.Config) {
mount := map[string]struct{}{"/root/host": {}}
return &container.Config{Image: "leadis_image", Volumes: mount, Cmd: strslice.StrSlice{"/root/server.py"}, AttachStdout: true}}
Also here is my dockerfile
FROM debian
MAINTAINER Leadis Journey
LABEL Description="This docker image is used to compile and execute user's program."
LABEL Version="0.1"
VOLUME /root/host/
RUN apt-get update && yes | apt-get upgrade
RUN yes | apt-get install gcc g++ python3 make
COPY container.py /root/server.py
EDIT
Just tried to test it with a simpler program
package main
import (
"fmt"
"os"
"io/ioutil"
"github.com/docker/docker/client"
"github.com/docker/docker/api/types"
"github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/strslice"
"golang.org/x/net/context"
)
func initConfig() (config *container.Config) {
mount := map[string]struct{}{"/root/host": {}}
return &container.Config{Image: "leadis_image", Volumes: mount, Cmd: strslice.StrSlice{"/root/server.py"}, AttachStdout: true}
}
func main() {
client, _ := client.NewEnvClient()
cwd, _ := os.Getwd()
ctx, err := os.Open(cwd+"/Dockerfile.tar.gz")
if err != nil {
fmt.Println(err)
return
}
build, err := client.ImageBuild(context.Background(), ctx, types.ImageBuildOptions{Tags: []string{"leadis_image"}, Context: ctx, SuppressOutput: false})
if err != nil {
fmt.Println(err)
return
}
b, _ := ioutil.ReadAll(build.Body)
fmt.Println(string(b))
_, err = client.ContainerCreate(context.Background(), initConfig(), nil, nil, "")
if err != nil {
fmt.Println(err)
}
}
Same dockerfile, but I still get the same error:
error during connect: Post
https://%2Fvar%2Frun%2Fdocker.sock/v1.23/containers/create: http:
server gave HTTP response to HTTPS client
client.NewEnvClient()
Last time I tried, this API expects environment variables like DOCKER_HOST in a different syntax from than the normal docker client.
From the client.go:
// NewEnvClient initializes a new API client based on environment variables.
// Use DOCKER_HOST to set the url to the docker server.
// Use DOCKER_API_VERSION to set the version of the API to reach, leave empty for latest.
// Use DOCKER_CERT_PATH to load the TLS certificates from.
// Use DOCKER_TLS_VERIFY to enable or disable TLS verification, off by default.
To use this, you need DOCKER_HOST to be set/exported in one of these formats:
unix:///var/run/docker.sock
http://localhost:2375
https://localhost:2376