I've already tried the solution in this stackoverflow thread. It did not work for me, as shown below.
Here's my docker-compose.yml file:
version: "3.7"
services:
db:
image: mysql
volumes:
- data:/var/lib/mysql
volumes:
data:
driver_opts:
type: none
device: /usr/local/opt/mysql/mywebsite.com
o: bind
Here's the result of docker-compose up:
$ docker-compose up
Creating volume "mycontainer_data" with default driver
Recreating 45bc03b2c674_mycontainer_db_1 ... error
ERROR: for 45bc03b2c674_mycontainer_db_1 Cannot create container for service db: error while mounting volume with options: type='none' device='/usr/local/opt/mysql/mywebsite.com' o='bind': no such file or directory
ERROR: for db Cannot create container for service db: error while mounting volume with options: type='none' device='/usr/local/opt/mysql/mywebsite.com' o='bind': no such file or directory
ERROR: Encountered errors while bringing up the project.
Instead of type: none, I have also tried type: volume and type: bind, but I got the same error.
The directory clearly exists:
$ ls -al /usr/local/opt/mysql/
...
drwxr-xr-x 2 cameronhudson staff 64 Jun 10 10:32 mywebsite.com
...
There were 2 reasons why my configuration wasn't working.
First, Docker does not follow symlinks, and the mysql directory is a symlink:
$ ls -al /usr/local/opt | grep mysql
lrwxr-xr-x 1 cameronhudson admin 22 Jan 23 17:42 mysql -> ../Cellar/mysql/8.0.13
However, even after using a path without any symlinks, /databases/mywebsite.com, I continued the experience the same no such file or directory error.
I found that if I changed my docker-compose.yml file to mount this directory as a nameless volume, I got a different, more reasonable error:
version: "3.7"
services:
db:
image: mysql
volumes:
- /databases/mywebsite.com:/var/lib/mysql
The new error:
ERROR: for mycontainer_db_1 Cannot start service db: b'Mounts denied: \r\nThe path /databases/mywebsite.com\r\nis not shared from OS X and is not known to Docker.\r\nYou can configure shared paths from Docker -> Preferences... -> File Sharing.\r\nSee https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.\r\n.'
After I added this path in Docker Desktop, I was able to up the services using my original configuration. It also worked with type: none, type: volume, type: bind, or leaving type out altogether.
Related
I spun up a Debian 11 EC2 on AWS, and installed postgres 14.5 on it and docker and docker compose on it.I added a superuser to postgres of "admin' with a password. I created my docker-compose.yml file and a .env file.
When I try to use the docker-compose.yml file, I get:
sudo docker compose up -d
services.database.environment must be a mapping
When I build my docker container with
sudo docker build . -t tvappbuilder:latest
and then try to run it with:
sudo docker run -p 8080:8080 tvappbuilder:latest --env-file .env -it
Config Path .
4:47PM INF server/utils/logging.go:105 > logging configured fileLogging=true fileName=app-builder-logs logDirectory=./logs maxAgeInDays=0 maxBackups=0 maxSizeMB=0
4:47PM FTL server/cmd/video_conferencing/server.go:71 > Error initializing database error="pq: Could not detect default username. Please provide one explicitly"
Here are the dockers so far:
sudo docker image list
REPOSITORY TAG IMAGE ID CREATED SIZE
<none> <none> 6e5f035abda5 18 hours ago 1.82GB
tvappbuilder latest 6166e24a47e0 21 hours ago 21.8MB
<none> <none> cedcaf2facd1 21 hours ago 1.82GB
hello-world latest feb5d9fea6a5 12 months ago 13.3kB
golang 1.15.1 9f495162f677 2 years ago 839MB
Here is the docker-compose.yml:
version: 3.7
services:
server:
container_name: server
build: .
depends_on:
- database
ports:
- 8080:8080
environment:
- APP_ID: $APP_ID
- APP_CERTIFICATE: $APP_CERTIFICATE
- CUSTOMER_ID: $CUSTOMER_ID
- CUSTOMER_CERTIFICATE: $CUSTOMER_CERTIFICATE
- BUCKET_NAME: $BUCKET_NAME
- BUCKET_ACCESS_KEY: $BUCKET_ACCESS_KEY
- BUCKET_ACCESS_SECRET: $BUCKET_ACCESS_SECRET
- CLIENT_ID: $CLIENT_ID
- CLIENT_SECRET: $CLIENT_SECRET
- PSTN_USERNAME: $PSTN_USERNAME
- PSTN_PASSWORD: $PSTN_PASSWORD
- SCHEME: $SCHEME
- ALLOWED_ORIGIN: ""
- ENABLE_NEWRELIC_MONITORING: false
- RUN_MIGRATION: true
- DATABASE_URL: postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD#database:5432/$POSTGRES_DB?sslmode=disable
database:
container_name: server_database
image: postgres-14.5
restart: always
hostname: database
environment:
- POSTGRES_USER: $POSTGRES_USER
- POSTGRES_PASSWORD: $POSTGRES_PASSWORD
- POSTGRES_DB: $POSTGRES_DB
Here is the Dockerfile:
## Using Dockerfile from the following post: https://medium.com/#petomalina/using-go-mod-download-to-speed-up-golang-docker-builds-707591336888
FROM golang:1.15.1 as build-env
# All these steps will be cached
RUN mkdir /server
WORKDIR /server
COPY go.mod .
COPY go.sum .
# Get dependancies - will also be cached if we won't change mod/sum
RUN go mod download
# COPY the source code as the last step
COPY . .
# Build the binary
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o /go/bin/server /server/cmd/video_conferencing
# Second step to build minimal image
FROM scratch
COPY --from=build-env /go/bin/server /go/bin/server
COPY --from=build-env /server/config.json config.json
ENTRYPOINT ["/go/bin/server"]
and here is the .env file:
ENCRYPTION_ENABLED=0
POSTGRES_USER=admin
POSTGRES_PASSWORD=<correct pswd for admin>
POSTGRES_DB=tvappbuilder
APP_ID=<my real app ID>
APP_CERTIFICATE=<my real app cert>
CUSTOMER_ID=<my real ID>
CUSTOMER_CERTIFICATE=<my real cert>
RECORDING_REGION=0
BUCKET_NAME=<my bucket name>
BUCKET_ACCESS_KEY=<my real key>
BUCKET_ACCESS_SECRET=<my real secret>
CLIENT_ID=
CLIENT_SECRET=
PSTN_USERNAME=
PSTN_PASSWORD=
PSTN_ACCOUNT=
PSTN_EMAIL=
SCHEME=esports1_agora
ENABLE_SLACK_OAUTH=0
SLACK_CLIENT_ID=
SLACK_CLIENT_SECRET=
GOOGLE_CLIENT_ID=
ENABLE_GOOGLE_OAUTH=0
GOOGLE_CLIENT_SECRET=
ENABLE_MICROSOFT_OAUTH=0
MICROSOFT_CLIENT_ID=
MICROSOFT_CLIENT_SECRET=
APPLE_CLIENT_ID=
APPLE_PRIVATE_KEY=
APPLE_KEY_ID=
APPLE_TEAM_ID=
ENABLE_APPLE_OAUTH=0
PAPERTRAIL_API_TOKEN=<my real token>
According to this: https://pkg.go.dev/github.com/lib/pq
I probably should not need to use pq, and instead use postgres directly, but it appears it
was set up this way.
Many thanks for any pointers!
As per the comments there are a number of issues with your setup.
The first is the error services.database.environment must be a mapping when running docker compose up -d. This is caused by lines like - APP_ID: $APP_ID in your docker-compose.yml - use either APP_ID: $APP_ID or - APP_ID=$APP_ID as per the documentation.
A further issue is that you installed Postgres on the bare OS and are then using a postgres container. You only need to do one or the other (but if using docker you will want to use a volume or mount for the Postgres data (otherwise it will be lose when the container is rebuilt).
There are probably further issues but the above should get you started.
I am trying to spin and connect two containers (mongo and scrapy spider) using docker-compose. Being new to Docker I've had a hard time troubleshooting networking ports (inside and outside the container). To respect your time I'll keep it short.
The problem:
Can't connect the spider to the mongo db container and get a timeout error. I think it has to with the IP address that I am trying to connect to from the container is incorrect. However, the spider works locally (non-dockerized version) and can pass data to a running mongo container.
small edit to remove name and email from code.
error:
pymongo.errors.ServerSelectionTimeoutError: 127.0.0.1:27017: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 5feb8bdcf912ec8797c25497, topology_type: Single
pipeline code:
from scrapy.exceptions import DropItem
# scrappy log is deprecated
#from scrapy.utils import log
import logging
import scrapy
from itemadapter import ItemAdapter
import pymongo
class xkcdMongoDBStorage:
"""
Class that handles the connection of
Input:
MongoDB
Output
"""
def __init__(self):
# requires two arguments(address and port)
#* connecting to the db
self.conn = pymongo.MongoClient(
'127.0.0.1',27017) # works with spider local and container running
# '0.0.0.0',27017)
# connecting to the db
dbnames = self.conn.list_database_names()
if 'randallMunroe' not in dbnames:
# creating the database
self.db = self.conn['randallMunroe']
#if database already exists we want access
else:
self.db = self.conn.randallMunroe
#* connecting to the table
dbCollectionNames = self.db.list_collection_names()
if 'webComic' not in dbCollectionNames:
self.collection = self.db['webComic']
else:
# the table already exist so we access it
self.collection = self.db.webComic
def process_item(self, item, spider):
valid = True
for data in item:
if not data:
valid = False
raise DropItem("Missing {0}!".format(data))
if valid:
self.collection.insert(dict(item))
logging.info(f"Question added to MongoDB database!")
return item
Dockerfile for the spider
# base image
FROM python:3
# metadata info
LABEL maintainer="first last name" email="something#gmail.com"
# exposing container port to be the same as scrapy default
EXPOSE 6023
# set work directly so that paths can be relative
WORKDIR /usr/src/app
# copy to make usage of caching
COPY requirements.txt ./
#install dependencies
RUN pip3 install --no-cache-dir -r requirements.txt
# copy code itself from local file to image
COPY . .
CMD scrapy crawl xkcdDocker
version: '3'
services:
db:
image: mongo:latest
container_name: NoSQLDB
restart: always
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: password
volumes:
- ./data/bin:/data/db
ports:
- 27017:27017
expose:
- 27017
xkcd-scraper:
build: ./scraperDocker
container_name: xkcd-scraper-container
volumes:
- ./scraperDocker:/usr/src/app/scraper
ports:
- 5000:6023
expose:
- 6023
depends_on:
- db
Thanks for the help
Try:
self.conn = pymongo.MongoClient('NoSQLDB',27017)
Within docker compose you reference other containers based on the service name.
So I have created a node/mongo app and I am trying to run everything on docker.
I can get everything to run just fine until I try and add the init file for the mongo instance into the entry-point.
Here is my docker file for mongo: (Called mongo.dockerfile in /MongoDB)
FROM mongo:4.2
WORKDIR /usr/src/mongo
VOLUME /docker/volumes/mongo /user/data/mongo
ADD ./db-init /docker-entrypoint-initdb.d
CMD ["mongod", "--auth"]
The db-init folder contains an init.js file that looks like so (removed the names of stuff):
db.createUser({
user: '',
pwd: '',
roles: [ { role: 'readWrite', db: '' } ]
})
Here is my docker-compose file:
version: "3.7"
services:
web:
container_name: web
env_file:
- API/web.env
build:
context: ./API
target: prod
dockerfile: web.dockerfile
ports:
- "127.0.0.1:3000:3000"
depends_on:
- mongo
links:
- mongo
restart: always
mongo:
container_name: mongo
env_file:
- MongoDB/mongo.env
build:
context: ./MongoDB
dockerfile: mongo.dockerfile
restart: always
The exact error I get when running a docker-compose up is:
ERROR: for mongo Cannot start service mongo: OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \"docker-entrypoint-initdb.d\": executable file not found in $PATH": unknown
I had this working at one point with another project but cannot seem to get this on to work at all.
Any thoughts on what I am doing wrong?
Also note I have seen other issues like this saying to chmod +x the path (tried that didnt work)
Also tried to chmod 777 also didnt work. (Maybe I did this wrong and I dont know exactly what to run this on?)
Your entrypoint has been modified from the upstream image, and it's not clear how from the input you've provided. You may have modified the mongo image itself and need to pull a fresh copy with docker-compose build --pull. Otherwise, you can force the entrypoint back to the upstream value:
ENTRYPOINT ["docker-entrypoint.sh"]
when trying to pull a git resource we are getting an error
runc run: exit status 1: container_linux.go:345: starting container process caused "process_linux.go:303: getting the final child's pid from pipe caused \"EOF\""
we are using oracle linux - release 7.6. Docker version 18.03.1-ce.
we have followed the instructions on https://github.com/concourse/concourse-docker. we have tried with older versions of concourse (4.2.0 & 4.2.3). we can see the workers are up using fly.
we found this: https://github.com/concourse/concourse/issues/4021 on github which had a similar issue but couldn't find the relating story on stack overflow which the answerer had mentioned.
our docker compose file:
version: '3'
services:
db:
image: postgres
environment:
POSTGRES_DB: concourse
POSTGRES_USER: concourse_user
POSTGRES_PASSWORD: concourse_pass
web:
image: concourse/concourse
command: web
links: [db]
depends_on: [db]
ports: ["61111:8080"]
volumes: ["<path to repo folder>/keys/web:/concourse-keys"]
environment:
CONCOURSE_EXTERNAL_URL: <our url>
CONCOURSE_POSTGRES_HOST: db
CONCOURSE_POSTGRES_USER: concourse_user
CONCOURSE_POSTGRES_PASSWORD: concourse_pass
CONCOURSE_POSTGRES_DATABASE: concourse
CONCOURSE_ADD_LOCAL_USER: test:test
CONCOURSE_MAIN_TEAM_LOCAL_USER: test
worker:
image: concourse/concourse
command: worker
privileged: true
depends_on: [web]
volumes: ["<path to repo folder>/keys/worker:/concourse-keys"]
links: [web]
stop_signal: SIGUSR2
environment:
CONCOURSE_TSA_HOST: web:2222
we expected the resource to pull as the connectivity to the repo is in place and verified.
Not sure about your second issue with volumes, but I solved the original problem by setting user.max_user_namespaces parameter to 15000:
sysctl -w user.max_user_namespaces=15000
The solution was found here: https://github.com/docker/docker.github.io/issues/7962
This issue was fixed by updating the kernal from 3.1.x to 4.1.x. we have a new issue: failed to create volume on all our pipelines. i will update if i find a solution to this too
I'm building Iroha for which i'm running a script for environment setup which is internally calling the docker-compose.yml, where i"m getting the error:
ERROR: yaml.parser.ParserError: while parsing a block mapping
in "/home/cdac/iroha/docker/docker-compose.yml", line 3, column 5
expected <block end>, but found '<scalar>'
in "/home/cdac/iroha/docker/docker-compose.yml", line 13, column 6
docker-compose.yml file is showing below.
services:
node:
image: hyperledger/iroha:develop-build
ports:
- "${IROHA_PORT}:50051"
- "${DEBUGGER_PORT}:20000"
environment:
- IROHA_POSTGRES_HOST=${COMPOSE_PROJECT_NAME}_postgres_1
- IROHA_POSTGRES_PORT=5432
- IROHA_POSTGRES_USER=iroha
- IROHA_POSTGRES_PASSWORD=helloworld
- CCACHE_DIR=/tmp/ccache
export G_ID=$(id -g $(whoami))
export U_ID=$(id -g $(whoami))
user: ${U_ID:-0}:${G_ID:-0}
depends_on:
- postgres
tty: true
volumes:
- ../:/opt/iroha
- ccache-data:/tmp/ccache
working_dir: /opt/iroha
cap_add:
- SYS_PTRACE
security_opt:
- seccomp:unconfined
postgres:
image: postgres:9.5
environment:
- POSTGRES_USER=iroha
- IROHA_POSTGRES_PASSWORD=helloworld
command: -c 'max_prepared_transactions=100'
volumes:
ccache-data:
any help will be appreciate, thanks in advance.
These lines are not belongs to the docker-compose syntax
export G_ID=$(id -g $(whoami))
export U_ID=$(id -g $(whoami))
Also this line wont be able to work as expected
user: ${U_ID:-0}:${G_ID:-0}
You should write your own shell script and use it as an entry point for the docker container (this should be done in the Dockerfile step) then run a container directly from the image that you have created without the need to assign a user or export anything within the docker-compose as it will be executed once your container is running.
Check the following URL which contains more explanation about the allowed keywords in docker-compose: Compose File: Service Configuration Reference
#MostafaHussein I removed the above 3 lines then executed the run-iroha-dev.sh script, and it started to work. it attached me to /opt/iroha in docker container and downloaded hyperledger/iroha:develop-build and iroha images and launched two containers.
is it same what you are suggesting?