Timeout Mongodb with Docker - mongodb

I'm having trouble working my code in docker. Could you please help me?
I'm putting my application in docker together with mongo in docker, but when I run the file inside docker it doesn't connect with the mongo of the other docker and accuses Timeout.
My Docker:
version: "3.4"
services:
mongo_db:
image: mongo:6.0
ports:
- "27017:27017"
volumes:
- ./mongo_db:/data/db
container_name: mongo_db
mongo_app:
image: mongo_img_db:latest
links:
- mongo_db
command: python3 /app/main.py
container_name: mongo_app
Error:
### Connection: Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True), 'app')
Traceback (most recent call last):
File "/app/main.py", line 216, in <module>
db_calc.update_storage_stats(db_calc.calculate_artifacts_size())
File "/app/main.py", line 43, in calculate_artifacts_size
artifacts_size = self.db_client.builds.aggregate(
File "/usr/local/lib/python3.9/site-packages/pymongo/collection.py", line 2428, in aggregate
with self.__database.client._tmp_session(session, close=False) as s:
File "/usr/local/lib/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1757, in _tmp_session
s = self._ensure_session(session)
File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1740, in _ensure_session
return self.__start_session(True, causal_consistency=False)
File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1685, in __start_session
self._topology._check_implicit_session_support()
File "/usr/local/lib/python3.9/site-packages/pymongo/topology.py", line 538, in _check_implicit_session_support
self._check_session_support()
File "/usr/local/lib/python3.9/site-packages/pymongo/topology.py", line 554, in _check_session_support
self._select_servers_loop(
File "/usr/local/lib/python3.9/site-packages/pymongo/topology.py", line 238, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 63542415a868649d12f2a966, topology_type: Unknown, servers: [<ServerDescription ('localhost', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27017: [Errno 111] Connection refused')>]>

Related

Can't establish a connection between Flask application and Mongo using Docker and Docker-Compose

I'm having trouble getting flask to connect to the mongo database. I was able to connect to the base via Pycharm - Database, additionally I downloaded MongoDB Compass to check if I can also connect from there and I did it. I installed the mongo server in containers, not locally.
This is my docker and docker-compose:
dockerfile:
FROM python:3.10
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONBUFFERED 1
WORKDIR /code
COPY Pipfile Pipfile.lock /code/
RUN pip install pipenv && pipenv install --system
CMD python main.py
docker-compose.yaml:
version: "3.8"
services:
db_mongo:
image: mongo:5.0
container_name: mongo
ports:
- "27018:27017"
volumes:
- ./init-mongo.js:/docker-entrypoint-initdb.d/init-mongo.js:ro
- ./mongo-volume:/data/db
env_file:
- ./env/.env_database
backend:
build: .
volumes:
- .:/code/
ports:
- '8100:5000'
container_name: flask_api_container
depends_on:
- db_mongo
init-mongo.js:
db.log.insertOne({"message": "Database created."});
db = db.getSiblingDB('admin');
db.auth('mk', 'adminPass')
db = db.getSiblingDB('flask_api_db');
db.createUser(
{
user: 'flask_api_user',
pwd: 'dbPass',
roles: [
{
role: 'dbOwner',
db: 'flask_api_db'
}
]
}
)
db.createCollection('collection_test');
I copied the uri address from MongoDB Compass (I connected to the base with this address). I have tried various combinations of this address.
main.py:
import pymongo
from flask import Flask, Response, jsonify
from flask_pymongo import PyMongo
app = Flask(__name__)
uri = 'mongodb://flask_api_user:dbPass#db_mongo:27018/?authMechanism=DEFAULT&authSource=flask_api_db'
client = pymongo.MongoClient(uri)
client.admin.command('ismaster') # to check if the connection has been established - show errors in terminal
# try:
# mongodb_client = PyMongo(
# app=app,
# uri='mongodb://flask_api_user:dbPass#db_mongo:27018/?authMechanism=DEFAULT&authSource=flask_api_db'
# # uri='mongodb://flask_api_user:dbPass#localhost:27018/?authMechanism=DEFAULT&authSource=flask_api_db'
# )
# db = mongodb_client.db
# print('DB: ', db, flush=True)
# # client.admin.command('ismaster')
# # print('OK', flush=True)
# except:
# print('Error', flush=True)
#app.route('/')
def index():
return 'Test2'
# #app.route("/add_one/")
# def add_one():
# db.my_collection.insert_one({'title': "todo title", 'body': "todo body"})
# return jsonify(message="success")
if __name__ == '__main__':
app.run(debug=True, host='0.0.0.0', port=8100)
Errors:
when I did this, it returned None:
# db = mongodb_client.db
# print('DB: ', db, flush=True)
below errors from this method:
uri = 'mongodb://flask_api_user:dbPass#db_mongo:27018/?authMechanism=DEFAULT&authSource=flask_api_db'
client = pymongo.MongoClient(uri)
client.admin.command('ismaster') # to check if the connection has been established - show errors in terminal
flask_api_container | Traceback (most recent call last):
flask_api_container | File "/code/main.py", line 9, in <module>
flask_api_container | client.admin.command('ismaster')
flask_api_container | File "/usr/local/lib/python3.10/site-packages/pymongo/database.py", line 721, in command
flask_api_container | with self.__client._socket_for_reads(read_preference, session) as (
flask_api_container | File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1235, in _socket_for_reads
flask_api_container | server = self._select_server(read_preference, session)
flask_api_container | File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1196, in _select_server
flask_api_container | server = topology.select_server(server_selector)
flask_api_container | File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 251, in select_server
flask_api_container | servers = self.select_servers(selector, server_selection_timeout, address)
flask_api_container | File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 212, in select_servers
flask_api_container | server_descriptions = self._select_servers_loop(selector, server_timeout, address)
flask_api_container | File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 227, in _select_servers_loop
flask_api_container | raise ServerSelectionTimeoutError(
flask_api_container | pymongo.errors.ServerSelectionTimeoutError: db_mongo:27018: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 62a9baa2cc5e740091e0a60b, topology_type: Unknown, servers: [<ServerDescription ('db_mongo', 27018) server_type: Unknown, rtt: None, error=AutoReconnect('db_mongo:27018: [Errno 111] Connection refused')>]>
flask_api_container exited with code 1
None of the uri used worked.
Thanks for any help in resolving this issue.
main.py
MongoDB Compass
PyCharm - database
main.py 2
The mapped port (27018) is used when you connect from the host to the database. Containers on the same bridge network as the database connect directly to the container and should use the container port (27017).
So your connection string should be
uri = 'mongodb://flask_api_user:dbPass#db_mongo:27017/?authMechanism=DEFAULT&authSource=flask_api_db'

Couldn't resolve module/action 'k8s_exec' on Ansible Playbook

I am trying to make a reboot to a pod on a Rancher Cluster through Ansible and I am having this error:
ERROR! couldn't resolve module/action 'k8s_exec'. This often indicates a misspelling, missing collection, or incorrect module path.
The error appears to be in '/home/ansible/ansible/GetKubectlPods': line 27, column 7, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
-
name: Reboot Machine
^ here
I don't know why I am getting an error since I am also using k8s_info and he is working be fine.
Here is the playbook I am running:
---
- hosts: localhost
connection: local
remote_user: root
vars:
ansible_python_interpreter: '{{ ansible_playbook_python }}'
tasks:
-
name: Get the pods in the specific namespace
k8s_info:
kubeconfig: '/etc/ansible/RCCloudConfig'
kind: Pod
namespace: redmine
register: pod_list
-
name: Print pod names
debug:
msg: "pod_list: {{ pod_list | json_query('resources[*].status.podIP') }} "
- set_fact:
pod_names: "{{pod_list|json_query('resources[*].metadata.name')}}"
-
name: Reboot Machine
k8s_exec:
kubeconfig: '/etc/ansible/RCCloudConfig'
namespace: redmine
pod: redmine_quick-testing-6c57cc5d65-lwkww #pod name
command: reboot
Ansible Version:
ansible 2.9.9
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3.6/site-packages/ansible
executable location = /usr/bin/ansible
python version = 3.6.8 (default, Apr 16 2020, 01:36:27) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
In case I edit my playbook and add the community.kubernetes collection, I get the following error:
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/__init__.py:12: DeprecationWarning: The package kubernetes.client.apis is renamed and deprecated, use kubernetes.client.api instead (please note that the trailing s was removed).\n DeprecationWarning\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py\", line 296, in websocket_call\n client = WSClient(configuration, get_websocket_url(url), headers, capture_all)\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py\", line 94, in __init__\n self.sock.connect(url, header=header)\n File \"/usr/local/lib/python3.6/site-packages/websocket/_core.py\", line 226, in connect\n self.handshake_response = handshake(self.sock, *addrs, **options)\n File \"/usr/local/lib/python3.6/site-packages/websocket/_handshake.py\", line 80, in handshake\n status, resp = _get_resp_headers(sock)\n File \"/usr/local/lib/python3.6/site-packages/websocket/_handshake.py\", line 165, in _get_resp_headers\n raise WebSocketBadStatusException(\"Handshake status %d %s\", status, status_message, resp_headers)\nwebsocket._exceptions.WebSocketBadStatusException: Handshake status 404 Not Found\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/home/ansible/.ansible/tmp/ansible-tmp-1592530746.4685414-46123-84159696554463/AnsiballZ_k8s_exec.py\", line 102, in <module>\n _ansiballz_main()\n File \"/home/ansible/.ansible/tmp/ansible-tmp-1592530746.4685414-46123-84159696554463/AnsiballZ_k8s_exec.py\", line 94, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/home/ansible/.ansible/tmp/ansible-tmp-1592530746.4685414-46123-84159696554463/AnsiballZ_k8s_exec.py\", line 40, in invoke_module\n runpy.run_module(mod_name='ansible_collections.community.kubernetes.plugins.modules.k8s_exec', init_globals=None, run_name='__main__', alter_sys=True)\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_k8s_exec_payload_q6oom26c/ansible_k8s_exec_payload.zip/ansible_collections/community/kubernetes/plugins/modules/k8s_exec.py\", line 148, in <module>\n File \"/tmp/ansible_k8s_exec_payload_q6oom26c/ansible_k8s_exec_payload.zip/ansible_collections/community/kubernetes/plugins/modules/k8s_exec.py\", line 135, in main\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/stream.py\", line 35, in stream\n return func(*args, **kwargs)\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/api/core_v1_api.py\", line 841, in connect_get_namespaced_pod_exec\n (data) = self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs) # noqa: E501\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/api/core_v1_api.py\", line 941, in connect_get_namespaced_pod_exec_with_http_info\n collection_formats=collection_formats)\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py\", line 345, in call_api\n _preload_content, _request_timeout)\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py\", line 176, in __call_api\n _request_timeout=_request_timeout)\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/stream.py\", line 30, in _intercept_request_call\n return ws_client.websocket_call(config, *args, **kwargs)\n File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py\", line 302, in websocket_call\n raise ApiException(status=0, reason=str(e))\nkubernetes.client.rest.ApiException: (0)\nReason: Handshake status 404 Not Found\n\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
Solution:
Find your k8s_exec.py file and edit your import settings from Kubernetes.apis to Kubernetes.api
Thank you in advance!

Error connecting from pyspark to mongodb with password

I'm running following pyspark code with connection to mongodb
sparkConf = SparkConf().setMaster("local").setAppName("MongoSparkConnectorTour").set("spark.app.id", "MongoSparkConnectorTour")
# If executed via pyspark, sc is already instantiated
sc = SparkContext(conf=sparkConf)
sqlContext = SQLContext(sc)
# create and load dataframe from MongoDB URI
df = sqlContext.read.format("com.mongodb.spark.sql.DefaultSource")\
.option("spark.mongodb.input.uri", config.MONGO_URL_AUTH + "/spark.times")\
.load()
inside Docker image with
CMD [ "spark-submit" \
, "--conf", "spark.mongodb.input.uri=mongodb://root:example#mongodb:27017/spark.times" \
, "--conf", "spark.mongodb.output.uri=mongodb://root:example#mongodb:27017/spark.output" \
, "--packages", "org.mongodb.spark:mongo-spark-connector_2.11:2.4.1" \
, "./spark.py" ]
config.MONGO_URL_AUTH is mongodb://root:example#mongodb:27017
but I'm getting exception on run:
db_1 | 2019-10-09T13:44:34.354+0000 I ACCESS [conn4] Supported SASL mechanisms requested for unknown user 'root#spark'
db_1 | 2019-10-09T13:44:34.378+0000 I ACCESS [conn4] SASL SCRAM-SHA-1 authentication failed for root on spark from client 172.22.0.4:49302 ; UserNotFound: Could not find user "root" for db "spark"
pyspark_1 | Traceback (most recent call last):
pyspark_1 | File "/home/ubuntu/./spark.py", line 35, in <module>
pyspark_1 | .option("spark.mongodb.input.uri", config.MONGO_URL_AUTH + "/spark.times")\
pyspark_1 | File "/home/ubuntu/spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/readwriter.py", line 172, in load
pyspark_1 | return self._df(self._jreader.load())
pyspark_1 | File "/home/ubuntu/spark-2.4.4-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
pyspark_1 | File "/home/ubuntu/spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/utils.py", line 63, in deco
pyspark_1 | return f(*a, **kw)
pyspark_1 | File "/home/ubuntu/spark-2.4.4-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
pyspark_1 | py4j.protocol.Py4JJavaError: An error occurred while calling o34.load.
pyspark_1 | : com.mongodb.MongoSecurityException: Exception authenticating MongoCredential{mechanism=SCRAM-SHA-1, userName='root', source='spark', password=<hidden>, mechanismProperties={}}
pyspark_1 | at com.mongodb.internal.connection.SaslAuthenticator.wrapException(SaslAuthenticator.java:173)
everything in this setup works flawlessly if I don't use user and password in mongodb docker and just connect with mongodb://mongodb:27017 address, and just with using pymongo I can connect with password, something is wrong with my spark to mongodb configuration when password is used and I can't understand what is wrong.
setup for mongodb (part of docker-compose file):
db:
image: mongo
restart: always
networks:
miasnet:
aliases:
- "miasdb"
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: example
MONGO_INITDB_DATABASE: spark
ports:
- "27017:27017"
volumes:
- /data/db:/data/db
https://hub.docker.com/_/mongo reads:
MONGO_INITDB_ROOT_USERNAME, MONGO_INITDB_ROOT_PASSWORD
These variables, used in conjunction, create a new user and set that user's password. This user is created in the admin authentication database and given the role of root, which is a "superuser" role.
You don't specify the authentication database, so mongo uses current database by default - spark in your case.
You need to specify "admin" auth database in the connection string:
spark.mongodb.input.uri=mongodb://root:example#mongodb:27017/spark.times?authSource=admin
spark.mongodb.output.uri=mongodb://root:example#mongodb:27017/spark.output?authSource=admin

How to open firewall port with ansible firewalld task on Centos 7

I have a task in my ansible-playbook script to open TCP port on a remote machine. but when I run my ansible playbook it throws an error. But when i run "firewall-cmd --permanent --zone=public --add-port=1234/tcp" and "firewalld-cmd --reload" I can see port is added in public zone.
Environment
Ansible local: OS x El Capitan
Ansible remote: AWS Centos 7 minimum version
Ansible version: 2.1.1.0
Remote python version: 2.7.5
My task
- name: open management console port
firewalld: port=1234/tcp zone=public permanent=true state=enabled immediate=yes
The error I am getting
fatal: [X.X.X.X]: FAILED! => {"changed": false, "failed": true, "module_stderr": "", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_MojhHQ/ansible_module_firewalld.py\", line 605, in <module>\r\n main()\r\n File \"/tmp/ansible_MojhHQ/ansible_module_firewalld.py\", line 456, in main\r\n is_enabled = get_port_enabled_permanent(zone, [port, protocol])\r\n File \"/tmp/ansible_MojhHQ/ansible_module_firewalld.py\", line 170, in get_port_enabled_permanent\r\n fw_zone = fw.config().getZoneByName(zone)\r\n File \"<string>\", line 2, in getZoneByName\r\n File \"/usr/lib/python2.7/site-packages/slip/dbus/polkit.py\", line 103, in _enable_proxy\r\n return func(*p, **k)\r\n File \"<string>\", line 2, in getZoneByName\r\n File \"/usr/lib/python2.7/site-packages/firewall/client.py\", line 52, in handle_exceptions\r\n return func(*args, **kwargs)\r\n File \"/usr/lib/python2.7/site-packages/firewall/client.py\", line 1505, in getZoneByName\r\n path = dbus_to_python(self.fw_config.getZoneByName(name))\r\n File \"/usr/lib64/python2.7/site-packages/dbus/proxies.py\", line 70, in __call__\r\n return self._proxy_method(*args, **keywords)\r\n File \"/usr/lib/python2.7/site-packages/slip/dbus/proxies.py\", line 50, in __call__\r\n return dbus.proxies._ProxyMethod.__call__(self, *args, **kwargs)\r\n File \"/usr/lib64/python2.7/site-packages/dbus/proxies.py\", line 145, in __call__\r\n **keywords)\r\n File \"/usr/lib64/python2.7/site-packages/dbus/connection.py\", line 651, in call_blocking\r\n message, timeout)\r\ndbus.exceptions.DBusException: org.fedoraproject.slip.dbus.service.PolKit.NotAuthorizedException.org.fedoraproject.FirewallD1.config: \r\n", "msg": "MODULE FAILURE", "parsed": false}
- name: Install firewalld
yum:
name: firewalld
state: latest
notify:
- start firewalld
- name: start firewalld
service:
name: firewalld
state: started
enabled: yes
become: yes
- name: enable 1234
firewalld:
zone: public
port: 1234/tcp
permanent: true
state: enabled
become: yes
Do it this way . It will work
dbus.exceptions.DBusException: org.fedoraproject.slip.dbus.service.PolKit.NotAuthorizedException.org.fedoraproject.FirewallD1.config indicates there's some sort of permissions error. The task probably needs to elevate its privileges with become: yes.
See the become documentation for more details.

socket.gaierror: [Errno 8] nodename nor servname provided, or not known in flask app

I followed the steps mentioned in https://developers.openshift.com/en/python-flask.html
to create a sample flask application.
When I am trying to run wsgi.py on my local mac machine. I am getting the following error.
Traceback (most recent call last):
File "wsgi.py", line 24, in <module>
httpd = make_server('localhost', 10000, application)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/simple_server.py",
line 144, in make_server
server = server_class((host, port), handler_class)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py",
line 419, in init
self.server_bind()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/simple_server.py",
line 48, in server_bind
HTTPServer.server_bind(self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py",
line 108, in server_bind
SocketServer.TCPServer.server_bind(self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py",
line 430, in server_bind
self.socket.bind(self.server_address)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py",
line 224, in meth
return getattr(self._sock,name)(*args)
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
Problem for me was using a non-standard port. Had to define my port separately from the host.
self.config = {
'user': 'root',
'password': 'root',
'host': '127.0.0.1',
'port': 8889,
'database': 'ET_Toolbox',
'charset': 'utf8'
}