I am trying to create a JupyterHub that uses an LDAP to authenticate users.
The JupyterHub is working but when I am trying log on the web page show a Error 500.
I must clarify that when I tried it with the PAM method it worked without problems, but when configuring it to use the LDAP it started to fail.
I am using docker-compose. I currently have three containers:
1. JupyterHub:
jupyter:
build:
context: ./jupyterhub
ports:
- "8380:8000"
environment:
VIRTUAL_HOST: jupyter.probandofran.eu
LETSENCRYPT_HOST: jupyter.probandofran.eu
VIRTUAL_PORT: 8000
restart: on-failure
depends_on:
- jupyterlab
- revproxy-letsencrypt
volumes:
- ${VOLUMES_BASE_PATH}/jupyter:/home
- /var/run/docker.sock:/var/run/docker.sock:ro
healthcheck:
test: curl --fail -s http://jupyter:8000/ || exit 1
interval: 10s # time between tests
timeout: 5s # time waiting for a response
start_period: 30s # time from launch when the failed tests are ignored
retries: 5 # countdown of failed tests
2.Jupyterlab:
jupyterlab:
build:
context: ./jupyterlab
image: ngd-jupyterlab
command: echo
healthcheck:
test: curl --fail -s http://jupyterlab:8080/ || exit 1
interval: 10s # time between tests
timeout: 5s # time waiting for a response
start_period: 30s # time from launch when the failed tests are ignored
retries: 5 # countdown of failed tests
3. OpenLDAP
openldap:
image: docker.io/bitnami/openldap:2.6
ports:
- '1389:1389'
- '1636:1636'
environment:
- LDAP_ENABLE_TLS=no
- LDAP_ADMIN_USERNAME=admin
- LDAP_ADMIN_PASSWORD=adminpassword
- LDAP_USERS=user1,user02
- LDAP_PASSWORDS=user1,password2
volumes:
- 'openldap_data:/bitnami/openldap'
I think the problem is in the JupyterHub configuration file
jupyterhub_config.py
from jupyter_client.localinterfaces import public_ips
ip = public_ips()[0]
c.Spawner.default_url = '/lab'
c.Authenticator.admin_users = {'fran'}
c.JupyterHub.admin_access = False
in_docker_compose = True # "False" for standalone testing (for instance, to check changes to Jupyterlab image)
c.JupyterHub.hub_ip = ip if not in_docker_compose else '0.0.0.0'
# 'jupyter' is the name of Jupyterhub service in "docker-compose" file
c.JupyterHub.hub_connect_ip = '' if not in_docker_compose else 'jupyter'
if in_docker_compose:
c.DockerSpawner.network_name = 'webproxy' # Should match the network name used in the docker compose file
c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
c.DockerSpawner.image = 'ngd-jupyterlab' # 'jupyter/datascience-notebook:r-4.0.3'
notebook_dir = "/home/jovyan/"
c.DockerSpawner.notebook_dir = notebook_dir
c.DockerSpawner.volumes = {'jupyterhub-user-{username}': dict(bind=notebook_dir, mode="rw")}
c.DockerSpawner.use_internal_ip = True
c.DockerSpawner.remove_containers = True
c.DockerSpawner.remove = True
# c.DockerSpawner.extra_host_config = { 'network_mode': network_name }
c.JupyterHub.authenticator_class = 'ldapauthenticator.LDAPAuthenticator'
c.LDAPAuthenticator.lookup_dn = False
c.LDAPAuthenticator.bind_dn_template = [
"uid={username},ou=people,dc=wikimedia,dc=org",
"cn={username},ou=users,dc=example,dc=org"
]
# c.JupyterHub.allow_named_servers = False
# c.JupyterHub.authenticator_class = 'jupyterhub.auth.PAMAuthenticator'
c.JupyterHub.cleanup_proxy = True
c.JupyterHub.cleanup_servers = True
c.LDAPAuthenticator.server_use_ssl = False
c.LDAPAuthenticator.use_ssl = False
c.LDAPAuthenticator.server_address = 'openldap'
c.LDAPAuthenticator.server_port = 1389
c.LDAPAuthenticator.user_search_base = 'dc=example,dc=org'
# c.JupyterHub.reset_db = True
When I use the docker-compose up it show:
jupyter_1 | [E 2022-04-20 11:57:07.496 JupyterHub web:1789] Uncaught exception POST /hub/login?next=%2Fhub%2F (192.168.112.1)
jupyter_1 | HTTPServerRequest(protocol='http', host='jupyter.probandofran.eu', method='POST', uri='/hub/login?next=%2Fhub%2F', version='HTTP/1.1', remote_ip='192.168.112.1')
jupyter_1 | Traceback (most recent call last):
jupyter_1 | File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1704, in _execute
jupyter_1 | result = await result
jupyter_1 | File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/login.py", line 151, in post
jupyter_1 | user = await self.login_user(data)
jupyter_1 | File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 804, in login_user
jupyter_1 | authenticated = await self.authenticate(data)
jupyter_1 | File "/usr/local/lib/python3.8/dist-packages/jupyterhub/auth.py", line 473, in get_authenticated_user
jupyter_1 | authenticated = await maybe_future(self.authenticate(handler, data))
jupyter_1 | File "/usr/local/lib/python3.8/dist-packages/ldapauthenticator/ldapauthenticator.py", line 382, in authenticate
jupyter_1 | conn = self.get_connection(userdn, password)
jupyter_1 | File "/usr/local/lib/python3.8/dist-packages/ldapauthenticator/ldapauthenticator.py", line 314, in get_connection
jupyter_1 | conn = ldap3.Connection(
jupyter_1 | File "/usr/local/lib/python3.8/dist-packages/ldap3/core/connection.py", line 355, in __init__
jupyter_1 | self.do_auto_bind()
jupyter_1 | File "/usr/local/lib/python3.8/dist-packages/ldap3/core/connection.py", line 374, in do_auto_bind
jupyter_1 | self.start_tls(read_server_info=False)
jupyter_1 | File "/usr/local/lib/python3.8/dist-packages/ldap3/core/connection.py", line 1264, in start_tls
jupyter_1 | if self.server.tls.start_tls(self) and self.strategy.sync: # for asynchronous connections _start_tls is run by the strategy
jupyter_1 | File "/usr/local/lib/python3.8/dist-packages/ldap3/core/tls.py", line 277, in start_tls
jupyter_1 | raise LDAPStartTLSError(connection.last_error)
jupyter_1 | ldap3.core.exceptions.LDAPStartTLSError: startTLS failed - protocolError
jupyter_1 |
It's my first time posting on stackOverflow, I'm sorry if I made any mistake posting.
I think it is bug from ldapauthenticator, we can hot fix by change the logic inside this lib (file jupyterhub_config.py)
As #jnishii suggestion in https://github.com/jupyterhub/ldapauthenticator/issues/211
Related
I'm having trouble working my code in docker. Could you please help me?
I'm putting my application in docker together with mongo in docker, but when I run the file inside docker it doesn't connect with the mongo of the other docker and accuses Timeout.
My Docker:
version: "3.4"
services:
mongo_db:
image: mongo:6.0
ports:
- "27017:27017"
volumes:
- ./mongo_db:/data/db
container_name: mongo_db
mongo_app:
image: mongo_img_db:latest
links:
- mongo_db
command: python3 /app/main.py
container_name: mongo_app
Error:
### Connection: Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True), 'app')
Traceback (most recent call last):
File "/app/main.py", line 216, in <module>
db_calc.update_storage_stats(db_calc.calculate_artifacts_size())
File "/app/main.py", line 43, in calculate_artifacts_size
artifacts_size = self.db_client.builds.aggregate(
File "/usr/local/lib/python3.9/site-packages/pymongo/collection.py", line 2428, in aggregate
with self.__database.client._tmp_session(session, close=False) as s:
File "/usr/local/lib/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1757, in _tmp_session
s = self._ensure_session(session)
File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1740, in _ensure_session
return self.__start_session(True, causal_consistency=False)
File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1685, in __start_session
self._topology._check_implicit_session_support()
File "/usr/local/lib/python3.9/site-packages/pymongo/topology.py", line 538, in _check_implicit_session_support
self._check_session_support()
File "/usr/local/lib/python3.9/site-packages/pymongo/topology.py", line 554, in _check_session_support
self._select_servers_loop(
File "/usr/local/lib/python3.9/site-packages/pymongo/topology.py", line 238, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 63542415a868649d12f2a966, topology_type: Unknown, servers: [<ServerDescription ('localhost', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27017: [Errno 111] Connection refused')>]>
I'm having trouble getting flask to connect to the mongo database. I was able to connect to the base via Pycharm - Database, additionally I downloaded MongoDB Compass to check if I can also connect from there and I did it. I installed the mongo server in containers, not locally.
This is my docker and docker-compose:
dockerfile:
FROM python:3.10
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONBUFFERED 1
WORKDIR /code
COPY Pipfile Pipfile.lock /code/
RUN pip install pipenv && pipenv install --system
CMD python main.py
docker-compose.yaml:
version: "3.8"
services:
db_mongo:
image: mongo:5.0
container_name: mongo
ports:
- "27018:27017"
volumes:
- ./init-mongo.js:/docker-entrypoint-initdb.d/init-mongo.js:ro
- ./mongo-volume:/data/db
env_file:
- ./env/.env_database
backend:
build: .
volumes:
- .:/code/
ports:
- '8100:5000'
container_name: flask_api_container
depends_on:
- db_mongo
init-mongo.js:
db.log.insertOne({"message": "Database created."});
db = db.getSiblingDB('admin');
db.auth('mk', 'adminPass')
db = db.getSiblingDB('flask_api_db');
db.createUser(
{
user: 'flask_api_user',
pwd: 'dbPass',
roles: [
{
role: 'dbOwner',
db: 'flask_api_db'
}
]
}
)
db.createCollection('collection_test');
I copied the uri address from MongoDB Compass (I connected to the base with this address). I have tried various combinations of this address.
main.py:
import pymongo
from flask import Flask, Response, jsonify
from flask_pymongo import PyMongo
app = Flask(__name__)
uri = 'mongodb://flask_api_user:dbPass#db_mongo:27018/?authMechanism=DEFAULT&authSource=flask_api_db'
client = pymongo.MongoClient(uri)
client.admin.command('ismaster') # to check if the connection has been established - show errors in terminal
# try:
# mongodb_client = PyMongo(
# app=app,
# uri='mongodb://flask_api_user:dbPass#db_mongo:27018/?authMechanism=DEFAULT&authSource=flask_api_db'
# # uri='mongodb://flask_api_user:dbPass#localhost:27018/?authMechanism=DEFAULT&authSource=flask_api_db'
# )
# db = mongodb_client.db
# print('DB: ', db, flush=True)
# # client.admin.command('ismaster')
# # print('OK', flush=True)
# except:
# print('Error', flush=True)
#app.route('/')
def index():
return 'Test2'
# #app.route("/add_one/")
# def add_one():
# db.my_collection.insert_one({'title': "todo title", 'body': "todo body"})
# return jsonify(message="success")
if __name__ == '__main__':
app.run(debug=True, host='0.0.0.0', port=8100)
Errors:
when I did this, it returned None:
# db = mongodb_client.db
# print('DB: ', db, flush=True)
below errors from this method:
uri = 'mongodb://flask_api_user:dbPass#db_mongo:27018/?authMechanism=DEFAULT&authSource=flask_api_db'
client = pymongo.MongoClient(uri)
client.admin.command('ismaster') # to check if the connection has been established - show errors in terminal
flask_api_container | Traceback (most recent call last):
flask_api_container | File "/code/main.py", line 9, in <module>
flask_api_container | client.admin.command('ismaster')
flask_api_container | File "/usr/local/lib/python3.10/site-packages/pymongo/database.py", line 721, in command
flask_api_container | with self.__client._socket_for_reads(read_preference, session) as (
flask_api_container | File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1235, in _socket_for_reads
flask_api_container | server = self._select_server(read_preference, session)
flask_api_container | File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1196, in _select_server
flask_api_container | server = topology.select_server(server_selector)
flask_api_container | File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 251, in select_server
flask_api_container | servers = self.select_servers(selector, server_selection_timeout, address)
flask_api_container | File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 212, in select_servers
flask_api_container | server_descriptions = self._select_servers_loop(selector, server_timeout, address)
flask_api_container | File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 227, in _select_servers_loop
flask_api_container | raise ServerSelectionTimeoutError(
flask_api_container | pymongo.errors.ServerSelectionTimeoutError: db_mongo:27018: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 62a9baa2cc5e740091e0a60b, topology_type: Unknown, servers: [<ServerDescription ('db_mongo', 27018) server_type: Unknown, rtt: None, error=AutoReconnect('db_mongo:27018: [Errno 111] Connection refused')>]>
flask_api_container exited with code 1
None of the uri used worked.
Thanks for any help in resolving this issue.
main.py
MongoDB Compass
PyCharm - database
main.py 2
The mapped port (27018) is used when you connect from the host to the database. Containers on the same bridge network as the database connect directly to the container and should use the container port (27017).
So your connection string should be
uri = 'mongodb://flask_api_user:dbPass#db_mongo:27017/?authMechanism=DEFAULT&authSource=flask_api_db'
I have a mongo container, started with the requireTLS TLS mode, and a mongo-express container. Mongo-express does not seem to manage to connect to mongo using TLS.
My docker-compose.yml:
version: '3.1'
services:
mongodb1:
image : "mongo:4.2"
container_name : "mongodb-001"
ports:
- '27017:27017'
environment:
MONGO_INITDB_ROOT_USERNAME : "admin"
MONGO_INITDB_ROOT_PASSWORD : "adminpasswd"
volumes:
- "./mongo-data:/data/db"
- "./etc_mongod.conf:/etc/mongod.conf"
- "./certificates:/etc/certificates:ro"
command:
- "--tlsMode"
- "preferTLS"
- "--tlsDisabledProtocols"
- "none"
- "--tlsCertificateKeyFile"
- "/etc/certificates/certificateKey.pem"
- "--tlsCAFile"
- "/etc/certificates/CA.crt"
- "--tlsAllowConnectionsWithoutCertificates"
mongo-express:
image : "mongo-express:latest"
container_name : "mongo-express-001"
ports:
- '8081:8081'
depends_on:
- mongodb1
volumes:
- "./certificates/CA.crt:/etc/certificates/CA.crt:ro"
environment:
ME_CONFIG_MONGODB_SERVER: "mongodb-001"
ME_CONFIG_MONGODB_PORT: "27017"
ME_CONFIG_MONGODB_ENABLE_ADMIN: "false"
ME_CONFIG_MONGODB_AUTH_DATABASE: "admin"
ME_CONFIG_MONGODB_AUTH_USERNAME: "admin"
ME_CONFIG_MONGODB_AUTH_PASSWORD: "adminpasswd"
ME_CONFIG_MONGODB_ADMINUSERNAME: "admin"
ME_CONFIG_MONGODB_ADMINPASSWORD: "adminpasswd"
ME_CONFIG_SITE_SSL_ENABLED: "true"
ME_CONFIG_MONGODB_CA_FILE: "/etc/certificates/CA.crt"
...and the error message I get:
mongodb-001 | 2020-10-09T14:16:13.299+0000 I NETWORK [listener] connection accepted from 172.31.0.3:44774 #2 (1 connection now open)
mongodb-001 | 2020-10-09T14:16:13.305+0000 I NETWORK [conn2] Error receiving request from client: SSLHandshakeFailed: The server is configured to only allow SSL connections. Ending connection from 172.31.0.3:44774 (connection id: 2)
mongodb-001 | 2020-10-09T14:16:13.305+0000 I NETWORK [conn2] end connection 172.31.0.3:44774 (0 connections now open)
mongo-express-001 |
mongo-express-001 | /node_modules/mongodb/lib/server.js:265
mongo-express-001 | process.nextTick(function() { throw err; })
mongo-express-001 | ^
mongo-express-001 | Error [MongoError]: connection 0 to mongodb-001:27017 closed
mongo-express-001 | at Function.MongoError.create (/node_modules/mongodb-core/lib/error.js:29:11)
mongo-express-001 | at Socket.<anonymous> (/node_modules/mongodb-core/lib/connection/connection.js:200:22)
mongo-express-001 | at Object.onceWrapper (events.js:422:26)
mongo-express-001 | at Socket.emit (events.js:315:20)
mongo-express-001 | at TCP.<anonymous> (net.js:674:12)
mongo-express-001 exited with code 1
Note that:
I can connect to MongoDB using a mongo shell with the same parameters I pass to mongo-express:
mongo "mongodb://admin:adminpasswd#mongodb-001:27017/admin?authSource=admin" --tls --tlsCAFile certificates/CA.crt
If I start MongoDB in preferTLS mode, the mongo-express connection works
tl;dr
Create a new config.js file with the following code
module.exports = {
mongodb: {
connectionOptions: {
ssl: true,
}
}
};
and mount that file in your docker compose file at /node_modules/mongo-express/config.js
Explanation
It appears to be an issue with their config.default.js file. In it, they have this
module.exports = {
mongodb: {
// if a connection string options such as server/port/etc are ignored
connectionString: mongo.connectionString || getConnectionStringFromEnvVariables(),
connectionOptions: {
// ssl: connect to the server using secure SSL
ssl: process.env.ME_CONFIG_MONGODB_SSL || mongo.ssl,
// sslValidate: validate mongod server certificate against CA
sslValidate: process.env.ME_CONFIG_MONGODB_SSLVALIDATE || true,
// sslCA: array of valid CA certificates
sslCA: sslCAFromEnv ? [sslCAFromEnv] : [],
// autoReconnect: automatically reconnect if connection is lost
autoReconnect: true,
// poolSize: size of connection pool (number of connections to use)
poolSize: 4,
}
}
You'll notice the existence of an env var that's not listed (with reason) in their documentation. ME_CONFIG_MONGODB_SSL
This is required to enable tls support. Setting the env var yourself however does nothing but throw an error and break express due to it not being cast to a Boolean. So it just reads as a string 'true' or 'false'.
This code is fixed in their npm package code, but they haven't updated their docker image since late 2021. So the only "fix" I've found for this is to create a new config.js file with the following code
module.exports = {
mongodb: {
connectionOptions: {
ssl: true,
sslValidate: true,
}
}
};
Then mount this file at /node_modules/mongo-express/config.js in your docker-compose file. It'll read these and overwrite the defaults.
Note: I added the sslValidate key:value pair as well due to the fact that it suffers from the same lack of type casting. So if you omit the ME_CONFIG_MONGODB_SSLVALIDATE env var entirely, it'll be set as true, but if you include the env var as either true or false, it'll just simply break (undefined behaviour).
I'm running following pyspark code with connection to mongodb
sparkConf = SparkConf().setMaster("local").setAppName("MongoSparkConnectorTour").set("spark.app.id", "MongoSparkConnectorTour")
# If executed via pyspark, sc is already instantiated
sc = SparkContext(conf=sparkConf)
sqlContext = SQLContext(sc)
# create and load dataframe from MongoDB URI
df = sqlContext.read.format("com.mongodb.spark.sql.DefaultSource")\
.option("spark.mongodb.input.uri", config.MONGO_URL_AUTH + "/spark.times")\
.load()
inside Docker image with
CMD [ "spark-submit" \
, "--conf", "spark.mongodb.input.uri=mongodb://root:example#mongodb:27017/spark.times" \
, "--conf", "spark.mongodb.output.uri=mongodb://root:example#mongodb:27017/spark.output" \
, "--packages", "org.mongodb.spark:mongo-spark-connector_2.11:2.4.1" \
, "./spark.py" ]
config.MONGO_URL_AUTH is mongodb://root:example#mongodb:27017
but I'm getting exception on run:
db_1 | 2019-10-09T13:44:34.354+0000 I ACCESS [conn4] Supported SASL mechanisms requested for unknown user 'root#spark'
db_1 | 2019-10-09T13:44:34.378+0000 I ACCESS [conn4] SASL SCRAM-SHA-1 authentication failed for root on spark from client 172.22.0.4:49302 ; UserNotFound: Could not find user "root" for db "spark"
pyspark_1 | Traceback (most recent call last):
pyspark_1 | File "/home/ubuntu/./spark.py", line 35, in <module>
pyspark_1 | .option("spark.mongodb.input.uri", config.MONGO_URL_AUTH + "/spark.times")\
pyspark_1 | File "/home/ubuntu/spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/readwriter.py", line 172, in load
pyspark_1 | return self._df(self._jreader.load())
pyspark_1 | File "/home/ubuntu/spark-2.4.4-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
pyspark_1 | File "/home/ubuntu/spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/utils.py", line 63, in deco
pyspark_1 | return f(*a, **kw)
pyspark_1 | File "/home/ubuntu/spark-2.4.4-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
pyspark_1 | py4j.protocol.Py4JJavaError: An error occurred while calling o34.load.
pyspark_1 | : com.mongodb.MongoSecurityException: Exception authenticating MongoCredential{mechanism=SCRAM-SHA-1, userName='root', source='spark', password=<hidden>, mechanismProperties={}}
pyspark_1 | at com.mongodb.internal.connection.SaslAuthenticator.wrapException(SaslAuthenticator.java:173)
everything in this setup works flawlessly if I don't use user and password in mongodb docker and just connect with mongodb://mongodb:27017 address, and just with using pymongo I can connect with password, something is wrong with my spark to mongodb configuration when password is used and I can't understand what is wrong.
setup for mongodb (part of docker-compose file):
db:
image: mongo
restart: always
networks:
miasnet:
aliases:
- "miasdb"
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: example
MONGO_INITDB_DATABASE: spark
ports:
- "27017:27017"
volumes:
- /data/db:/data/db
https://hub.docker.com/_/mongo reads:
MONGO_INITDB_ROOT_USERNAME, MONGO_INITDB_ROOT_PASSWORD
These variables, used in conjunction, create a new user and set that user's password. This user is created in the admin authentication database and given the role of root, which is a "superuser" role.
You don't specify the authentication database, so mongo uses current database by default - spark in your case.
You need to specify "admin" auth database in the connection string:
spark.mongodb.input.uri=mongodb://root:example#mongodb:27017/spark.times?authSource=admin
spark.mongodb.output.uri=mongodb://root:example#mongodb:27017/spark.output?authSource=admin
Im trying to save some data in a postgreSQL database using cygnus-ngsi, but nothing happens. Im running all services in a docker container using docker-compose.
docker-compose.yml:
...
cygnus:
image: fiware/cygnus-ngsi:latest
hostname: cygnus
container_name: cygnus_fiware
volumes:
- ./config/cygnus/cygnus.conf:/opt/apache-flume/conf/agent.conf
- ./config/cygnus/grouping_rules.conf:/opt/apache-flume/conf/grouping_rules.conf
links:
- orion
- postgres
expose:
- "5050"
ports:
- "5050:5050"
postgres:
restart: always
image: postgres:latest
container_name: postgres_fiware
volumes:
- ./data/db/postgres:/var/lib/postgresql/data
ports:
- "5432:5432"
expose:
- "5432"
environment:
- POSTGRES_USER=teste
- POSTGRES_DB=newdb
- POSTGRES_PASSWORD=123456789
agent.conf
cygnus-ngsi.sources = http-source
cygnus-ngsi.sinks = postgresql-sink
cygnus-ngsi.channels = postgresql-channel
#=============================================
# source configuration
# channel name where to write the notification events
cygnus-ngsi.sources.http-source.channels = postgresql-channel
# source class, must not be changed
cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming notifications
cygnus-ngsi.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
# URL target
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
# default service (service semantic depends on the persistence sink)
cygnus-ngsi.sources.http-source.handler.default_service = default
# default service path (service path semantic depends on the persistence sink)
cygnus-ngsi.sources.http-source.handler.default_service_path = /
# source interceptors, do not change
cygnus-ngsi.sources.http-source.interceptors = ts gi
# TimestampInterceptor, do not change
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
# GroupingInterceptor, do not change
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
# Grouping rules for the GroupingInterceptor, put the right absolute path to the file if necessary
# see the doc/design/interceptors document for more details
cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /opt/apache-flume/conf/grouping_rules.conf
# ============================================
# NGSIPostgreSQLSink configuration
# channel name from where to read notification events
cygnus-ngsi.sinks.postgresql-sink.channel = postgresql-channel
# sink class, must not be changed
cygnus-ngsi.sinks.postgresql-sink.type = com.telefonica.iot.cygnus.sinks.NGSIPostgreSQLSink
# true applies the new encoding, false applies the old encoding.
cygnus-ngsi.sinks.postgresql-sink.enable_encoding = false
# true if name mappings are enabled for this sink, false otherwise
cygnus-ngsi.sinks.postgresql-sink.enable_name_mappings = false
# true if the grouping feature is enabled for this sink, false otherwise
cygnus-ngsi.sinks.postgresql-sink.enable_grouping = false
# true if lower case is wanted to forced in all the element names, false otherwise
cygnus-ngsi.sinks.postgresql-sink.enable_lowercase = false
# the FQDN/IP address where the PostgreSQL server runs
cygnus-ngsi.sinks.postgresql-sink.postgresql_host = postgres
# the port where the PostgreSQL server listens for incomming connections
cygnus-ngsi.sinks.postgresql-sink.postgresql_port = 5432
# the name of the postgresql database
cygnus-ngsi.sinks.postgresql-sink.postgresql_database = newdb
# a valid user in the PostgreSQL server
cygnus-ngsi.sinks.postgresql-sink.postgresql_username = teste
# password for the user above
cygnus-ngsi.sinks.postgresql-sink.postgresql_password = 123456789
# how the attributes are stored, either per row either per column (row, column)
cygnus-ngsi.sinks.postgresql-sink.attr_persistence = row
# select the data_model: dm-by-service-path or dm-by-entity
cygnus-ngsi.sinks.postgresql-sink.data_model = dm-by-entity
# number of notifications to be included within a processing batch
cygnus-ngsi.sinks.postgresql-sink.batch_size = 100
# timeout for batch accumulation
cygnus-ngsi.sinks.postgresql-sink.batch_timeout = 30
# number of retries upon persistence error
cygnus-ngsi.sinks.postgresql-sink.batch_ttl = 10
# =============================================
# postgresql-channel configuration
# channel type (must not be changed)
cygnus-ngsi.channels.postgresql-channel.type = memory
# capacity of the channel
cygnus-ngsi.channels.postgresql-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnus-ngsi.channels.postgresql-channel.transactionCapacity = 100
Messages from docker-compose:
cygnus_fiware | time=2017-06-19T15:11:15.132Z | lvl=WARN | corr=4af15a0c-5501-11e7-aa0a-0242ac130004 | trans=508576db-1443-4c64-bfc9-629d1a0b250e | srv=espometeo | subsrv=/environment | comp=cygnus-ngsi | op=getEvents | msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[257] : [NGSIRestHandler] Unnecessary header
cygnus_fiware | time=2017-06-19T15:11:15.133Z | lvl=INFO | corr=89ac4c66-5501-11e7-850f-0242ac130004 | trans=3ae0dc99-de51-49a3-937f-42d887b7e7d7 | srv=espometeo | subsrv=/environment | comp=cygnus-ngsi | op=getEvents | msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[282] : [NGSIRestHandler] Starting internal transaction (3ae0dc99-de51-49a3-937f-42d887b7e7d7)
cygnus_fiware | time=2017-06-19T15:11:15.133Z | lvl=INFO | corr=89ac4c66-5501-11e7-850f-0242ac130004 | trans=3ae0dc99-de51-49a3-937f-42d887b7e7d7 | srv=espometeo | subsrv=/environment | comp=cygnus-ngsi | op=getEvents | msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[299] : [NGSIRestHandler] Received data ({ "subscriptionId" : "5947d328e143997a02b11008", "originator" : "localhost", "contextResponses" : [ { "contextElement" : { "type" : "EstacaoMeteo", "isPattern" : "false", "id" : "Estacao3", "attributes" : [ { "name" : "Humidity", "type" : "float", "value" : "35.3", "metadatas" : [ { "name" : "TimeInstant", "type" : "ISO8601", "value" : "2017-06-24T13:03:00" } ] }, { "name" : "Temperature", "type" : "float", "value" : "15.2", "metadatas" : [ { "name" : "TimeInstant", "type" : "ISO8601", "value" : "2017-06-24T13:03:00" } ] } ] }, "statusCode" : { "code" : "200", "reasonPhrase" : "OK" } } ]})
cygnus_fiware | time=2017-06-19T15:11:36.141Z | lvl=INFO | corr=89ac4c66-5501-11e7-850f-0242ac130004 | trans=3ae0dc99-de51-49a3-937f-42d887b7e7d7 | srv=espometeo | subsrv=/environment | comp=cygnus-ngsi | op=persistAggregation | msg=com.telefonica.iot.cygnus.sinks.NGSIPostgreSQLSink[479] : [postgresql-sink] Persisting data at NGSIPostgreSQLSink. Schema (espometeo), Table (environment_estacao3_estacaometeo), Fields ((recvTimeTs,recvTime,fiwareServicePath,entityId,entityType,attrName,attrType,attrValue,attrMd)), Values (('1497885075139','2017-06-19T15:11:15.139Z','/environment','Estacao3','EstacaoMeteo','Humidity','float','35.3','[{"name":"TimeInstant","type":"ISO8601","value":"2017-06-24T13:03:00"}]'),('1497885075139','2017-06-19T15:11:15.139Z','/environment','Estacao3','EstacaoMeteo','Temperature','float','15.2','[{"name":"TimeInstant","type":"ISO8601","value":"2017-06-24T13:03:00"}]'))
cygnus_fiware | time=2017-06-19T15:11:36.142Z | lvl=WARN | corr=89ac4c66-5501-11e7-850f-0242ac130004 | trans=3ae0dc99-de51-49a3-937f-42d887b7e7d7 | srv=espometeo | subsrv=/environment | comp=cygnus-ngsi | op=processNewBatches | msg=com.telefonica.iot.cygnus.sinks.NGSISink[541] :
Seems like cygnus is getting all the data from the Orion and putting it right, but when i go to the postgresql db, there nothing. Some one already have this problem?
Persistance ERROR message:
cygnus_fiware | time=2017-06-22T09:45:06.092Z | lvl=ERROR | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=processRollbackedBatches | msg=com.telefonica.iot.cygnus.sinks.NGSISink[398] : CygnusPersistenceError. -, null. Stack trace: [com.telefonica.iot.cygnus.sinks.NGSIPostgreSQLSink.persistAggregation(NGSIPostgreSQLSink.java:504), com.telefonica.iot.cygnus.sinks.NGSIPostgreSQLSink.persistBatch(NGSIPostgreSQLSink.java:231), com.telefonica.iot.cygnus.sinks.NGSISink.processRollbackedBatches(NGSISink.java:390), com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:372), org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68), org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147), java.lang.Thread.run(Thread.java:748)]
Cygnus start ERRORS:
cygnus_fiware | + exec /usr/lib/jvm/java-6-sun/bin/java -Xms512m -Xmx1g -Dcom.sun.management.jmxremote -Dflume.root.logger=INFO,console -Duser.timezone=UTC -Dfile.encoding=UTF-8 -cp '/opt/apache-flume/conf:/opt/apache-flume/lib/*:/opt/apache-flume/plugins.d/cygnus/lib/*:/opt/apache-flume/plugins.d/cygnus/libext/*' -Djava.library.path= com.telefonica.iot.cygnus.nodes.CygnusApplication -f /opt/apache-flume/conf/agent.conf -n cygnus-ngsi -p 8081
cygnus_fiware | /opt/apache-flume/bin/cygnus-flume-ng: line 232: /usr/lib/jvm/java-6-sun/bin/java: No such file or directory
cygnus_fiware | /opt/apache-flume/bin/cygnus-flume-ng: line 232: exec: /usr/lib/jvm/java-6-sun/bin/java: cannot execute: No such file or directory
There is a problem with PostgreSQL sink when cache is not enabled. It is described at this issue.
I solved the issue by changing false to true in agent.conf file: cygnus-ngsi.sinks.postgresql-sink.backend.enable_cache = true.