I am trying to integrate mongoDB and elasticsearch using monstache but I am facing this error. Please help me solve it out.
I will response with all the output you want.
config.toml file
mongo-url = "mongodb+srv://prince:mypassword#cluster0.mp297.mongodb.net/?retryWrites=true&w=majority"
elasticsearch-urls = ["http://127.0.0.1:9200"]
elasticsearch-max-conns = 10
replay = false
resume = true
enable-oplog = true
resume-name = "default"
namespace-regex = '^Satellite\.posts$'
direct-read-namespaces = ["Satellite.posts"]
change-stream-namespaces = ["Satellite."]
index-as-update = true
verbose = true
exit-after-direct-reads = false
[[mapping]]
namespace = "Satellite.posts"
index = "satellite"
So I have installed Cygnus and in the simple test configuration case which I took from here (https://github.com/telefonicaid/fiware-cygnus/blob/master/cygnus-ngsi/README.md) everything works fine.
But I need Postgresql as a backend for my application.
For this I adjusted the agent_1.conf file with all postgresql parameters found from http://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/installation_and_administration_guide/ngsi_agent_conf/
cygnus-ngsi.sources = http-source
cygnus-ngsi.sinks = postgresql-sink
cygnus-ngsi.channels = postgresql-channel
cygnus-ngsi.sources.http-source.channels = hdfs-channel mysql-channel ckan-channel mongo-channel sth-channel kafka-channel dynamo-channel postgresql-channel
cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.port = 5050
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
cygnus-ngsi.sources.http-source.handler.default_service = default
cygnus-ngsi.sources.http-source.handler.default_service_path = /
cygnus-ngsi.sources.http-source.interceptors = ts gi
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
cygnus-ngsi.sinks.postgresql-sink.channel = postgresql-channel
cygnus-ngsi.sinks.postgresql-sink.type = com.telefonica.iot.cygnus.sinks.NGSIPostgreSQLSink
cygnus-ngsi.sinks.postgresql-sink.postgresql_host = 127.0.0.1
cygnus-ngsi.sinks.postgresql-sink.postgresql_port = 5432
cygnus-ngsi.sinks.postgresql-sink.postgresql_database = myUser
cygnus-ngsi.sinks.postgresql-sink.postgresql_username = mydb
cygnus-ngsi.sinks.postgresql-sink.postgresql_password = xxxx
cygnus-ngsi.sinks.postgresql-sink.attr_persistence = row
cygnus-ngsi.sinks.postgresql-sink.batch_size = 100
cygnus-ngsi.sinks.postgresql-sink.batch_timeout = 30
cygnus-ngsi.sinks.postgresql-sink.batch_ttl = 10
# postgresql-channel configuration
cygnus-ngsi.channels.postgresql-channel.type = memory
cygnus-ngsi.channels.postgresql-channel.capacity = 1000
cygnus-ngsi.channels.postgresql-channel.transactionCapacity = 100
I didn'r really find any information about other files I am supposed to change and aren't really sure if all parameters are correct.
I also tried the sample configuration from here http://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/flume_extensions_catalogue/ngsi_postgresql_sink/index.html
Cygnus seems to start correctly but all if I try to send a notification I get connection refused
Befor doing anything, please, create the database, the user and the password in Postgresql.
The cygnus configuration is like this one. The /etc/cygnus/conf/cygnus_instance_1.conf file:
CYGNUS_USER=cygnus
CONFIG_FOLDER=/usr/cygnus/conf
CONFIG_FILE=/usr/cygnus/conf/agent_1.conf
AGENT_NAME=cygnus-ngsi
LOGFILE_NAME=cygnus.log
ADMIN_PORT=8081
POLLING_INTERVAL=30
So, the other file /usr/cygnus/conf/agent_1.conf is like this one (please change PostgreSQL parameters):
cygnus-ngsi.sources = http-source
cygnus-ngsi.sinks = postgresql-sink
cygnus-ngsi.channels = postgresql-channel
cygnus-ngsi.sources.http-source.channels = postgresql-channel
cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.port = 5050
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
cygnus-ngsi.sources.http-source.handler.default_service = default
cygnus-ngsi.sources.http-source.handler.default_service_path = /
cygnus-ngsi.sources.http-source.handler.events_ttl = 10
cygnus-ngsi.sources.http-source.interceptors = ts gi
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
#cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
# =============================================
# postgresql-channel configuration
# channel type (must not be changed)
cygnus-ngsi.channels.postgresql-channel.type = memory
# capacity of the channel
cygnus-ngsi.channels.postgresql-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnus-ngsi.channels.postgresql-channel.transactionCapacity = 100
# ============================================
# NGSIPostgreSQLSink configuration
# channel name from where to read notification events
cygnus-ngsi.sinks.postgresql-sink.channel = postgresql-channel
# sink class, must not be changed
cygnus-ngsi.sinks.postgresql-sink.type = com.telefonica.iot.cygnus.sinks.NGSIPostgreSQLSink
# true applies the new encoding, false applies the old encoding.
# cygnus-ngsi.sinks.postgresql-sink.enable_encoding = false
# true if the grouping feature is enabled for this sink, false otherwise
cygnus-ngsi.sinks.postgresql-sink.enable_grouping = false
# true if name mappings are enabled for this sink, false otherwise
cygnus-ngsi.sinks.postgresql-sink.enable_name_mappings = false
# true if lower case is wanted to forced in all the element names, false otherwise
# cygnus-ngsi.sinks.postgresql-sink.enable_lowercase = false
# the FQDN/IP address where the PostgreSQL server runs
cygnus-ngsi.sinks.postgresql-sink.postgresql_host = 127.0.0.1
# the port where the PostgreSQL server listens for incomming connections
cygnus-ngsi.sinks.postgresql-sink.postgresql_port = 5432
# the name of the postgresql database
cygnus-ngsi.sinks.postgresql-sink.postgresql_database = cygnusdb
# a valid user in the PostgreSQL server
cygnus-ngsi.sinks.postgresql-sink.postgresql_username = cygnus
# password for the user above
cygnus-ngsi.sinks.postgresql-sink.postgresql_password = cygnusdb
# how the attributes are stored, either per row either per column (row, column)
cygnus-ngsi.sinks.postgresql-sink.attr_persistence = row
# select the data_model: dm-by-service-path or dm-by-entity
cygnus-ngsi.sinks.postgresql-sink.data_model = dm-by-entity
# number of notifications to be included within a processing batch
cygnus-ngsi.sinks.postgresql-sink.batch_size = 1
# timeout for batch accumulation
cygnus-ngsi.sinks.postgresql-sink.batch_timeout = 30
# number of retries upon persistence error
cygnus-ngsi.sinks.postgresql-sink.batch_ttl = 0
# true enables cache, false disables cache
cygnus-ngsi.sinks.postgresql-sink.backend.enable_cache = true
We have been trying today to put a Cygnus container in production and we haven't been able to connect it to MongoDB. In our case, we have installed MongoDB with the Auth flag, and we created different users in order to test everything work.
However, we didn't find out the way to connect Cygnus. It tries to connect to the sth_default database, but the it requires enough privileges to create other databases.
The workaround was to start the MongoDB service without the Auth flag, allowing us to check that everything worked when the user can access with admin user without login in, which is not the way we would like to work, due to the fact that it is insecure.
Are we missing anything?
Thanks in advance!
UPDATE
I'm adding here the Cygnus agent.conf file. Moreover, I'm using the Docker Image (docker-ngsi: https://hub.docker.com/r/fiware/cygnus-ngsi/) in its latest version.
cygnus-ngsi.sources = http-source
# Using both, Mongo and Postgres sinks
cygnus-ngsi.sinks = mongo-sink postgresql-sink
cygnus-ngsi.channels = mongo-channel postgresql-channel
cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.channels = mongo-channel postgresql-channel
cygnus-ngsi.sources.http-source.port = 5050
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
cygnus-ngsi.sources.http-source.handler.default_service = default
cygnus-ngsi.sources.http-source.handler.default_service_path = /
cygnus-ngsi.sources.http-source.interceptors = ts gi
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /opt/apache-flume/conf/grouping_rules.conf
cygnus-ngsi.sinks.mongo-sink.type = com.telefonica.iot.cygnus.sinks.NGSIMongoSink
cygnus-ngsi.sinks.mongo-sink.channel = mongo-channel
#cygnus-ngsi.sinks.mongo-sink.enable_encoding = false
#cygnus-ngsi.sinks.mongo-sink.enable_grouping = false
#cygnus-ngsi.sinks.mongo-sink.enable_name_mappings = false
#cygnus-ngsi.sinks.mongo-sink.enable_lowercase = false
#cygnus-ngsi.sinks.mongo-sink.data_model = dm-by-entity
#cygnus-ngsi.sinks.mongo-sink.attr_persistence = row
cygnus-ngsi.sinks.mongo-sink.mongo_hosts = MyIP:MyPort
cygnus-ngsi.sinks.mongo-sink.mongo_username = MyUsername
cygnus-ngsi.sinks.mongo-sink.mongo_password = MyPassword
#cygnus-ngsi.sinks.mongo-sink.db_prefix = sth_
#cygnus-ngsi.sinks.mongo-sink.collection_prefix = sth_
#cygnus-ngsi.sinks.mongo-sink.batch_size = 1
#cygnus-ngsi.sinks.mongo-sink.batch_timeout = 30
#cygnus-ngsi.sinks.mongo-sink.batch_ttl = 10
#cygnus-ngsi.sinks.mongo-sink.data_expiration = 0
#cygnus-ngsi.sinks.mongo-sink.collections_size = 0
#cygnus-ngsi.sinks.mongo-sink.max_documents = 0
#cygnus-ngsi.sinks.mongo-sink.ignore_white_spaces = true
Thanks
The following configuration lines are missing:
cygnus-ngsi.sinks.mongo-sink.type = com.telefonica.iot.cygnus.sinks.NGSIMongoSink
cygnus-ngsi.sinks.mongo-sink.channel = mongo-channel
I.e. you have to specify the Java class implementing the MongoDB sink, and the channel that connects the source with such a sink.
If the configuration you are showing is the default one when Cygnus is installed through Docker, then the development team must be warned.
I am trying to create a query to dynamically update some values from a table Y from a table X.
I'm currenly dealing with a DB that hosts 10kk rows and I would like to optimize the amount of time of the processing.
Select:
SELECT cycle_completa.numcycle AS actual_cycle,
min(cycle_completa.pump1pre) AS min_pump1pre,
min(cycle_completa.pump2pre) AS min_pump2pre,
min(cycle_completa.pump3pre) AS min_pump3pre,
min(cycle_completa.slidevel) AS min_slidevel,
min(cycle_completa.slidepos) AS min_slidepos,
min(cycle_completa.slideforce) AS min_slideforce,
min(cycle_completa.slideservopilotpre) AS min_slideservopilotpre,
min(cycle_completa.slidecentralpre) AS min_slidecentralpre,
min(cycle_completa.slidelateralpre) AS min_slidelateralpre,
min(cycle_completa.slideringpre) AS min_slideringpre,
min(cycle_completa.slidepistonpre) AS min_slidepistonpre,
min(cycle_completa.slidey42pre) AS min_slidey42pre,
min(cycle_completa.cushionforce) AS min_cushionforce,
min(cycle_completa.cushionservopilotpre) AS min_cushionservopilotpre,
min(cycle_completa.cushionpistonpre) AS min_cushionpistonpre,
min(cycle_completa.cushionringpre) AS min_cushionringpre,
min(cycle_completa.pfillaccumpre) AS min_pfillaccumpre,
min(cycle_completa.pfillcentralpre) AS min_pfillcentralpre,
min(cycle_completa.pfilllateralpre) AS min_pfilllateralpre,
min(cycle_completa.pfillunlockingpre) AS min_pfillunlockingpre,
min(cycle_completa.reducedauxpumppre) AS min_reducedauxpumppre,
min(cycle_completa.tankoiltemp) AS min_tankoiltemp
FROM cycle_completa
WHERE slidestage = 17
GROUP BY cycle_completa.numcycle
UPDATE:
UPDATE cycle_parametros
SET seg3_min_valor_fmant_pump1pre = min_pump1pre,
seg3_min_valor_fmant_pump2pre = min_pump2pre,
seg3_min_valor_fmant_pump3pre = min_pump3pre,
seg3_min_valor_fmant_slidevel = min_slidevel,
seg3_min_valor_fmant_slidepos = min_slidepos,
seg3_min_valor_fmant_slideforce = min_slideforce,
seg3_min_valor_fmant_slideservopilotpre = min_slideservopilotpre,
seg3_min_valor_fmant_slidecentralpre = min_slidecentralpre,
seg3_min_valor_fmant_slidelateralpre = min_slidelateralpre,
seg3_min_valor_fmant_slideringpre = min_slideringpre,
seg3_min_valor_fmant_slidepistonpre = min_slidepistonpre,
seg3_min_valor_fmant_slidey42pre = min_slidey42pre,
seg3_min_valor_fmant_cushionforce = min_cushionforce,
seg3_min_valor_fmant_cushionservopilotpre = min_cushionservopilotpre,
seg3_min_valor_fmant_cushionpistonpre = min_cushionpistonpre,
seg3_min_valor_fmant_cushionringpre = min_cushionringpre,
seg3_min_valor_fmant_pfillaccumpre = min_pfillaccumpre,
seg3_min_valor_fmant_pfillcentralpre = min_pfillcentralpre,
seg3_min_valor_fmant_pfilllateralpre = min_pfilllateralpre,
seg3_min_valor_fmant_pfillunlockingpre = min_pfillunlockingpre,
seg3_min_valor_fmant_reducedauxpumppre = min_reducedauxpumppre,
seg3_min_valor_fmant_tankoiltemp = min_tankoiltemp
WHERE numcycle = actual_cycle
Thank you!
The following query should work:
WITH selected as (
SELECT cycle_completa.numcycle AS actual_cycle,
min(cycle_completa.pump1pre) AS min_pump1pre,
min(cycle_completa.pump2pre) AS min_pump2pre,
min(cycle_completa.pump3pre) AS min_pump3pre,
min(cycle_completa.slidevel) AS min_slidevel,
min(cycle_completa.slidepos) AS min_slidepos,
min(cycle_completa.slideforce) AS min_slideforce,
min(cycle_completa.slideservopilotpre) AS min_slideservopilotpre,
min(cycle_completa.slidecentralpre) AS min_slidecentralpre,
min(cycle_completa.slidelateralpre) AS min_slidelateralpre,
min(cycle_completa.slideringpre) AS min_slideringpre,
min(cycle_completa.slidepistonpre) AS min_slidepistonpre,
min(cycle_completa.slidey42pre) AS min_slidey42pre,
min(cycle_completa.cushionforce) AS min_cushionforce,
min(cycle_completa.cushionservopilotpre) AS min_cushionservopilotpre,
min(cycle_completa.cushionpistonpre) AS min_cushionpistonpre,
min(cycle_completa.cushionringpre) AS min_cushionringpre,
min(cycle_completa.pfillaccumpre) AS min_pfillaccumpre,
min(cycle_completa.pfillcentralpre) AS min_pfillcentralpre,
min(cycle_completa.pfilllateralpre) AS min_pfilllateralpre,
min(cycle_completa.pfillunlockingpre) AS min_pfillunlockingpre,
min(cycle_completa.reducedauxpumppre) AS min_reducedauxpumppre,
min(cycle_completa.tankoiltemp) AS min_tankoiltemp
FROM cycle_completa
WHERE slidestage = 17
GROUP BY cycle_completa.numcycle
)
UPDATE cycle_parametros cp
SET cp.seg3_min_valor_fmant_pump1pre = s.min_pump1pre,
cp.seg3_min_valor_fmant_pump2pre = s.min_pump2pre,
cp.seg3_min_valor_fmant_pump3pre = s.min_pump3pre,
cp.seg3_min_valor_fmant_slidevel = s.min_slidevel,
cp.seg3_min_valor_fmant_slidepos = s.min_slidepos,
cp.seg3_min_valor_fmant_slideforce = s.min_slideforce,
cp.seg3_min_valor_fmant_slideservopilotpre = s.min_slideservopilotpre,
cp.seg3_min_valor_fmant_slidecentralpre = s.min_slidecentralpre,
cp.seg3_min_valor_fmant_slidelateralpre = s.min_slidelateralpre,
cp.seg3_min_valor_fmant_slideringpre = s.min_slideringpre,
cp.seg3_min_valor_fmant_slidepistonpre = s.min_slidepistonpre,
cp.seg3_min_valor_fmant_slidey42pre = s.min_slidey42pre,
cp.seg3_min_valor_fmant_cushionforce = s.min_cushionforce,
cp.seg3_min_valor_fmant_cushionservopilotpre = s.min_cushionservopilotpre,
cp.seg3_min_valor_fmant_cushionpistonpre = s.min_cushionpistonpre,
cp.seg3_min_valor_fmant_cushionringpre = s.min_cushionringpre,
cp.seg3_min_valor_fmant_pfillaccumpre = s.min_pfillaccumpre,
cp.seg3_min_valor_fmant_pfillcentralpre = s.min_pfillcentralpre,
cp.seg3_min_valor_fmant_pfilllateralpre = s.min_pfilllateralpre,
cp.seg3_min_valor_fmant_pfillunlockingpre = s.min_pfillunlockingpre,
cp.seg3_min_valor_fmant_reducedauxpumppre = s.min_reducedauxpumppre,
cp.seg3_min_valor_fmant_tankoiltemp = s.min_tankoiltemp
FROM selected AS s
WHERE cp.numcycle = s.actual_cycle
Dears
I am using kannel 1.5.0 gateway with smpp on RHEL6 and when I receive an sms I get these errors:
2016-01-28 13:28:07 [8613] [6] WARNING: Could not convert GSM (0xd4) to Unicode.
2016-01-28 13:28:07 [8613] [6] WARNING: Could not convert GSM (0xf2) to Unicode.
.....
and I receive the messages incorrectly to my application, here is the request captured:
http://127.0.0.1:9091/services/smsReceive?msisdn=%2B353872849216&coding=0&smsText=%C3%85%3CH%C3%B9a%C3%91%C3%B9%25evM%C3%B9)zX%C3%ACp&DCS=-1&charset=UTF-8'
and this is my kannel configuration:
group = core
admin-port = 13001
smsbox-port = 13002
admin-password = bar
log-file = "/home/user/logs/kannellogs/SmscGateway.log"
log-level = 0
box-deny-ip = "*.*.*.*"
box-allow-ip = "127.0.0.1;172.*.*.*;192.*.*.*;10.*.*.*"
admin-allow-ip = "127.0.0.1;172.*.*.*;192.*.*.*;10.*.*.*"
admin-deny-ip = "*.*.*.*"
access-log = "/home/user/logs/kannellogs/access.log"
# SMSBOX SETUP
group = smsbox
bearerbox-host = localhost
sendsms-port = 13013
log-file="/home/user/logs/kannellogs/smsbox.log"
log-level = 0
access-log="/home/user/logs/kannellogs/sms_access.log"
reply-couldnotfetch = "Service is down, please try again later.(notfetch)"
reply-couldnotrepresent = "Service is down, please try again later.(notrepresent)"
reply-requestfailed = "Service is down, please try again later.(failed)"
reply-emptymessage = ""
mo-recode = true
# SEND-SMS USERS
group = sendsms-user
username=test
password=test
user-allow-ip = "*.*.*.*"
concatenation = true
split-chars = "#!^&*("
max-messages = 10
# SMPP PARAMETERS for SMSC account
group = smsc
smsc = smpp
smsc-id =Smsc12345
smsc-username = Voda
smsc-password = 12345678
host = 123.222.111.11
port = 1040
system-type = Vodafone403
interface-version = 34
source-addr-autodetect = false
source-addr-ton = 0
source-addr-npi = 1
dest-addr-ton = 1
dest-addr-npi = 1
reconnect-delay = false
reconnect-delay = 10
transceiver-mode = true
throughput = 10
address-range = "^12345$"
max-pending-submits = 3
group = sms-service
accepted-smsc = "Smsc12345"
keyword = default
get-url = "http://127.0.0.1:9091/services/smsReceive?msisdn=%p&coding=%c&smsText=%a&DCS=%m&charset=%C"
catch-all=true
max-messages = 0
I am new to kannel please help if i am doing anything wrong
You should check Kannel docs;
for a "normal" message, it will be "GSM" (coding=0), "binary" (coding=1) or "UTF-16BE" (coding=2)
What I see in url is that
&coding=0
what should be:
&coding=2
and also take care that it is url encoded correctly and about length of unicode message (if you are using aggregators not all support concatenation and long messages)
Hope it helps.
Vedran