I have installed Cygnus and when I do the test I have 4 errors, 2 in com.telefonica.iot.cygnus.utils.TestUtils.txt file and other 2 in com.telefonica.iot.cygnus.utils.TestConstants.txt.
The errors are as follows:
#more com.telefonica.iot.cygnus.utils.TestConstants.txt
------------------------------------------------------------------------------- Test set: com.telefonica.iot.cygnus.utils.TestConstants
------------------------------------------------------------------------------- Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.009 sec <<< FAILURE! initializationError(com.telefonica.iot.cygnus.utils.TestConstants) Time elapsed: 0.003 sec <<< ERROR! java.lang.Exception: Test class should have exactly one public constructor
at org.junit.runners.BlockJUnit4ClassRunner.validateOnlyOneConstructor(BlockJUnit4ClassRunner.java:134)
at org.junit.runners.BlockJUnit4ClassRunner.validateConstructor(BlockJUnit4ClassRunner.java:123)
at org.junit.runners.BlockJUnit4ClassRunner.collectInitializationErrors(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.ParentRunner.validate(ParentRunner.java:355)
at org.junit.runners.ParentRunner.<init>(ParentRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.<init>(BlockJUnit4ClassRunner.java:57)
at org.junit.internal.builders.JUnit4Builder.runnerForClass(JUnit4Builder.java:10)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:26)
at org.apache.maven.surefire.junit4.JUnit4TestSet.<init>(JUnit4TestSet.java:45)
at org.apache.maven.surefire.junit4.JUnit4DirectoryTestSuite.createTestSet(JUnit4DirectoryTestSuite.java:56)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.locateTestSets(AbstractDirectoryTestSuite.java:96)
at org.apache.maven.surefire.Surefire.createSuiteFromDefinition(Surefire.java:209)
at org.apache.maven.surefire.Surefire.run(Surefire.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:345)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1009)
initializationError(com.telefonica.iot.cygnus.utils.TestConstants) Time elapsed: 0 sec <<< ERROR! java.lang.Exception: No runnable methods
at org.junit.runners.BlockJUnit4ClassRunner.validateInstanceMethods(BlockJUnit4ClassRunner.java:169)
at org.junit.runners.BlockJUnit4ClassRunner.collectInitializationErrors(BlockJUnit4ClassRunner.java:104)
at org.junit.runners.ParentRunner.validate(ParentRunner.java:355)
at org.junit.runners.ParentRunner.<init>(ParentRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.<init>(BlockJUnit4ClassRunner.java:57)
at org.junit.internal.builders.JUnit4Builder.runnerForClass(JUnit4Builder.java:10)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:26)
at org.apache.maven.surefire.junit4.JUnit4TestSet.<init>(JUnit4TestSet.java:45)
at org.apache.maven.surefire.junit4.JUnit4DirectoryTestSuite.createTestSet(JUnit4DirectoryTestSuite.java:56)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.locateTestSets(AbstractDirectoryTestSuite.java:96)
at org.apache.maven.surefire.Surefire.createSuiteFromDefinition(Surefire.java:209)
at org.apache.maven.surefire.Surefire.run(Surefire.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:345)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1009)
# more com.telefonica.iot.cygnus.utils.TestUtils.txt
------------------------------------------------------------------------------- Test set: com.telefonica.iot.cygnus.utils.TestUtils
------------------------------------------------------------------------------- Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.003 sec <<< FAILURE! initializationError(com.telefonica.iot.cygnus.utils.TestUtils) Time elapsed: 0 sec <<< ERROR! java.lang.Exception: Test class should have exactly one public constructor
at org.junit.runners.BlockJUnit4ClassRunner.validateOnlyOneConstructor(BlockJUnit4ClassRunner.java:134)
at org.junit.runners.BlockJUnit4ClassRunner.validateConstructor(BlockJUnit4ClassRunner.java:123)
at org.junit.runners.BlockJUnit4ClassRunner.collectInitializationErrors(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.ParentRunner.validate(ParentRunner.java:355)
at org.junit.runners.ParentRunner.<init>(ParentRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.<init>(BlockJUnit4ClassRunner.java:57)
at org.junit.internal.builders.JUnit4Builder.runnerForClass(JUnit4Builder.java:10)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:26)
at org.apache.maven.surefire.junit4.JUnit4TestSet.<init>(JUnit4TestSet.java:45)
at org.apache.maven.surefire.junit4.JUnit4DirectoryTestSuite.createTestSet(JUnit4DirectoryTestSuite.java:56)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.locateTestSets(AbstractDirectoryTestSuite.java:96)
at org.apache.maven.surefire.Surefire.createSuiteFromDefinition(Surefire.java:209)
at org.apache.maven.surefire.Surefire.run(Surefire.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:345)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1009)
initializationError(com.telefonica.iot.cygnus.utils.TestUtils) Time elapsed: 0 sec <<< ERROR! java.lang.Exception: No runnable methods
at org.junit.runners.BlockJUnit4ClassRunner.validateInstanceMethods(BlockJUnit4ClassRunner.java:169)
at org.junit.runners.BlockJUnit4ClassRunner.collectInitializationErrors(BlockJUnit4ClassRunner.java:104)
at org.junit.runners.ParentRunner.validate(ParentRunner.java:355)
at org.junit.runners.ParentRunner.<init>(ParentRunner.java:76)
at org.junit.runners.BlockJUnit4ClassRunner.<init>(BlockJUnit4ClassRunner.java:57)
at org.junit.internal.builders.JUnit4Builder.runnerForClass(JUnit4Builder.java:10)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:26)
at org.apache.maven.surefire.junit4.JUnit4TestSet.<init>(JUnit4TestSet.java:45)
at org.apache.maven.surefire.junit4.JUnit4DirectoryTestSuite.createTestSet(JUnit4DirectoryTestSuite.java:56)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.locateTestSets(AbstractDirectoryTestSuite.java:96)
at org.apache.maven.surefire.Surefire.createSuiteFromDefinition(Surefire.java:209)
at org.apache.maven.surefire.Surefire.run(Surefire.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:345)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1009)
The agent.conf file is:
cygnusagent.sources = http-source
cygnusagent.sinks = ckan-sink mongo-sink
cygnusagent.channels = ckan-channel mongo-channel
#=============================================
# source configuration
# channel name where to write the notification events
cygnusagent.sources.http-source.channels = ckan-channel mongo-channel
# source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default service (service semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service = def_serv
# Default service path (service path semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service_path = def_servpath
# Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = 10
# Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts gi
# TimestampInterceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
# GroupinInterceptor, do not change
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
# Grouping rules for the GroupingInterceptor, put the right absolute path to the file if necessary
# See the doc/design/interceptors document for more details
###cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules_1.conf
# ============================================
# OrionCKANSink configuration
# channel name from where to read notification events
cygnusagent.sinks.ckan-sink.channel = ckan-channel
# sink class, must not be changed
cygnusagent.sinks.ckan-sink.type = com.telefonica.iot.cygnus.sinks.OrionCKANSink
#Â true if the grouping feature is enabled for this sink, false otherwise
cygnusagent.sinks.ckan-sink.enable_grouping = false
# the CKAN API key to use
cygnusagent.sinks.ckan-sink.api_key = ckanapikey
# the FQDN/IP address for the CKAN API endpoint
cygnusagent.sinks.ckan-sink.ckan_host = CKANIP
# the port for the CKAN API endpoint
cygnusagent.sinks.ckan-sink.ckan_port = CKANPORT
# Orion URL used to compose the resource URL with the convenience operation URL to query it
cygnusagent.sinks.ckan-sink.orion_url = http://localhost:1026
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.ckan-sink.attr_persistence = row
# enable SSL for secure Http transportation; 'true' or 'false'
cygnusagent.sinks.ckan-sink.ssl = false
# number of notifications to be included within a processing batch
cygnusagent.sinks.ckan-sink.batch_size = 100
# timeout for batch accumulation
cygunsagent.sinks.ckan-sink.batch_timeout = 30
# ============================================
# OrionMongoSink configuration
# sink class, must not be changed
cygnusagent.sinks.mongo-sink.type = com.telefonica.iot.cygnus.sinks.OrionMongoSink
# channel name from where to read notification events
cygnusagent.sinks.mongo-sink.channel = mongo-channel
#Â true if the grouping feature is enabled for this sink, false otherwise
cygnusagent.sinks.mongo-sink.enable_grouping = false
# FQDN/IP:port where the MongoDB server runs (standalone case) or comma-separated list of FQDN/IP:port pairs where the MongoDB replica set members run
cygnusagent.sinks.mongo-sink.mongo_hosts = ORIONIP:ORIONPORT
# a valid user in the MongoDB server (or empty if authentication is not enabled in MongoDB)
cygnusagent.sinks.mongo-sink.mongo_username =
# password for the user above (or empty if authentication is not enabled in MongoDB)
cygnusagent.sinks.mongo-sink.mongo_password =
# prefix for the MongoDB databases
###cygnusagent.sinks.mongo-sink.db_prefix = sth_
# prefix pro the MongoDB collections
###cygnusagent.sinks.mongo-sink.collection_prefix = sth_
# true is collection names are based on a hash, false for human redable collections
cygnusagent.sinks.mongo-sink.should_hash = false
# specify if the sink will use a single collection for each service path, for each entity or for each attribute
###cygnusagent.sinks.mongo-sink.data_model = collection-per-entity
# how the attributes are stored, either per row either per column (row, column)
###cygnusagent.sinks.mongo-sink.attr_persistence = column
#=============================================
# ckan-channel configuration
# channel type (must not be changed)
cygnusagent.channels.ckan-channel.type = memory
# capacity of the channel
cygnusagent.channels.ckan-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnusagent.channels.ckan-channel.transactionCapacity = 100
#=============================================
# mongo-channel configuration
# channel type (must not be changed)
cygnusagent.channels.mongo-channel.type = memory
# capacity of the channel
cygnusagent.channels.mongo-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnusagent.channels.mongo-channel.transactionCapacity = 100
Could someone help me?
Thanks
It is necessary update the Maven to the last version.
I have installed the Maven v.3.3.9 and now the error does not appear.
Related
I am trying to send and receive pickled versions of a random value generated by the producer. I am using the Multiprocess(not '-ing') and ForkingPickler module to to pickle and qeueue the generated value. However upon running the sample program below, i get the below error. Basis for using ForkingPickler is to pickle socket objects in future. I am now testing out with a sample version. Is this a feasible way to go about pickling socket objects?
rv = reduce(self.proto)
TypeError: cannot pickle 'memoryview' object
def producer(queue):
print('Producer: Running', flush=True)
# generate work
for i in range(10):
# generate a value
value = random()
# block
sleep(value)
# add to the queue
fork_value = ForkingPickler.dumps(value)
queue.put(fork_value)
# all done
queue.put(None)
print(f'Queue Size Consumer: {queue.qsize()}', flush=True)
print('Producer: Done', flush=True)
# consume work
def consumer(queue):
print('Consumer: Running', flush=True)
# consume work
while True:
print(f'Queue Size Consumer: {queue.qsize()}', flush=True)
# get a unit of work
fork_value = queue.get()
item = ForkingPickler.loads(fork_value)
# check for stop
if item is None:
break
# report
print(f'>got {item}', flush=True)
# all done
print('Consumer: Done', flush=True)
# entry point
if __name__ == '__main__':
# create the shared queue
queue = JoinableQueue()
# start the consumer
consumer_process = Process(target=consumer, args=(queue,))
consumer_process.start()
# start the producer
producer_process = Process(target=producer, args=(queue,))
producer_process.start()
# wait for all processes to finish
consumer_process.join()
producer_process.join()
I'm currently getting an error when I try to run my Play app. It says Failed to load data source but then it looks like it is indeed loading the data source. I'm very new to Play and Scala and the rest of my team is also new, so apologies if this is a silly error or if I'm missing some code samples. Database app-users with owner root exists on my local and I don't believe root has a password (created using the createuser tool).
Any ideas on what could cause this? Or what I am missing?
Error:
play.api.UnexpectedException: Unexpected exception[IllegalStateException: Failed to load data source for config: 'Config(SimpleConfigObject({"dataSource":"org.postgresql.ds.PGSimpleDataSource","database":"app-users","driver":"org.postgresql.Driver","host":"localhost","password":"","port":5432,"url":"jdbc:postgresql://localhost:5432/app-users","user":"root"}))']
at play.core.server.DevServerStart$$anon$1.reload(DevServerStart.scala:186)
at play.core.server.DevServerStart$$anon$1.get(DevServerStart.scala:124)
at play.core.server.AkkaHttpServer.modelConversion(AkkaHttpServer.scala:183)
at play.core.server.AkkaHttpServer.handleRequest(AkkaHttpServer.scala:189)
at play.core.server.AkkaHttpServer.$anonfun$createServerBinding$3(AkkaHttpServer.scala:106)
at akka.stream.impl.fusing.MapAsync$$anon$24.onPush(Ops.scala:1191)
at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:512)
at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:475)
at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:371)
at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:584)
Caused by: java.lang.IllegalStateException: Failed to load data source for config: 'Config(SimpleConfigObject({"dataSource":"org.postgresql.ds.PGSimpleDataSource","database":"app-users","driver":"org.postgresql.Driver","host":"localhost","password":"","port":5432,"url":"jdbc:postgresql://localhost:5432/app-users","user":"root"}))'
at io.getquill.JdbcContextConfig.dataSource(JdbcContextConfig.scala:24)
at io.getquill.PostgresJdbcContext.<init>(PostgresJdbcContext.scala:17)
at io.getquill.PostgresJdbcContext.<init>(PostgresJdbcContext.scala:18)
at io.getquill.PostgresJdbcContext.<init>(PostgresJdbcContext.scala:19)
at db.db.package$DBContext.<init>(package.scala:6)
at MyComponents.ctx$lzycompute(MyApplicationLoader.scala:19)
at MyComponents.ctx(MyApplicationLoader.scala:19)
at MyComponents.userService$lzycompute(MyApplicationLoader.scala:22)
at MyComponents.userService(MyApplicationLoader.scala:22)
at MyComponents.applicationController$lzycompute(MyApplicationLoader.scala:29)
Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: argument type mismatch
at com.zaxxer.hikari.util.PropertyElf.setProperty(PropertyElf.java:154)
at com.zaxxer.hikari.util.PropertyElf.lambda$setTargetFromProperties$0(PropertyElf.java:57)
at java.util.Hashtable.forEach(Hashtable.java:879)
at com.zaxxer.hikari.util.PropertyElf.setTargetFromProperties(PropertyElf.java:52)
at com.zaxxer.hikari.HikariConfig.<init>(HikariConfig.java:132)
at io.getquill.JdbcContextConfig.dataSource(JdbcContextConfig.scala:21)
at io.getquill.PostgresJdbcContext.<init>(PostgresJdbcContext.scala:17)
at io.getquill.PostgresJdbcContext.<init>(PostgresJdbcContext.scala:18)
at io.getquill.PostgresJdbcContext.<init>(PostgresJdbcContext.scala:19)
at db.db.package$DBContext.<init>(package.scala:6)
Caused by: java.lang.IllegalArgumentException: argument type mismatch
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.zaxxer.hikari.util.PropertyElf.setProperty(PropertyElf.java:149)
at com.zaxxer.hikari.util.PropertyElf.lambda$setTargetFromProperties$0(PropertyElf.java:57)
at java.util.Hashtable.forEach(Hashtable.java:879)
at com.zaxxer.hikari.util.PropertyElf.setTargetFromProperties(PropertyElf.java:52)
at com.zaxxer.hikari.HikariConfig.<init>(HikariConfig.java:132)
at io.getquill.JdbcContextConfig.dataSource(JdbcContextConfig.scala:21)
application.conf
play.db {
config = "db"
default = "default"
}
db.default {
driver = "org.postgresql.Driver"
dataSource = "org.postgresql.ds.PGSimpleDataSource"
url = "jdbc:postgresql://localhost:5432/app-users"
user = "root"
user = ${?DB_USER}
host = "localhost"
host = ${?DB_HOST}
port = 5432
port = ${?DB_PORT}
password = ""
password = ${?DB_PASSWORD}
database = "app-users"
}
db/package.scala
import io.getquill.{PostgresJdbcContext, SnakeCase}
package object db {
class DBContext(config: String) extends PostgresJdbcContext(SnakeCase, config)
trait Repository {
val ctx: DBContext
}
}
Using:
Scala 2.12.4
Quill 2.3.2
Play 2.6.6
Postgres JDBC Driver 42.2.1
PostgreSQL 10.2
UPDATE:
Added a password of "root" to the root user and switched to using the same format as the Quill docs, so now appliation.conf looks like this:
db.default {
dataSourceClassName = org.postgresql.ds.PGSimpleDataSource
dataSource.user = root
dataSource.password = root
dataSource.databaseName = app-users
dataSource.portNumber = 5432
dataSource.serverName = host
connectionTimeout = 30000
}
But the error message is still basically the same:
play.api.UnexpectedException: Unexpected exception[IllegalStateException: Failed to load data source for config: 'Config(SimpleConfigObject({"connectionTimeout":30000,"dataSource":{"databaseName":"app-users","password":"root","portNumber":5432,"serverName":"host","user":"root"},"dataSourceClassName":"org.postgresql.ds.PGSimpleDataSource"}))']
The following worked for me:
db.default {
dataSourceClassName = org.postgresql.ds.PGSimpleDataSource
dataSource.user = root
dataSource.password = root
dataSource.databaseName = app-users
dataSource.portNumber = 5432
dataSource.serverName = localhost
connectionTimeout = 30000
}
Basically, localhost instead of host. I'm guessing the first iteration didn't work because of the quotes.
So I have installed Cygnus and in the simple test configuration case which I took from here (https://github.com/telefonicaid/fiware-cygnus/blob/master/cygnus-ngsi/README.md) everything works fine.
But I need Postgresql as a backend for my application.
For this I adjusted the agent_1.conf file with all postgresql parameters found from http://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/installation_and_administration_guide/ngsi_agent_conf/
cygnus-ngsi.sources = http-source
cygnus-ngsi.sinks = postgresql-sink
cygnus-ngsi.channels = postgresql-channel
cygnus-ngsi.sources.http-source.channels = hdfs-channel mysql-channel ckan-channel mongo-channel sth-channel kafka-channel dynamo-channel postgresql-channel
cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.port = 5050
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
cygnus-ngsi.sources.http-source.handler.default_service = default
cygnus-ngsi.sources.http-source.handler.default_service_path = /
cygnus-ngsi.sources.http-source.interceptors = ts gi
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
cygnus-ngsi.sinks.postgresql-sink.channel = postgresql-channel
cygnus-ngsi.sinks.postgresql-sink.type = com.telefonica.iot.cygnus.sinks.NGSIPostgreSQLSink
cygnus-ngsi.sinks.postgresql-sink.postgresql_host = 127.0.0.1
cygnus-ngsi.sinks.postgresql-sink.postgresql_port = 5432
cygnus-ngsi.sinks.postgresql-sink.postgresql_database = myUser
cygnus-ngsi.sinks.postgresql-sink.postgresql_username = mydb
cygnus-ngsi.sinks.postgresql-sink.postgresql_password = xxxx
cygnus-ngsi.sinks.postgresql-sink.attr_persistence = row
cygnus-ngsi.sinks.postgresql-sink.batch_size = 100
cygnus-ngsi.sinks.postgresql-sink.batch_timeout = 30
cygnus-ngsi.sinks.postgresql-sink.batch_ttl = 10
# postgresql-channel configuration
cygnus-ngsi.channels.postgresql-channel.type = memory
cygnus-ngsi.channels.postgresql-channel.capacity = 1000
cygnus-ngsi.channels.postgresql-channel.transactionCapacity = 100
I didn'r really find any information about other files I am supposed to change and aren't really sure if all parameters are correct.
I also tried the sample configuration from here http://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/flume_extensions_catalogue/ngsi_postgresql_sink/index.html
Cygnus seems to start correctly but all if I try to send a notification I get connection refused
Befor doing anything, please, create the database, the user and the password in Postgresql.
The cygnus configuration is like this one. The /etc/cygnus/conf/cygnus_instance_1.conf file:
CYGNUS_USER=cygnus
CONFIG_FOLDER=/usr/cygnus/conf
CONFIG_FILE=/usr/cygnus/conf/agent_1.conf
AGENT_NAME=cygnus-ngsi
LOGFILE_NAME=cygnus.log
ADMIN_PORT=8081
POLLING_INTERVAL=30
So, the other file /usr/cygnus/conf/agent_1.conf is like this one (please change PostgreSQL parameters):
cygnus-ngsi.sources = http-source
cygnus-ngsi.sinks = postgresql-sink
cygnus-ngsi.channels = postgresql-channel
cygnus-ngsi.sources.http-source.channels = postgresql-channel
cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.port = 5050
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
cygnus-ngsi.sources.http-source.handler.default_service = default
cygnus-ngsi.sources.http-source.handler.default_service_path = /
cygnus-ngsi.sources.http-source.handler.events_ttl = 10
cygnus-ngsi.sources.http-source.interceptors = ts gi
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
#cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
# =============================================
# postgresql-channel configuration
# channel type (must not be changed)
cygnus-ngsi.channels.postgresql-channel.type = memory
# capacity of the channel
cygnus-ngsi.channels.postgresql-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnus-ngsi.channels.postgresql-channel.transactionCapacity = 100
# ============================================
# NGSIPostgreSQLSink configuration
# channel name from where to read notification events
cygnus-ngsi.sinks.postgresql-sink.channel = postgresql-channel
# sink class, must not be changed
cygnus-ngsi.sinks.postgresql-sink.type = com.telefonica.iot.cygnus.sinks.NGSIPostgreSQLSink
# true applies the new encoding, false applies the old encoding.
# cygnus-ngsi.sinks.postgresql-sink.enable_encoding = false
# true if the grouping feature is enabled for this sink, false otherwise
cygnus-ngsi.sinks.postgresql-sink.enable_grouping = false
# true if name mappings are enabled for this sink, false otherwise
cygnus-ngsi.sinks.postgresql-sink.enable_name_mappings = false
# true if lower case is wanted to forced in all the element names, false otherwise
# cygnus-ngsi.sinks.postgresql-sink.enable_lowercase = false
# the FQDN/IP address where the PostgreSQL server runs
cygnus-ngsi.sinks.postgresql-sink.postgresql_host = 127.0.0.1
# the port where the PostgreSQL server listens for incomming connections
cygnus-ngsi.sinks.postgresql-sink.postgresql_port = 5432
# the name of the postgresql database
cygnus-ngsi.sinks.postgresql-sink.postgresql_database = cygnusdb
# a valid user in the PostgreSQL server
cygnus-ngsi.sinks.postgresql-sink.postgresql_username = cygnus
# password for the user above
cygnus-ngsi.sinks.postgresql-sink.postgresql_password = cygnusdb
# how the attributes are stored, either per row either per column (row, column)
cygnus-ngsi.sinks.postgresql-sink.attr_persistence = row
# select the data_model: dm-by-service-path or dm-by-entity
cygnus-ngsi.sinks.postgresql-sink.data_model = dm-by-entity
# number of notifications to be included within a processing batch
cygnus-ngsi.sinks.postgresql-sink.batch_size = 1
# timeout for batch accumulation
cygnus-ngsi.sinks.postgresql-sink.batch_timeout = 30
# number of retries upon persistence error
cygnus-ngsi.sinks.postgresql-sink.batch_ttl = 0
# true enables cache, false disables cache
cygnus-ngsi.sinks.postgresql-sink.backend.enable_cache = true
I have installed Cygnus 0.8.2 in a VM with CentOS-6.5-x64.
In the config file agent.conf I only change the following:
cygnusagent.sinks.hdfs-sink.oauth2_token = xxxxxx
cygnusagent.sinks.hdfs-sink.hdfs_username = myuser
I run cygnus with this command:
/usr/cygnus/bin/cygnus-flume-ng agent --conf /usr/cygnus/conf/ -f /usr/cygnus/conf/agent.conf -n cygnusagent -Dflume.root.logger=INFO,console
But when I send a xml message occurs this error:
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)
2015-07-23 14:46:02,133 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - com.telefonica.iot.cygnus.sinks.OrionSink.process(OrionSink.java:163)] The event TTL has expired, it is no more re-injected in the channel (id=617320308, ttl=0)
2015-07-23 14:46:02,133 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - com.telefonica.iot.cygnus.sinks.OrionSink.process(OrionSink.java:193)] Finishing transaction (1437651766-108-0000000000)
2015-07-23 14:46:02,637 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - com.telefonica.iot.cygnus.sinks.OrionSink.process(OrionSink.java:128)] Event got from the channel (id=617320308, headers={timestamp=1437651946136, content-type=application/xml, transactionId=1437651766-108-0000000000, fiware-service=def_serv, fiware-servicepath=def_servpath, ttl=0, destination=sensorreading1_sensorreading}, bodyLength=891)
2015-07-23 14:46:02,643 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persist(OrionHDFSSink.java:356)] [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (def_serv/def_servpath/sensorreading1_sensorreading/sensorreading1_sensorreading.txt), Data ({"recvTime":"2015-07-23T11:45:46.136Z","nodeid":"1", "nodeid_md":[],"sensorid":"1", "sensorid_md":[],"systemid":"1", "systemid_md":[],"value":"-990.6", "value_md":[]})
2015-07-23 14:46:02,644 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - com.telefonica.iot.cygnus.sinks.OrionSink.process(OrionSink.java:143)] Persistence error (The /user/cristina.albaladejo/def_serv/def_servpath/sensorreading1_sensorreading directory could not be created in HDFS. HttpFS response: 503 Service unavailable)
2015-07-23 14:46:02,644 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - com.telefonica.iot.cygnus.sinks.OrionSink.process(OrionSink.java:163)] The event TTL has expired, it is no more re-injected in the channel (id=617320308, ttl=0)
2015-07-23 14:46:02,644 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - com.telefonica.iot.cygnus.sinks.OrionSink.process(OrionSink.java:193)] Finishing transaction (1437651766-108-0000000000)
Then, data are not stored in HDFS... How I can solve this?
The complete config is:
cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink mysql-sink ckan-sink
cygnusagent.channels = hdfs-channel mysql-channel ckan-channel
cygnusagent.sources.http-source.channels = hdfs-channel mysql-channel ckan-channel
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnusagent.sources.http-source.port = 5050
cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
cygnusagent.sources.http-source.handler.notification_target = /notify
cygnusagent.sources.http-source.handler.default_service = def_serv
cygnusagent.sources.http-source.handler.default_service_path = def_servpath
cygnusagent.sources.http-source.handler.events_ttl = 10
cygnusagent.sources.http-source.interceptors = ts gi
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
# OrionHDFSSink configuration
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
cygnusagent.sinks.hdfs-sink.type = com.telefonica.iot.cygnus.sinks.OrionHDFSSink
cygnusagent.sinks.hdfs-sink.hdfs_host = x1.y1.z1.w1,x2.y2.z2.w2
cygnusagent.sinks.hdfs-sink.hdfs_port = 14000
cygnusagent.sinks.hdfs-sink.hdfs_username = cristina.albaladejo
cygnusagent.sinks.hdfs-sink.oauth2_token = MYTOKEN
cygnusagent.sinks.hdfs-sink.attr_persistence = column
cygnusagent.sinks.hdfs-sink.hive_host = x.y.z.w
cygnusagent.sinks.hdfs-sink.hive_port = 10000
cygnusagent.sinks.hdfs-sink.krb5_auth = false
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_user = krb5_username
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_password = xxxxxxxxxxxxx
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_login_conf_file = /usr/cygnus/conf/krb5_login.conf
cygnusagent.sinks.hdfs-sink.krb5_auth.krb5_conf_file = /usr/cygnus/conf/krb5.conf
cygnusagent.channels.hdfs-channel.type = memory
cygnusagent.channels.hdfs-channel.capacity = 1000
cygnusagent.channels.hdfs-channel.transactionCapacity = 100
It seems you have not configured the following properties (putting the required values):
cygnusagent.sinks.hdfs-sink.hdfs_host = cosmos.lab.fiware.org
cygnusagent.sinks.hdfs-sink.hive_host = cosmos.lab.fiware.org
In addition, since you are only configuring the HDFS persistence, avoid any reference to mysql-sink and ckan-sink, i.e.:
cygnusagent.sinks = hdfs-sink
cygnusagent.channels = hdfs-channel
cygnusagent.sources.http-source.channels = hdfs-channel
Try with the proposed changes and let me know if it works.
I have a table row that is created, and periodically updated, with session.merge(). If the table in the DB (sqlite) is deleted, then I get an ObjectDeletedError, but I can't find a way to gracefully handle this.
On exception, I recreate the table, but the exception still occurs.
What do I need to do to get the session to recognise the new table, create the row and continue on as before ??
Here is the code ...
#!/usr/bin/env python
import sqlalchemy
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
Base = declarative_base()
##============================================================================
class Module(Base):
"""Schema for the modules table."""
##------------------------------------------------------------------------
__tablename__ = "modules"
id = sqlalchemy.Column(sqlalchemy.Integer, primary_key=True)
port = sqlalchemy.Column(sqlalchemy.Integer)
##------------------------------------------------------------------------
def __init__(self, id, port=None):
self.id = id
self.port = port
##============================================================================
class SessionMgr(object):
"""Class to manage the database session."""
##------------------------------------------------------------------------
def __init__(self, commitInterval=5, debug=False):
self.commitInterval = commitInterval
database = "merge-bug.db"
self.engine = sqlalchemy.create_engine("sqlite:///%s" % (database), echo=debug)
Session = sessionmaker(bind=self.engine)
self.session = Session()
self.create_tables()
##------------------------------------------------------------------------
def create_tables(self):
"""Create database tables if they do not exist."""
### List of tables to check/create.
tables = [ Module ]
### Create each table if it does not exist.
for t in tables:
tname = t.__tablename__
if not self.engine.dialect.has_table(self.engine.connect(), tname):
print "%s table didn't exist. Creating..." % (tname)
t.metadata.create_all(self.engine)
##------------------------------------------------------------------------
def commit(self):
"""Commit changes to the database."""
print "Committing to db"
try:
self.session.commit()
except Exception as ex:
print "ERROR: SessionMgr.commit():", ex
print "DEBUG: SessionMgr.commit(): Issuing a session rollback ..."
self.session.rollback()
## Check that tables still exist.
self.create_tables()
finally:
## Optional -- depends on use case.
#self.session.remove()
pass
##------------------------------------------------------------------------
##============================================================================
def main():
print "DEBUG: main():"
import time
sessmgr = SessionMgr(commitInterval=5, debug=False)
##
## Test adding a module and updating it using session.merge().
##
if 1:
errors = 0
m = Module(1234, 43210)
print "DEBUG: merge module"
mm = sessmgr.session.merge(m)
sessmgr.commit()
delay = 1
for i in range(10):
try:
print "DEBUG: sleeping %i second ..." % (delay)
time.sleep(delay)
port = mm.port + 1
print "DEBUG: updating port field to %i ..." % (port)
## Exception on the following statement if table is deleted !!
mm.port = port
sessmgr.commit()
print "DEBUG: updating id field ..."
mm.id = 1234
sessmgr.commit()
except Exception as ex:
print "DEBUG: caught exception", ex
if 0:
print "DEBUG: Issuing a session rollback ..."
sessmgr.session.rollback()
if 1:
print "DEBUG: Create tables ..."
sessmgr.create_tables()
if 1:
print "DEBUG: Remerge module ..."
m = Module(1234, 43210)
mm = sessmgr.session.merge(mm)
if 0:
print "DEBUG: Refresh merged module ..."
mm = sessmgr.session.merge(mm)
errors += 1
print "DEBUG: errors =", errors
if errors > 3:
raise
##============================================================================
if __name__ == "__main__":
main()
##============================================================================
Here is the output ...
DEBUG: main():
DEBUG: merge module
Committing to db
DEBUG: sleeping 2 second ...
DEBUG: updating port field to 43211 ...
Committing to db
DEBUG: updating id field ...
Committing to db
DEBUG: sleeping 2 second ...
DEBUG: caught exception (OperationalError) no such table: modules u'SELECT modules.id AS modules_id, modules.port AS modules_port \nFROM modules \nWHERE modules.id = ?' (1234,)
DEBUG: Create tables ...
modules table didn't exist. Creating...
Committing to db
DEBUG: Remerge module ...
DEBUG: errors = 1
DEBUG: sleeping 2 second ...
DEBUG: caught exception Instance '<Module at 0x102a586d0>' has been deleted, or its row is otherwise not present.
DEBUG: Create tables ...
Committing to db
DEBUG: Remerge module ...
DEBUG: errors = 2
DEBUG: sleeping 2 second ...
DEBUG: caught exception Instance '<Module at 0x102a586d0>' has been deleted, or its row is otherwise not present.
DEBUG: Create tables ...
Committing to db
DEBUG: Remerge module ...
DEBUG: errors = 3
DEBUG: sleeping 2 second ...
DEBUG: caught exception Instance '<Module at 0x102a586d0>' has been deleted, or its row is otherwise not present.
DEBUG: Create tables ...
Committing to db
DEBUG: Remerge module ...
DEBUG: errors = 4
Traceback (most recent call last):
File "merge_bug.py", line 135, in <module>
main()
File "merge_bug.py", line 103, in main
port = mm.port + 1
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/sqlalchemy/orm/attributes.py", line 316, in __get__
return self.impl.get(instance_state(instance), dict_)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/sqlalchemy/orm/attributes.py", line 611, in get
value = callable_(passive)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/sqlalchemy/orm/state.py", line 375, in __call__
self.manager.deferred_scalar_loader(self, toload)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/sqlalchemy/orm/loading.py", line 606, in load_scalar_attributes
raise orm_exc.ObjectDeletedError(state)
sqlalchemy.orm.exc.ObjectDeletedError: Instance '<Module at 0x102a586d0>' has been deleted, or its row is otherwise not present.
What do I need to do to get the session to recognise the new table, create the row and continue on as before ??
when an error occurs with the Session you need to roll it back to cancel the current transaction:
session.rollback()
now if you are still having this issue, then it is likely that SQLite can't handle the table being dropped (I assume by a different connection?) you may need to reconnect to the database entirely - disabling connection pooling using NullPool will achieve this, each time the Session closes its transaction, the connection itself will be closed.
It's also not really the appropriate use of a relational database to be creating and dropping tables as the application is running. If you need to delete all the data in a table, use "DELETE FROM table". The table structure itself should not be changing while the application is running (especially on a fragile system like SQLite).