Global Context Broker Federation - fiware-orion

Until now, I've been working with a context Broker Instance in stand-alone mode created in Fi-Lab/Cloud and using psb-orion-image. Now I would like to federate an instance of that CB with http//orion.lab.fi-ware.org. I use XML forms to create, update... and the name of the instance to federate is "UPCT:TEMPERATURE:SENSOR" wich sensor type is "UPCT:SENSOR".
So, connecting by SSH I send the next form:
(curl localhost:1026/NGSI10/subscribeContext -s -S --header 'Content-Type: application/xml' -d #- | xmllint --format -) <<EOF
<?xml version="1.0"?>
<subscribeContextRequest>
<entityIdList>
<entityId type="UPCT:SENSOR" isPattern="false">
<id>UPCT:TEMPERATURE:SENSOR</id>
</entityId>
</entityIdList>
<reference>http://orion.lab.fi-ware.eu:1026/ngsi10/notifyContext</reference>
<duration>P1M</duration>
<notifyConditions>
<notifyCondition>
<type>ONCHANGE</type>
<condValueList>
<condValue>temperature</condValue>
</condValueList>
</notifyCondition>
</notifyConditions>
<throttling>PT5S</throttling>
</subscribeContextRequest>
EOF
And I get a right reply, with and subscription ID. However, if I update de contextValue of my instance and try to send a query to http://orion.lab.fi.ware.eu:1026 I receive an error:
-:1: parser error : Start tag expected, '<' not found
Auth-token not found in request header
^
I think I should get the same value that I update to my Instance as indicate in
https://forge.fi-ware.org/plugins/mediawiki/wiki/fiware/index.php/Publish/Subscribe_Broker_-_Orion_Context_Broker_-_User_and_Programmers_Guide#Context_Broker_Federation
I need to know what's wrong and how could I do federete to the global CB.
Thank you

That error message is due to any request sent to Orion instance at orion.lab.fi-ware.org (including the notifications sent by other Orion instances) has to use autentication. At the present moment (i.e. version 0.14.1), Orion doesn't include the X-Auth-Token needed for authentication (see the quick start for programmers) in notifications.
However, the usual use case is federating Orion at orion.lab.fi-ware.org with private Orion user instances (i.e. orion.lab.fi-ware.org -> your Orion) not in the opposite way (i.e. your Orion -> orion.lab.fi-ware.org) as your are trying. This is given that typically what you want to do is to merge public information (e.g. Santander city sensors) with private information (e.g. the information produced by your sensors). Federating orion.lab.fi-ware.org -> your Orion in that way should work perfectly.
EDIT: the limitation in Orion 0.14.1 has been overcomed an current Orion version (2.0.0) propagates X-Auth-Token header in notifications (I don't remember in which exact version between 0.14.1 and 2.0.0, sorry...)

Related

wso2 API Endpoint creation failed: 404 ressource not found

I am trying to create a REST API with wso2 API_Manager to gather data from a Postgres database (learning purpose). I struggle doing so and I would like to know whether:
I did not understand wso2 components' roles properly (new techno and subject for me)
or there is an error in the way I configured the manager.
System setup
I used this official docker image, added postgres jdbc jar in /repository/components/lib/ and added the following in /repository/conf/datasources/master-datasources.xml:
<datasource>
<name>s0m3dAtabas3</name>
<description>The db used for testing purposes</description>
<definition type="RDBMS">
<configuration>
<url>jdbc:postgresql://sandor_postgres:5432/s0m3dAtabas3</url>
<driverClassName>org.postgresql.Driver</driverClassName>
<username>s0m3us3rfr0mdAtAMaj0r</username>
<password>N0t5uchAs1mple1</password>
<maxActive>80</maxActive>
<minIdle>5</minIdle>
<maxWait>60000</maxWait>
<defaultAutoCommit>false</defaultAutoCommit>
<testOnBorrow>true</testOnBorrow>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
I made sure than the postgres' container named sandor_postgres is accessible from wso2's with these credentials. In this database, I have a table called something. The image comes with the following UIs:
admin
publisher
store
Graphical API creation
I first followed the WorldBank tutorial which seemed crystal clear (though I am not quite sure where the data came from). I then tried to adapt it.
Step 1: Design
I used the database name as context (s0m3dAtabas3) v.1.0.0. Since the table is called something, the url pattern I end up with is /s0m3dAtabas3/1.0.0/something
Step 2: Implement
This is where things start to be confusing. No matter the resource path I use in the Endpoint (end point type REST), I get a 404 and the logs are not very helpful
http://192.168.8.111:8280 -> 404
http://192.168.8.111:8280/something -> 404
http://192.168.8.111:9443/tried_several -> Invalid - Error connecting to backend
http://192.168.8.111:8243/tried_several -> Invalid - Error connecting to backend
INFO - InboundDBSyncRequestEvent Running DB sync task.
INFO - LogMediator STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = /s0m3dAtabas3/1.0.0
INFO - CarbonAuthenticationUtil 'admin#carbon.super [-1234]' logged in at [2019-10-29 11:42:31,030+0000]
INFO - CarbonAuthenticationUtil 'admin#carbon.super [-1234]' logged in at [2019-10-29 11:42:31,197+0000]
INFO - LogMediator STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = /s0m3dAtabas3/bullshit
INFO - CarbonAuthenticationUtil 'admin#carbon.super [-1234]' logged in at [2019-10-29 11:48:30,649+0000]
INFO - CarbonAuthenticationUtil 'admin#carbon.super [-1234]' logged in at [2019-10-29 11:48:30,790+0000]
INFO - LogMediator STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = /
INFO - InboundDBSyncRequestEvent Running DB sync task.
INFO - LogMediator STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = /
Did I miss some important configuration step or wso2 API Manager is not the standalone component I thought it was and requires another component to achieve what I am looking for?
It seems there is a misunderstanding in the concept-wise.
Here is the basic idea of a typical API Management solution.
You have a web service (REST, SOAP, etc...) which you need to expose as a managed API. Now, you can front your service with API Manager and expose it as a managed API with security, rate limiting, managed life cycle etc.
In your case, it seems you don't have such a service, but only have a database table. So, before using API Manager to front your service, you first need to expose your table as a service. For that purpose, I'd suggest you use the data service component of WSO2 EI 7.0.0. See [1] for how to do that. Once you have your service ready, you can use API Manager to expose it as a managed API.
[1] https://ei.docs.wso2.com/en/latest/micro-integrator/use-cases/tutorials/sending-a-simple-message-to-a-datasource/

Intermittent problems with querying data from druid using SQL

I query data from druid via SQL. Sometimes it succeeds, but sometimes it fails. My query uses curl; it is:
curl --negotiate -u:srvadmin -X POST -H'Content-Type: application/json' http://du-s12-idc:8082/druid/v2/sql -d #query.json.
When it fails, I get this response:
{"error":"Unknown exception","errorMessage":"Failure getting results for query[6639c357-441f-456c-9a01-0f7ffd0758b7] url[http://du-s28-idc:8083/druid/v2/] because of [Invalid type marker byte 0x3c for expected value token\n at [Source: (SequenceInputStream); line: -1, column:
1]]","errorClass":"io.druid.java.util.common.RE","host":null}
The file query.json is simple:
{"query":"select * from bds_dsp_media_run_info_h_1016 limit 3"}
The data was loaded from hadoop to druid and succeeded. My druid version was 0.11 and built in a cluster with Kerberos.
Does anyone have this problem?
I think Invalid type marker byte 0x3c... exception is just uninformative response that tells you that the server has an internal error, but doesn't give you a clue on what is actually happening. It would help a lot if you could check broker logs when the request is happening.
But, to play a guessing game - I would expect it to be a Kerberos issue. Do you have KRB5_CLIENT_KTNAME env variable populated with the path to your key file?

Custom Spring cloud application - Kafka embeddedheader issue

I am trying to build a custom transformer application using the guidelines provided here
https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#streams-dev-guide
I have started kafka on my windows machine.
I have http source running on windows machine it writes to destination transformData.
Command: java -Dserver.port=8123 -Dhttp.path-pattern=/data -Dspring.cloud.stream.bindings.output.destination=transformData -jar http-source-kafka-10-1.3.1.RELEASE.jar
I have transform application running that reads input from transformData and outputs to destination transformedData
Command
java -Dserver.port=8090 -Dspring.cloud.stream.bindings.input.destination=transformData -Dspring.cloud.stream.bindings.output.destination=transformedData -jar transformer-0.0.1-SNAPSHOT.jar
I have log sink running that reads from destination transformedData
Command
java -Dserver.port=8888 -Dspring.cloud.stream.bindings.input.destination=transformedData -jar log-sink-kafka-10-1.3.1.RELEASE.jar
Problem:
When I try to send this curl request:
curl -H "Content-Type: application/json" -X POST -d '{"id":"1", "temp":"400"}' http://172.20.24.47:8123/data
On the custom Transformer console I see errors:
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized
token '▒': was expecting ('true', 'false' or 'null') at [Source:
(byte[])"?
contentType
"text/plain"originalContentType "application/json;charset=UTF-8"{"id":"1", "temp":"400"}"; line: 1,
column: 4]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804)
~[jackson-core-2.9.6.jar!/:2.9.6]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:679)
~[jackson-core-2.9.6.jar!/:2.9.6]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3526)
~[jackson-core-2.9.6.jar!/:2.9.6]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2621)
~[jackson-core-2.9.6.jar!/:2.9.6]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:826)
~[jackson-core-2.9.6.jar!/:2.9.6]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:723)
~[jackson-core-2.9.6.jar!/:2.9.6]
at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4141)
~[jackson-databind-2.9.6.jar!/:2.9.6]
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4000)
~[jackson-databind-2.9.6.jar!/:2.9.6]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3121)
~[jackson-databind-2.9.6.jar!/:2.9.6]
at org.springframework.cloud.stream.converter.ApplicationJsonMessageMarshallingConverter.convertParameterizedType(ApplicationJsonMessageMarshallingConverter.java:114)
~[spring-cloud-stream-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]
... 37 common frames omitted
Can any one help?
I got this to finally work. When building the custom application using the Spring initializr instead of selecting 2.0.4 release as starter I reverted back to 1.5.15 Release. Now I have no more need to pass properties on the subscriber end that is the custom app and the logger sink app using headerModes set to embeddedHeaders.
It appears that you are using Spring Cloud Stream 2.0.0.RELEASE, but your app is 1.3.x. Can you set spring.cloud.stream.bindings.input.consumer.headerMode to embeddedHeaders in the processor app where it is failing? In 2.0, the default is not to embed headers as Kafka supports it out of the box. However, in versions prior to 2.0 (which is used by 1.3.x apps), the default is to embed the headers. You need to explicitly set that when using that combination.

Howto add a connector to rest-service-component

In my Mule 2.X configuration I need to post a message as a URL parameter. Therefore I created a service that has an inbound and then tries to send the message using rest-service-component, as follows:
<service name="myService">
<inbound>
<vm:inbound-endpoint path="path/inbound" synchronous="true" connector-ref="myVmConnector"/>
</inbound>
<http:rest-service-component serviceUrl="http://www.domain.com/path/insert.asp" httpMethod="POST">
<http:payloadParameterName value="data_xml"/>
</http:rest-service-component>
</service>
But when I process a message through it I receive the following information:
Message : There are at least 2 connectors matching protocol "http", so the connector to use must be specified on the endpoint using the 'connector' property/attribute (java.lang.IllegalStateException)
Type : org.mule.transport.service.TransportFactoryException
Code : MULE_ERROR--2
http://www.mulesource.org/docs/site/current2/apidocs/org/mule/transport/service/TransportFactoryException.html
Normally this error can occur when you have multiple HTTP connectors configured and then you need to specify the connector on the endpoint (connector-ref). But the rest-service-component does not have such an attribute or child elements (http://www.mulesoft.org/documentation-3.2/display/MULE2USER/HTTP+Transport)
This is a very old bug that remained unresolved:
https://www.mulesoft.org/jira/browse/MULE-4272
Try to use http:outbound-endpoint instead.
UPDATE:
Try using a Groovy script component in place of the rest component to create calls to dynamic URLs in Mule 2. Something like this might work:
eventContext.sendEvent(message,"http://www.domain.com/path/insert.asp?data_xml=${payload}")
Depending on your payload, you may also have to URL encode that.

Atmosphere Jersey - Server messages

Is it possible to let the server send messages to all connected clients without waiting for any action from them? Let me explain it :-) I've been reading the docs/examples and I have found nothing that satifies my needs: the flow is always the same; a client connects (e.g: a GET call to a Rest API), the connection is suspendend and until a new API call is received (e.g.: a POST call) the server simply waits (or at least this is what I have understood). My use case is pretty different: I want the server to send some "notifications" once new data become available. This would be my use case (pretty simplifed):
Client A connects to server
Connection is suspended since no new data is available at the moment
The server gets notified new data is available from an external
source and broadcasts it to client A
Go to step 2
What I have achieved so far is getting the connection successfully established. The next step is to solve this server issue. I must say this technology is completely new to me so it is possible I misunderstood how something works. If that's the case, let me know!
This is my stack:
Spring 3.2.0 RELEASE
Jersey 1.8
Atmosphere Jersey 1.0.13
Tomcat 7.0.40
Thank you all in advance!
UPDATE: After following this I get this warning, which I have no idea how to get rid of:
2013-06-04 09:40:36,284 WARN [org.atmosphere.cpr.AtmosphereFramework] - Failed using comet support: org.atmosphere.container.Tomcat7AsyncSupportWithWebSocket, error: Tomcat failed to detect this is a Comet application because context.xml is missing or the Http11NioProtocol Connector is not enabled.
If that's not the case, you can also remove META-INF/context.xml and WEB-INF/lib/atmosphere-compat-tomcat.jar Is the Nio or Apr Connector enabled?
2013-06-04 09:40:36,285 WARN [org.atmosphere.cpr.AtmosphereFramework] - Using org.atmosphere.container.Tomcat7BIOSupportWithWebSocket
I followed the app structure commented here, so this should not be a problem. I have noticed that by changing the transport to "websocket" instead of "long-polling" shows no errors. The server finally sends data tough :)
I followed your link and modified the code a little.
When you are in the step 3 "The server gets notified new data is available from an external source and broadcasts it to client A", you have to write a line like this:
BroadcasterFactory.getDefault().lookup("/*").broadcast(response);
At first I used the TextMessage received from my ActiveMQ Queue but I get this error, so I used a Jackson class as an object response and everything worked fine.
SEVERE: A message body writer for Java class org.apache.activemq.command.ActiveMQTextMessage, and Java type class org.apache.activemq.command.ActiveMQTextMessage, and MIME media type application/json was not found
jun 03, 2014 11:32:21 AM com.sun.jersey.spi.container.ContainerResponse write
SEVERE: The registered message body writers compatible with the MIME media type are:
application/json (JSONJAXBElementProvider, JSONArrayProvider, JSONObjectProvider, JSONRootElementProvider, JSONListElementProvider, ...)