I want to use continuous queries to KSQL Server using web socket queries.
I heard that the KSQL CLI and Control Center interact with KSQL Server using web socket requests, but I did not find any information on using web sockets in the KSQL REST API reference. How to configure and use this feature?
Websockets is not a supported API, and not documented.
I guess you could run Confluent Control Center yourself and sniff its behaviour, but there'd be no guarantee that the API wouldn't change.
Related
I'm using Kafka connect to pull data from different places: mysql, mongodb, etc. And send to elastic search.
I would like to pull data where the origin is a webhook. So, can I configure some Kafka URL to send post http requests to it? What is a good practice to send to my Kafka http post requests?
Confluent maintains a source-available Kafka REST Proxy (be sure to read the Confluent Community License of the project). This would allow you to "send POST requests" to something that forwards this data to Kafka. Outside of this, write a simple HTTP endpoint on your own that does the same.
Personally, I have used Nifi's ListenHTTP handler to accept webhooks, then parse, route, filter, etc into a ProduceKafka request
Otherwise "pulling data" isn't a web hook, and there are a handful of "Kafka Connect http" source projects on Github.
I'm trying to connect broker with authorization( BlueMix IoT Foundation), but don't know how to provide credentials? How do I do it with Spark?
There is no way to connect to brokers with authorization using default MQTT connector from SparkStreaming. You have to redefine MQTTInputDstream in your own way to enable authentication.
I have customized the spark-streaming-mqtt-connector library from Apache Spark and added the following,
Added TLS v1.2 security such that the communication is always secured.
Stored topic along with the payload in the RDD.
The library is available here - https://github.com/sathipal/spark-streaming-mqtt-with-security_2.10-1.3.0. Also, added more detail in how to use the library to connect to IBM Watson IoT Platform. Let me know if you have any queries.
I'm working with Django and Loggly, and I need to decide between using Loggly with rsyslog or with its RESTful API. For the second option, I'd use grequests, sending a single request at a time (i.e., just to make the calls non-blocking, but I wouldn't send requests in bulk).
What are the advantages of using rsyslog over the RESTful API and vice versa?
Haven't tested it yet, but using the syslog approach has several advantages:
You can centralize logs at a system level, without particular
configurations on the django app
Logging is decoupled from the django app, you can set it to log to file,
a remote syslog server or loggly, without touching the django app
It should be faster if using UDP
If using a centralized syslog server, you only have to set the loggly
agent there
On the other hand, using the RestAPI would couple the app to the loggly implementation, and it could raise some errors while trying to report errors (DNS resolution failures, network problems, etc)
I am trying to get some simple web statistics using a wso2 BAM server, I have installed it, I understand how the events are defined, stored on the casssandra database etc. but I lack the most important thing: how to pass the events themselves to the BAM server?
I have programmed systems where json files are stored in a file ("queue") and my java process reads and processes them every 30 seconds, I thought this would be similar but the truth is I am pretty lost.
The port 7611 is TCP, listening.
Do I have to send the BAM server the events throught it?
Any help would be greatly apprecciated!
AFAIK, WSO2 BAM is not supported for apache mq directly. you have to use ESB or CEP to send events to BAM through thrift (Then you can use apache mq to send the events to ESB/CEP and through any of these product, you can send the events to BAM) .
But You can use the REST api to send the events to BAM which is more simple approach. Please refer the doc [1]. I thing future releases of BAM will support for jms (FYI, refer [2]).
[1] http://docs.wso2.org/wiki/display/BAM220/REST+API
[2] WSO2 BAM with JMS
Thnaks,
Mohan
BAM can get events either through REST API or through Thrift API.
7611 is the unsecured Thrift port and 7711 is the secured Thrift port by default.
As currently BAM does not have out-of-the-box Data Agent for Apache MQ, you have to write a one yourself.
If you want to publish events directly to BAM using Java you can use the DataBridge component in WSO2 code base. (i.e., use https://svn.wso2.org/repos/wso2/carbon/platform/branches/4.1.0/components/data-bridge/org.wso2.carbon.databridge.agent.thrift/4.1.1)
I'm looking to start using javascript on the server, most likely with node.js, as well as use websockets to communicate with clients. However, there doesn't seem to be a lot of information about encrypted websocket communication using TLS and the wss:// handler. In fact the only server that I've seen explicitly support wss:// is Kaazing.
This TODO is the only reference I've been able to find in the various node implementations. Am I missing something or are the websocket js servers not ready for encrypted communication yet?
Another option could be using something like lighttpd or apache to proxy to a node listener, has anyone had success there?
TLS/SSL support works for this websocket implementation in Node.js, I just tested it: https://github.com/Worlize/WebSocket-Node/issues/29
Well you have stream.setSecure() and server.setSecure().
I'm guessing you should be able to use one of those (specially the last one) to use TLS in websockets since in the end a websocket is just a normal http connection "upgraded" to websocket.
Using TLS in the normal http server object should theorically also secure the websocket, only by testing this can be confirmed.