I'm using Kafka connect to pull data from different places: mysql, mongodb, etc. And send to elastic search.
I would like to pull data where the origin is a webhook. So, can I configure some Kafka URL to send post http requests to it? What is a good practice to send to my Kafka http post requests?
Confluent maintains a source-available Kafka REST Proxy (be sure to read the Confluent Community License of the project). This would allow you to "send POST requests" to something that forwards this data to Kafka. Outside of this, write a simple HTTP endpoint on your own that does the same.
Personally, I have used Nifi's ListenHTTP handler to accept webhooks, then parse, route, filter, etc into a ProduceKafka request
Otherwise "pulling data" isn't a web hook, and there are a handful of "Kafka Connect http" source projects on Github.
Related
Is this more recommended than having the Kafka producer events done directly by whoever needs to emit it?
I assume by "API", you mean HTTP.
Depends on how easily you can configure Kafka or HTTP Authz & Authn. But if anyone exposes your HTTP endpoints / credentials, you'll end up with a bunch of questionable data in the topic.
With an HTTP interface, you can at least consolidate input validation. But you could also do the same with a TCP proxy, or gRPC endpoint. So, it solves the case where no one should be able to send random binary nonsense into your topics with a direct producer api call, if your producer credentials were to get leaked.
If you need simple HTTP passthough, that can be provided by Confluent REST Proxy, for example, but you still should setup an API gateway + auth / firewall to restrict access to it.
I use Confluent Kafka REST Proxy to send messages to Apache Kafka.
I set up basic authentication on the REST Proxy and whenever I submit a HTTP request to the proxy, I get the 403 HTTP Error !role.
The proxy requires Zookeeper, Kafka and Schema Registry to be running. I didn't configure any security on these services.
Without authentication, the proxy works and delivers messages to Kafka successfully.
How to I troubleshoot this problem? I spent multiple hours on that problem and I still can't fix it.
Check following:
Firewall allow the service or port
Is there any antivirus block the service or port
Rights given on kafka, confluent folder & respective log directory to kafka user.
I'm starting out with Kafka.
I see that I'm able to pass headers when producing messages.
Traditionally one would have a web client (single page app) where to user logs in via some remote oidc idp and receives a token. That token is then sent via Authentication: Bearer token-here header to some RESTful backend where the token is checked for validity and the payload is processed, saved to database or other and something is returned or not.
Now there's Apache Kafka. It has a REST proxy. I can pass headers to the REST proxy and produce messages, or consume them, but I'm interested in the "secure my RESTful JSON API" part.
Currently, without Kafka, I have either a oidc proxy (using keycloak, that's keycloak-gatekeeper) that does the filtering of which request makes it to the backend, or I have a oidc client that does token validation as some middleware function inside the backend. In any case invalid requests doesn't get "logged" as they would in Kafka, I assume.
Where does oidc token validation and request filtering fit in the Kafka/Confluent ecosystem?
Assume we have a SPA that talks to the Confluent REST Proxy. Some logged in user wants to post messages and some non-logged in user should not be able to.
How does Kafka and/or its tools deal with that scenario?
Kafka commonly uses SASL and other Authorization plugins to prevent access.
Certificates would be distributed amongst clients (here, that is the REST Proxy). You would need other proxies or plugins around that to prevent further access or audit the requests, as with any other web server.
HTTPS certificates would be used to secure traffic to the REST proxy, but seems you're asking about something more specific.
There is no reference to OpenID in the documentation, only LDAP RBAC, as a commercial offering
I want to use continuous queries to KSQL Server using web socket queries.
I heard that the KSQL CLI and Control Center interact with KSQL Server using web socket requests, but I did not find any information on using web sockets in the KSQL REST API reference. How to configure and use this feature?
Websockets is not a supported API, and not documented.
I guess you could run Confluent Control Center yourself and sniff its behaviour, but there'd be no guarantee that the API wouldn't change.
I have used local JIRA Server and i am using Webhook integration concept in JIRA. Here i am giving the Webhook URL is "localhost:8081/webhook" and Event is Create and Update issue. when i am create or update issue means it will be pass to "localhost:8081/webhook". The same address only i gave in mule also but its not working in my MULE. Mule didn't listen the webservice. How can i USE REST API in mule. Which flow is used for receive the JSON format REST API data. It's possible means how can i do this work.
By far the easiest way to do this in Mule now is to use the new Api Kit and write your service description in RAML.
Intro material with docs and examples at: ww.mulesoft.org/documentation/display/current/APIkit
The mule http components may be outboud e.g call your rest api or inbound where the mule server will listen for an event on a http port and url
You say mule is not listening, you should be able to quite simply setup a mule inbound http component flow .. and trigger this flow by pointing your browser at it. I think you will find mule is listening correctly, and you may have mis-configured some of your settings?