Is this more recommended than having the Kafka producer events done directly by whoever needs to emit it?
I assume by "API", you mean HTTP.
Depends on how easily you can configure Kafka or HTTP Authz & Authn. But if anyone exposes your HTTP endpoints / credentials, you'll end up with a bunch of questionable data in the topic.
With an HTTP interface, you can at least consolidate input validation. But you could also do the same with a TCP proxy, or gRPC endpoint. So, it solves the case where no one should be able to send random binary nonsense into your topics with a direct producer api call, if your producer credentials were to get leaked.
If you need simple HTTP passthough, that can be provided by Confluent REST Proxy, for example, but you still should setup an API gateway + auth / firewall to restrict access to it.
Related
I'm using Kafka connect to pull data from different places: mysql, mongodb, etc. And send to elastic search.
I would like to pull data where the origin is a webhook. So, can I configure some Kafka URL to send post http requests to it? What is a good practice to send to my Kafka http post requests?
Confluent maintains a source-available Kafka REST Proxy (be sure to read the Confluent Community License of the project). This would allow you to "send POST requests" to something that forwards this data to Kafka. Outside of this, write a simple HTTP endpoint on your own that does the same.
Personally, I have used Nifi's ListenHTTP handler to accept webhooks, then parse, route, filter, etc into a ProduceKafka request
Otherwise "pulling data" isn't a web hook, and there are a handful of "Kafka Connect http" source projects on Github.
I use Confluent Kafka REST Proxy to send messages to Apache Kafka.
I set up basic authentication on the REST Proxy and whenever I submit a HTTP request to the proxy, I get the 403 HTTP Error !role.
The proxy requires Zookeeper, Kafka and Schema Registry to be running. I didn't configure any security on these services.
Without authentication, the proxy works and delivers messages to Kafka successfully.
How to I troubleshoot this problem? I spent multiple hours on that problem and I still can't fix it.
Check following:
Firewall allow the service or port
Is there any antivirus block the service or port
Rights given on kafka, confluent folder & respective log directory to kafka user.
I'm starting out with Kafka.
I see that I'm able to pass headers when producing messages.
Traditionally one would have a web client (single page app) where to user logs in via some remote oidc idp and receives a token. That token is then sent via Authentication: Bearer token-here header to some RESTful backend where the token is checked for validity and the payload is processed, saved to database or other and something is returned or not.
Now there's Apache Kafka. It has a REST proxy. I can pass headers to the REST proxy and produce messages, or consume them, but I'm interested in the "secure my RESTful JSON API" part.
Currently, without Kafka, I have either a oidc proxy (using keycloak, that's keycloak-gatekeeper) that does the filtering of which request makes it to the backend, or I have a oidc client that does token validation as some middleware function inside the backend. In any case invalid requests doesn't get "logged" as they would in Kafka, I assume.
Where does oidc token validation and request filtering fit in the Kafka/Confluent ecosystem?
Assume we have a SPA that talks to the Confluent REST Proxy. Some logged in user wants to post messages and some non-logged in user should not be able to.
How does Kafka and/or its tools deal with that scenario?
Kafka commonly uses SASL and other Authorization plugins to prevent access.
Certificates would be distributed amongst clients (here, that is the REST Proxy). You would need other proxies or plugins around that to prevent further access or audit the requests, as with any other web server.
HTTPS certificates would be used to secure traffic to the REST proxy, but seems you're asking about something more specific.
There is no reference to OpenID in the documentation, only LDAP RBAC, as a commercial offering
There is an external web-service (REST based) which provides logistics service. In order to consume it from our application (which is java stack), I created a client library (using jersey and jackson, which is a self-contained jar file, and gives simple getters and setters to work with). All that is needed to consume this REST service is this jar file (which is being used as re-usable asset).
However, one of our clients asked, "since we already have ESB (a middleware from IBM) for our existing integrations, why can we not have this integration also through ESB ?"
Is it possible to consume the REST based service using a middleware such as ESB(or MQ) ? If so, how ?
Will it be wise to do it ?
What arguments can I posit to my client to tell them this may not work ?
Thanks.
Is it possible to consume the REST based service using a middleware such as ESB(or MQ) ? If so, how ? -
Yes. Every ESB has its syntax/APIs. Should not be difficult to understand if you are already working with REST/http requests. Note, MQ is a message broker that will allow asynchronous communication. For example, the application that wants to consume the REST service can send a message to the broker and not wait for a response from the external service. The message broker in turn can talk REST to the external service. The message broker might expose a REST interface to consume messages. But it all depends on the use case. You may want to call REST service synchronously.
Most ESBs support calling external web services REST/SOAP. So an ESB does not necessarily mean MQ.
Will it be wise to do it ?
It depends. Many organizations set up ESB/Gateway as a policy and to route/filter/govern all external routings. Also to manage load, HA etc.
What arguments can I posit to my client to tell them this may not work ?
It will most certainly work. You should assess if using an ESB is an overhead based on your requirements.
This one may be simple, but can't find it. I understand that flows with SOAP nodes (SOAP Input for example) are exposed as web services in Message Broker. How can I consume them with soapUI?
As you should have a WSDL file specifying your web services in your soap nodes, you should take that WSDL, import it into soapUI, then specify the endpoint to your WMB machine (I think default port is 7080 or something like that). It should be pretty straight forward.