I use Confluent Kafka REST Proxy to send messages to Apache Kafka.
I set up basic authentication on the REST Proxy and whenever I submit a HTTP request to the proxy, I get the 403 HTTP Error !role.
The proxy requires Zookeeper, Kafka and Schema Registry to be running. I didn't configure any security on these services.
Without authentication, the proxy works and delivers messages to Kafka successfully.
How to I troubleshoot this problem? I spent multiple hours on that problem and I still can't fix it.
Check following:
Firewall allow the service or port
Is there any antivirus block the service or port
Rights given on kafka, confluent folder & respective log directory to kafka user.
Related
Is this more recommended than having the Kafka producer events done directly by whoever needs to emit it?
I assume by "API", you mean HTTP.
Depends on how easily you can configure Kafka or HTTP Authz & Authn. But if anyone exposes your HTTP endpoints / credentials, you'll end up with a bunch of questionable data in the topic.
With an HTTP interface, you can at least consolidate input validation. But you could also do the same with a TCP proxy, or gRPC endpoint. So, it solves the case where no one should be able to send random binary nonsense into your topics with a direct producer api call, if your producer credentials were to get leaked.
If you need simple HTTP passthough, that can be provided by Confluent REST Proxy, for example, but you still should setup an API gateway + auth / firewall to restrict access to it.
I'm using Kafka connect to pull data from different places: mysql, mongodb, etc. And send to elastic search.
I would like to pull data where the origin is a webhook. So, can I configure some Kafka URL to send post http requests to it? What is a good practice to send to my Kafka http post requests?
Confluent maintains a source-available Kafka REST Proxy (be sure to read the Confluent Community License of the project). This would allow you to "send POST requests" to something that forwards this data to Kafka. Outside of this, write a simple HTTP endpoint on your own that does the same.
Personally, I have used Nifi's ListenHTTP handler to accept webhooks, then parse, route, filter, etc into a ProduceKafka request
Otherwise "pulling data" isn't a web hook, and there are a handful of "Kafka Connect http" source projects on Github.
I am trying to use Envoy proxy to proxy all the hadoop gcs connector requests to google storage/auth servers. As per the logs I can see GCS connector uses https. So when I use Envoy proxy it doesn't work as there is some issue in CONECT call handshake. Is there any way to disable the client side SSL? Thanks.
I've set-up a divolte clickstream script on a page which gets consumed by kafka.
The setup works when I host divolte scripts using http but when I source scripts via https and set up a reverseproxy in .htaccess to send requests to a http URL, kafka doesn't consume any content.
Any ideas on how to fix this?
The REST API for Kafka Connect is not secured and authenticated.
Since its not authenticated, the configuration for a connector or Tasks are easily accessible by anyone. Since these configurations may contain about how to access the Source System [in case of SourceConnector] and destination system [in case of SinkConnector], Is there a standard way to restrict access to these APIs?
In Kafka 2.1.0, there is possibility to configure http basic authentication for REST interface of Kafka Connect without writing any custom code.
This became real due to implementation of REST extensions mechanism (see KIP-285).
Shortly, configuration procedure as follows:
Add extension class to worker configuration file:
rest.extension.classes = org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension
Create JAAS config file (i.e. connect_jaas.conf) for application name 'KafkaConnect':
KafkaConnect {
org.apache.kafka.connect.rest.basic.auth.extension.PropertyFileLoginModule required
file="/your/path/rest-credentials.properties";
};
Create rest-credentials.properties file in above-mentioned directory:
user=password
Finally, inform java about you JAAS config file, for example, by adding command-line property to java:
-Djava.security.auth.login.config=/your/path/connect_jaas.conf
After restarting Kafka Connect, you will be unable to use REST API without basic authentication.
Please keep in mind that used classes are rather examples than production-ready features.
Links:
Connect configuratin
BasicAuthSecurityRestExtension
JaasBasicAuthFilter
PropertyFileLoginModule
This is a known area in need of improvement in the future but for now you should use a firewall on the Kafka Connect machines and either an API Management tool (Apigee, etc) or a Reverse proxy (haproxy, nginx, etc.) to ensure that HTTPS is terminated at an endpoint that you can configure access control rules on and then have the firewall only accept connections from the secure proxy. With some products the firewall, access control, and SSL/TLS termination functions can be all done in a fewer number of products.
As of Kafka 1.1.0, you can set up SSL and SSL client authentication for the Kafka Connect REST API. See KIP-208 for the details.
Now you are able to enable certificate based authentication for client access to the REST API of Kafka Connect.
An example here https://github.com/sudar-path/kc-rest-mtls