I am trying to use Envoy proxy to proxy all the hadoop gcs connector requests to google storage/auth servers. As per the logs I can see GCS connector uses https. So when I use Envoy proxy it doesn't work as there is some issue in CONECT call handshake. Is there any way to disable the client side SSL? Thanks.
Related
I am trying to use Envoy proxy to route requests to GCS. I am able to route the actual storage requests using through proxy. But when I set this fs.gs.proxy.address parameter in GCS connector settings, OAuth token generation requests fail.
Is there any way to disable OAuth requests and let them go directly to the server?
GCS connector has a global proxy settings for all requests (storage and OAuth), so you need to configure your proxy server to also pass through OAuth request.
It is not possible to completely disable OAuth in GCS Connector.
I use Confluent Kafka REST Proxy to send messages to Apache Kafka.
I set up basic authentication on the REST Proxy and whenever I submit a HTTP request to the proxy, I get the 403 HTTP Error !role.
The proxy requires Zookeeper, Kafka and Schema Registry to be running. I didn't configure any security on these services.
Without authentication, the proxy works and delivers messages to Kafka successfully.
How to I troubleshoot this problem? I spent multiple hours on that problem and I still can't fix it.
Check following:
Firewall allow the service or port
Is there any antivirus block the service or port
Rights given on kafka, confluent folder & respective log directory to kafka user.
Can we use Two Way SSL feature using AWS API Gateway ? We want to use API Gateway as proxy for kinesis in our real-time streaming application.
Below is my requirement
The client make request to apigateway and apigateway needs to put the data in kinesis streams.
The only way to authenticate the clients is using two way SSL. our clients doesnt support other options.
Currently on-premise F5 loadbalancer does this work for us and we have tomcats running behind F5 placing data into kinesis.
Will i be able to achieve the same using API Gateway ? looks like even aws ELB seems to be not supporting this option.
I have taken a look at below link but this to authenticate API Gateway at server not apigateway authenticating the client.
https://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started-client-side-ssl-authentication.html
Regards
Kalyan
API Gateway can authenticate itself to your back-end with TLS, as you have found, but it does not work in the opposite direction -- it does not support authenticating clients with TLS.
The REST API for Kafka Connect is not secured and authenticated.
Since its not authenticated, the configuration for a connector or Tasks are easily accessible by anyone. Since these configurations may contain about how to access the Source System [in case of SourceConnector] and destination system [in case of SinkConnector], Is there a standard way to restrict access to these APIs?
In Kafka 2.1.0, there is possibility to configure http basic authentication for REST interface of Kafka Connect without writing any custom code.
This became real due to implementation of REST extensions mechanism (see KIP-285).
Shortly, configuration procedure as follows:
Add extension class to worker configuration file:
rest.extension.classes = org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension
Create JAAS config file (i.e. connect_jaas.conf) for application name 'KafkaConnect':
KafkaConnect {
org.apache.kafka.connect.rest.basic.auth.extension.PropertyFileLoginModule required
file="/your/path/rest-credentials.properties";
};
Create rest-credentials.properties file in above-mentioned directory:
user=password
Finally, inform java about you JAAS config file, for example, by adding command-line property to java:
-Djava.security.auth.login.config=/your/path/connect_jaas.conf
After restarting Kafka Connect, you will be unable to use REST API without basic authentication.
Please keep in mind that used classes are rather examples than production-ready features.
Links:
Connect configuratin
BasicAuthSecurityRestExtension
JaasBasicAuthFilter
PropertyFileLoginModule
This is a known area in need of improvement in the future but for now you should use a firewall on the Kafka Connect machines and either an API Management tool (Apigee, etc) or a Reverse proxy (haproxy, nginx, etc.) to ensure that HTTPS is terminated at an endpoint that you can configure access control rules on and then have the firewall only accept connections from the secure proxy. With some products the firewall, access control, and SSL/TLS termination functions can be all done in a fewer number of products.
As of Kafka 1.1.0, you can set up SSL and SSL client authentication for the Kafka Connect REST API. See KIP-208 for the details.
Now you are able to enable certificate based authentication for client access to the REST API of Kafka Connect.
An example here https://github.com/sudar-path/kc-rest-mtls
Reading the documentation from http://kubernetes.io/docs/admin/kube-proxy/ it doesn't look like I can connect to the apiserver via https.
The same goes for scheduler and controller-manager, but those two maybe are supposed to run on the same machine as the apiserver and can connect via localhost.
But what about the proxy? This is supposed to connect remotely to the apiserver. Is it possible to configure authentication and/or authorization from the proxy to the apiserver?
This is embarrassing. Looks like all daemons support a "--kubeconfig" flag that gives access to public and private key for authorization.