Is it possible to setup federation using pure AMQP protocol/headers? Right now I'm using REST API to setup federation and un-federate an autogenerated queue which is a bit cumbersome.
Update:
I'm using RabbitMQ 3.7.8 which is AMQP version 0.9 of the protocol
Related
At present we have a lot of ASP.net WebAPI service applications hosted on premises. We are planning to move these to Azure AKS. We've identified a lot of common code across these applications which is mostly implemented as ASP.Net reusable middleware components so that the logic is not duplicated in code.
In a K8s environment it makes sense to offload this common functionality to one or more proxy applications which intercepts the requests being forwarded from the ingress to the services (assuming this is the correct approach). Some of the request inspection / manipulation logic is based on the service host and path to be defined in the ingress and even on the headers in the incoming requests.
For e.g. I considered using OAuth2_proxy but found that even though authentication is quite easy to implement, Azure AD group based authorization is impossible to do out of the box with that. So what's the idiomatic way one goes about setting up such a custom proxy application? (I'm familiar with using libraries such as ProxyKit middleware in ASP.Net to develop http proxies.)
One approach that comes to mind is to deploy such proxies as sidecar containers in each service application pod but that would mean there'd be unnecessary resource usage by all such duplicate container instances in each pod. I don't see the benefit over the use of middleware components as mentioned previously. :(
The ideal setup would be ingress --> custom proxy 1 --> custom proxy 2 --> custom proxy n --> service where custom proxies would be separately deployable and scalable.
So after a lot of reading and googling I found that the solution was to use API Gateways that are available as libraries (preferrably based on .Net):
Ocelot placed behind the nginx ingress fits the bill perfectly
Ocelot is a .NET API Gateway. This project is aimed at people using .NET running a micro services / service oriented architecture that need a unified point of entry into their system. However it will work with anything that speaks HTTP and run on any platform that ASP.NET Core supports.
Ocelot is currently used by Microsoft and Tencent.
The custom middleware and header/query/claims transformation solves my problem. Here are some worthy links
Microsoft Docs: Implement API Gateways with Ocelot
Ocelot on Github
Ocelot Documentation
Features
A quick list of Ocelot's capabilities for more information see the documentation.
Routing
Request Aggregation
Service Discovery with Consul & Eureka
Service Fabric
Kubernetes
WebSockets
Authentication
Authorisation
Rate Limiting
Caching
Retry policies / QoS
Load Balancing
Logging / Tracing / Correlation
Headers / Query String / Claims Transformation
Custom Middleware / Delegating Handlers
Configuration / Administration REST API
Platform / Cloud Agnostic
I am looking for an open source API Gateway that can do protocol transformations (Spring Cloud offers Zuul but it accepts only REST connections and does not do protocol transformations). Any idea (or clever solution) given that most of the following protocols must be supported?
CoAP
AMQP
MQTT (must be supported)
STOMP
REST (must be supported)
Sine the next hop after the API Gateway, I was thinking Spring Cloud Stream with custom inbound/outbound adapters. There is also the Spring Cloud Data Flow but I think that is it a different beast.
PS: Oracle API Gateway seems to support multiple protocols but I am looking for an open source solution.
The REST API for Kafka Connect is not secured and authenticated.
Since its not authenticated, the configuration for a connector or Tasks are easily accessible by anyone. Since these configurations may contain about how to access the Source System [in case of SourceConnector] and destination system [in case of SinkConnector], Is there a standard way to restrict access to these APIs?
In Kafka 2.1.0, there is possibility to configure http basic authentication for REST interface of Kafka Connect without writing any custom code.
This became real due to implementation of REST extensions mechanism (see KIP-285).
Shortly, configuration procedure as follows:
Add extension class to worker configuration file:
rest.extension.classes = org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension
Create JAAS config file (i.e. connect_jaas.conf) for application name 'KafkaConnect':
KafkaConnect {
org.apache.kafka.connect.rest.basic.auth.extension.PropertyFileLoginModule required
file="/your/path/rest-credentials.properties";
};
Create rest-credentials.properties file in above-mentioned directory:
user=password
Finally, inform java about you JAAS config file, for example, by adding command-line property to java:
-Djava.security.auth.login.config=/your/path/connect_jaas.conf
After restarting Kafka Connect, you will be unable to use REST API without basic authentication.
Please keep in mind that used classes are rather examples than production-ready features.
Links:
Connect configuratin
BasicAuthSecurityRestExtension
JaasBasicAuthFilter
PropertyFileLoginModule
This is a known area in need of improvement in the future but for now you should use a firewall on the Kafka Connect machines and either an API Management tool (Apigee, etc) or a Reverse proxy (haproxy, nginx, etc.) to ensure that HTTPS is terminated at an endpoint that you can configure access control rules on and then have the firewall only accept connections from the secure proxy. With some products the firewall, access control, and SSL/TLS termination functions can be all done in a fewer number of products.
As of Kafka 1.1.0, you can set up SSL and SSL client authentication for the Kafka Connect REST API. See KIP-208 for the details.
Now you are able to enable certificate based authentication for client access to the REST API of Kafka Connect.
An example here https://github.com/sudar-path/kc-rest-mtls
Almost every spring-cloud guide suggests Ribbon with Feign for RPC.
I wonder why Http Rest takes precedence over binary transport protocol for inner micro-services communication behind the API gateway?
What are the binary alternatives to http that support async invocation and all great features that Netflix OSS provides ? (auto-discovery, load-balancing, circuit breaker, retry policy etc.)
Thanks
There is nothing preventing you from using binary RPC like thrift or protobuffs or msgpack with spring cloud. You can use LoadBalancerClient.choose(<serviceId>) to get a host and port you could supply to any network client. Our integrations were the simple rest clients.
Here is a guide that integrates Spring MVC with google protocol buffers.
I'd like to connect to the JMX server on a WebSphere application server instance using a SOAP over HTTP connector but I don't know where to find a compatible SOAP protocol provider.
I get a MalformedURLException when using a JMXServiceURL starting service:jmx:soap:... that says Unsupported protocol: soap. What jar(s)/protocol provider string do I need to include?
Update: I am playing with a standalone Java client.
Unfortunately, IBM only provides a proprietary API to connect with SOAP over HTTP. They don't provide an implementation of the standard JMX API for that protocol. I faced the same issue and I developed such a connector (which is basically a thin JMX compliant wrapper around IBM's proprietary API), which is now available as Open Source. You can find more information here:
https://github.com/kszbcss/xm4was/wiki/JmxClientConnector
The following document describes how to set up a standard JMX tool (VisualVM) to use that connector:
https://github.com/kszbcss/xm4was/wiki/VisualVMHowTo
As you can see from these documents, after adding the relevant JARs to the class path and setting the necessary system properties, you would simply connect using wssoap as protocol.
Check that in the path Application servers > <server_name> > Administration services > JMX connectors the relative SOAPConnector exists and is Enabled.
Refer at Infocenter for more information.