I want to use ready made kafka connector for fetching the data from the REST API. I found kafka-connect-http connector on the confluent hub but this connector does not support pre-authentication of the API.
I raised this as an issue in the (https://github.com/castorm/kafka-connect-http) and got the response that unfortunately this feature is not supported in the existing code of the connector. So if you have the implementation of the API without authentication then this is the readymade solution for you else you can go for streams etc.
Although the author had agreed that he will look into this feature in the coming future.
Related
I need to ingest data into my application. There is a Kafka stream already build for data ingestion. One of the client's requirement needs an API interface for accepting the data. So I need an API management solution which provides API gateway functionality and is able to produce message on a Kafka topic.
I have analyzed Kong + Kong-upstream plugin. But looking for any other similar solution
You can check WSO2 Api Manager.
check this article for wso2 kafka integration. https://ei.docs.wso2.com/en/7.2.0/micro-integrator/references/connectors/kafka-connector/kafka-connector-producer-example/
Also you can write your custom mediator for WSO2.
Another Solution;
If you need different solution else that api gateway. You can check Apache Nifi. It's application that you create your flow. You can do anything what you want.
Apache Nifi's Documentation
For your problem you need to check this Apache Nifi's processors;
-PublishKafka + ConsumeKafka
-ListenHTTP
Edit after #OneCricketeer's comment:
Kafka Solution Without any Integration
Kafka REST Proxy
I am using a JBoss based vault to secure sensitive data such as the database credentials.
I use a Java based HTTP REST client to create distributed Kafka connectors but ended up with a security concern such that a request for the connector's "config" exposes the sensitive credentials in the response.
I referred this official documentation but could not get much help in the context of JBoss vault.
Any pointers or references that directly addresses this specific problem is very much appreciated.
Any references to alternate open source (and free to use) Vault based solutions would also be of great help.
You'd have to write code that implements the ConfigProvider interface of the Connect API, mentioned there.
You can browse Kafka source code on Github to see the existing File one, but that KIP (which references Hashicorp Vault) and the source files are the only such documentation for now.
Connect doesn't use JBoss, either, so you'd have to find a way around that
Wanted to check if ibm-eventstreams that I can deploy on IBM Cloud Private (ICP) 2.1.0.3, supports Kafka SASL authentication and ACLs applied to specific topics?
I was referring to this developerWorks article about Kafka ACLs:
https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/
Wondering whether it is available and supported with ibm-eventstreams.
If it is supported, are there any changes/enhancements to the ACL support that I see in the above doc? Is there any further documentation available?
Sorry, no - the current Tech Preview doesn't include any security or auth. (We're thinking hard about what the best way to do this would be though!)
I am looking to access a local instance of Couchbase Server through its REST API. The HTTP GET requests are sent from Java.
The problem I am currently running into has to do with authorization.
Specifically, I have managed to use Couchbase's Basic Authorization, but only by obtaining the hashed credentials (bG9jYWw6dHdlZXRzOnBBc3Mx in the example linked to above) by monitoring a Couchbase browser session using Chrome's developer tools and inspecting the request headers.
Now, another Couchbase article mentions that Couchbase uses SHA-1 in compliance with SCRAM. However, no mention is made of how to obtain the 'salt' and 'iterations' parameters from Couchbase. Which, I assume I need to go from the credentials to the challenge solution (i.e. hashed string)
So, the question is as follows: how to get from the credentials (user="local:tweets", pass="pAss1") (from the example of the first link) to bG9jYWw6dHdlZXRzOnBBc3Mx?
Thanks in advance,
Thomas
SCRAM SHA-1 support is only for the Data (K/V) service and only through certain Couchbase SDKs which use the memcached binary protocol. The Java SDK does have support for SCRAM SHA-1.
The REST interface you appear to be using is N1QL's API. That does not support SCRAM SHA auth.
If you're looking to give some other application HTTP access, my recommendation would be to write a small Java app with Spring Boot or the like and use the Java SDK from there. Then you have complete control over how auth is done at the REST interface. A colleague wrote one of these just the other day. Note that even in this case, the Java SDK won't be using SCRAM when running N1QL queries, but you can use that as a point of control.
As I want to set up a BAM solution, I'm wondering if there's any Open Source BAM solution that accepts REST requests as events and provide a REST interface to query for events ?
I found a BAM Open Source solution created by Jos Dirksen author of the SOA Governance in Action but I couldn't get the maven dependencies from http://mvn.riptano.com/content/repositories/public/ repository (ref: mvn.riptano.com not authorized connection, connection require authentication).
In [1] you can find how to publish data to WSO2 BAM 2.4.0. There, first you should create a stream (which is a predefined set of fields to be published) using REST API. Then you can send data to BAM via REST API. All the data will be saved in Cassandra where you can analyze data in it using a Hadoop cluster and summarize data. In most cases summarized data is stored in RDB where these data can be visualizes in a dashboard or generate a report.
[1] http://docs.wso2.org/display/BAM240/Sending+Events+through+the+REST+API