I need to ingest data into my application. There is a Kafka stream already build for data ingestion. One of the client's requirement needs an API interface for accepting the data. So I need an API management solution which provides API gateway functionality and is able to produce message on a Kafka topic.
I have analyzed Kong + Kong-upstream plugin. But looking for any other similar solution
You can check WSO2 Api Manager.
check this article for wso2 kafka integration. https://ei.docs.wso2.com/en/7.2.0/micro-integrator/references/connectors/kafka-connector/kafka-connector-producer-example/
Also you can write your custom mediator for WSO2.
Another Solution;
If you need different solution else that api gateway. You can check Apache Nifi. It's application that you create your flow. You can do anything what you want.
Apache Nifi's Documentation
For your problem you need to check this Apache Nifi's processors;
-PublishKafka + ConsumeKafka
-ListenHTTP
Edit after #OneCricketeer's comment:
Kafka Solution Without any Integration
Kafka REST Proxy
Related
I'm currently trying to download data from an infinte streaming API endpoint (without closure) using the InvokeHTTP processor. I'm using NiFi 1.20. I'm able to connect to the API, but I'm not able to download any data. However when I'm connecting to other endpoints in the same API with a http-response that ends the connection, data streams through the processor.
Am I missing some config parameters or do I have to use another approach?
I,ve tested most of the settings in the processor. In addition I have verified that the Oauth2 token is valid by downloading from the streaming endpoint using Curl.
PS! I'm trying to dowload data from the Barentswatch API
I want to use ready made kafka connector for fetching the data from the REST API. I found kafka-connect-http connector on the confluent hub but this connector does not support pre-authentication of the API.
I raised this as an issue in the (https://github.com/castorm/kafka-connect-http) and got the response that unfortunately this feature is not supported in the existing code of the connector. So if you have the implementation of the API without authentication then this is the readymade solution for you else you can go for streams etc.
Although the author had agreed that he will look into this feature in the coming future.
In our project, we need to work on multiple services. so started implementing Spring cloud contract testing between two services. (internal services ex: BookService, Employee Service)
Our Requriment:
How Can I implement Spring cloud contact for Third-party service( ex: AccessService) where we can access only API's but no control on Dev Code Base.
The base idea of contracts is that you can ensure that the producer API matches the requirements codified in the contract files. If that is not possible (you don't own the producer code base and you cannot do contract PRs to producer codebase), this approach will not be helpful for you. So you can use contracts for the services in which you can verify producer code against them and use a different approach for external services.
I am using a JBoss based vault to secure sensitive data such as the database credentials.
I use a Java based HTTP REST client to create distributed Kafka connectors but ended up with a security concern such that a request for the connector's "config" exposes the sensitive credentials in the response.
I referred this official documentation but could not get much help in the context of JBoss vault.
Any pointers or references that directly addresses this specific problem is very much appreciated.
Any references to alternate open source (and free to use) Vault based solutions would also be of great help.
You'd have to write code that implements the ConfigProvider interface of the Connect API, mentioned there.
You can browse Kafka source code on Github to see the existing File one, but that KIP (which references Hashicorp Vault) and the source files are the only such documentation for now.
Connect doesn't use JBoss, either, so you'd have to find a way around that
As I want to set up a BAM solution, I'm wondering if there's any Open Source BAM solution that accepts REST requests as events and provide a REST interface to query for events ?
I found a BAM Open Source solution created by Jos Dirksen author of the SOA Governance in Action but I couldn't get the maven dependencies from http://mvn.riptano.com/content/repositories/public/ repository (ref: mvn.riptano.com not authorized connection, connection require authentication).
In [1] you can find how to publish data to WSO2 BAM 2.4.0. There, first you should create a stream (which is a predefined set of fields to be published) using REST API. Then you can send data to BAM via REST API. All the data will be saved in Cassandra where you can analyze data in it using a Hadoop cluster and summarize data. In most cases summarized data is stored in RDB where these data can be visualizes in a dashboard or generate a report.
[1] http://docs.wso2.org/display/BAM240/Sending+Events+through+the+REST+API