Im trying to install Kafka REST Proxy from the following URL however it has complete package - I just want to install REST client alone as other components I have installed already
https://docs.confluent.io/current/installation/installing_cp.html#zip-and-tar-archives
Is there any place where I can find instructions?
Check out the individual packages available - you should be able to install just confluent-kafka-rest.
Or, just install full Confluent Platform and only start the REST Proxy, same net effect :)
Related
I have Confluent 5.0 on my local machine and trying to reading data from Rest API using Rest API Source Connect which is not part of confluent. till now i have used confluent inbuilt connectors only. Rest API source connect is open source and available on github https://github.com/llofberg/kafka-connect-rest
I have downloaded this connector from github and got stuck here.
Can anybody tell me the process to integrate this connector with confluent or how can i use this to pull the data from Rest API?
Disclaimer: There is no single answer to add an external Kafka Connect plugin; Confluent provides the Kafka Connect Maven plugin, but that doesn't mean people use it or even Maven to package their code.
If it is not on the Confluent Hub, then you'll have to build it by hand.
1) Clone the repo, and build it (install Git and Maven first)
git clone https://github.com/llofberg/kafka-connect-rest && cd kafka-connect-rest
mvn clean package
2) Create a directory for it on all Connect workers, similar to the other Connectors of Confluent Platform
mkdir $CONFLUENT_HOME/share/java/kafka-connect-rest
3) Find each of the shaded JARs (this connector happens to make multiple JARs, I don't know why...)
find . -iname "*shaded.jar" -type f
./kafka-connect-transform-from-json/kafka-connect-transform-from-json-plugin/target/kafka-connect-transform-from-json-plugin-1.0-SNAPSHOT-shaded.jar
./kafka-connect-transform-add-headers/target/kafka-connect-transform-add-headers-1.0-SNAPSHOT-shaded.jar
./kafka-connect-transform-velocity-eval/target/kafka-connect-transform-velocity-eval-1.0-SNAPSHOT-shaded.jar
./kafka-connect-rest-plugin/target/kafka-connect-rest-plugin-1.0-SNAPSHOT-shaded.jar
4) Copy each of these files into the $CONFLUENT_HOME/share/java/kafka-connect-rest folder created in step 2 for each Connect worker
5) Make sure your plugin.path of the connect-*.properties file points at the full path to $CONFLUENT_HOME/share/java
At this point, you've done all the steps that are listed in the README to build the thing and setup the plugin path, just not in Docker.
6) Start Connect (Distributed)
7) Hit GET /connector-plugins to verify the thing loaded.
8) Configure and send JSON payload to POST /connectors
I have not used this connector before, so I do not know how to configure it. Maybe see the examples or follow along with #rmoff's blog post before the KSQL stuff
I am getting the error as could not find file while trying to install from main menu help and install new software and adding the appropriate site.
Please check if you are using proxy and if the proxy details are correct. if it fails even with correct proxy. You can try offline install
https://docs.jboss.org/author/display/SWITCHYARD/Installing+Eclipse+Tooling
I want to use the camunda rest api (local), but I realy can't figure out how to set up the environment... .
First I downloaded the tomcat distribution from here (V. 7.2.0) and the prepackaged eclipse with BPMN 2.0 Modeler from here.
I modelled a process, startet the server (start-camunda.bat) and deployed it (copied .war to ...camunda\server\apache-tomcat-7.0.50\webapps).
It went well on local tasklist http://localhost:8080/camunda/app/tasklist/default/#/login and cockpit http://localhost:8080/camunda/app/cockpit/default/).
I also downloaded the engine-rest from "Maven Nexus Server" (Install the REST API web application). Now, if I call the engine (http://localhost:8080/engine-rest/engine), I got the following .json: [{"name":"default"}]
What to do next? I realy don't know (I'm new in camunda...)
Your setup seems to be fine. But please note that the camunda REST API is a backend which you can access with a client. Currently there doesn't exists a camunda Java REST client. So you have to implement it by your self. If you are aware of that a good starting point is the camunda REST documentation. Besides Java there exists a Javascript SDK which you could use to access the REST API.
May I suggest an alternative: use a docker image for Camunda. It's official and works like a charm. You can set it up in 5 mins tops and then use an API client like postman to access Camunda's API via http://localhost:8080/engine-rest.
I had the same problem as yours and found a docker solution much easier than installing Camunda and all related dependencies myself only having to troubleshoot them later. Here's the link to the official docker image if you're interested:
https://hub.docker.com/r/camunda/camunda-bpm-platform
Or enter these 2 commands from terminal and you'll be golden:
docker pull camunda/camunda-bpm-platform:latest
docker run -d --name camunda -p 8080:8080 camunda/camunda-bpm-platform:latest
Open URL with browser: http://localhost:8080/camunda-welcome/index.html
username/password: admin/admin
Is there any other way of building the packages other than from Package manager console in AEM or CQ5 ?
In CQ 5.4 apart from the CRX package Manager console, there used to be another console within CQ, which can be used for the same purpose. But with CQ 5.5, the removed it, and made the CRX package manager the standard one for managing packages.
The package manager does expose an HTTP service interface, which allows managing packages using through Command Line Interface. You can use curl or any other command line clients to achieve this.
The Package Manager Documentation provides the complete details on the various operations that are supported, the request and the response format, and some sample commands.
For e.g, the following is the curl command to create a new package.
curl -u <uid>:<pwd> -X POST
http://localhost:7402/crx/packmgr/service/.json/etc/packages/mycontent.zip?cmd=create
-d packageName=<name> -d groupName=<name>
Apart from the HTTP service mentioned by rakhi4110, you may also use Java API to manage packages. Start with this OSGi service: com.day.jcr.vault.packaging.Packaging.
You may also upload and install packages using maven crx plugin however this is just about upload and install programmatically https://github.com/Cognifide/Maven-CRX-Plugin
I read about using haproxy and keepalived to avoid spof for haproxy. Is it possible to do this in ec2?
Say I have 2 instances. Each with haproxy and keepalived installed. The VIP shall be aws elastic IP.
In theory this should be possible, there are blog posts dotted around with instructions on how to do it. However I have been trying to test this over the past few days but not had any luck with it.
Amazon blocks multicast on EC2 (Classic) so the primary communication method for Keepalived will not work. You need to install the latest version (at time of writing this is 1.2.13) which has unicast support built in. This should allow you to bypass the multicast restrictions that Amazon puts in place. I think the version of the keepalived package is 1.2.7 in the repos (yum install keepalived) which does not have the unicast patch included..
You should be able to use wget to download the latest TAR, unpackage it and build it from source (./configure prefix=/, make, make install). Make sure you have the GCC package and openssl-devel package installed before trying to configure as it will fail with errors otherwise.
If I get it working in the meantime I will come back and put a link to my blog with the exact steps needed :)