Setting mirror maker 2 using kafka connect rest api put method not allowed - apache-kafka

I am trying to do the setup for mirror maker 2 using my current connect cluster.
Based on this documentation it can be done via connect rest api.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-RunningMirrorMakerinaConnectcluster
I followed the sample sending this PUT request :
PUT /connectors/us-west-source/config HTTP/1.1
{
"name": "us-west-source",
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"source.cluster.alias": "us-west",
"target.cluster.alias": "us-east",
"source.cluster.bootstrap.servers": "us-west-host1:9091",
"topics": ".*"
}
but i am getting a method not allowed response error response.
{
"error_code": 405,
"message": "HTTP 405 Method Not Allowed"
}
The api looks ok if I do a simple GET from the / , returning the version
{
"version": "2.1.0-cp1",
"commit": "bda8715f42a1a3db",
"kafka_cluster_id": "VBo-j1OAQZSN8tO4lMJ0Gg"
}

the PUT method doesnt work, using POST works as the api's documentation shows:
https://docs.confluent.io/current/connect/references/restapi.html#get--connectors
Remove the name of the connector from the url as #cricket_007 suggested, and wrap the config with new element like this:
curl --noproxy "*" -XPOST -H 'Content-Type: application/json' -H 'Accept: application/json' http://localhost:8083/connectors -d'{
"name": "dc-west-source",
"config": {
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"source.cluster.alias": "dc-west",
"target.cluster.alias": "dc-east",
"source.cluster.bootstrap.servers": "dc-west-cp-kafka-0.domain:32721,dc-west-cp-kafka-1.domain:32722,dc-west-cp-kafka-2.dc.domain:32723",
"topics": ".*"
}
}
' | jq .

Related

OpenSearch 1.2 - Index pattern programmatically

I am trying to create an index_pattern for dashboard in Opensearch programmatically.
Due to the fact that I didn't found anything related to it on Opensearch documentation, I tried the elastic search saved_objects api:
POST /api/saved_objects/index-pattern/my_index_pattern
{
"attributes":{
"title": "my_index"
}
}
But when I call it I got the following error:
{
"error" : "no handler found for uri [/api/saved_objects/index-pattern/my_index_pattern?pretty=true] and method [POST]"
}
How can I solve it? Are there a different call for Opensearch to create an index_pattern?
curl -k -X POST http://<host>:5601/api/saved_objects/index-pattern/logs-* -H "osd-xsrf:true" -H "content-type:application/json" -d '
{
"attributes": {
"title": "logs-*",
"timeFieldName": "#timestamp"
}
}'
this works for me

Kafka REST API source connector with authentication header

I need to create kafka source connector for REST API with Header authentication like
curl -H "Authorization: Basic " -H "clientID: " "https:< url for source> " .
I am using apache kafka , I used connector class com.github.castorm.kafka.connect.http.HttpSourceConnector
Here is my json file for connector
{
"name": "rest_data6",
"config": {
"key.converter":"org.apache.kafka.connect.json.JsonConverter",
"value.converter":"org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable":"true",
"value.converter.schemas.enable":"true",
"connector.class": "com.github.castorm.kafka.connect.http.HttpSourceConnector",
"tasks.max": "1",
"http.request.headers": "Authorization: Basic <key1>",
"http.request.headers": "clientID: <key>",
"http.request.url": "https:<url for source ?",
"kafka.topic": "mysqltopic2"
}
}
Also I tried with "connector.class": "com.tm.kafka.connect.rest.RestSourceConnector", My joson file as below
"name": "rest_data2",
"config": {
"key.converter":"org.apache.kafka.connect.json.JsonConverter",
"value.converter":"org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable":"true",
"value.converter.schemas.enable":"true",
"connector.class": "com.tm.kafka.connect.rest.RestSourceConnector",
"rest.source.poll.interval.ms": "900",
"rest.source.method": "GET",
"rest.source.url":"URL of source ",
"tasks.max": "1",
"rest.source.headers": "Authorization: Basic <key> , clientId :<key2>",
"rest.source.topic.selector": "com.tm.kafka.connect.rest.selector.SimpleTopicSelector",
"rest.source.destination.topics": "mysql1"
}
}
But no hope . Any idea how to GET REST API data with authentication . My authentication parameter is
Authorization: Basic and Authorization: Basic .
Just for mention both the file are working with REST API without authentication , once I added authentication parameter then wither connector status is failed or It produce ":"Cannot route. Codebase/company is invalid"" message in topic.
Can any one suggest what is way to solve it
I mailed to original developer to Cástor Rodríguez. As per his solution I modified my json
Put header into a single form and it works
{
"name": "rest_data6",
"config": {
"key.converter":"org.apache.kafka.connect.json.JsonConverter",
"value.converter":"org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable":"true",
"value.converter.schemas.enable":"true",
"connector.class": "com.github.castorm.kafka.connect.http.HttpSourceConnector",
"tasks.max": "1",
"http.request.headers": "Authorization: Basic <key1>, clientID: <key>"
"http.request.url": "https:<url for source ?",
"kafka.topic": "mysqltopic2"
}
}

Request body matching with XML declaration in Wiremock-standalone

I have a stub described in the ./mappings/*.json file.
"request": {
"method": "POST",
"url": "/some/thing",
"bodyPatterns" : [ {
"matchesXPath" : {
"expression": "//nodeA/text()",
"contains": "999"
}
} ]
}
Wiremock (ver.2.26.2) is started in standalone mode.
When I call the service like this:
curl -d "<request><nodeA>999</nodeA></request>" -X POST http://localhost:8888/some/thing
I am getting the response from stub as expected. The problem is that the request has to be sent with XML declaration tag , e.g.
curl -d "<?xml version="1.0" encoding="UTF-8"?><request><nodeA>999</nodeA></request>" -X POST http://localhost:8888/some/thing
and in this case the request isn't matched. I tried to find smth in the documentation regarding it, but so far no luck
I found out that the problem was with curl that I used. It was malformed since I used same double quotes in XML declaration. Now I load the request body from file and everything works just fine
curl -H "Content-Type: application/xml" -d "#path_to_request_file" -X POST http://localhost:8888/some/thing

Why does DESCRIBE EXTENDED in Kafka KSQL return error ShowColumns not supported?

I have a simple KTABLE in KSQL called DIMAGE
When I run the following code
{
"ksql": "DESCRIBE EXTENDED DIMAGE ;"
}
I receive the following error
{
"#type": "generic_error",
"error_code": 40000,
"message": "Statement type `io.confluent.ksql.parser.tree.ShowColumns' not supported for this resource",
"stackTrace": []
}
I also receive a similar error message trying to describe a stream. I also receive the same error message if I remove the EXTENDED attribute.
You're using the wrong REST endpoint. If you use query endpoint query you'll get your error:
$ curl -s -X "POST" "http://localhost:8088/query" \
-H "Content-Type: application/vnd.ksql.v1+json; charset=utf-8" \
-d '{
"ksql": "DESCRIBE EXTENDED COMPUTER_T;"
}'
{"#type":"generic_error","error_code":40000,"message":"Statement type `io.confluent.ksql.parser.tree.ShowColumns' not supported for this resource","stackTrace":[]}⏎
If you use the statement endpoint ksql it works fine:
$ curl -s -X "POST" "http://localhost:8088/ksql" \
-H "Content-Type: application/vnd.ksql.v1+json; charset=utf-8" \
-d '{
"ksql": "DESCRIBE EXTENDED COMPUTER_T;"
}'|jq '.'
[
{
"#type": "sourceDescription",
"statementText": "DESCRIBE EXTENDED COMPUTER_T;",
"sourceDescription": {
"name": "COMPUTER_T",
"readQueries": [
{
"sinks": [
"COMP_WATCH_BY_EMP_ID_T"
],
"id": "CTAS_COMP_WATCH_BY_EMP_ID_T_0",
[...]
I've logged #2362 so that we can improve the UX of this.

wso2am API manager 2.1 publisher change-lifecycle issue

I deployed API Manager 2.1.0 and setup the api-import-export-2.1.0 war file described here. After importing my API endpoint by uploading a zip file the status=CREATED.
To actually publish the API I am calling the Publisher's change-lifecycle API but I am getting this exception:
TID: [-1234] [] [2017-07-06 11:11:57,289] ERROR
{org.wso2.carbon.apimgt.rest.api.util.exception.GlobalThrowableMapper}
- An Unknown exception has been captured by global exception mapper.
{org.wso2.carbon.apimgt.rest.api.util.exception.GlobalThrowableMapper}
java.lang.NoSuchMethodError:
org.wso2.carbon.apimgt.api.APIProvider.changeLifeCycleStatus(Lorg/wso2/carbon/apimgt/api/model/APIIdentifier;Ljava/lang/String;)Z
Any ideas on why?
I can get an access token (scope apim:api_view) and call this
:9443/api/am/publisher/v0.10/apis
to list the api's just fine.
I get a different acces_token (for scope: apim:api_publish) and then call
:9443/api/am/publisher/v0.10/apis/change-lifecycle
but get the above Exception. Here's the example:
[root#localhost] ./publish.sh
View APIs (token dc0c1497-6c27-3a10-87d7-b2abc7190da5 scope: apim:api_view)
curl -k -s -H "Authorization: Bearer dc0c1497-6c27-3a10-87d7-b2abc7190da5" https://gw-node:9443/api/am/publisher/v0.10/apis
{
"count": 1,
"next": "",
"previous": "",
"list": [
{
"id": "d214f784-ee16-4067-9588-0898a948bb17",
"name": "Health",
"description": "health check",
"context": "/api",
"version": "v1",
"provider": "admin",
"status": "CREATED"
}
] }
Publish API (token b9a31369-8ea3-3bf2-ba3c-7f2a4883de7d scope: apim:api_publish)
curl -k -H "Authorization: Bearer b9a31369-8ea3-3bf2-ba3c-7f2a4883de7d" -X POST https://gw-node:9443/api/am/publisher/v0.10/apis/change-lifecycle?apiId=d214f784-ee16-4067-9588-0898a948bb17&action=Publish
{
"code":500,
"message":"Internal server error",
"description":"The server encountered an internal error. Please contact administrator.",
"moreInfo":"",
"error":[]
}
Issue resolved. In apim 2.1 the publisher & store API versions changed.
In apim 2.0 I was using:
:9443/api/am/publisher/v0.10/apis
:9443/api/am/store/v0.10/apis
but in apim 2.1 they are:
:9443/api/am/publisher/v0.11/apis
:9443/api/am/store/v0.11/apis