Why does DESCRIBE EXTENDED in Kafka KSQL return error ShowColumns not supported? - apache-kafka

I have a simple KTABLE in KSQL called DIMAGE
When I run the following code
{
"ksql": "DESCRIBE EXTENDED DIMAGE ;"
}
I receive the following error
{
"#type": "generic_error",
"error_code": 40000,
"message": "Statement type `io.confluent.ksql.parser.tree.ShowColumns' not supported for this resource",
"stackTrace": []
}
I also receive a similar error message trying to describe a stream. I also receive the same error message if I remove the EXTENDED attribute.

You're using the wrong REST endpoint. If you use query endpoint query you'll get your error:
$ curl -s -X "POST" "http://localhost:8088/query" \
-H "Content-Type: application/vnd.ksql.v1+json; charset=utf-8" \
-d '{
"ksql": "DESCRIBE EXTENDED COMPUTER_T;"
}'
{"#type":"generic_error","error_code":40000,"message":"Statement type `io.confluent.ksql.parser.tree.ShowColumns' not supported for this resource","stackTrace":[]}⏎
If you use the statement endpoint ksql it works fine:
$ curl -s -X "POST" "http://localhost:8088/ksql" \
-H "Content-Type: application/vnd.ksql.v1+json; charset=utf-8" \
-d '{
"ksql": "DESCRIBE EXTENDED COMPUTER_T;"
}'|jq '.'
[
{
"#type": "sourceDescription",
"statementText": "DESCRIBE EXTENDED COMPUTER_T;",
"sourceDescription": {
"name": "COMPUTER_T",
"readQueries": [
{
"sinks": [
"COMP_WATCH_BY_EMP_ID_T"
],
"id": "CTAS_COMP_WATCH_BY_EMP_ID_T_0",
[...]
I've logged #2362 so that we can improve the UX of this.

Related

Setting mirror maker 2 using kafka connect rest api put method not allowed

I am trying to do the setup for mirror maker 2 using my current connect cluster.
Based on this documentation it can be done via connect rest api.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-RunningMirrorMakerinaConnectcluster
I followed the sample sending this PUT request :
PUT /connectors/us-west-source/config HTTP/1.1
{
"name": "us-west-source",
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"source.cluster.alias": "us-west",
"target.cluster.alias": "us-east",
"source.cluster.bootstrap.servers": "us-west-host1:9091",
"topics": ".*"
}
but i am getting a method not allowed response error response.
{
"error_code": 405,
"message": "HTTP 405 Method Not Allowed"
}
The api looks ok if I do a simple GET from the / , returning the version
{
"version": "2.1.0-cp1",
"commit": "bda8715f42a1a3db",
"kafka_cluster_id": "VBo-j1OAQZSN8tO4lMJ0Gg"
}
the PUT method doesnt work, using POST works as the api's documentation shows:
https://docs.confluent.io/current/connect/references/restapi.html#get--connectors
Remove the name of the connector from the url as #cricket_007 suggested, and wrap the config with new element like this:
curl --noproxy "*" -XPOST -H 'Content-Type: application/json' -H 'Accept: application/json' http://localhost:8083/connectors -d'{
"name": "dc-west-source",
"config": {
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"source.cluster.alias": "dc-west",
"target.cluster.alias": "dc-east",
"source.cluster.bootstrap.servers": "dc-west-cp-kafka-0.domain:32721,dc-west-cp-kafka-1.domain:32722,dc-west-cp-kafka-2.dc.domain:32723",
"topics": ".*"
}
}
' | jq .

What is the format of the JSON for a Jenkins REST buildWithParameters to override the default parameters values

I am able to build a Jenkins job with its parameters' default values by sending a POST call to
http://jenkins:8080/view/Orion_phase_2/job/test_remote_api_triggerring/buildWithParameters
and I can override the default parameters "product", "suites" and "markers by sending to this URL:
http://jenkins:8080/view/Orion_phase_2/job/test_remote_api_triggerring/buildWithParameters?product=ALL&suites=ALL&markers=ALL
But I saw examples were the parameters can be override by sending a JSON body with new values. I am trying to do that by sending the following json bodies. Neither of them works for me.
{
'product': 'ALL',
'suites': 'ALL',
'markers': 'ALL'
}
and
{
"parameter": [
{
"name": "product",
"value": "ALL"
},
{
"name": "suites",
"value": "ALL"
},
{
"name": "markers",
"value": "ALL"
}
]
}
What JSON to send if I want to override the values of parameters "product", "suites" & "markers"?
I'll leave the original question as is and elaborate here on the various API calls to trigger parameterized builds. These are the calls options that I used.
Additional documentation: https://wiki.jenkins.io/display/JENKINS/Remote+access+API
The job contains 3 parameters named: product, suites, markers
Send the parameters as URL query parameters to /buildWithParameters:
http://jenkins:8080/view/Orion_phase_2/job/test_remote_api_triggerring/buildWithParameters?product=ALL&suites=ALL&markers=ALL
Send the parameters as JSON data\payload to /build:
http://jenkins:8080/view/Orion_phase_2/job/test_remote_api_triggerring/build
The JSON data\payload is not sent as the call's json_body (which is what confused me), but rater in the data payload as:
json:'{
"parameter": [
{"name":"product", "value":"123"},
{"name":"suites", "value":"high"},
{"name":"markers", "value":"Hello"}
]
}'
And here are the CURL commands for each of the above calls:
curl -X POST -H "Jenkins-Crumb:2e11fc9...0ed4883a14a" http://jenkins:8080/view/Orion_phase_2/job/test_remote_api_triggerring/build --user "raameeil:228366f31...f655eb82058ad12d" --form json='{"parameter": [{"name":"product", "value":"123"}, {"name":"suites", "value":"high"}, {"name":"markers", "value":"Hello"}]}'
curl -X POST \
'http://jenkins:8080/view/Orion_phase_2/job/test_remote_api_triggerring/buildWithParameters?product=234&suites=333&markers=555' \
-H 'authorization: Basic c2hsb21pb...ODRlNjU1ZWI4MjAyOGFkMTJk' \
-H 'cache-control: no-cache' \
-H 'jenkins-crumb: 0bed4c7...9031c735a' \
-H 'postman-token: 0fb2ef51-...-...-...-6430e9263c3b'
What to send to Python's requests
In order to send the above calls in Python you will need to pass:
headers = jenkins-crumb
auth = tuple of your (user_name, user_auth_token)
data = dictionary type { 'json' : json string of {"parameter":[....]} }
curl -v POST http://user:token#host:port/job/my-job/build --data-urlencode json='{"parameter": [{"name":"xx", "value":"xxx"}]}
or use Python request:
import requests
import json
url = " http://user:token#host:port/job/my-job/build "
pyload = {"parameter": [
{"name":"xx", "value":"xxx"},
]}
data = {'json': json.dumps(pyload)}
rep = requests.post(url, data)

Parse: Creating a New Class Programmatically

Is it possible to create a new Class programmatically (i.e. not from the dashboard) via any of the API's or the Parse CLI?
The REST API appears to have functionality to fetch, modify and delete individual Schemas (classes) but not to add them. (https://parse.com/docs/rest/guide#schemas).
Hoping for something like the following:
curl -X ADD \
-H "X-Parse-Application-Id: XXXXXX" \
-H "X-Parse-Master-Key: XXXXXXXX" \
-H "Content-Type: application/json" \
https://api.parse.com/1/schemas/City
You seem to have skipped the part which deals with adding schema in the documentation. To create a new class, according to documentation, You use following method in cURL:
curl -X POST \
-H "X-Parse-Application-Id: Your APP Id" \
-H "X-Parse-Master-Key: Your master key" \
-H "Content-Type: application/json" \
-d '
{
"className": "Your Class name goes here",
"fields": {
"Your field name here": {
"type": "Your field's data type e.g. String, Int etc. Add multiple fields if you want"
}
}
}' \
https://api.parse.com/1/schemas/[Your class name]
Or in Python:
import json,httplib
connection = httplib.HTTPSConnection('api.parse.com', 443)
connection.connect()
connection.request('POST', '/1/schemas/Game', json.dumps({
"className":"[Your class name]","fields":{"Your field name":{"type":"your field's data type"} }
}), {
"X-Parse-Application-Id": "7Lo3U5Ei75dragCphTineRMoCfwD7UJjd1apkPKX",
"X-Parse-Master-Key": "ssOXw9z1ni1unx8tW5iuaHCmhIObOn4nSW9GHj5W",
"Content-Type": "application/json"
})
result = json.loads(connection.getresponse().read())
print result

Index parameterization in Cypher REST query

I have this query which works well but without parametrization in index. emp is an index, and NUM_OFC_CA is an emp number (key in emp index), to simplify i want to return NME_CA.
curl -X POST http://xyzhost:7474/db/data/ext/CypherPlugin/graphdb/execute_query -H "Content-Type: application/json" --data-binary '{
"query": "START ca=node:emp(\"NUM_OFC_CA: 997015\") RETURN distinct ca.NME_CA as `CA Name`",
"params": {
}
}'
How can i parametrize above REST query, i have tried something like this:
curl -X POST http://xyzhost:7474/db/data/ext/CypherPlugin/graphdb/execute_query -H "Content-Type: application/json" --data-binary '{
"query": "START ca=node:emp(\"NUM_OFC_CA: {num_ofc_ca}\") RETURN distinct ca.NME_CA as `CAName`",
"params": {
"num_ofc_ca": "997015"
}
}'
I am getting this error:
{
"message" : "org.apache.lucene.queryParser.ParseException: Cannot parse 'NUM_OFC_CA: {num_ofc_ca}': Encountered \" \"}\" \"} \"\" at line 1, column 23.\nWas expecting one of:\n \"TO\" ...\n <RANGEEX_QUOTED> ...\n <RANGEEX_GOOP> ...\n ",
"exception" : "BadInputException",
"stacktrace" : [ "org.neo4j.server.plugin.cypher.CypherPlugin.executeScript(CypherPlugin.java:61)", "java.lang.reflect.Method.invoke(Method.java:597)", "org.neo4j.server.plugins.PluginMethod.invoke(PluginMethod.java:57)", "org.neo4j.server.plugins.PluginManager.invoke(PluginManager.java:168)", "org.neo4j.server.rest.web.ExtensionService.invokeGraphDatabaseExtension(ExtensionService.java:300)", "org.neo4j.server.rest.web.ExtensionService.invokeGraphDatabaseExtension(ExtensionService.java:122)", "java.lang.reflect.Method.invoke(Method.java:597)" ]
}
Need help to resolve this issue.
Thanks!
If you need a plain index query, the syntax is as follows:
curl -X POST http://<host>:7474/db/data/ext/CypherPlugin/graphdb/execute_query -H "Content-Type: application/json" --databinary '{
"query": ca=node:emp(NUM_OFC_CA = {num_ofc_ca}) RETURN distinct ca.NME_CA as `CAName`",
"params": {
"num_ofc_ca": "997015"
}
}'
For a general lucene index query, you need to parametrize the full query:
curl -X POST http://<host>:7474/db/data/ext/CypherPlugin/graphdb/execute_query -H "Content-Type: application/json" --databinary '{
"query": ca=node:emp({num_ofc_ca_query}) RETURN distinct ca.NME_CA as `CAName`",
"params": {
"num_ofc_ca_query": "NUM_OFC_CA:997015"
}
}'

Parameterize collection: IN Operator WHERE clause Cypher REST

How to specify parameters in something like this - WHERE a.name IN ["Peter", "Tobias"]. I am trying to pass the collection after IN operator as a parameter in Cypher. I am using Cypher through REST API.
This is my example:
curl -X POST http://localhost:7474/db/data/ext/CypherPlugin/graphdb/execute_query -H "Content-Type: applicatio/json" --data-binary '{
"query": "start ca=node:ca({search_ca_query}) MATCH ca_club-[:has]-ca WHERE (ca_club.CA_CLUB IN {CA_CLUB}) RETURN distinct ca.NUM_OFC_CA, ca.NME_CA, ca_club.CA_CLUB",
"params": {
"search_ca_query": "NUM_OFC_CA:(\"000333\", \"111033\", \"222197\")",
"CA_CLUB": "[\"Driad\", \"No-Club\"]"
}
}'
I have also tried swapping square brackets in query, but even that didn't worked. (i.e. i am not getting any error but getting an empty list - "data" : [ ].
Any suggestions on how to do this?
Your in parameter needs to be a list:
curl -X POST http://localhost:7474/db/data/ext/CypherPlugin/graphdb/execute_query -H "Content-Type: applicatio/json" --data-binary '{
"query": "start ca=node:ca({search_ca_query}) MATCH ca_club-[:has]-ca WHERE (ca_club.CA_CLUB IN {CA_CLUB}) RETURN distinct ca.NUM_OFC_CA, ca.NME_CA, ca_club.CA_CLUB",
"params": {
"search_ca_query": "NUM_OFC_CA:(\"000333\", \"111033\", \"222197\")",
"CA_CLUB": ["Driad", "No-Club"]
}
}'