CloudSQL - Export data - Using google-api-services-sqladmin - v1beta4 - google-cloud-storage

I'm new to cloudSQL, trying to build a small console application to test the functionalities of CLoudSQL API. (google-api-services-sqladmin - v1beta4)
Can somebody help me to get start with some sample code?
For example, i want to export data from cloudSQL to GCS using - select query?

Here's a sample HTTP request that will export the table mysql.user to a CSV file in Cloud Storage:
POST https://www.googleapis.com/sql/v1beta4/projects/<project>/instances/<instance>/export
content-type: application/json
content-length: <length-of-request-body>
Authorization: Bearer <access-token>
{
"exportContext": {
"csvExportOptions": {
"selectQuery": "SELECT * FROM mysql.user"
},
"uri": "gs://<bucket>/users.csv",
"fileType": "CSV"
}
}
Note that you need to set the values for <project>, <instance>, <access-token>, and <bucket>.
Once you have those parameters, you can easily try this either using the API Explorer, at the bottom where it says "try it".
Or simply by using curl:
CURL command:
$ curl -X POST \
https://www.googleapis.com/sql/v1beta4/projects/<project>/instances/<instance>/export \
-H'content-length: <length-of-request-body>'
-H'content-type: application/json'
-H'Authorization: Bearer <access-token>'
-d'{"exportContext": {"csvExportOptions": {"selectQery": "SELECT * FROM mysql.user"}, "fileType": "CSV", "uri": "gs://<bucket>/users.csv"}}'

Related

Jenkins http plugin to upload file using rest

I am trying to upload a file to a rest server from jenkins using http plugin. I have a jenkins pipelie where a step involves loading a file(type formData)
to a server using rest.
the server side method uses two parameters:
(#FormDataParam("file") InputStream file, #FormDataParam("fileName") String fileName)
I am using the below method
def filename = "${WORKSPACE}/Test.txt"
data="""{ \"fileName\" : \"Test.txt\" }"""
resp3 = httpRequest consoleLogResponseBody: true,url: "http://<url>",contentType:'APPLICATION_OCTETSTREAM',customHeaders:[[name:'Authorization', value:"Basic ${auth}"]],httpMode: 'POST',multipartName: 'Test.txt',uploadFile: "${filename}",requestBody:data,validResponseCodes: '200'
but when I run the status code is 400 and in the server logs the message is that no filestream and no filename is received i.e not able to get both the arguments.
Please let me know where it is getting wrong
Regards
You can try using curl instead of built-in Jenkins methods:
curl -XPOST http://<url> -H 'Content-Type: application/octet-stream' -H 'Authorization: Basic ${auth}' --data-binary '{\"fileName\" : \"Test.txt\" }'
You can debug it first from within shell. Once it's working, wrap it in sh directive:
sh "curl ..."
Since I was running on windows so bat + curl worked for me .With this workaround I was able to transfer files using jenkins and rest
However using httpRequest from jenkins inbuild library is still not working.

Cloud Functions REST API: Creating a new action from a zip file

I'm trying to create a nodejs function from a zip file and through the REST API using the following curl:
curl --request PUT --url 'https://my:credentials#openwhisk.eu-gb.bluemix.net/api/v1/namespaces/mynamespace/actions/my_action_name?overwrite=true' --header 'accept: application/json' --header 'content-type: application/json' --data '{"annotations":[{"key":"exec","value":"nodejs:10"},{"key":"web-export","value":true}],"exec":{"kind":"nodejs:10","init":"./action.zip"},"parameters":[{"key":"message","value":"Hello World"}]}'
As a result I get an error:
"error":"The request content was malformed:\n'code' must be a string or attachment object defined in 'exec' for 'nodejs:10' actions"
Is it possible to get an example of how to create a new action from a zip file and through the REST API? Thank you.
You have to base64 encode your .zip file and then pass it as a code parameter. I have written a shell script(bash) to encode and also create an action called 'action'. Save the script as create.sh and execute the script ./create.sh
#!/bin/sh
ACTION=action
ZIP=$ACTION.zip
base64 $ZIP | echo "\"$(cat)\"" | jq "{namespace:\"_\", name:\"$ACTION\", exec:{kind:\"nodejs:10\", code:., binary:true, main:\"main\"}}" | curl -X PUT -H "Content-Type:application/json" -d #- https://USERNAME:PASSWORD#openwhisk.ng.bluemix.net/api/v1/namespaces/_/actions/$ACTION?overwrite=true
Complete code
app.js or index.js code
function myAction(args) {
const leftPad = require("left-pad")
const lines = args.lines || [];
return { padded: lines.map(l => leftPad(l, 30, ".")) }
}
exports.main = myAction;
package.json
{
"name": "node",
"version": "1.0.0",
"description": "",
"main": "app.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "",
"license": "ISC",
"dependencies": {
"left-pad" : "1.1.3"
}
}
Run npm install and zip the file zip -r action.zip *.
To test the action
ibmcloud fn action invoke --result action --param lines "[\"and now\", \"for something completely\", \"different\" ]"
The REST API for creating or updating a Cloud Functions actions is documented in the IBM Cloud Functions API docs. A good way to find out the exact curl / request syntax is to use the IBM Cloud Functions CLI in verbose mode (-v). The CLI is just a wrapper around the REST API and in verbose mode all the REST details are printed.
Here is the relevant part for what could be printed:
Req Body
Body exceeds 1000 bytes and will be truncated
{"namespace":"_","name":"mytest/myaction","exec":{"kind":"nodejs:8","code":"UEsDBBQAAAAIAHJPhEzjlkxc8wYAAH8VAAALABwAX19tYWluX18ucHlVVAkAA+iFxFrohcRadXgLAAEE9AEAAAT0AQAAxVhtb9s2EP6uX8HRCCLBipb002DA69YkbYo17dZ0GwbDMGSKlrXJokfSToNh/313R+rNL2labJiK1iJ578/x7tTBgJ7A/QzYq8IuN3NmdbpYFIIZm9rC2EKYmiIYsB+1ynW6Ykqz1y9u2WWpNhl7uamELVTFrGJClaUUtha2LeQ9S6uMiVJVspYNgnDPWKVhb5lalqU2ZUXFUqZlmaKwtKTNeWpkzKp0JcsHdj
You would need to set the binary field to true and include the zip content as code. The curl docs suggest to use #filename to reference your zip file:
If you want the contents to be read from a file, use <#filename> as
contents.

how to do a put for camel-example-swagger-cdi?

I can do get on camel-example-swagger-cdi. But how do I do a put ?
I tried the following on browser, it didn't work at all.
http://localhost:8080/user/{'id':'222','name':'Richard'}.
And I tried the following curl command, it didn't work either
curl -X PUT -d id=222 -d name="Richard" localhost:8080/user
Please advise how I can issue a post
public class UserRouteBuilder extends RouteBuilder {
#Override
public void configure() throws Exception {
// configure we want to use servlet as the component for the rest DSL
// and we enable json binding mode
restConfiguration().component("netty4-http").bindingMode(RestBindingMode.json)
// and output using pretty print
.dataFormatProperty("prettyPrint", "true")
// setup context path and port number that netty will use
.contextPath("/").port(8080)
// add swagger api-doc out of the box
.apiContextPath("/api-doc")
.apiProperty("api.title", "User API").apiProperty("api.version", "1.2.3")
// and enable CORS
.apiProperty("cors", "true");
// this user REST service is json only
rest("/user").description("User rest service")
.consumes("application/json").produces("application/json")
.get("/{id}").description("Find user by id").outType(User.class)
.param().name("id").type(path).description("The id of the user to get").dataType("int").endParam()
.to("bean:userService?method=getUser(${header.id})")
.put().description("Updates or create a user").type(User.class)
.param().name("body").type(body).description("The user to update or create").endParam()
.to("bean:userService?method=updateUser")
.get("/findAll").description("Find all users").outTypeList(User.class)
.to("bean:userService?method=listUsers");
}
}
I used Postman to do a put:
localhost:8080/user?body={'id':'222' , 'name':'Richard' }
Got
Used Postman to do a put. Got
java.net.URISyntaxException: Illegal character in query at index 11: /user?body=
{%27id%27:%272222%27%20,%20%27name%27:%27Richard%27%20}
I got the following exception when I run curl to put.
$ curl -X PUT -d "body={'id':'222','name':'Richard'}" http://localhost:8080/user
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3142 100 3108 100 34 33419 365 --:--:-- --:--:-- --:--:-- 50129com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'body': was expecting ('true', 'false' or 'null')
at [Source: java.io.ByteArrayInputStream#17b2613c; line: 1, column: 6]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1581)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:533)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3451)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2610)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:841)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:737)
at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3776)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3721)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2796)
at org.apache.camel.component.jackson.JacksonDataFormat.unmarshal(JacksonDataFormat.java:173)
at org.apache.camel.processor.UnmarshalProcessor.process(UnmarshalProcessor.java:69)
at org.apache.camel.processor.binding.RestBindingProcessor.process(RestBindingProcessor.java:168)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.component.netty4.handlers.ServerChannelHandler.processAsynchronously(ServerChannelHandler.java:134)
at org.apache.camel.component.netty4.handlers.ServerChannelHandler.channelRead0(ServerChannelHandler.java:105)
at org.apache.camel.component.netty4.http.handlers.HttpServerChannelHandler.channelRead0(HttpServerChannelHandler.java:211)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at org.apache.camel.component.netty4.http.handlers.HttpServerMultiplexChannelHandler.channelRead0(HttpServerMultiplexChannelHandler.java:113)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:42)
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:309)
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:36)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
at java.lang.Thread.run(Thread.java:724)
If you can use GUI tools you should take a look at Postman, otherwise you can run:
curl -X PUT -d "{'id':'222','name':'Richard'}" http://localhost:8080/user
I would also recommend you to take a look at PUT vs POST in REST, and Swagger's Parameter Object docs.
I figured it out. And it is working now. curl -X PUT --header "Content-Type: application/json" --header "Accept: application/json" -d "{ \"id\": 222,\"name\": \"Richard\"}" http://localhost:8080/user

Run a MapReduce job via rest api

I use hadoop2.7.1's rest apis to run a mapreduce job outside the cluster. This example "http://hadoop-forum.org/forum/general-hadoop-discussion/miscellaneous/2136-how-can-i-run-mapreduce-job-by-rest-api" really helped me. But when I submit a post response, some strange things happen:
I look at "http://master:8088/cluster/apps" and a post response produce two jobs as following picture:
strange things: a response produces two jobs
After wait a long time, the job which I defined in the http response body fail because of FileAlreadyExistsException. The reason is another job creates the output directory, so Output directory hdfs://master:9000/output/output16 already exists.
This is my response body:
{
"application-id": "application_1445825741228_0011",
"application-name": "wordcount-demo",
"am-container-spec": {
"commands": {
"command": "{{HADOOP_HOME}}/bin/hadoop jar /home/hadoop/hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount /data/ /output/output16"
},
"environment": {
"entry": [{
"key": "CLASSPATH",
"value": "{{CLASSPATH}}<CPS>./*<CPS>{{HADOOP_CONF_DIR}}<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/*<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/lib/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/lib/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/lib/*<CPS>./log4j.properties"
}]
}
},
"unmanaged-AM": false,
"max-app-attempts": 2,
"resource": {
"memory": 1024,
"vCores": 1
},
"application-type": "MAPREDUCE",
"keep-containers-across-application-attempts": false
}
and this is my command:
curl -i -X POST -H 'Accept: application/json' -H 'Content-Type: application/json' http://master:8088/ws/v1/cluster/apps?user.name=hadoop -d #post-json.txt
Can anybody help me? Thanks a lot.
When you run the map reduce, see that you do not have output folder as the job will not run if it is present. You can write program so that you can delete the folder is it exists, or manually delete it before calling the rest api. This is just to prevent the data loss and avoid overwriting the output of other job.

How to deploy a report (jrxml file) through the Jasper PHP/REST API?

How to deploy a report (jrxml file) through the Jasper PHP/REST API
to the Jasper Reports Server?
For deploying Jasper reports on Jasper Server using Rest-API you can use following method:
Uploading JRXML file
JRXML_DATA=$(cat $PATHTOJRXMLFILE/$JRXML_FILE)
curl -X POST $JASPER_REST_HOST/jasperserver/rest_v2/resources/reports \
-H "Content-Type:application/jrxml" \
-H "Content-Disposition:attachment; filename=test" \
-H "Content-Description:test file" \
-d "$JRXML_DATA" \
--user $JASPER_USERNAME:$JASPER_USERNAME
Creating ReportUnit
RESOURCEDESCRIPTOR_JSON=$(cat $REPORT/deployable/reportunit.json)
curl -X POST $JASPER_REST_HOST/jasperserver/rest_v2/resources/reports \
-H "Content-Type:application/repository.reportUnit+json" \
-d "$RESOURCEDESCRIPTOR_JSON" \
--user $JASPER_USERNAME:$JASPER_PASSWORD
ResourceDescriptor for reportUnit example
{
"uri": "/reports/test_report",
"label": "test_report",
"description": "description",
"permissionMask": "0",
"version": "0" ,
"alwaysPromptControls": "true",
"controlsLayout": "popupScreen",
"jrxml": {
"jrxmlFileReference": {
"uri": "/reports/test"
}
}
}
Resource Descriptor references
http://community.jaspersoft.com/documentation/jasperreports-server-web-services-guide/v56/v2-resource-descriptor-types
To upload a jrxml file, create a ResourceDescriptor with PROP_HAS_DATA = true and insert the jrxml content in a multipart PUT request.
After a while of researching and investigating I got it running and developed a PHP class that's easy to use.
http://blog.flowl.info/2013/jasper-php-library-on-github/
To upload a jrxml file this code does the job:
// Init the Jasper connection
require_once('Jasper/Jasper.php');
$jasper = new \Jasper\Jasper();
$jasper->login('jasperadmin', 'jasperadmin', 'jasper.host.com:8080');
// Create a Resource Descriptor object for the jrxml file
$jrxml = new \Jasper\JasperJrxml('/reports/test.jrxml');
// Upload the Resource Descriptor object with content
$jasper->createContent($jrxml, file_get_contents('templates/test.jrxml'));
To create a report unit, go on with the following lines:
// Datasource Resource Descriptor
$mongo = new \Jasper\JasperDatasource();
$mongo->setPropIsReference('true');
$mongo->setPropReferenceUri('/datasources/mongo_local_test');
// Put everything together and deploy the report
$report->addChildResource($mongo);
$report->addChildResource($jrxml);
// Want to see the Resource Descriptor of the Report Unit?
// true = pretty print
print_r($report->getXml(true));
// Create a the Report Unit
$jasper->createResource($report);