This is yet another variation of "binary data not working", slightly different from existing questions.
I have defined a Lambda-Proxy Integration:
And I have enabled the option in the settings:
And I use curl -H to make a request with the proper Accept header, and you can also see the proper header in the response:
So why is the final response still base-64 encoded?
Related
I am following the docs here https://docs.github.com/en/rest/actions/artifacts#download-an-artifact to use Github actions rest API to download artifacts. Given an ARTIFACT_ID and access token if the repo is private, one can call the API via cURL or the github CLI to get a response from github. The response header contains Location:... which provides a temporary URL lasting 1 minute from which the artifact can be downloaded. The artifact can then be downloaded via a second call to cURL.
I would like to know the reason for this design decision on the part of Github. In particular, why not just return the artifact in response to the first call to cURL? Additionally, given that the first call to cURL is intended to return a temporary URL from which the artifact can be retrieved, why not have this temporary URL returned directly by call to cURL rather than having it only contained in the header. Other information such as if the credentials are bad, or if the object has been moved are returned in json when this cURL command is run, so why can't the temporary URL also be contained here?
To help clarify my question, here is some relevant code:
# The initial cURL command looks something like this:
curl -v \
-H "Accept: application/vnd.github+json" \
-H "Authorization: token <TOKEN>" \
https://api.github.com/repos/OWNER/REPO/actions/artifacts/ARTIFACT_ID/ARCHIVE_FORMAT
# the temporary URL, which can be curled to retrieve the artifact, looks like something like this:
curl https://pipelines/actions/githubusercontent.com/serviceHosts/<HEXSTRING>/_apis/pipelines/1/runs/16/\
signedartifactscontent?artifactName=<artName>&urlExpires=<date>&urlSigningMethod=HMACV2&urlSignature=<SIGNATURE>
Additionally, I am currently capturing the standard error of the cURL command and then running regex on it so as to extract the temporary URL. Is there a better way to do this? For example, is there a flag I could pass to cURL that would give me the value of Location directly?
Additionally, it is stated that The archive_format must be zip. Given this is the case, what is the benefit of having this parameter. Is it not redundant? If so, what is the benefit of this redundency?
This is a consequence of a 2011 design decision regarding https://github.blog/2011-08-02-nodeload2-downloads-reloaded/
When implementing a proxy of any kind, you have to deal with clients that can’t read content as fast as you can send it.
When an HTTP server response stream can’t send any more data to you, write() returns false.
Then, you can pause the proxied HTTP request stream, until the server response emits a drain event.
The drain event means it’s ready to send more data, and that you can now resume the proxied HTTP request stream.
TO avoid DDOS, it is better to manage that stream from a temporary URL, rather than a fixed one.
You can use -D to display response header, but you would still need to post-process its answer to get the redirection URL.
I'm currently learning to use REST API (from WooCommerce in this case) and got some basic questions:
How to see complete request string in Postman software?
I'm testing a simple GET request which works great with for example:
<host>/wp-json/wc/v3/products
to receive the product list. In this case I use the authorization tab to enter my user/pass as Basic Auth.
I also tested curl.exe using another simple Windows command prompt. This also returned product list:
curl.exe <host>/wp-json/wc/v3/products -u mykey:mysecret
What is the difference between them? The last example is a simple GET, i assume, although it's not stated. How about POST or DELETE etc? This is what i don't understand: A https request can only have an address and eventual parameters. Where and how does "GET" come into the picture?!
If possible, I would like the see the complete URL request (as one string) from the working Postman example?
My last question is about testing the same method on another server/service which is not WooCommerce. Afaik this service is created with something called swagger:
curl "<host>/orderapi/item" -H "accept: application/json" -H "X-Customer: <customer>" -H "X-ApiKey: <mykey>" -H "X-ApiSecret: <mysecret>" -H "Content-Type: application/json"
This also returns a list of, in this case orders instead of products. All good.
But for this example I haven't figured out how to achieve the same request in Postman. What auth method should I use?
And again, I don't understand the GET/POST/DELETE thing. And I also would like to see the complete request as one-string.
1) How to see complete request string in Postman software? I would like the see the complete URL request (as one string) from the working Postman example
On version 9.x.x:
The code window(image) shows the choosen method (yellow mark) and the code window(red arrow), where you get the actual
curl code(image)
2) What is the difference between them? The last example is a simple GET, i assume, although it's not stated. How about POST or DELETE etc? Where and how does "GET" come into the picture?
From the curl documentation:
-X, --request
(HTTP) Specifies a custom request method to use when communicating
with the HTTP server. The specified request method will be used
instead of the method otherwise used (which defaults to GET). Read the
HTTP 1.1 specification for details and explanations. Common additional
HTTP requests include PUT and DELETE, but related technologies like
WebDAV offers PROPFIND, COPY, MOVE and more.
GET is the default method for curl, which means:
curl.exe <host>/wp-json/wc/v3/products -u mykey:mysecret
is the same as:
curl.exe <host>/wp-json/wc/v3/products -u mykey:mysecret -X "GET"
so, for a POST/DELETE/... you should change your '-X' parameter for example:
curl.exe <host>/wp-json/wc/v3/products -u mykey:mysecret -X "POST" [...otherOptions]
(Assuming that you can receive a POST on the url above)
3) [On another server/service] I haven't figured out how to achieve the same request in Postman. What auth method should I use?
The -H specify the header parameter you are passing. You have those in your example:
accept: application/json
X-Customer:
X-ApiKey:
X-ApiSecret:
Content-Type: application/json
You need to add those in your postman on the headers(image) tab. In this case you don't need to specify a auth method, once you're sending the ApiKey on the header. In addition to that, you can specify the authorization Type to be "Api Key" and put X-ApiKey as key and your apikey value on the value field(image). It'll generate the same request as shown in the headers image.
curl, at least the GNU one on Linux, uses GET method by default. If you want to change a HTTP method in your request, there's -X option, for example:
$ curl -X DELETE https://example.com
Postman has something called Postman Console which you can open by pressing Alt + Ctrl + C:
and where you can see more details about requests and responses.
Postman also lets you import curl commands, so you don't need to manually prepare the request, you can only paste the curl command in Postman.
There are many resources online on the specifics, e.g. how to import a curl command.
I'm trying to call a simple 'Hello World' REST API via the GET verb. The API is only expecting one HTTP Header, Accept application/json which i've set in the Advanced Settings>HTTP Headers of my tRestClient component. However, looking at the code tab, Talend seems to automatically create another entry for the contents of the Accept Type dropdown. When i run I’m getting HTTP 406 Not Acceptable back because the API is not expecting 2 headers.
I've tested this API with other software and it responds correctly so it must be down to Talend configuration. Anybody know a way around this or had a similar issue they've resolved?
I have screenshots but unfortunately they're being blocked by my firewall at work.
Thanks
tRESTClient defines its http headers based on the parameters you supply in the component settings. It has an "Accept Type" setting, which you can set to "JSON", this adds the http header "Accept: application/json" (this way you don't have to add it in the http headers section).
For your use case, you can also use tREST, which allows you to have complete control over http headers, it only sends those you set in the http headers section.
I am using postman client to make REST calls to JIRA API. It says "POST your credentials to http://jira.example.com:8090/jira/rest/auth/1/session" to get SESSION. I tried posting with Form-data, application/x-www-form-urlencoded, raw etc. Nothing worked. which is the right way to do that.
Here is the tutorial i am following: https://developer.atlassian.com/jiradev/jira-apis/jira-rest-apis/jira-rest-api-tutorials/jira-rest-api-example-cookie-based-authentication
Since you're using postman, I'm assuming you're in a dev environment. In this case, it might be simpler to get going with the auth header, which is a base-64 encoded username/password. From the documentation here:
Supplying Basic Auth headers
If you need to you may construct and send basic auth headers yourself. To do this you need to perform the following steps:
Build a string of the form username:password
Base64 encode the string
Supply an "Authorization" header with content "Basic " followed by the encoded string. For example, the string "fred:fred" encodes to "ZnJlZDpmcmVk" in base64, so you would make the request as follows.
curl -D- -X GET -H "Authorization: Basic ZnJlZDpmcmVk" -H "Content-Type: application/json" "http://kelpie9:8081/rest/api/2/issue/QA-31"
In the Headers section of Postman, add Authorization with Basic <base64-encoded-username:password>
Don't forget to also add the header Content-Type as application/json
(You can use base64encode.org to quickly encode your username/password).
Don't forget to put the string in as username-colon-password (username:password)
If you are on the same UI as I for postman, click Authorization, select an auth type (I used basic auth succesfully), and then enter your credentials. Next click over to the body tab, select raw, and on the drop down menu on the right choose JSON(applications/json), and supply the body as normal.
That is the first hurdle. The next hurdle which may be hit (and the one I am stuck on) is that once your basic-auth gets accepted, JIRA will deny access as part of Cross Site Request Forgery checks (XSRF) with a code 403. I have a ticket open right now seeing if there is a possible workaround to post and put from postman, because using postman and newman would be much much simpler than building an entire plugin which I have to jump through a bunch of hoops to access.
With Postman can simply add withCredentials:true to your request header section.
I am new in API gateways. I have python based API deployed on an EC2 server. I can access this as URL http://xxx.xxxxxxx.com/RPC2/. I can see objects, methods in this URL. I am trying to use API gateway for same.
Created API. (ExampleAPI)
Created POST method. (given path http://xxx.xxxxxxx.com/RPC2/ as end point URL ). I have not created the resource since I am expecting HTTP Proxy for all Methods of the resource. Its looks fine when I put my content in the request body. I get a response.
Now I have deployed it to one stage dev1. Got a new endpoint URL.
Also created an API key and attached it with dev1. Also Set API key required true in POST Method Request.
Questions.
1. When I hit dev1 URL (https://xxxxxxxxxxx-api.us-east-1.amazonaws.com/dev1), it does not give me same page as http://xxx.xxxxxxx.com/RPC2/. It gives me {"message":"Missing Authentication Token"} error. Am I missing some fundamentals here?
http://xxx.xxxxxxx.com/RPC2/ do have several methods, so how can I use it? All of them are POST methods. Can I set some parameters or some request body, or some templates? How can I improve this process?
How can I use API key here? Or it won`t work in POST method?
If i do curl -H "Content-Type: application/JSON" -X POST -d "{\"method\": \"app.menu\",\"params\":[] }" https://xxxxxxxxxxx-api.us-east-1.amazonaws.com/dev1 i get same response as i curl http://xxx.xxxxxxx.com/RPC2/. is it the only way to access my dev1 URL or I can create individual methods or string parameters.
Regards,
Ashish
See answers posted to this forum:
When i hit dev1 URL (https://xxxxxxxxxxx-api.us-east-1.amazonaws.com/dev1), it do not give
me same page as http://xxx.xxxxxxx.com/RPC2/. It gives me
{"message":"Missing Authentication Token"} error. Am i missing some
fundamentals here ?
If you are hitting the URL in a browser with a GET method, it will not
work. You have to specify all of the HTTP methods on a resource that
you want the client to access. If you hit a method that is not
defined, you get that message.
http://xxx.xxxxxxx.com/RPC2/ do have several methods, so how can i use it? all of them are POST methods. Can i set some parameters or
some request body, or some templates. how can i improve this process.
Yes if you are mapping to an RPC API then you can build the REST
methods/resources in API Gateway and set a static value for the header
or in the body, wherever the RPC action is expected by the backend.
How can i use API key here? or it won`t work in POST method? Because while accessing from curl, it works fine without API key.
First you should set API Key Required on the method (Method Request
page), then you'll have to add the API Stage to the API Key and make
sure it's enabled. After all that, if you send the API Key in a header
called 'x-api-key' it should work, otherwise you should get a 403
response saying "Forbidden".
If i do "curl -H "Content-Type: application/json" -X POST -d "{\"method\": \"app.menu\",\"params\":[] }"
https://xxxxxxxxxxx-api.us-east-1.amazonaws.com/dev1" i get same
response as i curl http://xxx.xxxxxxx.com/RPC2/. is it the only way to
access my dev1 URL or i can create individual methods or string
parametrs.
You will have to create each method in API Gateway (like GET) and they
can all point to the same backend url but specify a different RPC
action in the header or wherever it is specified.
I'd encourage you to check the public developer guides for parameter
mapping and payload transformation to learn what tools we have in API
Gateway.