Github Webhook to AWS API Gateway integrated with SQS Bug? - aws-api-gateway

I have this mostly working. All events sent by Github end up on my SQS Queue. The webhook is sending application/json with "Send me everthing" at the org-level. I'm using the following template mapping in my API Gateway:
Action=SendMessage&MessageBody={
"bodyJson":"$util.base64Encode($util.json('$'))",
"requestId":"$context.requestId",
"resourcePath":"$context.resourcePath",
"apiId":"$context.apiId",
"stage":"$context.stage",
"resourceId":"$context.resourceId",
"path":"$context.path",
"protocol":"$context.protocol",
"requestTimeEpoch":"$context.requestTimeEpoch",
"X-GitHub-Event":"$method.request.header.X-GitHub-Event",
"X-GitHub-Delivery":"$method.request.header.X-GitHub-Delivery",
"X-Hub-Signature":"$method.request.header.X-Hub-Signature"
}
(I've also tried '"bodyJson":"$util.base64Encode($util.body)",')
About 5% of event notifications (of various event types, e.g.: status, push) from Github fail to transform properly. When I go to base64 decode messages from the queue, I get part message/part random junk. Have noticed that resending the same event fails on every retry. (The event looks like well-formed json in the Github UI though.) I'm logging to cloudwatch, but the logs are truncated. So I can't tell how much of the original message gets through. I've also tried setting application/json as a binary media type. That just caused all events to fail on transform and return 500 to Github. Anyone know what I'm doing wrong or if this is a bug?
Update
Need to test a bit more, but think I've figured it out. We need to urlEncode the base64Encoded bodyJson because base64 encoding output includes characters (+ and /) that are not application/x-www-form-urlencoded compatible without being url encoded:
Action=SendMessage&MessageBody={
"bodyJson":"$util.urlEncode($util.base64Encode($input.body))",
"requestId":"$context.requestId",
"resourcePath":"$context.resourcePath",
"apiId":"$context.apiId",
"stage":"$context.stage",
"resourceId":"$context.resourceId",
"path":"$context.path",
"protocol":"$context.protocol",
"requestTimeEpoch":"$context.requestTimeEpoch",
"X-GitHub-Event":"$method.request.header.X-GitHub-Event",
"X-GitHub-Delivery":"$method.request.header.X-GitHub-Delivery",
"X-Hub-Signature":"$method.request.header.X-Hub-Signature"
}

Related

How do I translate the following POST request into ESP8266 AT-command format?

I've got a working local website that takes in HTML form data.
The fields are:
Temperature
Humidity
The server successfully receives the data and spits out a graph updated with the new entries.
Using a browser tool, I was able to capture the actual POST request as follows:
http://127.0.0.1:5000/add_data
Temperature=25.4&Humidity=52.2
Content-Length:30
Now, I want to migrate from using the human interface browser with manual entries to an ESP01 device using AT commands.
According to the ESP AT-commands documentation, a POST request is performed using the following command:
AT+HTTPCPOST=
Find the link below for the full description of the command.
I cannot seem to get this POST request working. The ESP01 device immediately returns an "ERROR" message without any delay, as though it did not even try to send the request, that the syntax might be wrong.
Among many variations, the following is my best attempt:
AT+HTTPCPOST="http://MYIPADDR:5000/add_data",30,2,"Temperature: 25.4","Humidity: 52.2"
With MYIPADDR above replaced with my IP address.
How do I translate a post request into ESP01 AT command format, and are there any prerequisites needed to be in place to perform such a request?
I did connect the ESP01 device to the WiFi network.
Here's the link to the POST AT command description:
https://docs.espressif.com/projects/esp-at/en/release-v2.2.0.0_esp8266/AT_Command_Set/HTTP_AT_Commands.html#cmd-httpcpost
The documentation says:
AT+HTTPCPOST=url,length[,<http_req_header_cnt>][,<http_req_header>..<http_req_header>]
Response:
OK
The symbol > indicates that AT is ready for receiving serial data, and you can enter the data now. When the requirement of message length
determined by the parameter is met, the transmission starts.
...
Parameters
: HTTP URL. : HTTP data length to POST. The maximum
length is equal to the system allocable heap size.
<http_req_header_cnt>: the number of <http_req_header> parameters.
[<http_req_header>]: you can send more than one request header to the
server.
You're sending:
AT+HTTPCPOST="http://MYIPADDR:5000/add_data",30,2,"Temperature: 25.4","Humidity: 52.2"
The length is 30. The problem is that everything after the length is HTTP header fields; you need to send the variables in the body. So the command is:
AT+HTTPCPOST="http://MYIPADDR:5000/add_data",30
followed on the next line by after the ESP-01 send the > character:
Temperature=25.4&Humidity=52.2
Because you passed 30 as the body length, the ESP-01 will read exactly 30 characters after the end of the AT command and send that data as the post body. If the size of that data changes (for instance, maybe the temperature is 2.2, so one digit less), you'll need to send the new length rather than 30.

I need simple proxy between 2 rest APIs

My code is working ok for GET/POST/PUT to/from restApi1 and restApi2.
However, my problem I need to implement HEAD/OPTIONS (no body!) and GET uri1
HEAD/OPTIONS could return 204 or 200 depends on a process status. I am getting error "Stream closed". Sounds like Camel want body bytes, but I don't indend to have it. Even I set ExchangePattern.InOnly or optional etc error occur...
What is correct way to see responses and handle requests WITHOUT body, just statuses exchange?
How to see response from restApi2 on Camel rest("/restApi1").head().route().routeId("id1")
.to("direct:restApi2").routeId("/id1").setHeader(Exchange.HTTP_METHOD,constant("HEAD"))
setExchanggePattern(ExchangePattern.OutOptionalIn).recepientList(simple(restApi2));
I figured it out. Need to set '.convertBodyTo(String.class)' even I don't have a body.

MATLAB Not properly sending HTTP POST requests

I'm currently trying to build a MATLAB based system to interface with the API of my stock broker. I'm however running into quite some issues with sending the http post requests to the server.
I already have it working perfectly when testing with POSTMAN, but for some reason it keeps refusing my MATLAB send requests. I now testing the actual requests through PIPEDREAM which lets me view the http request.
Image of the good and bad requests:
The Left is an image of my postman requests which it perfectly processes as JSON strings. However my MATLAB requests are not processed properly and also are 10 characters longer than the actual string value.
The (trimmed) code to send the requests can be seen here.
% http request classes
import matlab.net.*
import matlab.net.http.*
% prepare payload
username = "usr";
password = "XXXXXXXXXXXXX";
login_payload = struct("username", username, "password", password);
request = RequestMessage('POST', [ bunchOfHeaders ], jsonencode(login_payload));
% Send request to login api
[login_resp, c, h] = request.send("https://trading.somebroker.com/login/secure/login");
Does anyone have any clue what could be happening here? If I set the content-lenght to the "correct" length (same as length(login_payload)) it says my length is wrong even though my postman requests seem to not struggle with this.
Found the answer... Matlabs http stuff is absolutely braindead.
I had a closer look into the raw intercepted messages (pipedream just sends your request back to you and you can view it with string(login_resp)).
For some god darn reason matlab encases the json string with "s which makes the receiver treat the whole body as a string. This is caused by setting the "content-type" to "application/json". changing the content-type to "text/plain" did not encase it in "s and completely solved my issue

Camel does not set 'Transfer-Encoding chunked' in case response is prepared by exception handler processor

I am implementing REST services using Apache-CXF running on servicemix and for that I have a camel route that does some processing, sends the message over queue, process some more and send back the reply. Something like this:
from("direct:start")
.process(A)
.process("activemq:abc")
.process(B);
On this route I have applied some basic validation and exception handler and when I have to stop the route in both cases, I use something like this:
exchange.getOut().setBody(response);
exchange.setProperty(Exchange.ROUTE_STOP, Boolean.TRUE);
I use soap UI, restclient-UI and putty to make http requests and I get proper response body displayed in all of them. Now I wanted to preserve request headers so I made a little change everywhere in the code so that response bodies are set in exchange.getIn() only. For example: in case of validation failure I do:
exchange.getIn().setBody(response);
exchange.setProperty(Exchange.ROUTE_STOP, Boolean.TRUE);
Just with this little change, the rest clients I am using to make request stopped displaying the response body. As per the server logs, response is being generated and also as per the logs in rest client, I am getting the proper response but they are unable to display the response body only in case when I stop the route in between. Normal response is displaying just fine. Only the restclient-UI was considerate enough to show the error as to why they are not displaying body and the error is:
Byte array conversion from response body stream failed.
Diggin deeper, I found the only response header which was there in success response but missing in error response:
Transfer-Encoding chunked
Error response is around 1000 characters long and contains a header called content-length. I am not sure but I think the problem has something to do with this itself. I would really like to play with exchange.getIn but these different kind of responses prepared by camel are confusing me. How can I make sure my camel responses are always displayed properly?
The Content-Length header will be preserved from the original request so you need to remove it so that camel cxf can work out the new body length on the response and set Content-Length with that.

Play framework - retrieving the Date header in the request

I need to access the Date: header when I handle the request, but this seems to be "swallowed" by the framework; any other header (even made up FooBar ones) show up and I can get them, but this gives me None (I'm using Postman to send a simple GET request - everything else works just fine):
println("Date: " + request.headers.get("Date").getOrElse("no date!"))
returns "no date!" no matter how I try to send something sensible.
I'm wondering whether this gets processed before the request object reaches my Action.
I need the actual string value sent, as this should be part of the request's signature - so an equivalent Date object representing the same value would not be of much use (as it needs to be part of the hash, to avoid replay attacks).
Just as a test, I replaced the Date header with a Date-Auth one, and this one shows up just fine:
ArrayBuffer((Date-Auth, ArrayBuffer(Wed, 15 Nov 2014 06:25:24 GMT))
Any ideas or suggestions greatly appreciated!
Are you sure there is a Date Header in your request (tested with tools like firebug or wireshark)?
Browsers do not need to send a Date header.
RFC 2616 (HTTP 1.1) from the Date section (14.18)
Clients SHOULD only send a Date header field in messages that include an entity-body, as in the case of the PUT and POST requests, and even then it is optional. A client without a clock MUST NOT send a Date header field in a request.
I stand corrected - it turns out that Chrome blocks a whole bunch of headers:
http://www.getpostman.com/docs/requests
I wrote a Python Flask test server and, in fact, the Date header is not there.
That page has also a fix, which works just fine with Postman Version 0.10.4.3 and Interceptor(1).
sorry for wasting everyone's time!
1 Incidentally, IMO Postman is the best REST client and has now also some awesome looks, beyond incredible functionality. If you're working with REST APIs, I highly recommend it.