I have an HAProxy configuration with a custom JSON log-format. I want to capture a specific response header and log it.
However, no matter how I try to capture it, I cannot make it appear in the log.
In my log format I use %[capture.res.hdr(0)] but it only comes up as -. I've also tried %[res.hdr(0)] and %[res.hdr(MyHeader)] but they were not valid configuration and HAProxy failed to start.
I've tried capturing using:
capture response header MyHeader len 50
But it doesn't work. I also tried:
declare capture response len 50
http-response capture res.hdr(MyHeader) id 0
With no success. The %hs format variable works - all the captured headers are logged in a delimited string. But I want to log the headers separately as JSON properties.
What am I doing wrong?
I'm currently using HAProxy 1.8.
It seems like the combination of having capture response header MyHeader len 50 in the frontend section and %[capture.res.hdr(0)] in the log-format actually works. Turns out I had multiple instances of HAProxy running and only reloaded some of them, so the changes only came through for some requests.
Related
I've got a working local website that takes in HTML form data.
The fields are:
Temperature
Humidity
The server successfully receives the data and spits out a graph updated with the new entries.
Using a browser tool, I was able to capture the actual POST request as follows:
http://127.0.0.1:5000/add_data
Temperature=25.4&Humidity=52.2
Content-Length:30
Now, I want to migrate from using the human interface browser with manual entries to an ESP01 device using AT commands.
According to the ESP AT-commands documentation, a POST request is performed using the following command:
AT+HTTPCPOST=
Find the link below for the full description of the command.
I cannot seem to get this POST request working. The ESP01 device immediately returns an "ERROR" message without any delay, as though it did not even try to send the request, that the syntax might be wrong.
Among many variations, the following is my best attempt:
AT+HTTPCPOST="http://MYIPADDR:5000/add_data",30,2,"Temperature: 25.4","Humidity: 52.2"
With MYIPADDR above replaced with my IP address.
How do I translate a post request into ESP01 AT command format, and are there any prerequisites needed to be in place to perform such a request?
I did connect the ESP01 device to the WiFi network.
Here's the link to the POST AT command description:
https://docs.espressif.com/projects/esp-at/en/release-v2.2.0.0_esp8266/AT_Command_Set/HTTP_AT_Commands.html#cmd-httpcpost
The documentation says:
AT+HTTPCPOST=url,length[,<http_req_header_cnt>][,<http_req_header>..<http_req_header>]
Response:
OK
The symbol > indicates that AT is ready for receiving serial data, and you can enter the data now. When the requirement of message length
determined by the parameter is met, the transmission starts.
...
Parameters
: HTTP URL. : HTTP data length to POST. The maximum
length is equal to the system allocable heap size.
<http_req_header_cnt>: the number of <http_req_header> parameters.
[<http_req_header>]: you can send more than one request header to the
server.
You're sending:
AT+HTTPCPOST="http://MYIPADDR:5000/add_data",30,2,"Temperature: 25.4","Humidity: 52.2"
The length is 30. The problem is that everything after the length is HTTP header fields; you need to send the variables in the body. So the command is:
AT+HTTPCPOST="http://MYIPADDR:5000/add_data",30
followed on the next line by after the ESP-01 send the > character:
Temperature=25.4&Humidity=52.2
Because you passed 30 as the body length, the ESP-01 will read exactly 30 characters after the end of the AT command and send that data as the post body. If the size of that data changes (for instance, maybe the temperature is 2.2, so one digit less), you'll need to send the new length rather than 30.
I am trying to communicate with kdb+ via HTTP. I succeeded to get the the response from the DB when I did it from the browser, but for some reason, the response is neither JSON nor anything that looks machine readable.
What I did:
Opened a port on q console (8080)
Defined a function getData that gives me the data I want.
The above steps can be presented in this .q file I created:
\p 8080
system["l db"]
getData: {[a;b] ?[table;((>=;`start;$[`long;a]);(<=;`end;$[`long;b]));0b;()]}
h:hopen `:localhost:8080:user:pass
I then typed http://localhost:8080/?getData[1;2] in the browser to get the results
Got the results as text with spaces inside <pre> tag.
The Problem
I have no I idea how to parse it to JSON. How can I solve this? is there a way to tell kdb+ to send response in JSON format?
The solution I found:
If you add .json before the ? in the url (i.e. http://localhost:8080/.json?getData[1;2]) the response will be in JSON.
It looks like, based on your code, the following may work to return data:
http://localhost:8080/?getData[a;b]
Where a and b are start and end times (or dates, this will depend on your data).
e.g.
http://localhost:8080/?getData[2021.07.23;2021.07.30]
In order to convert a KDB+ object to json you can make use of the .j.j operator.
For example you can change your code to something like:
getData:{[a;b] t:?[table;((>=;`start;$[`long;a]);(<=;`end;$[`long;b]));0b;()];.j.j t}
If you want to see what the incoming request looks like on your Q process you could add some logging to .z.ph (the default http message handler), e.g. something like:
.z.ph:{[zph;x]zph 0N!x}[.z.ph]
This will log any incoming requests out to your q process using 0N! and then run the usual .z.ph message handing code on it
I'm trying to add a new content type to a REST endpoint. Currently it only returns json but I now need to be able to return also a CSV file.
As far as I know, the best way to do this is by using the Accept header with value text/csv and then add a converter that is able to react to this and convert the returned body to the proper CSV representation.
I've been able to do this but then I have a problem handling exceptions. Up until know, all the errors returned are in json. The frontend expects any 500 status code to contain a specific body with the error. But now, by adding the option to return either application/json or text/csv to my endpoint, in case of an error, the converter to be used to transform the body is going to be either the jackson converter or my custom one depending on the Accept header passed. Moreover, my frontend is going to need to read the content-type returned and parse the value based on the type of representation returned.
Is this the normal approach to handle this situation?
A faster workaround would be to forget about the Accept header and include a url parameter indicating the format expected. Doing it this way, I'd be able to change the content-type of the response and the parsing of the data directly in the controller as the GET request won't include any Accept header and it will be able to accept anything. There are some parts of the code already doing this where the only expected response format is CSV so I'm going to have a difficult time defending the use of the Accept header unless there is a better way of handling this.
my frontend is going to need to read the content-type returned and parse the value based on the type of representation returned.
Is this the normal approach to handle this situation?
Yes.
For example, RFC 7807 describes a common format for describing problems. So the server would send an application/problem+json or an application/problem+xml representation of the issue in the response, along with the usual meta data in the headers.
Consumers that understand application/problem+json can parse the data with in, and forward a useful description of the problem to the user/logs whatever. Consumers that don't understand that representation are limited to acting on the information in the headers.
A faster workaround would be to forget about the Accept header and include a url parameter indicating the format expected.
That's also fine -- more precisely, you can have a different resource responsible for the each of the different media-types that you support.
It may be useful to review section 3.4 of RFC 7231, which describes the semantics of content negotiation.
I created my own “404 Page not found” error page on a TYPO3 website and implemented it via the /typo3conf/LocalConfiguration.php as follows, using the page’s Speaking URL path:
return [
...
'FE' => [
...
'pageNotFound_handling' => '/page-not-found/',
]
]
Now when I call a non-existing page, the error page gets displayed but there is a 4-digit alphanumeric number (hexadecimal as far as I’ve seen by now) BEFORE the HTML source code and a “0” AFTER it. Example (the number in the beginning is different after most of the reloads):
37b3
<!DOCTYPE html>
...
</html>
0
When calling the error page URL itself the page is returned correctly without those numbers.
Having the RealURL extension activated or deactivated does not make a difference.
Thanks a lot in advance!
I added the full description from the install tool and I guess we might find the solution there.
How TYPO3 should handle requests for non-existing/accessible pages.
empty (default)
The next visible page upwards in the page tree is shown.
'true' or '1'
An error message is shown.
String
Static HTML file to show (reads content and outputs with correct headers), e.g. notfound.html or http://www.example.org/errors/notfound.html.
Prefix "REDIRECT:"
If prefixed with "REDIRECT:" it will redirect to the URL/script after the prefix.
Prefix "READFILE:"
If prefixed with "READFILE" then it will expect the remaining string to be a HTML file which will be read and outputted directly after having the marker "###CURRENT_URL###" substituted with REQUEST_URI and ###REASON### with reason text, for example: READFILE:fileadmin/notfound.html.
Prefix "USER_FUNCTION:"
If prefixed with "USER_FUNCTION:" a user function is called, e.g. USER_FUNCTION:fileadmin/class.user_notfound.php:user_notFound->pageNotFound where the file must contain a class user_notFound with a method pageNotFound() inside with two parameters $param and $ref.
What you configured:
You're passing a string, thus TYPO3 expects to find a file - which you don't have, because it's more like an URL.
From what you try to achieve I'd go with REDIRECT:/page-not-found/.
Thanks for pointing this one out btw, I will remove the string configuration from the core since it does not make sense to have more people trip into this pitfall.
In short: change the following line in the FE section of your LocalConfiguration.php:
'pageNotFound_handling' => '/your404page.html',
to
'pageNotFound_handling' => 'REDIRECT:/your404page.html',
Cause
The actual cause is a combination of chunked Content-Encoding and the TYPO3 not being able to decode that in some cases. In your case the page not found handler eventually uses GeneralUtility::getUrl() to retrieve the error page.
If you have [SYS][curlUse] enabled it will use cUrl to retrieve the page and there is no problem.
If you don't have [SYS][curlUse] enabled it will open a socket, read the headers and then read the rest of the body. If the webserver uses "chunked" Content-Encoding the body will contain blocks of data and each block starts with a line with the length in hexadecimal format. The content ends with an empty block (with of course a line with the length "0").
cUrl apparently knows how to decode chunked data.
getUrl() itself does not know how to handle chunked data and uses the content as is as the page content.
In TYPO3 8 LTS the guzzle library is used to handle HTTP requests. In the guzzle code I can't find anything about handling chunked data. Guzzle will check if the cUrl PHP extension is present and use that as preferred transport. In most installations cUrl is present and since this decodes chunked data automagically no problem is visible. I have to test guzzle with PHP that has cUrl disabled to see if the issue is also present in v8/master.
Workaround/solution
If the PHP extension cUrl is enabled in your installation you can simply set [SYS][curlUse] in the Install Tool. The numbers around the 404 page content will disappear.
I need to update an order which is done via PUT method passing the order id as part of the https url string and a single parameter, the status_id.
https://mystore.mybigcommerce.com/orders/12345.json
I have tried several methods to pass the status_id value but no matter what I try "status_id=12" or formatted as JSON "{"status_id": 12,}" I always get the same response:
[{"status":415,"message":"The specified input content type is not valid."}]
I have also tried as a POST request passing the JSON or XML code as raw data but that method is not supported.
How am I supposed to pass that field=value pair? can I embed it in the url string?
I also tried it but it wouldn't work for me.
Any ideas?
In case you are wondering I am doing it within FileMaker with TROIUrl plugIn, not a very popular technology, but the GET method retrieving orders works like a charm
TURL_Put( ""; $url ;"status_id=12") (I have also tried other FM plugIns to no avail)
Don't get too caught up in the Filemaker part, I don't expect many people out there to be familiar with BigCommerce and Filemaker. I just need a generic answer.
Thanks
Commandline tool curl is worth a try. It supports put and https.
Mac OS X: curl already installed, call from FileMaker via AppleScript do shell script.
Windows: must be installed, call via Powershell.
It works for me using { "status_id": "3" } which means you probably need to put quotes around the actual number.
Also, it is a PUT operation and application/json which is part of the request content.
The error message received by the OP:
[{"status":415,"message":"The specified input content type is not valid."}]
Is saying that he did not supply the 'Content-Type' header in his request or that the header supplied is for a content type that is not allowed. For the OP's case using JSON he would need to include the header:
Content-Type: application/json
in his HTTPS request. This description can be found along with those of the other status codes you may see here:
https://developer.bigcommerce.com/api/status-codes