I am getting some cURL data using Guzzle Framework.
Code
$client = new Client();
$request = $client->get($url);
$response = $request->send();
$status = $response->getStatusCode();
After X requests I am getting 403 forbidden error code, mostly more than 150.
Any solutions?
Is the API/URI that you're calling subject to rate limits? I know that the Twitter API used to implement a maximum rate of approximately 150 requests per hour before it started to return an error message. The solution is to implement caching at your end if the situation is appropriate for the data to be cached.
Without knowing the service that you're calling it is difficult to offer much more help currently.
Related
I'm currently trying to build a MATLAB based system to interface with the API of my stock broker. I'm however running into quite some issues with sending the http post requests to the server.
I already have it working perfectly when testing with POSTMAN, but for some reason it keeps refusing my MATLAB send requests. I now testing the actual requests through PIPEDREAM which lets me view the http request.
Image of the good and bad requests:
The Left is an image of my postman requests which it perfectly processes as JSON strings. However my MATLAB requests are not processed properly and also are 10 characters longer than the actual string value.
The (trimmed) code to send the requests can be seen here.
% http request classes
import matlab.net.*
import matlab.net.http.*
% prepare payload
username = "usr";
password = "XXXXXXXXXXXXX";
login_payload = struct("username", username, "password", password);
request = RequestMessage('POST', [ bunchOfHeaders ], jsonencode(login_payload));
% Send request to login api
[login_resp, c, h] = request.send("https://trading.somebroker.com/login/secure/login");
Does anyone have any clue what could be happening here? If I set the content-lenght to the "correct" length (same as length(login_payload)) it says my length is wrong even though my postman requests seem to not struggle with this.
Found the answer... Matlabs http stuff is absolutely braindead.
I had a closer look into the raw intercepted messages (pipedream just sends your request back to you and you can view it with string(login_resp)).
For some god darn reason matlab encases the json string with "s which makes the receiver treat the whole body as a string. This is caused by setting the "content-type" to "application/json". changing the content-type to "text/plain" did not encase it in "s and completely solved my issue
I'm writing a C# Web API server application, and will send JSON to it via a Mirth HTTP Sender destination. This post is about how to handle error conditions. Specifically, there are three scenarios I want to handle:
Sometimes we take the C# application server offline for a short period for system upgrade or maintenance, and Mirth is unable to connect at all. I want Mirth to queue all messages in order, and when the server is available, process them in the order they were received.
The server receives the request, but rejects it due to a problem with the content of the request, e.g., missing a required field. In accordance with REST conventions, the server will return a 400-level HTTP response. This message would be rejected every time it's submitted, so it should not be re-sent; just log the failure and move on to the next message.
The server receives the request, but something goes wrong on the server, and the server returns an HTTP 500 Server Error response. This would be the appropriate response, for example, when something in the server environment has gone wrong. One real-world example was the time the Web API server was running, but somebody rebooted the database server. REST conventions would suggest we continue to resend the message until the transient problem has been resolved.
For #1, initially I had it queue on failure/always, but it appears the response transformer never runs for messages that were queued (at least, the debug statements never showed in the log). I have turned queueing off, and set it to retry every ten seconds for an hour, and that seems to give the desired behavior. Am I on the right track here, or missing something?
For #2 and #3, returning any HTTP 400 or 500 error invokes the 1-hour retries. What I want is to apply the 1-hour retries for the 500 errors, but not the 400 errors. I’ve tried responseStatus = SENT in the response transformer, but the response transformer only runs once, after the hour has expired, and not for each retry.
This seems like a common problem, yet I’m not finding a solution. How are the rest of you handling this?
You're close!
So by default, the response transformer will only run if there's a response payload to transform. For connection problems, or possibly for 4xx/5xx responses that contain no payload, the response transformer won't execute.
However, if you set your response data types (From the Summary -> Set Data Types dialog, or from the Destinations -> Edit Response, Message Templates tab) to Raw, then the response transformer will execute all the time. The reason being that the Raw data type considers even an empty payload to be "transformable".
So turn queuing back on, and set your response data types to Raw. Then in the response transformer, if you look at the Reference tab there's a category for HTTP Sender:
You'll want the "response status line", that's the "HTTP/1.1 200 OK" line of the response that contains the response code. Here's a response transformer script that forces 4xx responses to error:
if (responseStatus == QUEUED) {
var statusLine = $('responseStatusLine');
if (statusLine) {
var parts = statusLine.split(' ');
if (parts.length >= 2) {
var responseCode = parseInt(parts[1], 10);
// Force 4xx responses to error
if (responseCode >= 400 && responseCode < 500) {
responseStatus = ERROR;
responseStatusMessage = statusLine;
}
}
}
}
The Watson Speech-to-Text asynchronous HTTP interface allows one to register a callback url through a call to register_callback. This call is clearly not working; for illustration, please see these six lines of code.
# Illustration of how I can't get the Watson Speech-to-Text
# register_callback call to work.
r = requests.post(
"https://stream.watsonplatform.net/speech-to-text/api/v1/register_callback?{0}".format(
urllib.urlencode({ "callback_url": callback_url })),
auth=(watson_username, watson_password),
data="{}")
print(r.status_code)
print(pprint.pformat(r.json()))
# This outputs:
# 400
# {u'code': 400,
# u'code_description': u'Bad Request',
# u'error': u"unable to verify callback url 'https://xuyv2beqpj.execute-api.us-east-1.amazonaws.com/prod/SpeechToTextCallback' , server responded with status code: 400"}
# and no http call is logged on the server.
r = requests.get(
callback_url, params=dict(challenge_string="what does redacted mean?"))
print(r.status_code)
print(r.text)
# This outputs:
# 200
# what does redacted mean?
# and an HTTP GET is logged on the server.
I first call register_callback with a perfectly valid callback_url parameter, in exactly the way the documentation describes. This call returns with a 400 and, according to my callback URL server logs, the callback URL never receives an HTTP request. Then I GET the callback URL myself with a challenge_string. Not only is the callback URL responding with the right output, but a log appears on my server indicating the URL received an HTTP request. I conclude that register_call is not working.
Answer:
We identified the issue on our end: the server that makes the outbound calls to your URL did not support the SSL encryption method that your callback server uses. We have fixed that and we are in the process of pushing to the production environment very soon.
Also FYI:
The error message with 400 indicates the callback URL does not meet
request or does not exist. Please refer to the detail in
Speech-To-Text service API document,
http://www.ibm.com/watson/developercloud/speech-to-text/api/v1/?curl#register_callback
If the service does not receive a response with a response code of 200
and a body that echoes a random alphanumeric challenge string from the
callback URL within 5 seconds, it does not whitelist the URL; it
sends response code 400 in response to the registration request.
we just fixed the issue you reported. The problem was on our end, the servers responsible for making the callback to the server you set up did not support the cipher suites needed for establishing the SSL connection. We just updated the servers and we are happy to learn that it is now working for you: )
Dani
I have a REST API where the GET request can take 10-20 seconds. So I usually return a 202 code with a location like http://fakeserver/pending/blah where the client can check the status of this request. pending/blah returns a 200 code with "Status: pending" if the request is still pending, and a 303 code when it's done, with a final location for the result: http://fakeserver/finished/blah .
But what if the server crashes during the request processing? Should pending/blah return a 303 code, and then finished/blah returns a 404? How can I alert the client that the resource may be available at a location, but I'm not sure? Assume the requests are persistent, so that when the server reboots, it continues processing the request.
First of all I'll make the state of processed resource an internal field of this resource. This way you can avoid using strange endpoints like: /finished/blah/ or /pending/blah/ and instead of it introduce a single endpoint /resources/blah/ which will among other fields return the state it's currently in.
After changing architecture to the endpoint mentioned above if you ask for blah and server has crashed you can:
return 200 with pending status - client doesn't have necessarily to know about the crash
return 404, simple not found with and extra message that server has crashed.
return 500 and inform the client explicitly what the problem is.
Other useful codes may be also 409 or 503. Returning any 3XX is not a good idea IMO since no redirection applies here. Personally I'd go for 200 or 500(3).
I am wondering what status code would I response with in my else statement from the code below:
if (album) {
res.status(200).json({error: false, data: {channel_id: album.attributes.channel_id, id: album.id}});
} else {
res.status(200).json({error: false, data: {message: 'There is not album found with this name'}});
}
I don't want to leave both of them 200 as I want from front end to manage messaged thru status code, for ex if it returns 200 I would say "Created Successfully" while in else case I would display "No data found".
What is your suggestion?
"No data found" should be 404.
"Created Successfully" should be 201.
For the 201 you should also specify a Location header for where another request can access the new resource.
Refs:
201 http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.2
404 http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5
UPDATE:
I thought I'd expand on this, because the comments below point to thought processes I've battled with myself.
GET /some/thing responding 404 Not Found may mean a database entity not found, but could also mean there is no such API end point. HTTP itself doesn't do much to help differentiate these cases. The URL represents a resource. It's not a thing in itself to be considered differently from the thing it represents.
Some APIs respond 400 when a request is made to a non-existant endpoint, but personally I don't like this as it seems to contradict the way web servers respond to normal resource requests. It could also confuse developers building client applications, as it suggests something is wrong in the HTTP transport rather than in their own code.
Suppose you decide to use 404 for missing database entities and 400 for what you consider bad requests. This may work for you, but as your application grows you'll uncover more and more scenarios that simple status codes just can't communicate on their own. And this is my point..
My suggestion is that all API responses carry meaningful data in the body. (JSON or XML, or whatever you're using). A request for an invalid url may look like:
HTTP/1.1 404 Not Found
{ "error": "no_endpoint", "errorText":"No such API end point" }
Although, I agree with above post, I would also consider HTTP status 200 for some cases.
For example, you have Post and Post_Comments entities. When you request comments for give Post Id, you can have either have 404 (an error which you then need to handle on your REST API consumer side) or 200 which means that everything is OK and an empty array is returned. In the HTTP status 200 case, you do not need to handle an error. As an example, see how FB treats HTTP codes https://apigee.com/about/blog/technology/restful-api-design-what-about-errors