How to match a result to a request when sending multiple requests? - guzzle

Question

A. Summary
As its title, Guzzle allows to send multiple requests at once to save time, as in documentation.
$responses = $client->send(array(
$requestObj1,
$requestObj2,
...
));
(given that each request object is an instance of
Guzzle\Http\Message\EntityEnclosingRequestInterface)
When responses come back, to identify which response is for which request, we can loop through each request and get the response (only available after executing the above command):
$response1 = $requestObj1->getResponse();
$response2 = $requestObj2->getResponse();
...
B. Problem
If the request object contains the same data. It's impossible to identify the original request.
Assume we have the following scenario where we need to create 2 articles: A and B on a distance server: something.com/articles/create.json
Each request has same POST data:
subject: This is a test article
After created, the Guzzle responses with 2 location come back:
something.com/articles/223.json
something.com/articles/245.json
Using the above method to link response-to-its-request, we still don't know which response is for which article, because the request object is exactly the same.
Hence in my database I cannot write down the result:
article A -> Location: 245.json
article B -> Location: 223.json
because it can be the other way arround:
article A -> Location: 223.json
article B -> Location: 245.json
A solution is to put some extra parameter in the POST request, e.g.
subject: This is a test article
record: A
However, the distance server will return error and does not create article because it does not understand the key "record". The distance server is a third party server and I cannot change the way it works.
Another proper solution for this is to set some specific id/tag on the request object, so we can identify it afterwards. However, I've looked through the documentation but there is no method to uniquely identity the request like
$request->setID("id1")
or
$request->setTag("id1")
This has been bugging me for months and still cannot resolve this issue.
If you have solution, please let me know.
Many many thanks and you've saved me!!!!
Thanks for reading this long post.

Answer 1

I've found a proper way to do it, Guzzle allow to add callback once a request is completed. So we can achieve this by setting it on each request in the batch
Each request by default can be created like this
$request = $client->createRequest('GET', 'http://httpbin.org', [
'headers' => ['X-Foo' => 'Bar']
]);
So, to achieve what we want:
$allRequests = [];
$allResults = [];
for($k=0; $k<=10; $k++){
$allRequests['key_'.$k] = $client->createRequest('GET', 'http://httpbin.org?id='.$k, [
'headers' => ['X-Foo' => 'Bar'],
'events' => [
'complete' => function ($e) use (&$allResults, $k){
$response = $e->getResponse();
$allResults['key_'.$k] = $response->getBody().'';
}
]
]);
}
$client->sendAll(array_values($allRequests));
print_r($allResults);
So now the $allResults has result for each corresponding request.
e.g. $allResults['key_1'] is the result of $allRequests['key_1']

Answer 2

I was having the same problem with this.
I solved it by adding a custom query parameter with a unique id generated for each request and add it to the request url (you will need to remember this ids for each one of them to address it after).
After $responses = $client->send($requests) you could iterate through the responses and retrieve the effective url $response->getEffectiveUrl() and parse it (see parse_url and parse_str) to get the custom parameter (with the unique id) and search in your array of requests which one has it.

Answer 3

I found a much better answer.
I was sending batches of 20 requests at a time, 4 concurrently, and used the pooling technique where I got fulfilled, and rejected back, as in the documentation.
I found that I could add this code to the end of my requestAsync() function calls, when yielding / building the array (I do both in different places).
$request = $request->then(function (\GuzzleHttp\Psr7\Response $response) use ($source_db_object) {
$response->_source_object = $source_db_object;
return $response;
});
And then in the clousures on the pool, I can just access the _source_object on the response normally, and it works great.
I find it a little hacky, but if you are just sure to use a name that NEVER clashes with anything in Guzzle, this should be fine.
Here is a full example:
use GuzzleHttp\Client;
use GuzzleHttp\Pool;
use GuzzleHttp\Psr7\Response as GuzzleResponse;
$client = new Client();
$requests = [];
// Simple set-up here, generate some random async requests
for ($i = 0; $i < 10; $i++) {
$request = $client->requestAsync('GET', 'https://jsonplaceholder.typicode.com/todos/1');
// Here we can attach any identifiable data
$request->_source_object = $i;
array_push($requests, $request);
}
$generator = function () use($requests) {
while ($request = array_pop($requests)) {
yield function() use ($request) {
return $request->then(function (GuzzleResponse $response) use ($request) {
// Attach _source_object from request to the response
$response->_source_object = $request->_source_object ?? [];
return $response;
});
};
}
};
$requestPool = new Pool($client, $generator(), [
'concurrency' => 5,
'fulfilled' => function ($response) {
// Then we can properly access the _source_object data once response has arrived here!
echo $response->_source_object . "\n";
}
]);
$requestPool->promise()->wait();

Answer 4

I do it this way :
// create your requests
$requests[] = $client->createRequest('GET', '/endpoint', ['config' => ['order_id' => 123]]);
...
// in your success callback get
$id = $event->getRequest()->getConfig()['order_id']

Answer 5

An update related to the new GuzzleHttp guzzlehttp/guzzle
Concurrent/parallel calls are now run through a few different methods including Promises.. Concurrent Requests
The old way of passing a array of RequestInterfaces will not work anymore.
See example here
$newClient = new \GuzzleHttp\Client(['base_uri' => $base]);
foreach($documents->documents as $doc){
$params = [
'language' =>'eng',
'text' => $doc->summary,
'apikey' => $key
];
$requestArr[$doc->reference] = $newClient->getAsync( '/1/api/sync/analyze/v1?' . http_build_query( $params) );
}
$time_start = microtime(true);
$responses = \GuzzleHttp\Promise\unwrap($requestArr); //$newClient->send( $requestArr );
$time_end = microtime(true);
$this->get('logger')->error(' NewsPerf Dev: took ' . ($time_end - $time_start) );
In this example you will be able to refer to each of the Responses using $requestArr[$doc->reference] . In short give an index to your array and run the Promise::unwrap call.

Answer 6

I also had come across this issue. This was the first thread coming up. I know this is a resolved thread, but I have eventually come up with a better solution. This is for all those who might encounter the issue.
One of the options is to use Guzzle Pool::batch.
What batch does is, it pushed the results of pooled requests into an array and returns the array. This ensures that the response and requests are in the same order.
$client = new Client();
// Create the requests
$requests = function ($total) use($client) {
for ($i = 1; $i <= $total; $i++) {
yield new Request('GET', 'http://www.example.com/foo' . $i);
}
};
// Use the Pool::batch()
$pool_batch = Pool::batch($client, $requests(5));
foreach ($pool_batch as $pool => $res) {
if ($res instanceof RequestException) {
// Do sth
continue;
}
// Do sth
}