Hi I have a requirement to use play framework 2.5 (scala) to receive a large request body, then transform it and then stream it straight back out.
So far I've been unable to get the request stream to be sent out correctly using a chunked response (even untransformed).
Code example:
def endpointA = EssentialAction { requestHeader =>
Accumulator.source.map { source: Source[ByteString, _] =>
Ok.chunked(source)
}
}
POSTing data to the endpoint with curl does not output the posted data as expected and just results in the error below. I confirmed with wireshark that no response body is sent.
curl -v --data 'hello' -H "Connection: Keep-Alive" -H "Keep-Alive: 300" -H "Content-type: text/plain" http://localhost:9584/binding-tariff-admin/upload-csv
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9584 (#0)
> POST /binding-tariff-admin/endpoint-a HTTP/1.1
> Host: localhost:9584
> User-Agent: curl/7.64.1
> Accept: */*
> Connection: Keep-Alive
> Keep-Alive: 300
> Content-type: text/plain
> Content-Length: 5
>
* upload completely sent off: 5 out of 5 bytes
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Cache-Control: no-cache,no-store,max-age=0
< Content-Security-Policy: default-src 'self' 'unsafe-inline' *.s3.amazonaws.com www.google-analytics.com data:
< X-Permitted-Cross-Domain-Policies: master-only
< Content-Type: application/octet-stream
< Date: Wed, 19 Feb 2020 10:15:36 GMT
<
* transfer closed with outstanding read data remaining
* Closing connection 0
curl: (18) transfer closed with outstanding read data remaining
Also, if I change the code to return a stream I create myself, it works fine:
val testStream: Source[ByteString, NotUsed] = Source(List("hello")).map(ByteString.apply)
Is there something fundamentally wrong with what I'm trying to do here? I have seen other stack overflow examples with people suggesting this should be possible, e.g.
Play Framework Scala: How to Stream Request Body
I also tried using the verbatimBodyParser method described in the link but got the same results.
Thanks!
NFV
Related
Hi there I want get Last_modified from http header with dart.io HttpClient()
code sample is:
var client = new HttpClient();
HttpClientRequest req = await client.getUrl(Uri.parse("sayagh.asnafhormozgan.ir/wp-content/tables/essentials.csv"));
var a = req.headers.value("lastModifiedHeader");
but a returns null;
how I can get Last modified?
but when I get it with curl:
curl -v "sayagh.asnafhormozgan.ir/wp-content/tables/drawer.csv"
* Trying 51.89.173.235:80...
* TCP_NODELAY set
* Connected to sayagh.asnafhormozgan.ir (51.89.173.235) port 80 (#0)
> GET /wp-content/tables/drawer.csv HTTP/1.1
> Host: sayagh.asnafhormozgan.ir
> User-Agent: curl/7.65.3
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx
< Date: Thu, 19 Sep 2019 10:05:37 GMT
< Content-Type: text/csv
< Content-Length: 599
< Connection: keep-alive
< Last-Modified: Thu, 19 Sep 2019 09:38:30 GMT
< Accept-Ranges: bytes
< Cache-Control: max-age=0
< Expires: Thu, 19 Sep 2019 10:05:37 GMT
<
You'll need to wait for the second future to get the response, i.e.:
A getUrl request is a two-step process, triggered by two Futures. When
the first future completes with a HttpClientRequest, the underlying
network connection has been established, but no data has been sent. In
the callback function for the first future, the HTTP headers and body
can be set on the request. Either the first write to the request
object or a call to close sends the request to the server.
See https://api.dartlang.org/stable/2.5.0/dart-io/HttpClient-class.html
Also it should be 'last-modified' and not 'lastModifiedHeader' (or even better use the static const variable HttpHeaders.lastModifiedHeader), e.g.:
HttpClient client = HttpClient();
HttpClientRequest req = await client.getUrl(Uri.parse(
'http://sayagh.asnafhormozgan.ir/wp-content/tables/essentials.csv'));
HttpClientResponse response = await req.close();
print(response.headers.value(HttpHeaders.lastModifiedHeader));
I am trying to develop a rest api for my service wherein I set custom http status code depending on authorisation failure. But when I test it out using curl I am receiving 404 instead of 403. I am perplexed as to what might be causing this? Please help.
This is what I see from curl output or swagger UI:
root#ubuntu:~# curl -X GET http://localhost:8082/mr/v1/topic/bhakk -v
Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8082 (#0)
> GET /mr/v1/topic/bhakk HTTP/1.1
> Host: localhost:8082
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Mon, 07 May 2018 22:00:10 GMT
< Exception: serviceBlockedException
< Content-Type: application/vnd.kafka.v1+json
< Content-Length: 83
< Server: Jetty(9.2.z-SNAPSHOT)
<
* Connection #0 to host localhost left intact
{"error_code":40301,"message":"This service does not have access to the resource."}
Here is the code:
public Collection<String> list(#HeaderParam("x-nssvc-serviceid") String serviceID) {
Date now = new java.util.Date();
if (! ctx.getSecurityRestrictions().isServiceAllowed(uri, httpHeaders, "Describe", "Cluster", "kafka-cluster"))
throw Errors.serviceBlockedException(ctx,httpServletResponse);
List<String> topicsCopy = new ArrayList<String>(topics);
for (Iterator<String> iterator = topicsCopy.iterator(); iterator.hasNext();) {
String topic = iterator.next();
if (! ctx.getSecurityRestrictions().hasAccess (serviceId, "Describe", "Topic", topic)) {
iterator.remove();
}
}
return topicsCopy;
}
public static RestException serviceBlockedException(Context ctx,HttpServletResponse httpServletResponse) {
httpServletResponse.setHeader("Exception","serviceBlockedException");
httpServletResponse.setStatus(Status.FORBIDDEN.getStatusCode()); <----// here i am setting status code.
return new RestNotFoundException(SERVICE_ID_BLOCKED_MESSAGE, SERVICE_ID_BLOCKED_ERROR_CODE);
}
Kafka sets the Response 404 status in its RestNotFoundException
See: https://github.com/confluentinc/rest-utils/blob/master/core/src/main/java/io/confluent/rest/exceptions/RestNotFoundException.java
I am trying to figure out how to sync particular outlook account emails. I am using the http rest endpoints (here is a link to the documentation I have been following: https://msdn.microsoft.com/en-us/office/office365/api/mail-rest-operations#synchronize-messages).
I found out that if a user has "ported" their existing email (lets say gmail for now) to outlook, outlook assigns them a special email. Usually in the form of the following:
testaccount#gmail.com -> outlook_some hash#outlook.com
Now, I have been having some trouble syncing these "ported" users (i'll call them that for now) and every time I hit the sync api https://outlook.office.com/api/v2.0/Users/*insert email address here*/MailFolders/AllItems/messages, I get the following error:
{ error:
{ code: 'ErrorServiceUnavailable',
message: 'Active Directory operation did not succeed. Try again later.' } }
and the status code of the request is 503.
My request is pretty simple:
export ACCESS_TOKEN=here; export EMAIL=here; curl -X GET -H "Authorization: Bearer $ACCESS_TOKEN" \
-H "Accept: application/json" \
-H "Cache-Control: no-cache" \
-H "Postman-Token: 4455c965-0d49-610a-48e8-7ab37a20c111" \
"https://outlook.office.com/api/v2.0/Users/$EMAIL/MailFolders/AllItems/messages"
Response:
< HTTP/1.1 503 Service Unavailable
< Cache-Control: private
< Transfer-Encoding: chunked
< Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
< Server: Microsoft-IIS/8.5
< Set-Cookie: exchangecookie=**cookie**; expires=Fri, 16-Feb-2018 19:02:08 GMT; path=/; HttpOnly
< request-id: **request id**
< X-CalculatedFETarget: BN3PR0601CU001.internal.outlook.com
< X-BackEndHttpStatus: 503
< X-FEProxyInfo: BN3PR0601CA0025.NAMPRD06.PROD.OUTLOOK.COM
< X-CalculatedFETarget: namprd13-provisioning.internal.outlook.com
< X-BackEndHttpStatus: 503
< X-FEProxyInfo: DM3PR13CA0019.NAMPRD13.PROD.OUTLOOK.COM
< X-CalculatedBETarget: CY4PR13MB1078.namprd13.prod.outlook.com
< X-BackEndHttpStatus: 503
< OData-Version: 4.0
< X-AspNet-Version: 4.0.30319
< X-DiagInfo: CY4PR13MB1078
< X-BEServer: CY4PR13MB1078
< X-FEServer: DM3PR13CA0019
< X-FEServer: BN3PR0601CA0025
< X-Powered-By: ASP.NET
< X-FEServer: BY2PR11CA0016
< X-MSEdge-Ref: Ref A: 1DD7FE1BB0584157A03AB3615CBC3EB6 Ref B: BY1EDGE0414 Ref C: Thu Feb 16 11:02:08 2017 PST
< Date: Thu, 16 Feb 2017 19:02:07 GMT
<
* Curl_http_done: called premature == 0
* Connection #0 to host outlook.office.com left intact
{"error":{"code":"ErrorServiceUnavailable","message":"Active Directory operation did not succeed. Try again later."}}
This error only happens on these "ported" outlook accounts. These users have given me access by oauth2 (i have an access token and a refresh token).
any ideas how to actually sync these accounts? Seems like a bug on outlook's side seeing as my code works great for normal outlook accounts (like jerry_smith#hotmail.com, or ilovecats#outlook.com).
thanks in advance
Found the answer, if you change the url to
"https://outlook.office.com/api/v2.0/me/MailFolders/AllItems/messages"
(replace Users/$EMAIL with me)
it works, no 503.
I'm trying the Reverse routing sample code
Here are my routes
GET /hello/:name controllers.Application.hello(name)
GET /bob controllers.Application.helloBob
and my codes
def helloBob = Action {
Redirect(routes.Application.hello("Bob"))
}
def hello(name: String) = Action {
Ok("Hello " + name + "!")
}
I can get hello response
$ curl -v localhost:9001/hello/play
Hello play!
But, can't get "Bob" response after redirect?
$ curl -v localhost:9001/bob
* Trying ::1...
* Connected to localhost (::1) port 9001 (#0)
> GET /bob HTTP/1.1
> Host: localhost:9001
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 303 See Other
< Location: /hello/Bob
< Date: Fri, 18 Sep 2015 03:19:04 GMT
< Content-Length: 0
<
* Connection #0 to host localhost left intact
The path component of a URI is case sensitive. Check it.
try
curl -v localhost:9001/hello/Bob
Update
You code is correct (verified on my project) and you show the correct log - it prints 303 code. I think, you just need to say curl to follow redirect, like this
curl -L localhost:9000/bob
I want to write a curl command for a POST request equivalent to this Perl code:
use strict;
use warnings;
use LWP::UserAgent;
my $base = 'http://www.uniprot.org/mapping/';
my $params = {
from => 'ACC',
to => 'P_REFSEQ_AC',
format => 'tab',
query => 'P13368'
};
my $agent = LWP::UserAgent->new();
push #{$agent->requests_redirectable}, 'POST';
my $response = $agent->post($base, $params);
$response->is_success ?
print $response->content :
die 'Failed, got ' . $response->status_line .
' for ' . $response->request->uri . "\n";
I tried with (and many other variants) :
curl -X POST -H "Expect:" --form "from=ACC;to=P_REFSEQ_AC;format=tab; query=P13368" http://www.uniprot.org/mapping/ -o out.tab
The Perl code retrieves the expected result, but the curl command line does not. It retrieves the web page from "http://www.uniprot.org/mapping/" but does not make the POST request.
I looked for an error in the response header, but didn't find anything suspicious.
> POST http://www.uniprot.org/mapping/ HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: www.uniprot.org
> Accept: */*
> Proxy-Connection: Keep-Alive
> Content-Length: 178
> Content-Type: multipart/form-data; boundary=----------------------------164471d8347f
>
} [data not shown]
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: Apache-Coyote/1.1
< Vary: User-Agent
< Vary: Accept-Encoding
< X-Hosted-By: European Bioinformatics Institute
< Content-Type: text/html;charset=UTF-8
< Date: Wed, 05 Aug 2015 20:32:00 GMT
< X-UniProt-Release: 2015_08
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: origin, x-requested-with, content-type
< X-Cache: MISS from localhost
< X-Cache-Lookup: MISS from localhost:3128
< Via: 1.0 localhost (squid/3.1.20)
< Connection: close
<
I spent almost three days looking for a solution in the web, but nothing is working for me.
It looks like the server expects the data as application/x-www-form-urlencoded and not as multipart/form-data as you do with the --form argument. The following should work:
curl -v -L --data \
"from=ACC&to=P_REFSEQ_AC&format=tab&query=P13368" \
http://www.uniprot.org/mapping/ -o out.tab
With --data you get the expected content-type header, but you must do the encoding yourself. With -L curl follows a redirect which is needed here to get the resulting data.
The -X POST option is not needed since POST is the default method when sending data. And -H "Expect:" is not needed either.