curl command line equivalent to this perl code - perl

I want to write a curl command for a POST request equivalent to this Perl code:
use strict;
use warnings;
use LWP::UserAgent;
my $base = 'http://www.uniprot.org/mapping/';
my $params = {
from => 'ACC',
to => 'P_REFSEQ_AC',
format => 'tab',
query => 'P13368'
};
my $agent = LWP::UserAgent->new();
push #{$agent->requests_redirectable}, 'POST';
my $response = $agent->post($base, $params);
$response->is_success ?
print $response->content :
die 'Failed, got ' . $response->status_line .
' for ' . $response->request->uri . "\n";
I tried with (and many other variants) :
curl -X POST -H "Expect:" --form "from=ACC;to=P_REFSEQ_AC;format=tab; query=P13368" http://www.uniprot.org/mapping/ -o out.tab
The Perl code retrieves the expected result, but the curl command line does not. It retrieves the web page from "http://www.uniprot.org/mapping/" but does not make the POST request.
I looked for an error in the response header, but didn't find anything suspicious.
> POST http://www.uniprot.org/mapping/ HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: www.uniprot.org
> Accept: */*
> Proxy-Connection: Keep-Alive
> Content-Length: 178
> Content-Type: multipart/form-data; boundary=----------------------------164471d8347f
>
} [data not shown]
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: Apache-Coyote/1.1
< Vary: User-Agent
< Vary: Accept-Encoding
< X-Hosted-By: European Bioinformatics Institute
< Content-Type: text/html;charset=UTF-8
< Date: Wed, 05 Aug 2015 20:32:00 GMT
< X-UniProt-Release: 2015_08
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: origin, x-requested-with, content-type
< X-Cache: MISS from localhost
< X-Cache-Lookup: MISS from localhost:3128
< Via: 1.0 localhost (squid/3.1.20)
< Connection: close
<
I spent almost three days looking for a solution in the web, but nothing is working for me.

It looks like the server expects the data as application/x-www-form-urlencoded and not as multipart/form-data as you do with the --form argument. The following should work:
curl -v -L --data \
"from=ACC&to=P_REFSEQ_AC&format=tab&query=P13368" \
http://www.uniprot.org/mapping/ -o out.tab
With --data you get the expected content-type header, but you must do the encoding yourself. With -L curl follows a redirect which is needed here to get the resulting data.
The -X POST option is not needed since POST is the default method when sending data. And -H "Expect:" is not needed either.

Related

Play framework 2.5 will not stream the request into the response

Hi I have a requirement to use play framework 2.5 (scala) to receive a large request body, then transform it and then stream it straight back out.
So far I've been unable to get the request stream to be sent out correctly using a chunked response (even untransformed).
Code example:
def endpointA = EssentialAction { requestHeader =>
Accumulator.source.map { source: Source[ByteString, _] =>
Ok.chunked(source)
}
}
POSTing data to the endpoint with curl does not output the posted data as expected and just results in the error below. I confirmed with wireshark that no response body is sent.
curl -v --data 'hello' -H "Connection: Keep-Alive" -H "Keep-Alive: 300" -H "Content-type: text/plain" http://localhost:9584/binding-tariff-admin/upload-csv
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9584 (#0)
> POST /binding-tariff-admin/endpoint-a HTTP/1.1
> Host: localhost:9584
> User-Agent: curl/7.64.1
> Accept: */*
> Connection: Keep-Alive
> Keep-Alive: 300
> Content-type: text/plain
> Content-Length: 5
>
* upload completely sent off: 5 out of 5 bytes
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Cache-Control: no-cache,no-store,max-age=0
< Content-Security-Policy: default-src 'self' 'unsafe-inline' *.s3.amazonaws.com www.google-analytics.com data:
< X-Permitted-Cross-Domain-Policies: master-only
< Content-Type: application/octet-stream
< Date: Wed, 19 Feb 2020 10:15:36 GMT
<
* transfer closed with outstanding read data remaining
* Closing connection 0
curl: (18) transfer closed with outstanding read data remaining
Also, if I change the code to return a stream I create myself, it works fine:
val testStream: Source[ByteString, NotUsed] = Source(List("hello")).map(ByteString.apply)
Is there something fundamentally wrong with what I'm trying to do here? I have seen other stack overflow examples with people suggesting this should be possible, e.g.
Play Framework Scala: How to Stream Request Body
I also tried using the verbatimBodyParser method described in the link but got the same results.
Thanks!
NFV

ErrorServiceUnavailable when hitting the outlook http sync-messages endpoint

I am trying to figure out how to sync particular outlook account emails. I am using the http rest endpoints (here is a link to the documentation I have been following: https://msdn.microsoft.com/en-us/office/office365/api/mail-rest-operations#synchronize-messages).
I found out that if a user has "ported" their existing email (lets say gmail for now) to outlook, outlook assigns them a special email. Usually in the form of the following:
testaccount#gmail.com -> outlook_some hash#outlook.com
Now, I have been having some trouble syncing these "ported" users (i'll call them that for now) and every time I hit the sync api https://outlook.office.com/api/v2.0/Users/*insert email address here*/MailFolders/AllItems/messages, I get the following error:
{ error:
{ code: 'ErrorServiceUnavailable',
message: 'Active Directory operation did not succeed. Try again later.' } }
and the status code of the request is 503.
My request is pretty simple:
export ACCESS_TOKEN=here; export EMAIL=here; curl -X GET -H "Authorization: Bearer $ACCESS_TOKEN" \
-H "Accept: application/json" \
-H "Cache-Control: no-cache" \
-H "Postman-Token: 4455c965-0d49-610a-48e8-7ab37a20c111" \
"https://outlook.office.com/api/v2.0/Users/$EMAIL/MailFolders/AllItems/messages"
Response:
< HTTP/1.1 503 Service Unavailable
< Cache-Control: private
< Transfer-Encoding: chunked
< Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
< Server: Microsoft-IIS/8.5
< Set-Cookie: exchangecookie=**cookie**; expires=Fri, 16-Feb-2018 19:02:08 GMT; path=/; HttpOnly
< request-id: **request id**
< X-CalculatedFETarget: BN3PR0601CU001.internal.outlook.com
< X-BackEndHttpStatus: 503
< X-FEProxyInfo: BN3PR0601CA0025.NAMPRD06.PROD.OUTLOOK.COM
< X-CalculatedFETarget: namprd13-provisioning.internal.outlook.com
< X-BackEndHttpStatus: 503
< X-FEProxyInfo: DM3PR13CA0019.NAMPRD13.PROD.OUTLOOK.COM
< X-CalculatedBETarget: CY4PR13MB1078.namprd13.prod.outlook.com
< X-BackEndHttpStatus: 503
< OData-Version: 4.0
< X-AspNet-Version: 4.0.30319
< X-DiagInfo: CY4PR13MB1078
< X-BEServer: CY4PR13MB1078
< X-FEServer: DM3PR13CA0019
< X-FEServer: BN3PR0601CA0025
< X-Powered-By: ASP.NET
< X-FEServer: BY2PR11CA0016
< X-MSEdge-Ref: Ref A: 1DD7FE1BB0584157A03AB3615CBC3EB6 Ref B: BY1EDGE0414 Ref C: Thu Feb 16 11:02:08 2017 PST
< Date: Thu, 16 Feb 2017 19:02:07 GMT
<
* Curl_http_done: called premature == 0
* Connection #0 to host outlook.office.com left intact
{"error":{"code":"ErrorServiceUnavailable","message":"Active Directory operation did not succeed. Try again later."}}
This error only happens on these "ported" outlook accounts. These users have given me access by oauth2 (i have an access token and a refresh token).
any ideas how to actually sync these accounts? Seems like a bug on outlook's side seeing as my code works great for normal outlook accounts (like jerry_smith#hotmail.com, or ilovecats#outlook.com).
thanks in advance
Found the answer, if you change the url to
"https://outlook.office.com/api/v2.0/me/MailFolders/AllItems/messages"
(replace Users/$EMAIL with me)
it works, no 503.

Illegal response header Scala Spray pipelining

I'm getting the follow error when I am trying to pipeline a request from Scala spray
[play-akka.actor.default-dispatcher-14] INFO application - Pipelining chain request
[WARN] [03/19/2015 11:08:49.115] [application-akka.actor.default-dispatcher-2] [akka://application/user/IO-HTTP/group-0/0] Illegal response header: Illegal 'Access-Control-Allow-Origin' header: Unexpected end of input, expected $timesAccess$minusControl$minusAllow$minusOrigin (line 1, pos 1):
^
and here is where I am building the request:
val pipeline =
addCredentials(BasicHttpCredentials("API_KEY",
"API_SECRET")) ~> sendReceive
val response: Future[HttpResponse] = pipeline(Post(api,notification))
Logger.info("Pipelining chain request")
response
I don't really know much about Access Control Allow Origin. Do I need to add some sort of header to this request to get it to work?
The error itself means Access-Control-Allow-Origin header wasn't parsed correctly (see grammar). This header is pretty new and allows Cross Origin Resource Sharing. Examples of normal Access-Control-Allow-Origin(from here):
"Access-Control-Allow-Origin" in {
"Access-Control-Allow-Origin: *" =!= `Access-Control-Allow-Origin`(AllOrigins)
"Access-Control-Allow-Origin: null" =!= `Access-Control-Allow-Origin`(SomeOrigins(Nil))
"Access-Control-Allow-Origin: http://spray.io" =!= `Access-Control-Allow-Origin`(SomeOrigins(Seq("http://spray.io")))
}
I can guess that you may use some old version of spray, which doesn't support multiple origins or maybe it's related to this. Anyway, server returns a response with this header only if Origin header is specified in request (which means CORS initiation), so the problem should be solved by removing Origin header from it.
UPDATE: This is a bug of chain.com API, you use. If Origin header is not specified, they return Access-Control-Allow-Origin: (empty string) to you, so it's not parsable:
> curl -v https://api.chain.com/v2/notifications -X POST
> POST /v2/notifications HTTP/1.1
> User-Agent: curl/7.41.0
> Host: api.chain.com
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Methods: GET,POST,PATCH,PUT,DELETE,OPTIONS,HEAD
< Access-Control-Allow-Origin:
< Content-Type: text/plain; charset=utf-8
< Date: Sun, 22 Mar 2015 01:38:07 GMT
< Strict-Transport-Security: max-age=25920000; includeSubDomains
< Vary: Accept-Encoding
< Www-Authenticate: Basic realm="chain-api"
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Xss-Protection: 1
< Content-Length: 47
< Connection: keep-alive
<
{"code":"CH004","message":"Must authenticate"}
You have to specify some Origin as workaround:
>curl -v https://api.chain.com/v2/notifications -X POST -H "Origin: http://google.com"
> POST /v2/notifications HTTP/1.1
> User-Agent: curl/7.41.0
> Host: api.chain.com
> Accept: */*
> Origin: http://google.com
< HTTP/1.1 401 Unauthorized
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Methods: GET,POST,PATCH,PUT,DELETE,OPTIONS,HEAD
< Access-Control-Allow-Origin: http://google.com
< Content-Type: text/plain; charset=utf-8
< Date: Sun, 22 Mar 2015 01:39:10 GMT
< Strict-Transport-Security: max-age=25920000; includeSubDomains
< Vary: Accept-Encoding
< Www-Authenticate: Basic realm="chain-api"
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Xss-Protection: 1
< Content-Length: 47
< Connection: keep-alive

Zend Rest Api: putAction on POST method

I got a weird issue. I hope someone could help me.
I am new to Zend and I'm writing a RESTfull API.
When I run the curl command as POST method, it calls the putAction() function.
For example, I am run a curl command:
curl -X POST http://localhost/ws/user/post -v
Here is the response:
* About to connect() to localhost port 80 (#0)
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 80 (#0)
> POST /ws/user/post HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
> Host: localhost
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 22 Oct 2012 13:37:56 GMT
< Server: Apache/2.2.14 (Ubuntu)
< X-Powered-By: PHP/5.3.2-1ubuntu4.18
< Vary: Accept-Encoding
< Content-Length: 53
< Content-Type: text/html
<
* Connection #0 to host localhost left intact
"From putAction() updating the requested article"
Method = POST
* Closing connection #0
Here is the code:
[...]
public function postAction() {
echo 'Method = ' . $_SERVER['REQUEST_METHOD'];
if (!$id = $this->_getParam('id', false)) {
$this->sendResponse("Bad request. Id is missing", 400);
}
$this->sendResponse("From postAction() creating the requested article", 201);
}
public function putAction() {
echo 'Method = ' . $_SERVER['REQUEST_METHOD'];
$this->sendResponse("From putAction() updating the requested article");
}
[...]
Any ideas ?
EDIT:
I realized that I put this code in my bootstrap:
public function _initRestRoute() {
$front = Zend_Controller_Front::getInstance();
$router = $front->getRouter();
$restRoute = new Zend_Rest_Route($front, array(), array(
'ws',
));
$router->addRoute('rest', $restRoute);
}
When I comment it, it works.
Any explanations ? Thanks!
You shouldn't be having http://localhost/ws/user/post in your request while using the Zend_Rest_Route. This route interprets /user/post as parameters and changes action from post to put. Try http://localhost/ws instead.
See Zend/Rest/Route.php line 208.

adding a response header to 302 response using perl

i am trying to write a perl page that returns an http 302 response to a different location and adds a custom header to that response.
so my desired http response should be something like this:
HTTP/1.1 302 Moved
Date: Sun, 15 Apr 2012 10:59:02 GMT
Server: Apache
Location: http://www.google.com
Content-Length: 396
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1
CUSTOM_HEADER: CUSTOM_VALUE
i tried using CGI:
#!/bin/perl
use strict;
use APR::Request::Apache2;
my $r = shift;
$r->content_type('text/html; charset=utf-8');
$r->headers_out()->add("CUSTOM_HEADER", "CUSTOM_VALUE");
$r->headers_out()->add("Location", "http://www.google.com");
$r->status(302);
and i do get 302 response to google but no CUSTOM_HEADER. once i change the status to 200 by $r->status(200); i do get the CUSTOM_HEADER.
so whats up with this behavior ? how can i add my header to the 302 response ?
Use $r->err_headers_out->set or $r->err_headers_out->add
my $r = shift;
$r->content_type('text/html; charset=utf-8');
$r->err_headers_out->set(Location => "http://www.google.com");
$r->status(302);
You should use err_headers_out(). These will be printed even on errors and redirects.