I'm currently trying to use cassandra_snapshotter with cloud storage in interoperability mode. However event the first request to upload a file fails with a signature mismatch.
Did a small testcase, as even the first request is failing:
import boto
from boto.s3.connection import S3Connection
connection = S3Connection(
aws_access_key_id='GOOGY',
aws_secret_access_key='SECRET',
host='storage.googleapis.com'
)
bucket = connection.get_bucket('foo-cassandra-backups', validate=False)
mp = bucket.initiate_multipart_upload('/g/mnt/cassandra-data2/data/bar/baz-63b205e0618711e7a41cf5e393c7464c/snapshots/20170705150900/manifest.json.lzo')
This fails with:
send: 'POST /g/mnt/cassandra-data2/data/foo/threshold-63b205e0618711e7a41cf5e393c7464c/snapshots/20170705150900/manifest.json.lzo?uploads HTTP/1.1\r\nHost: blubb-cassandra-backups.storage.googleapis.com\r\nAccept-Encoding: identity\r\nDate: Thu, 06 Jul 2017 11:18:46 GMT\r\nContent-Length: 0\r\nAuthorization: AWS BLUBB:F6Ml3vSzDphY7UTGNuf3m+fe19I=\r\nUser-Agent: Boto/2.47.0 Python/2.7.12 Linux/4.4.0-83-generic\r\n\r\n'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: X-GUploader-UploadID: BLUBB
header: Content-Type: application/xml; charset=UTF-8
header: Content-Length: 447
header: Vary: Origin
header: Date: Thu, 06 Jul 2017 11:18:46 GMT
header: Server: UploadServer
Traceback (most recent call last):
File "foo.py", line 11, in <module>
mp = bucket.initiate_multipart_upload('/g/mnt/cassandra-data2/data/foo/threshold-63b205e0618711e7a41cf5e393c7464c/snapshots/20170705150900/manifest.json.lzo')
File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 1767, in initiate_multipart_upload
response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
<?xml version='1.0' encoding='UTF-8'?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message><StringToSign>POST
Thu, 06 Jul 2017 11:18:46 GMT
/blubb-cassandra-backups/g/mnt/cassandra-data2/data/foo/threshold-63b205e0618711e7a41cf5e393c7464c/snapshots/20170705150900/manifest.json.lzo</StringToSign></Error>
So i guess the API is not compatible in this case and i'm relatively lost here on how to properly fix it. Additionally i'd like to avoid big changes to the uploader, as we would need to deploy the patches as well.
EDIT: This is with boto 2.47 and i guess it's because AWS requires the ?uploads param but googles API is different and therefore it's not expected in the signature.
Google Cloud Storage does not support the multi-part upload feature of the S3 API, so I would suggest using the regular upload approach (ex. set_contents_from_file()).
Related
I have a server using the google Drive API. I tried with a curl PUT request to upload a simple file (test.txt) at http://myserver/test.txt. As you can see, I did the PUT request at the root of my server. The response I get is the following:
HTTP/1.1 200 OK
X-GUploader-UploadID: AEnB2UqANa4Bj6ilL7z5HZH0wlQi_ufxDiHPtb2zq1Gzcx7IxAEcOt-AOlWsbX1q_lsZUwWt_hyKOA3weAeVpQvPQTwbQhLhIA
ETag: "6e809cbda0732ac4845916a59016f954"
x-goog-generation: 1548877817413782
x-goog-metageneration: 1
x-goog-hash: crc32c=jwfJwA==
x-goog-hash: md5=boCcvaBzKsSEWRalkBb5VA==
x-goog-stored-content-length: 6
x-goog-stored-content-encoding: identity
Content-Type: text/html; charset=UTF-8
Accept-Ranges: bytes
Via: 1.1 varnish
Content-Length: 0
Accept-Ranges: bytes
Date: Wed, 30 Jan 2019 19:50:17 GMT
Via: 1.1 varnish
Connection: close
X-Served-By: cache-bwi5139-BWI, cache-cdg20732-CDG
X-Cache: MISS, MISS
X-Cache-Hits: 0, 0
X-Timer: S1548877817.232336,VS0,VE241
Vary: Origin
Access-Control-Allow-Methods: POST,PUT,PATCH,GET,DELETE,OPTIONS
Access-Control-Allow-Headers: Cache-Control,X-Requested-With,Authorization,Content-Type,Location,Range
Access-Control-Allow-Credentials: true
Access-Control-Max-Age: 300
I know you're not supposed to use the API that way. I did that for testing purposes. I understand every headers returned but can't figure out if my file has been uploaded because I don't have enough knowledge of this API.
My question is very simple :
Just by looking at the response, can you tell me if my file has been uploaded ?
If yes can I retrieve it and how ?
The HTTP status code traditionally indicates, for any given request, if it was successful. The status code in the response is always on the first line:
HTTP/1.1 200 OK
200 type status codes mean success. You should take some time to familiarize yourself with HTTP status codes if you intend to work with HTTP APIs.
I'm trying to communicate with IBM Rational Quality Manager server using its REST API. I'm using RESTClient browser plugin, and while the browser is logged in, everything works as expected. For the record, my requests look like
https://server/qm/service/com.ibm.rqm.integration.service.IIntegrationService/resources/project/testscript/urn:com.ibm.rqm:testscript:42
However, if I wait long enough for RQM to logout, REST API says I need to login back to proceed (see below). I'm pretty sure this is possible to do via the API itself, because RQM ships with RQMUrlUtility which accepts username and password and runs basically the same REST requests I'm using:
java -jar RQMUrlUtility.jar -command GET -user JazzUserID -password JazzPassword -filepath pathtoFile -url REST_URL
So far, I have found this topic explaining how to login using HTTP basic authentication. Following this advice, I have added Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ= (not my real password) to the request, but RQM still fails to login. I have also tried setting User-Agent to a bogus value, as well as sending the value from JSESSIONID in X-Jazz-CSRF-Prevent header as described here, but regardless of all these headers being present or not, I get the same response:
Status Code: 200 OK
Cache-Control: no-cache="set-cookie, set-cookie2"
Connection: Keep-Alive
Content-Encoding: gzip
Content-Language: en-US
Content-Type: text/html; charset=UTF-8
Date: Tue, 26 Jan 2016 15:48:02 GMT
Expires: Thu, 01 Dec 1994 16:00:00 GMT
Keep-Alive: timeout=10, max=100
Set-Cookie: JazzFormAuth=Form; Path=/qm; Secure
x-com-ibm-team-scenario=ac55f959-c738-4ef0-854d-6e37648edcba%3Bname%3DInitial+Page+Load%3Bextras%3D%2Fqm%2Fauth%2Fauthrequired%2C1453823282026; Path=/
Transfer-Encoding: chunked
X-Powered-By: Servlet/3.0
X-com-ibm-team-repository-web-auth-msg: authrequired
Can anyone with experience with RQM API tell me what's wrong? Or perhaps I'm missing something basic, common to most RESP APIs out there?
Could it be your header name?
Authorisation: Basic dXNlcm5hbWU6cGFzc3dvcmQ=
Should probably be:
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=
Notice the "z".
I'm in Amazon's API Gateway, and any change to the Mapping Templates section of the Integration Response breaks the Resource/Method (causes the Test to return an error) and cannot be fixed (you must delete the Resource/Method and create a new one).
I create a new Resource, then create a new Method (POST) under that.
I map this to a simple Lambda function (it doesn't require any parameters and only returns/logs 'hi').
I test this, and it succeeds.
I go into Integration Response, and I change the Mapping Templates ... I change application/json to application/xml and I change 'Output passthrough' to 'Mapping template'.
I enter this as the template:
#set($inputRoot = $input.path('$'))
<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Message>
<Body>
$inputRoot
</Body>
</Message>
</Response>
I save that by clicking the checkbox and by clicking the Save button.
I go back to test it
This is the result:
{
"message": "Internal server error"
}
This is the content of the Logs output (I replaced potentially sensitive information with [explanation here] since I'm not sure what's sensitive or not):
Execution log for request test-request
Sun Dec 06 17:33:50 UTC 2015 : Starting execution for request: test-invoke-request
Sun Dec 06 17:33:50 UTC 2015 : API Key: test-invoke-api-key
Sun Dec 06 17:33:50 UTC 2015 : Method request path: {}
Sun Dec 06 17:33:50 UTC 2015 : Method request query string: {}
Sun Dec 06 17:33:50 UTC 2015 : Method request headers: {}
Sun Dec 06 17:33:50 UTC 2015 : Method request body before transformations: null
Sun Dec 06 17:33:50 UTC 2015 : Endpoint request URI: [lambda uri here]
Sun Dec 06 17:33:50 UTC 2015 : Endpoint request headers: {Authorization=[lots of * here], X-Amz-Date=20151206T173350Z, X-Amz-Source-Arn=[arn here], Accept=application/json, User-Agent=AmazonAPIGateway_[string here], Host=lambda.us-east-1.amazonaws.com}
Sun Dec 06 17:33:50 UTC 2015 : Endpoint request body after transformations:
Sun Dec 06 17:33:50 UTC 2015 : Endpoint response body before transformations: "hi"
Sun Dec 06 17:33:50 UTC 2015 : Endpoint response headers: {x-amzn-Remapped-Content-Length=0, x-amzn-RequestId=[data here], Connection=keep-alive, Content-Length=12, Date=Sun, 06 Dec 2015 17:33:50 GMT, Content-Type=application/json}
Sun Dec 06 17:33:50 UTC 2015 : Execution failed due to configuration error: No match for output mapping and no default output mapping configured
Sun Dec 06 17:33:50 UTC 2015 : Method completed with status: 500
I also tried going into Method Response, and changing the Content Type for Response Models for 200 from application/json to application/xml ... That produced the same error.
I also tried, at this point, to revert my changes ... Method Response back to application/json & Integration Response back to application/json and 'Output passthrough' ... That produced the same error - it's like this API Resource/Method is now permanently broken.
I also tested another new Resource, changing only the Content Type for Response Models for 200 in Method Response from 'application/json' to 'application/xml' ... This resulted in a successful test.
I also tried a more minor change to the Mapping Templates in Integration Response ... Rather than a full switch from 'Output passthrough' to 'Mapping Template', I just changed the content type from application/json to application/xml ... This resulted in the same error.
So it seems like the root cause is changing from Output Passthrough to Mapping Template ... Once that change is made, the test will fail & you will not be able to return it to a passing state - you must delete the Resource/Method entirely & start a new one.
Also, to be clear, there are no deploys throughout any of this process - I'm strictly working in the AWS console itself, using their 'Test' link in the web interface.
Anyone know what's going on here?
I'm also trying to get an answer in their Discussion Forum, but those threads usually aren't nearly as active as here...
Added Note
I do have a functional deploy running, which uses this mapping template. That deploy is from 18:35 12-05-2015, so it's possible that this is a new error/change in the Amazon API Gateway...
This must have been a temporary issue with API Gateway, because the issue is gone now.
There's one caveat:
NEVER click the big Save button on the Integration Response page. That seems to cause issues, at least as of today (2015-12-05).
I spoke with amazon's support staff and its a known issue. As long as you don't press the Save button you should be fine but once you do there is no going back.
Just press the checkbox thing when making changes to the template and refresh the page. That seems to work for me.
There was an issue with APIs when saving the default integration response mapping. The bug caused requests to your API methods that were saved incorrectly to return a 500 error, the CloudWatch logs should say "Execution failed due to configuration error: No match for output mapping and no default output mapping configured".
The issue is now resolved. If you are experiencing this, please re-deploy your API configuration.
For more information, please refer to this AWS forums entry: https://forums.aws.amazon.com/thread.jspa?threadID=221197&tstart=0
Regards,
Jurgen
The issue is now resolved. It's now safe to save the default integration response. If your deployed APIs are having issues, a redeploy should resolve the problem. Thanks for your patience.
Ryan
I'm trying to search through repositories, but I can't seem to figure it out with github enterprise edition. I have tried the following with no results. Any suggestions?
curl -i http://my.domain.com/api/v3/repositories "If-Modified-Since: Mon, 16 Jun 2014 01:01:01 CST"
curl -i http://my.domain.com/api/v3/search/repos?q=pushed:2014-06-17
HTTP/1.1 404 Not Found
Server: GitHub.com
Date: Wed, 18 Jun 2014 16:45:58 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Status: 404 Not Found
X-GitHub-Media-Type: github.beta
X-Content-Type-Options: nosniff
Content-Length: 29
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: ETag, Link, X-RateLimit-Limit, X-RateLimit-Remaining, X- RateLimit-Res
et, X-OAuth-Scopes, X-Accepted-OAuth-Scopes
Access-Control-Allow-Origin: *
X-GitHub-Request-Id: b4eec0e7-1b1a-48b7-81d8-d63c28b55b37
{
"message": "Not Found"
}
One of the nice things of Github's API both public and Enterprise, is if you go to the API root, it will tell you what endpoints are available. On an enterprise instance it is: http://my.domain.com/api/v3/. Looking at my company's enterprise instance (sorry not sure of the version), I only see the legacy search API endpoints.
As a result: http://my.domain.com/api/v3/legacy/repos/search/pushed:2014-06-17 is likely the search URL you are wanting.
The Google Cloud Storage Developer Guide explains how to set Cache-Control headers, and explains their critical impact on the consistency behavior of the api, yet the Expires headers aren't mentioned nor did they appear to be inheriting from the Cache-Control configuration.
The Expires header appeared to always be equal to request time plus 1 year, regardless of Cache-Control setting, eg.
$ gsutil setmeta -h "Cache-Control:300" gs://example-bucket/doc.html
A request was made to a document (doc.html) in the Google Cloud Storage bucket (example-bucket) via
$ curl -I http://example-bucket.storage.googleapis.com/doc.html
which produced the following headers
HTTP/1.1 200 OK
Server: HTTP Upload Server Built on Oct 3 2012 16:52:30 (1349308350)
Date: Sat, 13 Oct 2012 00:51:13 GMT
Cache-Control: 300, no-transform
Expires: Sun, 13 Oct 2013 00:51:13 GMT
Last-Modified: Fri, 12 Oct 2012 20:08:41 GMT
ETag: "28fafe4213ae34c7d3ebf9ac5a6aade8"
x-goog-sequence-number: 82
x-goog-generation: 1347601001449082
x-goog-metageneration: 1
Content-Type: text/html
Accept-Ranges: bytes
Content-Length: 7069
Vary: Origin
Not sure why you say the Expires header shows request time plus 1 year. In your example, the Expires header shows a timestamp one hour after the request date, which is to be expected.
I just did an experiment where I set an object's max age to 3600 and then 7200 via this command:
gsutil setmeta "Cache-Control:max-age=7200" gs://marc-us/xyz.txt
Then I retrieved the object using the gsutil cat command with the -D option to see the request/response details, like this:
gsutil -D cat gs://marc-us/xyz.txt
In both experiments, the Expires header produced the expected timestamp, as per the object's max-age setting (i.e. one hour after request time and two hours after request time).
Looks like this was caused by a malformed header. Duh.
Cache-Control: 300, no-transform
should be
Cache-Control: public, max-age=300, no-transform
When things are set correctly, they work. See RFC 2616 (HTTP/1.1) Section 14.9 (Cache-Control).