I have an application which is uploading objects to Google Cloud Storage using signed URLs and I'd like to know if it's possible to make the object public during the sign/upload step.
I know it's possible to make the object publicly visible by setting the policy on its bucket or by using the client library/making a REST request after it's been uploaded, but in order to minimize the impact on my workflow, I'd like to do it all in one go. Is this possible? If it can be done, I'm assuming it's by setting a header when signing the URL or when making the REST request using the signed URL but I haven't been able to find documentation which covers this.
UPDATE:
I've just found the Extension/Custom Headers section of the XML API docs which claims that this can be achieved using the x-goog-acl header (e.g. x-goog-acl: public-read). Unfortunately, this does not work. The object is not publicly visible after setting the header when signing the URL and when uploading the file.
Quoting the Cloud Storage documenation regarding Signed URLs:
When specifying the name:value pairs for headers, keep in mind the following:
Remove any whitespace around the colon that appears after the header name.
For example, using the custom header x-goog-acl: private without removing the space after the colon returns a 403 Forbidden error, because the request signature you calculate does not match the signature Google calculates.
So the solution could be setting the header value as x-goog-acl:public-read instead of x-goog-acl: public-read.
Related
The aim is to connect to a public REST api using ADF. It's my first stab at sending requests to a REST api in ADF. It is the Companies House ('CH') governement website's API in England.
I have created an account and obtained a key. Apparently, it is basic authentication and the user name is the API key and password will be ignored (CH note on authentication)
I want to explore the contents of the 'Search all' API (CH note on Search All) and want to copy the results to Blob Storage.
I therefore set the linked service to use REST as below, the obfuscated User Name is the key I obtained from CH, the password is jsut the key repeated as their documentation states they ignore the password:
[
I then have added a REST dataset referencing this linked service:
And the testing of the connection works fine.
Problems then arise in the copy data task, I am getting an error when previewing and also when I attempt a copy to blob of 'Invalid Authorization Header':
I'd be grateful for pointers on where I'm going wrong.
I can't reproduce your Auth error but i notice that you want to add some parameters with your GET request in the Request Body.
I think you need to add parameters in relativeUrl property:
A relative URL to the resource that contains the data. When this
property isn't specified, only the URL that's specified in the linked
service definition is used. The HTTP connector copies data from the
combined URL: [URL specified in linked service]/[relative URL
specified in dataset].
Also i suggest you checking the correct REST API format of Search Api you are using.There is no other special features in the ADF REST connector. Just make sure the GET request works locally and duplicate it.
I've managed to add custom metadata to my public file stored in Google Cloud Bucket, but that custom header is not returned in the HTTP response.
The image below shows that my custom metadata (X-Content-Type-Options) was added to my object. When I request that file from my browser, this custom header is not part of the response.
It is possible to add custom headers, but they will be prefixed with x-goog-meta-. AWS S3 suffers from the same limitation. It seems that this is due to security reasons. The leanest solution I've found to overcome this limitation is to use an edge such as AWS Lambda Edge or Cloudflare Edge Workers. The idea is to rewrite the headers on the fly. In my case, that would mean catching all headers that start with x-goog-meta-, and removing that prefix.
Here is an article of somebody who did that with AWS Lambda Edge: https://medium.com/#tom.cook/edge-lambda-cloudfront-custom-headers-3d134a2c18a2
You can use the x-goog-meta- for setting the metadata to the object (some examples here for adding a single metadata or for adding it in a cp operation).
You can get the custom metadata with the gsutil command and the -L param. You can also recover the custom metadata with the HTTP request API (try it out here).
But the custom metadata aren't provided in your browser when you access to the object via the URL https://storage.cloud.google.com/.... You have to build a proxy which requests the object with Storage API (for getting the content and the custom metadata) and which provides the object with the expected metadata.
Got a problem, using GCS signed url feature (https://cloud.google.com/storage/docs/access-control/signed-urls) and want to force response content disposition.
I've used it once before in the past, but now I cannot seem to get it work and cannot find any documentation about Google deprecating this feature.
My current example url: https://storage.googleapis.com/examplebucket/examplefile.pdf?GoogleAccessId=account&Expires=expire&Signature=xx&response-content-disposition=inline
Signing is working and I can reach the file, just cannot force disposition.
If I upload file as Content-Disposition=attachment, then it wont make it inline and vice versa.
Any ideas?
For signed URLs, the contentDisposition metadata property, if specified, overrides the response-content-disposition query parameter. This is because the response-content-disposition query parameter is not part of the signature, which means that end users could change it.
If you want to specify response-content-disposition in the signed URL, clear the contentDisposition metadata property, and the value in the query parameter will be used.
I'm just getting started with Cloudinary, and I'm attempting to Uploading with a direct call to the API. Using the DHC REST Client (chrome extension), I put my request together per the instructions found here at Creating API authentication signatures. Here's a screenshot of that request and response.
I also tried...
adding quotes around all values except timestamp as shown in the example
making the request a multi-part request and attaching the image to the body as a "file"
deleting timestamp, api_key, and signature and instead replacing them with upload_preset to try and upload an unsigned image (yes, I created the preset)
And finally, I did try adding public_id even though it says it would assign one if not provided.
In all cases, I get the same error response... Missing required parameter - file
Can anyone tell me what I'm missing?
Ok, figured it out. These name/value pairs need to be added to the body of the request rather than the header. Here's what that would look like in the DHC client. Note that the upload_preset will not work for you... I only created it to test with. Also note that doing a signed request is accomplished the same way but with different parameters.
I'm designing a public API to my company's data. We want application developers to sign up for an API key so that we can monitor use and overuse.
Since the API is REST, my initial thought is to put this key in a custom header. This is how I've seen Google, Amazon, and Yahoo do it. My boss, on the other hand, thinks the API is easier to use if the key becomes merely a part of the URL, etc. "http://api.domain.tld/longapikey1234/resource". I guess there is something to be said for that, but it violates the principle of the URL as a simple address of what you want, and not how or why you want it.
Would you find it logical to put the key in the URL? Or would you rather not have to manually set HTTP headers if writing a simple javascript frontend to some data?
It should be put in the HTTP Authorization header. The spec is here https://www.rfc-editor.org/rfc/rfc7235
If you want an argument that might appeal to a boss: Think about what a URL is. URLs are public. People copy and paste them. They share them, they put them on advertisements. Nothing prevents someone (knowingly or not) from mailing that URL around for other people to use. If your API key is in that URL, everybody has it.
It is better to use API Key in header, not in URL.
URLs are saved in browser's history if it is tried from browser. It is very rare scenario. But problem comes when the backend server logs all URLs. It might expose the API key.
In two ways, you can use API Key in header
Basic Authorization:
Example from stripe:
curl https://api.stripe.com/v1/charges -u sk_test_BQokikJOvBiI2HlWgH4olfQ2:
curl uses the -u flag to pass basic auth credentials (adding a colon after your API key will prevent it from asking you for a password).
Custom Header
curl -H "X-API-KEY: 6fa741de1bdd1d91830ba" https://api.mydomain.com/v1/users
passing api key in parameters makes it difficult for clients to keep their APIkeys secret, they tend to leak keys on a regular basis.
A better approach is to pass it in header of request url.you can set user-key header in your code .
For testing your request Url you can use Postman app in google chrome by setting user-key header to your api-key.
I would not put the key in the url, as it does violate this loose 'standard' that is REST. However, if you did, I would place it in the 'user' portion of the url.
eg: http://me#example.com/myresource/myid
This way it can also be passed as headers with basic-auth.