I'm working in an Asp.Net Core 2 web api for files hosted at Google Cloud Storage. The files hosted there are not public, so I can't use the MediaLink property of the object. I tried to make a download endpoint using MemoryStream but when there are many users downloading large files at once I run into memory issues.
My question is: is there a way to create something link a one-time download link for a file or something similar?
I'm also trying to implement what's described in this link but I'd need to give the bearer token to the user. I can't do that.
Any tips?
Yes. Google Cloud Storage offers a feature called "signed URLs" that is what you described: a URL that is only good for a short while to download a single file. The idea is that you craft a download URL, then use the private key of a service account to "sign" the URL. Anyone holding that final URL can use it to act as that service account for the purpose of downloading that one object.
Take a look: https://cloud.google.com/storage/docs/access-control/#Signed-URLs
Writing code to generate the signed URL is a bit tricky, but the client libraries provide helper methods in several languages to do it for you. You can also generate one with the gsutil command: gsutil signurl -d 10m privatekey.p12 gs://bucket/foo
There is a code sample for generating he signed URLs programatically on their GitHub project: Signed URLs
I managed to Create it using C#. I'm posting here because this will be useful to someone else:
1 - Create your private key
2 - Create and UrlSigner:
private readonly UrlSigner _urlSigner;
2 - In your class constructor:
using (var stream = File.OpenRead(_googleSettings.StorageAuthJson))
{
_urlSigner = UrlSigner.FromServiceAccountData(stream);
}
_googleSettings.StorageAuthJson has the physical path of the json file you downloaded when creating your key.
3 - Method to get the URL:
public string GetSignedUrl(string bucketName, string objectName, TimeSpan duration) {
var url = _urlSigner.Sign(bucketName, objectName, duration, null);
return url;
}
Related
Wondering if someone knows how to upload an image from flutter (not really relevant, javascript would be the same problem) but set the image path/name from the server side.
When I upload my image to my s3 bucket in AWS amplify, the name the client passes is taken to be put in the bucket.
I would like to change this behavior by passing the image in the api call but then on the server set the path and name.
Has anyone had any luck doing this?
Best regards
I'm assuming you have already gone through the official documentation on uploading via flutter client which looks something like this.
final UploadFileResult result = await Amplify.Storage.uploadFile(
local: exampleFile,
key: 'ExampleKey',
onProgress: (progress) {
print('Fraction completed: ${progress.getFractionCompleted()}');
}
);
here, you can change key to anything of your choice e.g.
key: 'folder1/image1.jpg'
If this is not what you want to do then you can always create a new REST endpoint on your server which accepts multipart/form-data. nodejs/express example
Simple file upload to S3 using aws-sdk and Node/Express
Problem:
I need to be able to remove all link decoration from the download URL that is generated for images in Firebase Storage.
However, when all link decoration is stripped away, the resulting link currently would return a JSON document of the image's metadata.
Context:
The flow goes as follows:
An image is uploaded to Firebase from an iOS app. Once that is done the download URL is then sent in a POST request to an external server.
The server that the URL is being sent to doesn't accept link decoration when submitting image URLs.
Goal:
Alter the Firebase Storage download URL such as it is stripped of all link decoration like so:
https://firebasestorage.googleapis.com/v0/b/example.appspot.com/o/[FOLDER_NAME]%[IMAGE_NAME].jpg
Notes:
The problem is twofold really, first the link needs to be manipulated to remove all the link decoration. Then the behavior of the link needs to changed, since in order to return an image, you need ?alt=media following the file extension, in this case .jpg. Currently, without link decoration, using the link with my desired structure would return a JSON document of the metadata.
The current link structure is as follows:
https://firebasestorage.googleapis.com/v0/b/example.appspot.com/o/[FOLDER_NAME]%[IMAGE_NAME].jpg?alt=media&token=[TOKEN]
Desired link structure:
https://firebasestorage.googleapis.com/v0/b/example.appspot.com/o/[FOLDER_NAME]%[IMAGE_NAME].jpg
The token is necessary for accessing the image depending security rules in place, but can be ignored with the proper read permissions. I can adjust the rules as needed, but I still need to be able to remove the ?alt=media and still return an image.
Building up on Frank's answer, if you access to your associated Google Cloud Platform project, find the bucket in the Storage tab and make this bucket public, you will be able to get the image from here with the format you wish. That is, you will not be accessing through Firebase
https://firebasestorage.googleapis.com/v0/b/example.appspot.com/o/[FOLDER_NAME]%[IMAGE_NAME].jpg
but through Google Cloud Storage, with a link like
https://storage.googleapis.com/[bucket_name]/[path_to_image]
Once in your GCP project Console, access the Storage bucket with the same name as the one you have in your Firebase project. They are the same bucket. Then make the bucket public by following these steps. After that, you will be able to construct your links as mentioned above and they will be accessible with no token and no alt=media param. If you do not want to make the public to everyone, you will be able to play around with the permissions there as you wish.
You could split the url string into two halves by using String.componentsSeparatedByString(_ separator:)
Storage.storage().reference().child(filePath).downloadURL(completion: { (url, error) in
let urlString = url.absoluteString
let urlStringWithoutQueryString = urlString.componentsSeparatedByString("?").first!
})
Calling .downloadURL on a StorageReference will return you that URL, but this method can be used to remove the query string from any URL. String.componentsSeparatedByString(_ separator:) breaks a String into an array of Strings, splitting the string by any occurrence of a given separator, in this case ?.
NOTE this method assumes that ? occurs only once within the url string, which I believe is the case for all Firebase Storage urls.
You should treat the download URL that you get back from Firebase as an opaque string. There's no way to strip the parameters from a download URL without breaking that download URL.
If you want to allow public access to the files in your bucket with simpler URLs, consider making the object in your (or even your entire) bucket public.
i would like to ask, if there's any way to retrieve files links of folder which is publicly shared. Like someone create random public folder(everyone can view it) and put some random files into it. So i need to get all files links from that folder. All i know is link to that folder in format: https://www.dropbox.com/sh/[code]/[code].
Can i do that by using dropbox api, or the only option is to scrape dropbox page directly?
Here is a copy paste example:
using Dropbox.Api;
using Dropbox.Api.Files;
...
// AccessToken - get it from app console
// FolderToDownload - https://www.dropbox.com/sh/{unicorn_string}?dl=0
using (var dbx = new DropboxClient(_dropboxSettings.AccessToken))
{
var sharedLink = new SharedLink(_dropboxSettings.FolderToDownload);
var sharedFiles = await dbx.Files.ListFolderAsync(path: "", sharedLink: sharedLink);
foreach (var file in sharedFiles.Entries)
{
}
}
The documentation wasn't clear about setting path to an empty string when using it with publicly shared folders.
The official way to get information about a particular shared link is to use the Dropbox API's /2/sharing/get_shared_link_metadata endpoint:
https://www.dropbox.com/developers/documentation/http/documentation#sharing-get_shared_link_metadata
In the official Dropbox .NET SDK that's the GetSharedLinkMetadataAsync method:
https://dropbox.github.io/dropbox-sdk-dotnet/html/M_Dropbox_Api_Sharing_Routes_SharingUserRoutes_GetSharedLinkMetadataAsync_1.htm
This unfortunately doesn't offer the list of files though. We'll consider that a feature request.
Note that scraping the site would be error prone and likely to break without warning. (It's also against the terms anyway.)
Edit:
Dropbox API v2 now supports listing the contents of a shared link for a folder. This can be accomplished using the same interface as listing a folder in a connected user's account, via the list_folder functionality. To list the contents of a shared link for a folder, you instead provide the shared link URL in the shared_link parameter to /2/files/list_folder:
https://www.dropbox.com/developers/documentation/http/documentation#files-list_folder
If you're using an official SDK, there will also be a corresponding method for this endpoint. In the .NET SDK that's available as ListFolderAsync:
https://dropbox.github.io/dropbox-sdk-dotnet/html/M_Dropbox_Api_Files_Routes_FilesUserRoutes_ListFolderAsync_1.htm
I am new to REST and I am trying to test a REST call to my Private azure blob storage. I downloaded a small REST call testing program that asks for a URL and Headers (both as string).
I need to list all the blobs in a container using the method described here: List Blobs (REST API)
I am basically wondering how to write the Headers (to include my Key to access the private container).
Thanks
Edit : The program I use to test REST calls is an Extension for Chrome named "Simple REST Client"
I think you should input it this way into that chrome plugin:
Authorization="SharedKey accountname:SharedKey" x-ms-date:2013-01-24T21:33:40 (Current Date)
At least according to this document:
http://msdn.microsoft.com/en-us/library/windowsazure/dd179428.aspx
If I'm using GWT File widget and form panel, can someone explain how to handle upload on blobstore on google application engine??
Take a look at gwtupload. There are examples on how to use it with GAE Blobstore.
Google blobstore is specifically designed to upload and serve blobs via http. Blobstore service (obtained using BlobstoreServiceFactory.getBlobstoreService()) generates http post action for you to use in the html form. By posting file to it you upload your blob to the blobstore. When you generate this action you provide a path to the handler (servlet) where you have access to uploaded blob key:
Map<String, BlobKey> blobs = blobstoreService.getUploadedBlobs(req);
BlobKey blobKey = blobs.get("data");
Note, that "data" is the file field in your form. All you have is a key to the blob (your file). From here you take control - you can save this key for later and/or immediately serve the blob on a page (using key):
BlobKey blobKey = new BlobKey(req.getParameter("blob-key"));
blobstoreService.serve(blobKey, res);
Of course, for details see Google documentation.
One nice feature of the blobstore that it's integrated with Google Mapper (rudimentary map-reduce) service (work in progress) which lets you process files uploaded as blobs line by line: http://ikaisays.com/2010/08/