I want to allow a client (browser) application to upload files directly to Google Cloud Storage while maintaining the ability to constrain the file size that the client could upload.
It's seems the best practice approach is to combine resumable uploads with signed urls created by a server with enough credentials (Service Account).
Is it possible, with the given flow, to limit the file size for a specific signed url or resumable upload id? Meaning that Google Cloud Storage would reject a file that exceeds a desired size.
It is possible to limit a signed URL to an exact allowed size by adding a Content-Length canonical extension header. This will make only uploads with that exact content-length allowed (and will not allow chunked encoding uploads which do not have that header. You could thus have your client pass in the desired length to the service that provides the signed URL, which then includes this limit.
Related
User Case: Customer can upload the file from the public REST api to our S3 bucket and then we can process the file using downstream services.
After doing some research I am able to find 3 ways to do it:
Uploading using OCTET-STREAM file type
Upload the file using form-data request
Upload the file using the pre-signed URL
In first 2 cases user will send the binary file and we will upload the file to S3 after file validation.
In the 3rd method user have to hit 3 apis. First API to get the S3 pre-signed URL which will give access to the user to upload the file to S3. In second hit user will upload the file to that s3 pre-signed URL. After the user complete the upload he will send the request to process the file.
Do we have any security issues with step 3? As user can misuse the pre-signed URL with malicious file.
Which of these method is best according to industry practice?
Details of each approach:
1. Uploading using OCTET-STREAM file type
Pros:
This method is good to upload file types which can be opened in some application such as xlsx.
1 API hit. Direct file upload
Cons:
This option is not suitable to upload multiple files. If in future we need to support multiple file upload this should be changed to multipart/form-data (A2).
No metadata can be send as body parameter. Metadata can be send in headers.
2. Upload the file using form-data request
User will upload the file with the API request by attaching it as multipart form.
Pros
We can send multiple files at the same time.
We can send extra parameters in the body.
3. Upload the file using the pre-signed URL
Cons
Customer have to hit the 3 APIs to upload the file. (2 API hits to upload and then 1 more API hit to check the process the file)
If you want them to load data into a bucket, the best way will almost always be the pre-signed URL. This gives you complete control over how you hand out access to the bucket, but also allows them to directly upload into the bucket when they have the access.
In the first two examples the user can send malicious data to your API, potentially DOSing the server / incurring costs on you to manage the payloads as you have no control over access (it is public).
In the third case they can request a URL from you, but that is it, other than spamming you for requests for URLs, unless you grant them a URL they can't access the bucket or do anything else. This seems much better than spamming your upload with large junk files and having you process them before you decide you didn't want them anyway.
Finally using the pre-signed URL is the pattern AWS would expect you to use, and so have a lot of support for managing the access, roles, logging and monitoring etc that you would want to put around this service. When you are standing up the API yourself this will all be up to you to manage.
I am implementing a web application in which I am allowing user's to upload there images and I am going to save those images directly to cloud storage, and I want to allow that specific user only to download or see the image, How can I set such restriction? (oauth protocol is certainly not helpful as it needs google login base authentication)
I want to allow user download image only through my web application, At the moment I am creating signedurl with validity of 10 minutes, but if someone in meantime sees network traffic, and notices the url then other person using that link during those 10 minutes can download the image. So how can I make my system fullproof?
Ohkey, So moral of the story is, If we send an HTTPS request to the google cloud storage then network administrator or intruders won't be able to see actual full URLs, They only will be able to see the host's IP address on which our request is going to send...
SO if we send our request to https://storage.googleapis.com/bucket/object? then what will actually happen is it will send request to storage.googleapis.com for secure HTTPS connection and once token is generated then it SSL will do its magic and pass remaining URL data to the server, SO, what happens here is, administrator or intruders will know that we are sending request to storage.googleapis.com but won't be able to see our bucketName and other signature stuff
So, our data file stays safe with downloading/uploading over HTTPS connection
This Stack Exchange question help me to understand a bit more https://security.stackexchange.com/questions/34794/if-ssl-encrypts-urls-then-how-are-https-messages-routed
https://security.stackexchange.com/questions/20803/how-does-ssl-tls-work/20847#20847
I am trying to figure out which is better: resumable upload or signed url. For upload only. Does anyone know if one uploadid can be used by multiple uploads? Or how can a user upload multiple files using one uploadid?
If your goal is to allow users without Google credentials to upload objects, signed URLs are your best bet. This is their intended purpose.
You can use uploadIds to accomplish the same goal, but they are much less featureful in this regard. For example, they do not support setting expiration times, and the server must set all parameters other than the data itself.
I have a Google Cloud Endpoints wich is using Cloud SQL to store data. I want to provide a file upload for Clients and the files should be stored in Cloud Storage but I also want to store file meta data and the file storage url in Cloud SQL.
What's the best was to do this?
Can I upload files through cloud endpoints or do I need an extra upload Servlet?
How can I update my database entities which needs a reference to the uploaded files.
Any examples on how to combine those 3 technologies?
Assuming your clients are not added to your google cloud project (which is typically the case), your users don't have write access to your GCS bucket. You can either submit files to your application and move to GCS from there (not recommended as consumes more network and CPU) or a better way is to submit to GCS directly.
To let the client write to your GCS bucket directly, you will need to either:
1. put your access key on client for write access (not recommended), if the client is used by limited trusted people.
2. generate a time-bound token and put it on the client as signed URL to upload directly.
Endpoints APIs themselves cannot do this, but you can generate the signed GCS URL at the server and get it using endpoints on client. then set it as form action (on web client, other clients have similar ways for signed upload) and submit the form to upload the file.
<form action="SIGNED_URL_FROM_ENDPOINTS" method="post" enctype="multipart/form-data">
I don't see an open-source code out there doing exactly this, but closest is this project that does generate the signed URL with a time-out (the only unintuitive part).
Best way to update the metadata in your database is to watch GCS bucket using 'Object Change Notifications'. Another way is to send the metadata to your server from client itself, which can be an endpoints call. You can also use a mix of both where the metadata goes to server using endpoints even before the the file is uploaded and the notification updates the record with confirmation that it is available to serve.
Say we want a REST API to support file uploads, and we want uploads to be done directly on S3.
According to this solution Amazon S3 direct file upload from client browser - private key disclosure, we have to create POLICY and SIGNATURE for user to be allowed to upload to S3.
However, we want a single entry point for the API, including uploads.
Can we:
1. in our API, catch POST https://www.example.org/users/1234/objects
2. calculate POLICY and SIGNATURE to allow direct upload to S3
3. return a 307 "Temporary Redirect" to https://s3-bucket.s3.amazonaws.com
How to pass POLICY and SIGNATURE in the redirect?
What is best practice here?
You dont redirect, instead your API should return the policy and signature in the response (say in JSON).
Then the browser can use these values to directly upload to S3 as in the document. This is a two step process.