I am trying to figure out which is better: resumable upload or signed url. For upload only. Does anyone know if one uploadid can be used by multiple uploads? Or how can a user upload multiple files using one uploadid?
If your goal is to allow users without Google credentials to upload objects, signed URLs are your best bet. This is their intended purpose.
You can use uploadIds to accomplish the same goal, but they are much less featureful in this regard. For example, they do not support setting expiration times, and the server must set all parameters other than the data itself.
Related
I want to build a tiny story system where users can upload videos.
I'm using Firebase and the frontend will be in flutter.
I'm struggling a bit to design the flow from frontend to my Go backend. What's the simplest way to achieve this ?
From what I understand I could use different flows:
Front ask for an upload signed url to Go backend
Backend generate a gcp storage signed url
Front uploads the video
Front send the link to backend
Backend transcode the video
Backend store the link in firestore
Or
Front use directly firebase storage
Front send the link to backend ?
What's the benefits of using an upload signed url vs directly firebase storage?
Thanks in advance
What's the benefits of using an upload signed url vs directly firebase storage?
Firebase storage offers simplicity of security rules to restrict access while using GCS directly will require you to have a backend to generate signed URLs. I would prefer signed URLs when it's the system does not use Firebase Authentication or you want some validation before the file is uploaded as first place. However most of that can be done using security rules as well.
When using Firebase storage, the upload is simpler just by using uploadBytes() function while signed URLs would require some additional code. An example can be found in this
I am not sure what you mean by 'transcode video' but you can use Cloud Storage Triggers for Cloud Functions and run any actions such as adding URL to Firestore or process video once a file is uploaded.
Okay, I was using Flutter and Firebase to upload data into Cloud Storage. I gained the downloadURL which can be accessible on web if people know the URL. I had enabled Public Access Prevention in Google Cloud Storage Console based on this doc and chose Access Control Uniform for this on doc.
I also had added Security Rule in Firebase Cloud Storage, so only Users with certain custom token can use it. But, it seems useless as everyone can get its downloaded URL. My question is why is that I still able to access the file if I am using the same URL which was I stored in Firestore? You can test it on this url.
Can hacker get the download URL I downloaded from Firestore?
Is there a secure way to download song from Firebase Cloud Storage so hacker won't get its URL?
Thank you for helping me out.
Updated v2:
I just found out that current audio file has its own AuthenticatedUrl as shown on this picture below. How can I get access to this url?
Updated v1:
I think I haven't activated Firebase App Check. Does this feature have ability to prevent it from being accessed publicly or maybe there is other things that I have to do to be able to prevent it being accessed publicly, beside all ways I described above???
Security rules only check if a user can get the download URL and do not restrict anyone from using it. You can use the getData() method instead. It doesn't return any URL and downloads the files directly and is controlled by security rules. So a user must be authenticated to fetch them.
As mentioned in the Answer :
If you're using the FlutterFire Storage library in your app, you can
call getData on a reference to the file to get its data. So with
that you just need to know the path to the data, and you won't need
the download URL in your application. Once you have the data locally,
you can create an image out of it with: Converting a byte array to
image in Flutter?
Unlike download URLs, the call to getData() is
checked by security rules, so you'll have to ensure that the user is
permitted to access the file.
You can also refer to this Answer :
For web apps: in the JavaScript/Web SDK using a download URL is the
only way to get at the data, while for the native mobile SDKs we also
have getData() and getFile() methods, which are enforced through
security rules.
Until that time, if signed URLs fit your needs
better, you can use those. Both signed URLs and download URLs are just
URLs that provide read-only access to the data. Signed URLs just
expire, while download URLs don't.
For more information, you can refer to this Github issue where a similar issue has been discussed.
User Case: Customer can upload the file from the public REST api to our S3 bucket and then we can process the file using downstream services.
After doing some research I am able to find 3 ways to do it:
Uploading using OCTET-STREAM file type
Upload the file using form-data request
Upload the file using the pre-signed URL
In first 2 cases user will send the binary file and we will upload the file to S3 after file validation.
In the 3rd method user have to hit 3 apis. First API to get the S3 pre-signed URL which will give access to the user to upload the file to S3. In second hit user will upload the file to that s3 pre-signed URL. After the user complete the upload he will send the request to process the file.
Do we have any security issues with step 3? As user can misuse the pre-signed URL with malicious file.
Which of these method is best according to industry practice?
Details of each approach:
1. Uploading using OCTET-STREAM file type
Pros:
This method is good to upload file types which can be opened in some application such as xlsx.
1 API hit. Direct file upload
Cons:
This option is not suitable to upload multiple files. If in future we need to support multiple file upload this should be changed to multipart/form-data (A2).
No metadata can be send as body parameter. Metadata can be send in headers.
2. Upload the file using form-data request
User will upload the file with the API request by attaching it as multipart form.
Pros
We can send multiple files at the same time.
We can send extra parameters in the body.
3. Upload the file using the pre-signed URL
Cons
Customer have to hit the 3 APIs to upload the file. (2 API hits to upload and then 1 more API hit to check the process the file)
If you want them to load data into a bucket, the best way will almost always be the pre-signed URL. This gives you complete control over how you hand out access to the bucket, but also allows them to directly upload into the bucket when they have the access.
In the first two examples the user can send malicious data to your API, potentially DOSing the server / incurring costs on you to manage the payloads as you have no control over access (it is public).
In the third case they can request a URL from you, but that is it, other than spamming you for requests for URLs, unless you grant them a URL they can't access the bucket or do anything else. This seems much better than spamming your upload with large junk files and having you process them before you decide you didn't want them anyway.
Finally using the pre-signed URL is the pattern AWS would expect you to use, and so have a lot of support for managing the access, roles, logging and monitoring etc that you would want to put around this service. When you are standing up the API yourself this will all be up to you to manage.
I'm quite new to Cloud Storage solutions, and I'm currently researching options to upgrade our current solution (we currently just upload on a SVN server).
What I have is a native application running on client computers, which will upload data to the Cloud Storage. Afterwards, client should be able to download and browse their data (source is not set in stone, could be a website or from other applications). They should not be able to access other user's data.
I'm not sure how I'm supposed to proceed. As far as I understand, the native application will upload using a Native Application Credential, using JSON.
Do I need multiple credentials to track multiple users? That seems wrong to me. Besides when they come back as 'users' through the web interface, they wouldn't be using that authentification, would they?
Do I need to change the ACL of the uploaded files afterwards?
Should I just not give write/read access to any particular users and handle read requests through Signed URLs, dealing with permission details by myself using something else on the side? (not forcing a Google Account is probably a requirement)
Sorry if this is too many questions, and thanks!
Benjamin
The "individual credentials per instance of an app" question has come up before, and unfortunately there's not a great answer. If you want every user to have different permissions, you need every user to be associated with a different account.
Like you point out, the best current answer, other than requiring users to have Google accounts, is to have a centralized service that vends signed URLs to the end applications. That service would be the only owner of all of the objects and would give out permission to read or upload as needed.
I have a newbie question regarding google cloud storage. I'm looking to create a website where people can upload files (similar to Flickr in concept) and view them. I was looking at google's cloud storage option and it seemed interesting. I got a little hung up on the authentication process. Do all users need a Google account or did I just misread it? I guess my question is can I create a site where everyone doesn't have to sign up for a google account?
Thank you for your patience and help
Nope, there's no need for customers to have their own Google accounts. Google Cloud Storage supports that, in case you want to say "these accounts have access to this data", but you can also let your app grant access on a per-request basis using whatever logic or authentication scheme you like.
This is usually accomplished with signed URLs. Basically, you'd use your credentials to sign a very specific request (download object X, upload an object with name Y) and pass that URL to the user for them to use. Signed URLs are only valid for as long as you like (one of the parameters is how long the URL is good for).
Documentation on signed URLs are here: https://developers.google.com/storage/docs/accesscontrol#Signed-URLs
It's not strictly necessary to require a user to be logged in in order to write to your bucket (e.g., we have a public-read-write canned ACL for buckets), but it's generally a bad idea. Any data stored in your bucket will ultimately be charged to you, so allowing anyone to write without authentication opens you up to a great deal of abuse.
Likewise, you could make all of your objects public-read, but then you get charged for the bandwidth costs and have no control over it (though this is a much more reasonable thing to do than public-write).
A safer option would be to proxy bytes for your users - i.e., only you/your app can write to your bucket, and your users hand their bytes to you.
Generally speaking, though, the only types of authentication we support are Google accounts, signed URLs, and anonymous users.