Is it possible to stream data(Upload) to store on bucket of Google cloud storage and allow to download at the same time? - google-cloud-storage

Is it possible to stream data(Upload) to store on bucket of Google cloud storage and allow to download at the same time?
I have tried to use the Cloud API to upload a 100MB file to the bucket by using the code as below, but during the upload, and i refresh the bucket in the Google cloud console, i cannot see the new uploading file until the upload is finished. I would like to upload realtime video encoded in H.264 to store on the Cloud storage, so the size is unknown and at the same time, other users can start downloading the file event it is uploading. So is it possible?
Test code:
File tempFile = new File("StorageSample");
RandomAccessFile raf = new RandomAccessFile(tempFile, "rw");
try
{
raf.setLength(1000 * 1000 * 100);
}
finally
{
raf.close();
}
uploadFile(TEST_FILENAME, "text/plain", tempFile, bucketName);
public static void uploadFile(
String name, String contentType, File file, String bucketName)
throws IOException, GeneralSecurityException
{
InputStreamContent contentStream = new InputStreamContent(
contentType, new FileInputStream(file));
// Setting the length improves upload performance
contentStream.setLength(file.length());
StorageObject objectMetadata = new StorageObject()
// Set the destination object name
.setName(name)
// Set the access control list to publicly read-only
.setAcl(Arrays.asList(
new ObjectAccessControl().setEntity("allAuthenticatedUsers").setRole("READER"))); //allUsers//
// Do the insert
Storage client = StorageFactory.getService();
Storage.Objects.Insert insertRequest = client.objects().insert(
bucketName, objectMetadata, contentStream);
insertRequest.getMediaHttpUploader().setDirectUploadEnabled(false);
insertRequest.execute();
}

Unfortunately it's not possible, as state in the documentation:
Objects are immutable, which means that an uploaded object cannot
change throughout its storage lifetime. An object's storage lifetime
is the time between successful object creation (upload) and successful
object deletion.
This means that an object in cloud storage starts to exist when the upload it's finished, so you cannot access the object until your upload it's not completed.

Related

Azure Media Services - Download Transient Error

I have a lot of audios in my database whose URLs are like:
https://mystorage.blob.core.windows.net/mycontainer/uploaded%2F735fe9dc-e568-4920-a3ed-67230ce01991%2F5998d1f8-1795-4776-a19c-f1bc4a0d4786%2F2020-08-13T13%3A09%3A13.0996703Z?sv=2020-02-10&se=2022-01-05T16%3A58%3A50Z&sr=b&sp=r&sig=hQBPyOE92%2F67MqU%2Fe5V2NsqGzgPxogVeXQT%2BOlvbayw%3D
I am using these URLs as my JobInput, and submitting a encoding job, because I want to migrate the audios distribution to a streaming approach.
However, every time I use this kind of URL, it fails with DownloadTransientError, and a message something like while trying to download the input files, the files were not acessible.
If I manually upload a file to the blob storage with a simpler URL (https://mystorage.blob.core.windows.net/mycontainer/my-audio.wav), and use it as the JobInput, it works seamlessly. I suspect it has something to do with the special characters on the bigger URL, but I am not sure. What could be the problem?
Here is the part of the code that submits the job:
var jobInput = new JobInputHttp(new[]
{
audio.AudioUrl.ToString()
});
JobOutput[] jobOutput =
{
new JobOutputAsset(outputAssetName),
};
var job = await client.Jobs.CreateAsync(
resourceGroupName: _azureMediaServicesSettings.ResourceGroup,
accountName: _azureMediaServicesSettings.AccountName,
transformName: TransformName,
jobName: jobName,
new Job
{
Input = jobInput,
Outputs = jobOutput
});
You need to include the file name in the URL you're providing. I'll use your URL as an example, but unescape it as well so that it is more clear. The URL should be something like https://mystorage.blob.core.windows.net/mycontainer/uploaded/735fe9dc-e568-4920-a3ed-67230ce01991/5998d1f8-1795-4776-a19c-f1bc4a0d4786/2020-08-13T13:09:13.0996703Z/my-audio.wav?sv=2020-02-10&se=2022-01-05T16:58:50Z&sr=b&sp=r&sig=hQBPyOE92/67MqU/e5V2NsqGzgPxogVeXQT+Olvbayw=
Just include the actual blob name of the input video or audio file with the associated file extension.

Getting DetailedApiRequestError when uploading a file to google drive with pre-generated file id

I want my users to be able to back-up their data on google drive. I used the "Try it now" tool of google drive to generate a file id, because if I don't set file id before uploading the file, the drive.files.create() method creates a new file id after every upload to google drive.
First I import google drive API.
import 'package:googleapis/drive/v3.dart' as ga;
This is how I create the file I want to upload:
// Create the file we want to upload.
ga.File fileToUpload = ga.File();
var file = await _localFile;
fileToUpload.parents = ["appDataFolder"];
fileToUpload.name = path.basename(file.absolute.path);
// Setting a generated id from the "Try it now" tool.
fileToUpload.id = "1azltlNpZEt6RyUBNwdYUAWPZv2o6w7bP";
And this is how I upload it to google drive:
var response = await drive.files.create(
fileToUpload,
uploadMedia: ga.Media(file.openRead(), file.lengthSync()),
);
The response I get is: "Unhandled Exception: DetailedApiRequestError(status: 403, message: Method not supported for files within the Application Data folder.)"
If I remove the next line when I create the file:
fileToUpload.parents = ["appDataFolder"];
And then I try to upload the file, I get: "DetailedApiRequestError(status: 403, message: The user does not have sufficient permissions for this file.)"
If I don't set the file id, which means I remove the next line:
fileToUpload.id = "1azltlNpZEt6RyUBNwdYUAWPZv2o6w7bP";
Then I can upload the file to google drive. But it's not helping me because it creates a new file id.
Is there anything I can do?
Edit: The problem with google drive generating a new file Id every time the user syncing the app data on google drive, is that it means that after every sync, I need to save the new file id on a server (so I can retrieve the file when needed). And when there are many active users that backing their data multiple times a day, all those update calls to the service would cost me too much money. So I prefer that the file Id will always stay the same.

How to create a Blob from a Google Cloud Storage URL in Python

I have a blob created using the google cloud storage API, and have saved its path using blob.path. The path is of the form
/b/bucketname/o/some%2Fobject%2Fid
How do I recreate the blob from this URL?
It's unfortunate that the GCS API doesn't provide a factory method to go from the path URL back to a blob, since saving blob paths in databases etc. is quite common.
Here is a factory method that allows you to go from a blob.path back to a blob:
def blob_from_blobpath(blob_path):
import google.cloud.storage as gcs
blob_path = blob_path[3:] # /b/
slash_loc = blob_path.index('/')
bucket_name = blob_path[:slash_loc]
blob_name = blob_path[(slash_loc+3):] # /o/
bucket = gcs.Client().get_bucket(bucket_name)
return bucket.blob(blob_name)

Jclouds swift api upload object directly from inputstream

The following snippet uploads files to object store without any problem
public void uploadObjectFromStream(String container, String name, InputStream stream) {
SwiftApi swiftApi = getApi();
createContainerIfAbsent(container, swiftApi);
ObjectApi objectApi = swiftApi.getObjectApiForRegionAndContainer(REGION, container);
Payload payload = new InputStreamPayload(stream);
objectApi.put(name, payload, PutOptions.Builder.metadata(ImmutableMap.of("X-Object-Meta-key1", "value3", "X-Object-Meta-key2", "test"))); // test
}
If I try to upload ~10Mb file I get error
o.j.h.i.HttpWire [SLF4JLogger.java:56] over limit 10485760/262144: wrote temp file
java.lang.OutOfMemoryError: Java heap space
The question is if I can upload object from input stream to object store without saving the stream in application memory or file system.
jclouds does not buffer InputStream unless you enable wire logging. Generally this should be disabled unless you are debugging an issue.

Multiple s3 buckets in Filepicker.io

I need to upload to multiple s3 buckets with filepicker.io. I found a tweet that indicated that there was a hacky, but possible, way to do this. Support hasn't gotten back to me yet, so I'm hoping that someone here already knows the answer!
Have you tried generating a second application/API key? It looks like they lock your S3/AWS credentials to an application/API key rather than directly to the account.
Support just got back to me. There's no way to do this besides creating multiple applications, which is okay if you are just switching between prod/staging/dev, but not a good solution if you have to upload to arbitrary buckets.
My solution is to execute a PUT request with the x-amz-copy-source header after the file has been uploaded, which copies it to the correct bucket.
This is pretty hacky as it request two extra requests per file -- one filepicker.stat and one more call to s3 (or your server).
#Ben
I am developing code with same issue of files needing to go into many buckets. I'm working in ASP.net.
What I have done is have one Filepicker 'application' with it's own S3 bucket.
I already had a callback to the server in the javascript onSuccess() function (which is passed as a parameter to filepicker.store()). This callback needed to be there to do some book-keeping anyway.
So I have just added in an extra bit to the server-side callback code which uses the AWS SDK to copy the object from the bucket filepicker uploades it to, to it's final destination bucket.
This is my C# code for moving, or rather copying, an object between buckets:
public bool MoveObject(string bucket1, string key1, string bucket2, string key2 = null)
{
bool success = false;
if (key2 == null) key2 = key1;
Logger logger = new Logger(); // my logging system
try
{
RegionEndpoint region = RegionEndpoint.EUWest1; // use your region here
using (AmazonS3Client s3Client = new AmazonS3Client(region))
{
// TODO: CheckForBucketFunction
CopyObjectRequest request = new CopyObjectRequest();
request.SourceBucket = bucket1;
request.SourceKey = key1;
request.DestinationBucket = bucket2;
request.DestinationKey = key2;
S3Response response = s3Client.CopyObject(request);
logger.Info2Log("response xml = \n{0}\n", response.ResponseXml);
response.Dispose();
success = true;
}
}
catch (AmazonS3Exception ex)
{
logger.Info2Log("Error copying file between buckets: {0} - {1}",
ex.ErrorCode, ex.Message);
success = false;
}
return success;
}
There are AWS SDKs for other server languages and the good news is Amazon doesn't charge for copying objects between buckets in the same region.
Now I just have to decide how to delete the object from the filepicker application bucket. I could do it on the server using more AWS SDK code but that will be messy as it leaves links to the object in the filepicker console. Or I could do it from the browser using filepicker code.