Store the path to uploaded file on client-side or the file outside the browser for offline - progressive-web-apps

Is there a way to store the path to file which user wants to upload, but doesn't have an internet connection (it's a PWA) and reupload it when a connection is back? Or maybe not store the path, but save the file outside browser storage, somewhere on the user's machine (even if it will require some acceptance from the user to allow the browser to read/write files), but I'm not sure if it's even allowed to do.
Currently, I'm storing the whole file as a base64 in IndexedDB, but it's crashing/slowing down the browser when it comes to reading big files (around 100MB). Also, I don't want to overload browser storage.

There's a couple of things to consider.
Storing the data you need to upload in IndexedDB and then reading that in later will be the most widely supported approach. As you say, though, it means taking up extra browser storage. One thing that might help is to skip the step of encoding the file in Base64 first, as in all modern browsers, IndexedDB will gladly store bytes directly for you as a Blob.
A more modern approach, but one that's not currently supported by non-Chromium browsers, would be to use the File System Access API. As described in this article, once you get the user's permission, you can save a handle to a file in IndexedDB, and then read the file later on (assuming the underlying file hasn't changed in the interim). This has the advantage of not duplicating the file's contents in IndexedDB, saving on storage space. Here's a code snippet, borrowed from the article:
import { get, set } from 'https://unpkg.com/idb-keyval#5.0.2/dist/esm/index.js';
const pre = document.querySelector('pre');
const button = document.querySelector('button');
button.addEventListener('click', async () => {
try {
// Try retrieving the file handle.
const fileHandleOrUndefined = await get('file');
if (fileHandleOrUndefined) {
pre.textContent =
`Retrieved file handle "${fileHandleOrUndefined.name}" from IndexedDB.`;
return;
}
// This always returns an array, but we just need the first entry.
const [fileHandle] = await window.showOpenFilePicker();
// Store the file handle.
await set('file', fileHandle);
pre.textContent =
`Stored file handle for "${fileHandle.name}" in IndexedDB.`;
} catch (error) {
alert(error.name, error.message);
}
});
Regardless of how you store the file, it would be helpful to use the Background Sync API when available (again, currently limited to Chromium browsers) to handle automating the upload once the network is available again.

Related

Attempting a Google Drive partial Download (Flutter) throws a header error

Here's my issue :
I am creating a small application based on audio files stored on Google Drive, in Flutter.
I am using the drive api to make my requests, with these scopes in my google sign in :
GoogleSignIn _googleSignIn = GoogleSignIn(
scopes: [
'email',
'https://www.googleapis.com/auth/userinfo.profile',
'https://www.googleapis.com/auth/contacts.readonly',
'https://www.googleapis.com/auth/drive',
'https://www.googleapis.com/auth/docs',
'https://www.googleapis.com/auth/drive.appdata',
],
);
I have an auth element and handle signing in and out. Until then, no issues.
I can also request my files with an implementation looking like this :
var api = widget.api.getAPI();
var files = await api.files.list($fields: '*');
This works perfectly, and so does :
var api = widget.api.getAPI();
var files = await api.files.get("myFileId"); (//does get a file instance)
But since I'd like to retrieve some of the Metadata included in my audio files, and since the drive API doesn't natively support extracting audio metadata and sending it as a google metadata, I thought I'd extract it with a partial download on the file itself.
Here's the catch : I can't seem to get the partial download to work.
Based on the doc, I thought the implementation would look something like this :
import 'package:googleapis/drive/v3.dart' as ga;
(...)
try {
var partiallyDownloadedFile = await api.files.get(
"myFileIdHere",
downloadOptions: ga.PartialDownloadOptions(ga.ByteRange(0, 10))); //should get a ga.Media instance
print("partial download succeeded");
print(partiallyDownloadedFile);
//(...do stuff...)
return;
} catch (err) {
print('Error occured : ');
print(err);
return;
}
But this always throws this error :
ApiRequestError(message: Attempting partial download but got invalid
'Content-Range' header (was: null, expected: bytes 0-10/).)
I tried using it on Wav files, but also MP4 files. The error is always the same, which leads me to believe it's my implementation that's somehow wrong, but I'm not sure what I'm supposed to do to fix it. Is it my request missing the header ? The response not including it ?
While very clear, that error doesn't help me troubleshoot my issue at all. I can't seem to find any documentation on how to conduct a partial media request. I haven't found any example projects to compare it with.
PartialDownloadOptions does not have much documentation.
I could handmake a partial request through the download links (which is how I can read the music to begin with) but the drive API supposedly allows this. Could anyone familiar with Flutter/the google APIs help me correct my implementation?
EDIT : This was due to an error within the commons library in the Dart google APIs, and was (at the very least superficially) fixed thanks to Kevmoo's efforts : https://github.com/google/googleapis.dart/issues/462
It was a Content-Range error happening due to browser specifications with access-control-expose-header compared to iOS/Android-type requests that typically expose every header.

Sharing a text file (csv) with share_plus: "unable to attach file"

I want to get data from Firestore, convert it into csv format, then share it using the user's method of choice (e.g. email). I'm using the csv package to convert the data to csv, the path_provider to get the directory to write the file to the phone, and the share_plus package to share the file.
However, when I tap a share method (e.g. Gmail, Outlook, Whatsapp), it opens the right app but then I get a message on the phone like "Unable to attach file" (but there is no error in my app). The file is definitely being written as I can read it and it comes back with the expected string. The ShareResultStatus that is returned by the share is 'successful' Can anyone figure it out what the problem is and how to fix it?
Here is my code:
Future exportData() async {
// Dummy data (in reality, this will come from Firestore)
List<List> dummyData = [
['Date', 'Piece', 'Rating'],
[DateTime.utc(2022, 05, 01), 'Sonata', 4],
[DateTime.utc(2022, 05, 02), 'Sonata', 2],
];
// Turn into CSV
String csvData = _convertToCsv(dummyData);
// Write to local disk
await _writeData(csvData);
// Share
String filePath = await _filePath;
final result =
await Share.shareFilesWithResult([filePath], mimeTypes: ['text/csv']);
print(result.status);
}
And here are the functions used in above code:
Future<String> get _filePath async {
final directory = await getApplicationDocumentsDirectory();
return '${directory.path}/my_practice_diary_data.csv';
}
Future<File> _writeData(String csvData) async {
String filePath = await _filePath;
final file = File(filePath);
return file.writeAsString(csvData);
}
String _convertToCsv(List<List> data) {
String result = const ListToCsvConverter().convert(data);
return result;
}
Note: I've tried doing it both as txt and csv files, got same result
*** EDIT (06/06/2022): After a lot of reading and watching youtube videos, I have come to realise that the problem is that the directory getApplicationDocumentsDirectory() is only accessible by my app (and hence the app that is is being shared to, like gmail, cannot read it).
For now I have worked around it by using the package mailer and using user's google credentials to send emails (I followed these helpful youtube tutorials: Flutter Tutorial - How To Send Email In Background [2021] Without Backend and Flutter Tutorial - Google SignIn [2021] With Firebase Auth - Android, iOS, Flutter Web.
However, it would still be nice to know a nice way to generate a file and share it using share_plus, as per my original question. I believe that this can be achieved via one of two ways:
Find a way to allow other apps to access this specific file in the app directory; or
Find a way to download the file into an external storage (like downloads folder), then share that. (I cannot find a way to do this on both Android and iOS).
Anyone who knows how to do these or any other solution to the problem, please share!

For PWA, what is the easiest way to get per-device settings (as in reading a .ini file or environment variables)?

For PWA, what is the easiest way to get per-device settings (as in reading a .ini file or environment variables)?
I'm making a very simple in-company react PWA for andriod-based tablets (only). I just want to store a couple of settings (the room number where the device is being used, and a device id) and read those in upon startup.
My experience in recently in Windows, and so I'm imaging a text file that I could place on each tablet with the settings. Does that make sense for our PWA?
Or is there a better/easier way to do app settings?
Thank you.
The answer depends on how that data is initially provisioned and what kind of guarantees you need about it being "tamper-proof."
Assuming you can provision the information during the web app's initial launch, and you're fine using storage that's exposed via a browser's Developer Tools (i.e. your threat model doesn't include a motivated user using DevTools to erase/modify the data), a simple approach would be to a) use the Cache Storage API to read/write that data as JSON, using a synthetic URL as the key and b) requesting persistent storage just for an added guarantee that it won't be purged if the device ends up running low on storage.
This could look like:
// Just use any URL that doesn't exist on your server.
const SETTINGS_KEY = '/_settings';
const SETTINGS_CACHE_NAME = 'settings';
async function getSettings() {
const cache = await caches.open(SETTINGS_CACHE_NAME);
const cachedSettingsResponse = await cache.match(SETTINGS_KEY);
if (cachedSettingsResponse) {
return await cachedSettingsResponse.json();
}
// This assumes a generateInitialSettings function that does provisioning
// and returns an object that can be stringified.
const newSettings = await generateInitialSettings();
await cache.put(SETTINGS_KEY, JSON.stringify(newSettings), {
headers: {
'content-type': 'application/json',
}
});
// Optional: request persistent storage.
// This call may trigger a permissions dialog in the local browser, so it is
// a good idea to explain to the user what's being stored first.
// See https://web.dev/persistent-storage/#when-should-i-ask-for-persistent-storage
if (navigator.storage && navigator.storage.persist) {
// This returns a promise, but we don't have to delay the
// rest of the program execution by await-ing it.
navigator.storage.persist();
}
return newSettings;
}

Azure Media Services - Download Transient Error

I have a lot of audios in my database whose URLs are like:
https://mystorage.blob.core.windows.net/mycontainer/uploaded%2F735fe9dc-e568-4920-a3ed-67230ce01991%2F5998d1f8-1795-4776-a19c-f1bc4a0d4786%2F2020-08-13T13%3A09%3A13.0996703Z?sv=2020-02-10&se=2022-01-05T16%3A58%3A50Z&sr=b&sp=r&sig=hQBPyOE92%2F67MqU%2Fe5V2NsqGzgPxogVeXQT%2BOlvbayw%3D
I am using these URLs as my JobInput, and submitting a encoding job, because I want to migrate the audios distribution to a streaming approach.
However, every time I use this kind of URL, it fails with DownloadTransientError, and a message something like while trying to download the input files, the files were not acessible.
If I manually upload a file to the blob storage with a simpler URL (https://mystorage.blob.core.windows.net/mycontainer/my-audio.wav), and use it as the JobInput, it works seamlessly. I suspect it has something to do with the special characters on the bigger URL, but I am not sure. What could be the problem?
Here is the part of the code that submits the job:
var jobInput = new JobInputHttp(new[]
{
audio.AudioUrl.ToString()
});
JobOutput[] jobOutput =
{
new JobOutputAsset(outputAssetName),
};
var job = await client.Jobs.CreateAsync(
resourceGroupName: _azureMediaServicesSettings.ResourceGroup,
accountName: _azureMediaServicesSettings.AccountName,
transformName: TransformName,
jobName: jobName,
new Job
{
Input = jobInput,
Outputs = jobOutput
});
You need to include the file name in the URL you're providing. I'll use your URL as an example, but unescape it as well so that it is more clear. The URL should be something like https://mystorage.blob.core.windows.net/mycontainer/uploaded/735fe9dc-e568-4920-a3ed-67230ce01991/5998d1f8-1795-4776-a19c-f1bc4a0d4786/2020-08-13T13:09:13.0996703Z/my-audio.wav?sv=2020-02-10&se=2022-01-05T16:58:50Z&sr=b&sp=r&sig=hQBPyOE92/67MqU/e5V2NsqGzgPxogVeXQT+Olvbayw=
Just include the actual blob name of the input video or audio file with the associated file extension.

Multiple s3 buckets in Filepicker.io

I need to upload to multiple s3 buckets with filepicker.io. I found a tweet that indicated that there was a hacky, but possible, way to do this. Support hasn't gotten back to me yet, so I'm hoping that someone here already knows the answer!
Have you tried generating a second application/API key? It looks like they lock your S3/AWS credentials to an application/API key rather than directly to the account.
Support just got back to me. There's no way to do this besides creating multiple applications, which is okay if you are just switching between prod/staging/dev, but not a good solution if you have to upload to arbitrary buckets.
My solution is to execute a PUT request with the x-amz-copy-source header after the file has been uploaded, which copies it to the correct bucket.
This is pretty hacky as it request two extra requests per file -- one filepicker.stat and one more call to s3 (or your server).
#Ben
I am developing code with same issue of files needing to go into many buckets. I'm working in ASP.net.
What I have done is have one Filepicker 'application' with it's own S3 bucket.
I already had a callback to the server in the javascript onSuccess() function (which is passed as a parameter to filepicker.store()). This callback needed to be there to do some book-keeping anyway.
So I have just added in an extra bit to the server-side callback code which uses the AWS SDK to copy the object from the bucket filepicker uploades it to, to it's final destination bucket.
This is my C# code for moving, or rather copying, an object between buckets:
public bool MoveObject(string bucket1, string key1, string bucket2, string key2 = null)
{
bool success = false;
if (key2 == null) key2 = key1;
Logger logger = new Logger(); // my logging system
try
{
RegionEndpoint region = RegionEndpoint.EUWest1; // use your region here
using (AmazonS3Client s3Client = new AmazonS3Client(region))
{
// TODO: CheckForBucketFunction
CopyObjectRequest request = new CopyObjectRequest();
request.SourceBucket = bucket1;
request.SourceKey = key1;
request.DestinationBucket = bucket2;
request.DestinationKey = key2;
S3Response response = s3Client.CopyObject(request);
logger.Info2Log("response xml = \n{0}\n", response.ResponseXml);
response.Dispose();
success = true;
}
}
catch (AmazonS3Exception ex)
{
logger.Info2Log("Error copying file between buckets: {0} - {1}",
ex.ErrorCode, ex.Message);
success = false;
}
return success;
}
There are AWS SDKs for other server languages and the good news is Amazon doesn't charge for copying objects between buckets in the same region.
Now I just have to decide how to delete the object from the filepicker application bucket. I could do it on the server using more AWS SDK code but that will be messy as it leaves links to the object in the filepicker console. Or I could do it from the browser using filepicker code.