Multiple s3 buckets in Filepicker.io - filepicker.io

I need to upload to multiple s3 buckets with filepicker.io. I found a tweet that indicated that there was a hacky, but possible, way to do this. Support hasn't gotten back to me yet, so I'm hoping that someone here already knows the answer!

Have you tried generating a second application/API key? It looks like they lock your S3/AWS credentials to an application/API key rather than directly to the account.

Support just got back to me. There's no way to do this besides creating multiple applications, which is okay if you are just switching between prod/staging/dev, but not a good solution if you have to upload to arbitrary buckets.
My solution is to execute a PUT request with the x-amz-copy-source header after the file has been uploaded, which copies it to the correct bucket.
This is pretty hacky as it request two extra requests per file -- one filepicker.stat and one more call to s3 (or your server).

#Ben
I am developing code with same issue of files needing to go into many buckets. I'm working in ASP.net.
What I have done is have one Filepicker 'application' with it's own S3 bucket.
I already had a callback to the server in the javascript onSuccess() function (which is passed as a parameter to filepicker.store()). This callback needed to be there to do some book-keeping anyway.
So I have just added in an extra bit to the server-side callback code which uses the AWS SDK to copy the object from the bucket filepicker uploades it to, to it's final destination bucket.
This is my C# code for moving, or rather copying, an object between buckets:
public bool MoveObject(string bucket1, string key1, string bucket2, string key2 = null)
{
bool success = false;
if (key2 == null) key2 = key1;
Logger logger = new Logger(); // my logging system
try
{
RegionEndpoint region = RegionEndpoint.EUWest1; // use your region here
using (AmazonS3Client s3Client = new AmazonS3Client(region))
{
// TODO: CheckForBucketFunction
CopyObjectRequest request = new CopyObjectRequest();
request.SourceBucket = bucket1;
request.SourceKey = key1;
request.DestinationBucket = bucket2;
request.DestinationKey = key2;
S3Response response = s3Client.CopyObject(request);
logger.Info2Log("response xml = \n{0}\n", response.ResponseXml);
response.Dispose();
success = true;
}
}
catch (AmazonS3Exception ex)
{
logger.Info2Log("Error copying file between buckets: {0} - {1}",
ex.ErrorCode, ex.Message);
success = false;
}
return success;
}
There are AWS SDKs for other server languages and the good news is Amazon doesn't charge for copying objects between buckets in the same region.
Now I just have to decide how to delete the object from the filepicker application bucket. I could do it on the server using more AWS SDK code but that will be messy as it leaves links to the object in the filepicker console. Or I could do it from the browser using filepicker code.

Related

For PWA, what is the easiest way to get per-device settings (as in reading a .ini file or environment variables)?

For PWA, what is the easiest way to get per-device settings (as in reading a .ini file or environment variables)?
I'm making a very simple in-company react PWA for andriod-based tablets (only). I just want to store a couple of settings (the room number where the device is being used, and a device id) and read those in upon startup.
My experience in recently in Windows, and so I'm imaging a text file that I could place on each tablet with the settings. Does that make sense for our PWA?
Or is there a better/easier way to do app settings?
Thank you.
The answer depends on how that data is initially provisioned and what kind of guarantees you need about it being "tamper-proof."
Assuming you can provision the information during the web app's initial launch, and you're fine using storage that's exposed via a browser's Developer Tools (i.e. your threat model doesn't include a motivated user using DevTools to erase/modify the data), a simple approach would be to a) use the Cache Storage API to read/write that data as JSON, using a synthetic URL as the key and b) requesting persistent storage just for an added guarantee that it won't be purged if the device ends up running low on storage.
This could look like:
// Just use any URL that doesn't exist on your server.
const SETTINGS_KEY = '/_settings';
const SETTINGS_CACHE_NAME = 'settings';
async function getSettings() {
const cache = await caches.open(SETTINGS_CACHE_NAME);
const cachedSettingsResponse = await cache.match(SETTINGS_KEY);
if (cachedSettingsResponse) {
return await cachedSettingsResponse.json();
}
// This assumes a generateInitialSettings function that does provisioning
// and returns an object that can be stringified.
const newSettings = await generateInitialSettings();
await cache.put(SETTINGS_KEY, JSON.stringify(newSettings), {
headers: {
'content-type': 'application/json',
}
});
// Optional: request persistent storage.
// This call may trigger a permissions dialog in the local browser, so it is
// a good idea to explain to the user what's being stored first.
// See https://web.dev/persistent-storage/#when-should-i-ask-for-persistent-storage
if (navigator.storage && navigator.storage.persist) {
// This returns a promise, but we don't have to delay the
// rest of the program execution by await-ing it.
navigator.storage.persist();
}
return newSettings;
}

Store the path to uploaded file on client-side or the file outside the browser for offline

Is there a way to store the path to file which user wants to upload, but doesn't have an internet connection (it's a PWA) and reupload it when a connection is back? Or maybe not store the path, but save the file outside browser storage, somewhere on the user's machine (even if it will require some acceptance from the user to allow the browser to read/write files), but I'm not sure if it's even allowed to do.
Currently, I'm storing the whole file as a base64 in IndexedDB, but it's crashing/slowing down the browser when it comes to reading big files (around 100MB). Also, I don't want to overload browser storage.
There's a couple of things to consider.
Storing the data you need to upload in IndexedDB and then reading that in later will be the most widely supported approach. As you say, though, it means taking up extra browser storage. One thing that might help is to skip the step of encoding the file in Base64 first, as in all modern browsers, IndexedDB will gladly store bytes directly for you as a Blob.
A more modern approach, but one that's not currently supported by non-Chromium browsers, would be to use the File System Access API. As described in this article, once you get the user's permission, you can save a handle to a file in IndexedDB, and then read the file later on (assuming the underlying file hasn't changed in the interim). This has the advantage of not duplicating the file's contents in IndexedDB, saving on storage space. Here's a code snippet, borrowed from the article:
import { get, set } from 'https://unpkg.com/idb-keyval#5.0.2/dist/esm/index.js';
const pre = document.querySelector('pre');
const button = document.querySelector('button');
button.addEventListener('click', async () => {
try {
// Try retrieving the file handle.
const fileHandleOrUndefined = await get('file');
if (fileHandleOrUndefined) {
pre.textContent =
`Retrieved file handle "${fileHandleOrUndefined.name}" from IndexedDB.`;
return;
}
// This always returns an array, but we just need the first entry.
const [fileHandle] = await window.showOpenFilePicker();
// Store the file handle.
await set('file', fileHandle);
pre.textContent =
`Stored file handle for "${fileHandle.name}" in IndexedDB.`;
} catch (error) {
alert(error.name, error.message);
}
});
Regardless of how you store the file, it would be helpful to use the Background Sync API when available (again, currently limited to Chromium browsers) to handle automating the upload once the network is available again.

How can I access a file of storage with dart Google Storage API

I'm trying to get information of a file that is inside of a folder in Google cloud Storage (not firebase storage). But the API of it in Dart language is not complete, it doesn't have a function to show the blob (file) information like we have in python same API. I just need to access the name of the file. Here's my code:
var credentials = auth.ServiceAccountCredentials.fromJson({
"type": "service_account",
...
});
List<String> scopes = []..addAll(Storage.SCOPES);
var client = await auth.clientViaServiceAccount(credentials, scopes);
var storage = Storage(client, "project_name");
var bucket = storage.bucket("Bucket_name");
var list = await bucket.read("folder_name/");
list.forEach((element) {
print(element.toString());
});
It has a lot of options like toList(),toSet(), asBroadcastStream() and etc. But any of these return me what I need. Some ones just return a empty list, that doesn't make sence for me.
Anyways, if someone know how to read data from a folder of GCP storage, please anwser me. Sorry for my english and Thanks!
The API docs: https://pub.dev/documentation/gcloud/latest/gcloud.storage/gcloud.storage-library.html
For you backend, you can use bucket.list(prefix: "folder_name/","") as described in the documentation:
Listing operates like a directory listing, despite the object namespace being flat. Unless delimiter is specified, the character / is being used to separate object names into directory components. To list objects recursively, the delimiter can be set to empty string.
For the front end, forget this! You can't provide a service account key file in your frontend! If you share the secret publicly, it's like if you set your bucket public!
So, for this, you need to have a backend that authenticated the user and generate a signed URL to read and write inside the bucket.
bucket.read() gets the contents of individual objects. If you want to get the object names, use bucket.list().

Firebase: Authenticate an existing user using REST API and Firebases hidden Auth URL

For the past 3 years we have used HTML/Js only with Firebase but now we are using Unity as well.
The current Unity/Firebase only works on Android/iOS when deployed and 99% of our work is on the windows store.
I've actually got a pretty decent Unity/Firebase codebase going but it requires me to use a full App Secret.
All the other libraries expose a method to login with Email/Password but the REST API only allows the use of a token or your app secret that it then states is ill advised to put into your client; I guess the thinking is if you're using a different library that you'll have your own auth/user method which we don't...
Now, I've pulled apart the web version and got this:
https://auth.firebase.com/v2/<myfirebase>/auth/password?&email=dennis%40<mysite>&password=<mypassword>v=js-2.2.9&transport=json&suppress_status_codes=true
So there IS an endpoint that I can send stuff to and I've tested it inside unity with good results.
Obviously the URL isn't guaranteed to stay working but I'm wondering if there is any reason NOT to use this?
Also, Why not just expose this endpoint in the official REST API?
As I understand it, that URL will continue to work for your Legacy Firebase project. You will have to do the same sort of reverse engineering if you want to update to the new Firebase 3.0 API. However, if you are still using a legacy Firebase project -- I encourage you to take a look at this. It has not been updated to work with Firebase 3.0 -- so I needed to do something similar to what you did to allow login to the new API.
I was able to do this with the new API using C# as follows (where FirebaseManager is a Singleton I wrote for Global variables and functions to write and read from/to the DB :
Hashtable loginData = new Hashtable();
loginData.Add ("email", <EMAIL-GOES-HERE>);
loginData.Add ("password", <PASSWORD-GOES-HERE>);
loginData.Add ("returnSecureToken", true);
UnityHTTP.Request loginRequest = new UnityHTTP.Request ("post",
"https://www.googleapis.com/identitytoolkit/v3/relyingparty/verifyPassword?key="
+ <YOUR-PROJECT-API-KEY-GOES-HERE>, loginData);
loginRequest.Send ((request) => {
Hashtable jsonResponse = (Hashtable)JSON.JsonDecode(request.response.Text);
if (jsonResponse == null) {
DisplayErrorMessage("Error logging in. Server returned null or malformed response");
}
FirebaseManager.Instance.idToken = (string)jsonResponse["idToken"]; // This is your auth token
FirebaseManager.Instance.uid = (string)jsonResponse["localId"]; // this is your "uid"
});
// I have a list of users in my db keyed by the "uid" -- I access them like this
UnityHTTP.Request fullnameRequest = new UnityHTTP.Request ("get",
<YOUR-DATABASE-ROOT-URL-HERE>
+ "/users/" + FirebaseManager.Instance.uid + ".json?auth=" + FirebaseManager.Instance.idToken);
fullnameRequest.Send ((request) => {
Debug.Log(request.response.Text);
Hashtable jsonResponse = (Hashtable)JSON.JsonDecode(request.response.Text);
if (jsonResponse == null) {
DisplayErrorMessage("Error getting user info. Server returned null or malformed response");
}
FirebaseManager.Instance.fullname = (string)jsonResponse["fullname"];
FirebaseManager.Instance.groupId = (string)jsonResponse["group"]; // just storing this in memory
});
So I don't think there is any harm in using the URL, just make sure you budget time for more work when things change.

Help with a Windows Service/Scheduled Task that must use a web browser and file dialogs

What I'm Trying To Do
I'm trying to create a solution of any kind that will run nightly on a Windows server, authenticate to a website, check a web page on the site for new links indicating a new version of a zip file, use new links (if present) to download a zip file, unzip the downloaded file to an existing folder on the server, use the unzipped contents (sql scripts, etc.) to build an instance of a database, and log everything that happens to a text file.
Forms App: The Part That Sorta Works
I created a Windows Forms app that uses a couple of WebBrowser controls, a couple of threads, and a few timers to do all that except the running nightly. It works great as a Form when I'm logged in and run it, but I need to get it (or something like it) to run on it's own like a Service or scheduled task.
My Service Attempt
So, I created a Windows Service that ticks every hour and, if the System.DateTime.Now.Hour >= 22, attempts to launch the Windows Forms app to do it's thing. When the Service attempts to launch the Form, this error occurs:
ActiveX control '8856f961-340a-11d0-a96b-00c04fd705a2' cannot be instantiated because the current thread is not in a single-threaded apartment.
which I researched and tried to resolve by either placing the [STAThread] attribute on the Main method of the Service's Program class or using some code like this in a few places including the Form constructor:
webBrowseThread = new Thread(new ThreadStart(InitializeComponent));
webBrowseThread.SetApartmentState(ApartmentState.STA);
webBrowseThread.Start();
I couldn't get either approach to work. In the latter approach, the controls on the Form (which would get initialized inside IntializeComponent) don't get initialized and I get null reference exceptions.
My Scheduled Task Attempt
So, I tried creating a nightly scheduled task using my own credentials to run the Form locally on my dev machine (just testing). It gets farther than the Service did, but gets hung up at the File Download Dialog.
Related Note: To send the key sequences to get through the File Download and File Save As dialogs, my Form actually runs a couple of vbscript files that use WScript.Shell.SendKeys. Ok, that's embarassing to admit, but I tried a few different things including SendMessage in Win32 API and referencing IWshRuntimeLibrary to use SendKeys inside my C# code. When I was researching how to get through the dialogs, the Win32 API seemed to be the recommended way to go, but I couldn't figure it out. The vbscript files was the only thing I could get to work, but I'm worried now that this may be the reason why a scheduled task won't work.
Regarding My Choice of WebBrowser Control
I have read about the System.WebClient class as an alternative to the WebBrowser control, but at a glance, it doesn't look like it has what I need to get this done. For example, I needed (or I think I needed) the WebBrowser's DocumentCompleted and FileDownload events to handle the delays in pages loading, files downloading, etc. Is there more to WebClient that I'm not seeing? Is there another class besides WebBrowser that is more Service-friendly and would do the trick?
In Summary
Geez, this is long. Sorry! It would help to even have a high level recommendation for a better way to do what I'm trying to do, because nothing I've tried has worked.
Update 10/22/09
Well, I think I'm closer, but I'm stuck again. I should end up with a decent-sized zip file with several files in it, but the zip file resulting from my code is empty. Here's my code:
// build post request
string targetHref = "http://wwwcf.nlm.nih.gov/umlslicense/kss/login.cfm";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(targetHref);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
// encoding to use
Encoding enc = Encoding.GetEncoding(1252);
// build post string containing authentication information and add to post request
string poststring = "returnUrl=" + fixCharacters(targetDownloadFileUrl);
poststring += getUsernameAndPasswordString();
poststring += "&login2.x=0&login2.y=0";
// convert to required byte array
byte[] postBytes = enc.GetBytes(poststring);
request.ContentLength = postBytes.Length;
// write post to request
Stream postStream = request.GetRequestStream();
postStream.Write(postBytes, 0, postBytes.Length);
postStream.Close();
// get response as stream
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
// writes stream to zip file
FileStream writeStream = new FileStream(fullZipFileName, FileMode.Create, FileAccess.Write);
ReadWriteStream(responseStream, writeStream);
response.Close();
responseStream.Close();
The code for ReadWriteStream looks like this.
private void ReadWriteStream(Stream readStream, Stream writeStream)
{
// taken verbatum from http://www.developerfusion.com/code/4669/save-a-stream-to-a-file/
int Length = 256;
Byte[] buffer = new Byte[Length];
int bytesRead = readStream.Read(buffer, 0, Length);
// write the required bytes
while (bytesRead > 0)
{
writeStream.Write(buffer, 0, bytesRead);
bytesRead = readStream.Read(buffer, 0, Length);
}
readStream.Close();
writeStream.Close();
}
The building of the post string is taken from my previous forms app that works. I compared the resulting values in poststring for both sets of code (my working forms app and this one) and they're identical.
I'm not even sure how to troubleshoot this further. Anyone see anything obvious as to why this isn't working?
Conclusion 10/23/09
I finally have this working. A couple of important hurdles I had to get over. I had some problems with the ReadWriteStream method code that I got online. I don't know why, but it wasn't working for me. A guy named JB in Claudio Lassala's Virtual Brown Bag meeting helped me to come up with this code which worked much better for my purposes:
private void WriteResponseStreamToFile(Stream responseStreamToRead, string zipFileFullName)
{
// responseStreamToRead will contain a zip file, write it to a file in
// the target location at zipFileFullName
FileStream fileStreamToWrite = new FileStream(zipFileFullName, FileMode.Create);
int readByte = responseStreamToRead.ReadByte();
while (readByte != -1)
{
fileStreamToWrite.WriteByte((byte)readByte);
readByte = responseStreamToRead.ReadByte();
}
fileStreamToWrite.Flush();
fileStreamToWrite.Close();
}
As Will suggested below, I did have trouble with the authentication. The following code is what worked to get around that issue. A few comments inserted addressing key issues I ran into.
string targetHref = "http://wwwcf.nlm.nih.gov/umlslicense/kss/login.cfm";
HttpWebRequest firstRequest = (HttpWebRequest)WebRequest.Create(targetHref);
firstRequest.AllowAutoRedirect = false; // this is critical, without this, NLM redirects and the whole thing breaks
// firstRequest.Proxy = new WebProxy("127.0.0.1", 8888); // not needed for production, but this helped in order to debug the http traffic using Fiddler
firstRequest.Method = "POST";
firstRequest.ContentType = "application/x-www-form-urlencoded";
// build post string containing authentication information and add to post request
StringBuilder poststring = new StringBuilder("returnUrl=" + fixCharacters(targetDownloadFileUrl));
poststring.Append(getUsernameAndPasswordString());
poststring.Append("&login2.x=0&login2.y=0");
// convert to required byte array
byte[] postBytes = Encoding.UTF8.GetBytes(poststring.ToString());
firstRequest.ContentLength = postBytes.Length;
// write post to request
Stream postStream = firstRequest.GetRequestStream();
postStream.Write(postBytes, 0, postBytes.Length); // Fiddler shows that post and response happen on this line
postStream.Close();
// get response as stream
HttpWebResponse firstResponse = (HttpWebResponse)firstRequest.GetResponse();
// create new request for new location and cookies
HttpWebRequest secondRequest = (HttpWebRequest)WebRequest.Create(firstResponse.GetResponseHeader("location"));
secondRequest.AllowAutoRedirect = false;
secondRequest.Headers.Add(HttpRequestHeader.Cookie, firstResponse.GetResponseHeader("Set-Cookie"));
// get response to second request
HttpWebResponse secondResponse = (HttpWebResponse)secondRequest.GetResponse();
// write stream to zip file
Stream responseStreamToRead = secondResponse.GetResponseStream();
WriteResponseStreamToFile(responseStreamToRead, fullZipFileName);
responseStreamToRead.Close();
sl.logScriptActivity("Downloading update.");
firstResponse.Close();
I want to underscore that setting AllowAutoRedirect to false on the first HttpWebRequest instance was critical to the whole thing working. Fiddler showed two additional requests that occurred when this was not set, and it broke the rest of the script.
You're trying to use UI controls to do something in a windows service. This will never work.
What you need to do is just use the WebRequest and WebResponse classes to download the contents of the webpage.
var request = WebRequest.Create("http://www.google.com");
var response = request.GetResponse();
var stream = response.GetResponseStream();
You can dump the contents of the stream, parse the text looking for updates, and then construct a new request for the URL of the file you want to download. That response stream will then have the file, which you can dump on the filesystem and etc etc.
Before you wonder, GetResponse will block until the response returns, and the stream will block as data is being received, so you don't need to worry about events firing when everything has been downloaded.
You definitely need to re-think your approach (as you've already begun to do) to eliminate the Forms-based application approach. The service you're describing needs to operate with no UI at all.
I'm not familiar with the details of System.WebClient, but since it
provides common methods for sending
data to and receiving data from a
resource identified by a URI,
it will probably be your answer.
At first glance, WebClient.DownloadFile(...) or WebClient.DownloadFileAsync(...) will do what you need.
The only thing I can add is that once you have scraped your screen and have the fully qualified name of the file you want to download, you could pass it along to the Windows/DOS command 'get' which will fetch files via HTTP. You can also script a command-line FTP client if desired. It's been a long time since I tried something like this in Windows, but I think you're almost there. Once you have fetched the correct file, building a batch file to do everything else should be pretty easy. If you are more comfortable with Unix, google "unix services for windows" just keep an eye on the services they start running (DHCP, etc). There are some nice utilities which will let your treat dos as a unix-like shell (ls -l, grep, etc) Finally, you could try another language like Perl or Python but I don't think that's the kind of advice you were looking for. :)