Upload file to S3 using filepicker.io and replace whitespace - filepicker.io

I've been uploading files to the amazon s3 using the javascript library filepicker.io and the implementation works ok, the problem I'm facing now is that when users upload files with white spaces or fancy characters those files are not accessible through http, i was wondering if there is any way to apply some kind of renaming to the file prior the uploading face.
I'm using the pickAndStore method from the filepicker.io api
filepicker.pickAndStore({
mimetype: ['video/*'],
services:['COMPUTER', 'DROPBOX', 'GOOGLE_DRIVE', 'BOX', 'URL', 'VIDEO', 'FTP'],
},{location: 's3'},function(fpfiles){
fpfile = fpfiles[0];
$('#temp_file_url').val('/' + fpfile.key);
});
Thanks.


You can specify a "path" parameter in both the filepicker.store() call and filepicker.pickAndStore(), so if you want to specifically strip the whitespace from the filenames but keep the rest the same, the structure would be:
filepicker.pick({...options...}, function(fpfile){
var new_name = process(fpfile.filename);
filepicker.store(new_name,
{path: new_name, filename: new_name}, function(){});
});


You can also use path on pickAndStore but you won't be able to preprocess the inkblob.

Related

How to download a file (csv.gz) from a url using Python 3.7

As with others who have posted in the past, I cannot figure out to download a csv.gz file from a URL in Python 3.7. I see posts but they only post a 2kb file.
I am a 100% newbie using Python. What follows is the code for one file that I am trying to obtain. I can't even do that. The final goal would be to request all files that start with 2019* using python. Please try the code below to save the file. As others stated, the file is just a name without the true content - Ref: Downloading a csv.gz file from url in Python
import requests
url = 'https://public.bitmex.com/?prefix=data/trade/20191026.csv.gz'
r = requests.get(url, allow_redirects=True)
open('20191026.csv.gz', 'wb').write(r.content)
Yields:
Out[40]:
1245
I've tried "wget" and urllib.request along with "urlretrieve" also.
I wish I could add a screenshot or attach a file. The file created is 2kb and not even a csv.gz file. But the true file that I can download from a web browser is 78mb. The file is 20191026.csv.gz not that it matters as they all do the same thing. The location is https://public.bitmex.com/?prefix=data/trade/
Again, if you know of a way to obtain all the files using a filter such that 2019*csv.gz would be fantastic.

You are trying to download the files from https://public.bitmex.com/?prefix=data/trade/.
To achieve your final goal of download all the files starting from 2019* you have to do in 3 steps
1) you read the content of https://public.bitmex.com/?prefix=data/trade/
2) convert the content into an list, from that filter out the file names which starting from 2019.
3) from the result list try to download the csv using the example which you referring.
Hope this approach will help you
Happy coding.

Filepicker.io gives "does not register as an accepted file" error when extension is allowed

I have a filepicker.io instance where I am using the pickAndStore function to allow users to upload various files, however while testing Microsoft Visio I found the files are being blocked / denied upload by a yellow error that states it does not register as an accepted file type (and lists out all the files it believes are allowed)?
In my logs of the arguments sent to the function, I can see the full array of file types I allow and the 4 variants of visio I added are clearly there:
The four I added:
".vss", ".vssx", ".vsd", ".vsdx"
Full array:
[".doc", ".dot", ".docx", ".docm", ".dotx", ".xls", ".xlt", ".xlsx", ".xltx", ".xlsm", ".xlsb", ".oft", ".msg", ".ppt", ".pptx", ".pptm", ".pps", ".ppsx", ".mpp", ".pub", ".pdf", ".html", ".mhtml", ".txt", ".rtf", ".csv", ".xml", ".css", ".zip", ".tar", ".rar", ".vss", ".vssx", ".vsd", ".vsdx", ".mp3", ".wav", ".swf", ".ics", ".srt", ".wmf", ".eps", ".ai", ".psd", ".gif", ".jpg", ".jpeg", ".png", ".bmp", ".m4v", ".mp4", ".flv", ".f4v", ".mov", ".wmv", ".wm", ".webm", ".3gp", ".3gpp", ".m2p", ".rv", ".rm", ".avi", ".3gp2", ".mpg", ".mpeg", ".ts", ".vp6", ".h264", ".arf", ".wrf", ".m2ts"]
However When I use "My Computer" as a source and upload any one of the twenty odd .vsd files I have to use as tests, all of them trigger the error to appear and deny upload:
The image i am seeing saying that .vsd does not register
I'm not sure what else I can do at this point to fix? I don't particularly want to have to use mimetype in this one instance as it suggests not to use this along side extension in the filepicker documentation.
Here is the link i used that provides various Visio files you can use to test. I would rather not use the files clients upload using our platform as I would need to ask permission and in case they are sensitive. I don't think there has been a single successful upload (of a visio file, others are fine) so I would be surprised if it was file specific.
https://www.microsoft.com/en-gb/download/confirmation.aspx?id=24023
Thanks!

All extensions are converted back to mimetype therefor you can't mixed extension & mimetype.
It appears that ".vss", ".vssx", ".vsd", ".vsdx" are in the database.
Could you post some of the files you are testing so we can check them ourselves.
Regards,
Dylan

Filepicker needs to include "application/vnd.ms-visio.viewer" in the mapping from those file extensions. It looks like that's what the browser is reporting the MIME type for those files to be.

Prevent Duplicate file uploads in FilePicker.io

I use FilePicker (now called FileStack) and I wanted to know if it's possible to prevent duplicate file uploads to a single container. For example, if I allow users to up upload some music files, how do I prevent them from adding the same music file twice in the same upload instance? The starter code is below:
filepicker.pickAndStore(
{
mimetype:"image/*",
multiple: true
},
{
location:"S3"
},
function(Blobs){
console.log(JSON.stringify(Blobs));
}
);

To only reliable way for finding duplicates would be compare files MD5 hashes.
MD5 hash is a unique identifier and it is available via filepicker API.
Example call:
https://www.filepicker.io/api/file/hFHUCB3iTxyMzseuWOgG/metadata?md5=true
Response:
{"md5": "f31dbf9b885e315d98e136f1db0daf52"}
To be even more reliable you can also compare file size.
So what I would recommend is storing md5 hash and files size together with file links in the database.

Change storage folder on file upload

I have form with fields like
'name, firstname, street, zip, city and image'.
The image was an upload
field. Everything works fine, i can upload the image as Filereferrence. But the image was uploaded every time into "fileadmin/". I want to upload the image into "/uploads/<extensionname>" or "fileadmin/user_uploads/<extensionname>".
The part where the upload and file moving was realized look like.
$storageRepository = $this->objectManager->get('TYPO3\\CMS\\Core\\Resource\\StorageRepository');
$storage = $storageRepository->findByUid('1');
$fileData = array();
$fileData['name'] = $_FILES['tx_oaevents_eventslisting']['name']['image'][0];
$fileData['type'] = $_FILES['tx_oaevents_eventslisting']['type']['image'][0];
$fileData['tmp_name'] = $_FILES['tx_oaevents_eventslisting']['tmp_name']['image'][0];
$fileData['size'] = $_FILES['tx_oaevents_eventslisting']['size']['image'][0];
$newFileObject = $storage->addFile(
$fileData['tmp_name'], $storage->getRootLevelFolder(), $fileData['name']
);
This line deines the storage folder form database:
$storage = $storageRepository->findByUid('1');
which is fileadmin as default. But what is the best way to change/modify this storage folder destination?

If you want to specify the destination folder for your upload, I suggest you take a look at the FAL File Upload example from Helmut Hummel. For his example, there is also a blogpost which describes how the file upload works and why you should use a TypeConverter for FAL file upload in TYPO3 6.2+
I have implemented the solution several times and it works just great, since it covers a lot of scenarios you may run into when implementing a file upload (e.g. validation, error handling, file replacement if file exists)

How can I rename file uploaded to s3 using javascript api?

'pickAndStore' method allows me to specify full path to the file, but I don't know it's extension at this point (file path has to be defined before file is uploaded, so it's not possible to provide a path with correct extension).
if I use 'pick' and then 'store' I have 2 files (because both methods uploads file to the s3). I can delete 'old' file, but it's not optimal and can be pain (take ages) with really big files.
Is there any better solution? Ideally to rename existing file.

Currently, there is no workaround for renaming file.
However, in our Javascript API v2 we are planing to add new callback function. onStart callback will be fired after user pick file but before file uploading. There could be option like renaming file based on original filename.
We will keep you updated.