Is there a way to make a form where it can simultaneously upload to several servers at once?
Currently in my web application, I am asking the users to type in some info + select a few files to upload.
Title, Description, Info, etc
File 0
File 1
File 2
File ...
On the backend, I'm using Pylons. Currently it accepts POST of (info + all files), processes the info and the first file (file 0), and uploads them again to Amazon S3. I only need to process info and the 'file 0' on my own server, the rest of the files I can pass through directly to S3 via a POST.
Is there a way to make a form where the info+file0 will be POST'ed to one server, and the rest of the files be POST'ed directly to S3?
While you could employ JavaScript trickery to make clicking on a submit button result in multiple POST requests to multiple servers, such a solution will almost certainly cause more problems than it solves. You should probably stick with your current method.
Check this and this out.
My solution would involve ajax upload(s) to S3 as a first step. Once you're done with the files, you can submit all the other fields (not ajax) to your webserver, eventually adding the generated S3 links.
Related
I am using CloverETL Designer for ETL operations and I want to load some csv files from GCS to my Clover graph. I used FlatFileReader and tried to get file using remote File URL but it is not working. Can someone please detail the entire process here??
The path for file in GCS is
https://storage.cloud.google.com/PATH/Write_to_a_file.csv
And I need to get this csv file into the FlatFileReader in CloverETL Designer
You should use the Google Cloud Storage API to GET the file; Clover's HTTPConnector component will allow you to pass in the appropriate parameters to make a GET request (you will presumably have to do an OAuth2 authentication first to get a token), and send the output to a local destination specified in "Output File URL." Then you can use a FlatFileReader to read from that local file.
GCS has several different ways to download files from your buckets. You can use the console and the Cloud Storage browser. Steps: open the storage browser, navigate to the object you want to download, right click, and save to your chosen local folder. If you use Chrome the save appears as “Save Link As…”.
To use the GS Utility, use this command:
`gsutil cp gs://[BucketName]/[ObjectName] [ObjectDestination]`.
Or you can use client libraries or the REST APIs to download files. With these last options you could work with a number of files or create a job to download them. Once they are in a location known to Clover ETL the process is straightforward.
Within Clover designer, under the navigation pane you can right click a folder and choose import. Pick the one in which you placed your GCS file. Once the file is imported then you can use data from it like any other datafile in Clover. Since this is a .csv file, remember to edit your metadata (right click the component, choose extract metadata then edit inside the Metadata Editor -- for data types, labels and such.) Assign metadata to the edges of your components so they know what is coming in/going out of that step. Depending on your file, this process may be repeated many times.
Even with an ETL tool, getting the data and data types correct can be tricky. If you have questions about how to configure data types or your edges in an ETL project, a wiki may help. The web has additional resources may help you get the end analysis you’re looking for.
We have a website, which allows users to upload documents (word, pdf, PPT, etc.).
We are uploading files to Amazon S3. So, all files will have it's own web URL.
For these uploaded documents, we would like to generate thumbnails. This thumbnail needs to be generated based on it's content (like Google document viewer).
Is there any Service/API, which generates thumbnails of documents by it's URL?
Thanks and Regards,
Ashish Shukla
You could roll your own solution. I'm evaluating 2JPEG and it appears to support 275 formats including Word, Excel, Publisher & Powerpoint files. fCoder recommends running 2JPEG as a scheduled background task. The command line syntax is pretty comprehensive. I don't think it has the ability to process remote AWS files, but you could retain it locally temporarily, generate the thumbnail and then delete the local source file.
Here's a sample snippet to generate a thumbnail for a specific file:
2jpeg.exe -src "c:\files\myfile.docx" -dst "c:\files" -oper Resize size:"100 200" fmode:fit_width -options pages:"1" scansf:no overwrite:yes template:"{Title}_thumb.jpg" silent:yes
You should also take a look at AWS Lambda. In fact, this presentation from the AWS re:Invent 2014 conference shows a live example of using Lambda to generate thumbnail images. This solution will be very reliable, and very cost-effective, but has the downside that you'll be responsible for maintaining the code, or debugging issues.
I am writing a web service using play 2.0 in scala.
After getting client's configuration parameters, then the web service will take a while to generate some files. (Note: some files can be short, some other files can take long).
I figured out how to get client's configurations and trigger my computation,
now I need to present the generated files to client side at the end of the session.
So questions are:
How to present the files to client?
I'm thinking about returning a static folder link to client, so that they can go into the folder to
see further what files are there, and so when clicking on the files, the details of the files can be shown, either jpg or text.
But how?
How to not to block client during the generation of the files?
For example, client can still click on the files that are available at that moment.
Then here comes the third question:
How to let the client side know that the long-time computation file is available and listed in the folder?
I'm pretty new to web application, thanks for any suggestion, advice, or little examples are greatly appreciated too!
I would create a web interface to show the created files to your clients and create a background job to for the long computation process. You can create a wait page where a javascript check if the file is created.
I am developing a simple iPad application that should submit some text from a form to a .csv file. I could manage to update the .csv file which is saved locally in the documents folder on my computer. However, I need to keep the file on a server, probably download the file, append data, and upload it again (Export a bulk of data to the file on server). Any idea how I could do something like that?
I guess ftp might be the easiest way, see this question
your other options likely involve writing a server-side service to post data to.
I have following need:
1) Users will upload .xls or .csv files in "uploads" folder.
2) "uploads" folder have to be constantly monitored, and with each new file added to him, a job has to be started.
3) Job will process data from .xls or .csv file so they meet DB table structure, and write this data into DB table.
This have to be automated process, and I'm looking for all-in-one solution tool.
You didn't tell on which operating system, and you didn't tell if the user upload the files on a different server, or not. If the upload goes thru a web application (using an HTTP POST request), it is also different.
And I'm not sure that your wish scales well with many users.
You should take a look at Pentaho Data Integration, a.k.a. Kettle: http://sourceforge.net/projects/pentaho/
With Kettle you can desing a Job that pools the upload directory and once a file is found makes all the needed transformation and input on the desired database table.