I'm trying to build a website on Squarespace, in which the site links to a database file. It's stored in a standard file system on a server tower with cluster. No SQL architecture or anything that I explicitly know of. Unfortunately Google Drive isn't an option due to the size of the file ( > 200 GB). I'm rather lost due to the size constraint -- does anyone have an idea about how to do this? Can I set up some sort of server query using a link on the site? Can I upload the file from my computer and store it somewhere in the backend? Thanks.
"...the size of the file ( > 200 GB)..."
Unfortunately, Squarespace's own upload limitations are far below this for the two places where files like that can be stored: file storage (20MB) and developer-mode '/assets' folder (10MB). The CSS-/Style-related storage only supports images (and likely has a limit of less than 20MB). Digital download products can be 300MB (still to small for your file) and likely can't be linked-to and accessed as you'd need for your application.
"...Can I set up some sort of server query using a link on the site?..."
If you mean a query on some other service besides Squarespace which connects to the file hosted on your Squarespace site, the answer is no simply because there's no way to upload the file to Squarespace due to its size. If, however, your mean a query from your Squarespace site to the file hosted elsewhere, then this must be done using JavaScript and done entirely client-side due to Squarespace's lack of support for server-side languages.
"...Can I upload the file from my computer and store it somewhere in the backend?..."
See the options mentioned above, though all have file size limits below that of your file.
If you are able to utilize the file on your site using client-size/front-end JavaScript only, then perhaps you could host the file on Amazon S3 or other such provider and access it that way.
Related
I have no experience with OpenText.
I have a client who hosts a large amount of documents on OpenText Content Server. They would like to do things such as:
Navigating through the file system and moving files between directories in the file system
Upload files (PDF, MS Word docs, etc.) from local machines to the cloud file system
Retrieve files from the cloud file system to be stored and opened on local machines, or read the content of files directly through the API
Changing access rights (read/write) of users on Content Server
Are these options possible through the AppWorks REST API?
Lastly, is there any reporting analytics data available about the Content Server, e.g.:
Number of uploads/downloads in a timeframe
Number of users logged in a timeframe
etc.
The first 4 items are possible out of the box with the newer versions.
The latter two would need reports to be created on the server side which in turn then be called via rest.
I want to upload physical files on Content with using GOS. So my BASIS consultant made some preferences for this in OAC0, OAC1 tcodes etc. But now, when I try to upload a file message appears like this:
system error when accessing knowledge provider.
Anyone have idea for this issue?
This means that your BASIS gut improperly set up knowledge provider in your system.
Binary GOS attachments are stored through SAPOffice, particularly through Knowledge Providers, and how they are stored is maintained in transaction KPRO, where certain type of content server can be specified. More info can be obtained in notes 904711 and 530297However, if you are not proficient in SAP administration it's better to assign this to your BASIS. If he is not able to conduct his work, you'd better find another one:)
I am asking for advice on possibly better solutions for the part of the project I'm working on. I'll first give some background and then my current thoughts.
Background
Our clients can use my company's products to generate potentially large data sets for use in their industry. When the data sets are generated, the clients will file a processing request to us.
We want to send the clients a summary email which contains some statistical charts as well as sampling points from the data sets so they can do some initial quality control work. If the data sets are of bad quality, they don't need to file any request.
One problem is that the charts and sampling points can be potentially too large to be sent in an email. The charts and the sampling points we want to include in the emails are pictures. Although we can use low-quality format such as JPEG to save space, we cannot control how many data sets would be included in the summary email, so the total size could still exceed the normal email size limit.
In terms of technologies, we are mainly developing in Python on Ubuntu 14.04.
Goals of the Solution
In general, we want to present a report-like thing to the clients to do some initial QA. The report may contains external links but does not need to be very interactive. In other words, a static report should be fine.
We want to reduce the steps or things that our clients must do to read the report. For example, if the report can be just an email, the user only needs to 1). log in and 2). open the email. If they use a client software, they may skip 1). and just open and begin to read.
We also want to minimize the burden of maintaining extra user accounts for both us and our clients. For example, if the solution requires us to register a new user account, this solution is, although still acceptable, not ranked very high.
Security is important because our clients don't want their reports to be read by unauthorized third parties.
We want the process automated. We want the solution to provide programming interface so that we can automate the report sending/sharing process.
Performance is NOT a critical issue. Our user base is not large. I think at most in hundreds. They also don't generate data that frequently, at most once a week. We don't need real-time response. Even a delay of a few hours is still acceptable.
My Current Thoughts of Solution
Possible solution #1: In-house web service. I can set up a server machine and develop our own web service. We put the report into our database and the clients can then query via the Internet.
Possible solution #2: Amazon Web Service. AWS is quite mature but I'm not sure if they could be expensive because so far we just wanna share a report with our remote clients which doesn't look like a big deal to use AWS.
Possible solution #3: Google Drive. I know Google Drive provides API to do uploading and sharing programmatically, but I think we need to register a dedicated Google account to use that.
Any better solutions??
You could possibly use AWS S3 and Cloudfront. Files can easily be loaded into S3 using the AWS SDK's and API. You can then use the API to generate secure links to the files that can only be opened for a specific time and optionally from a specific IP.
Files on S3 can also be automatically cleaned up after a specific time if needed using lifecycle rules.
Storage and transfer prices are fairly cheap with AWS and remember that the S3 storage cost indicated is by the month so if you only have an object loaded for a few days then you only pay for a few days.
S3: http://aws.amazon.com/s3/pricing
Cloudfront: https://aws.amazon.com/cloudfront/pricing/
Here's a list of the SDK's for AWS:
https://aws.amazon.com/tools/#sdk
Or you can use their command line tools for Windows batch or powershell scripting:
https://aws.amazon.com/tools/#cli
Here's some info on how the private content urls are created:
http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/PrivateContent.html
I will suggest to built this service using mix of your #1 and #2 options. You can do the processing and for transferring the data leverage AWS S3 which is quiet cheap.
Example: 100GB costs like approx $3.
Also AWS S3 will be beneficial as you are covered for any disaster on your local environment your data will be safe in S3.
For security you can leverage data encryption and signed URLS in AWS S3.
We've noticed that when trying to GET a shared file (doesn't matter if you're sharing, or you're being shared with) using the REST API, the FileSize field is simply non-existent, which is actually rather troublesome for our usages.
We can certainly download the file via one of the various links in the response, and then detect the size, but that would require processing on our backend, and we'd much prefer to fail out (if need be) on our frontend before reaching that point.
Per the API documentation:
fileSize long The size of the file in bytes. This is only populated for files with content stored in Drive.
Are shared files not technically "stored" in Google Drive?
Thanks all!
-Cory
I must provide a solution where user can upload files and they must be stored together with some metadata, and this may grow really big.
Access to these files must be controlled, so they want me to just store them in DB BLOBs, but I fear PostgreSQL won't handle it properly over time.
My first idea was use some NoSQL DB solution, but I couldn't find any that would replace a good RDBMS and elegantly store files together. Then I thought on just saving these files in HD somewhere WebServer won't serve them, name them their table ID, and just load them on RAM and print them with proper content-type.
Could anyone suggest me any better solution for this?
I had the requirement to store many images (with some meta data) and allow controlled access to them, here is what I did.
To the cloud™
I save the image files in Amazon S3. My local database holds the metadata with the S3 location of the file as one column. When an authenticated and authorized user needs to see the file they hit a URL in my system (where the authentication and authorization checks occur) which then generates a pre-signed, expiring URL for the image and sends a redirect back to the browser. The browser is then able to load the image for a given amount of time (as specified in the signature within the URL.)
With this solution I have user level access to the resources and I don't have to store them as BLOBs or anything like that which may grow unwieldy over time. I also don't use MY bandwidth to stream the files to the client and get cheap, redundant storage for them. Obviously the suitability of this solution will depend on the nature of the binary files you are looking to store and your level of trust in Amazon. The world doesn't end if there is a slip and someone sees an image from my system they shouldn't. YMMV.