The below documentation given for Sophos Intellix APIs does not mention the restrictions on file size or a number of submissions that can be made in a given time, are there any such restrictions in place or we can submit the file of any size and can we make any number of submissions in a given time:
https://api.labs.sophos.com/doc/index.html
The API docs says "The maximum size of the file is 4.5 MB (It will be increased in upcoming versions)."
The service is in Early Access Program (EAP) so do not expect so high performance.
Related
I'm a bit confused after reading this doc.
The doc says:
The fragments of the file must be uploaded sequentially in order. Uploading fragments out of order will result in an error.
Does that mean that, for one file divided into #1~10 fragments in order, I can only upload fragment 2 after I finish uploading fragment 1? If so, why is it possible to have multiple nextExpectedRanges? I mean, if you upload fragments one by one, you can make sure that previous fragments have already been uploaded.
According to the doc, byte range size has to be a multiple of 320 KB. Does that imply that the total file size has to be a multiple of 320 KB also?
There are currently some limitations that necessitate this sequencing requirement, however the long-term goal is to not. As a result, the API reflects this by supporting multiple nextExpectedRanges, but does not currently leverage it.
No, multiples of 320KiB are just the ideal size. You can choose others, and you can mix them. So for you scenario you could use all 320KiB chunks, except for the last one which would be whatever size is relevant to hit the overall size of your file.
We are getting conflicting reports on the maximum size of JSON blobs that DB2 will allow queries on (e.g. functions like JSON_VAL and JSON_TABLE.)
There is evidence that it is limited to 16M, but I have found nothing conclusive. For example, here is a link to an IBM tech note regarding DB2 11 for z/OS. The Setup & Configuration section shows DB2’s definitions for SYSTOOLS.BSON2JSON, which seems to declare the BSON value as a 16M CLOB.
On the other hand, one source told me that larger sizes ARE allowed but it will not perform well due to DB2's inability to cache a value larger than 16M. If true, this would at least allow us to run BSON queries in our development environment, or for one-time data extraction.
Can anyone point to a more definitive answer?
IBM confirmed in response to a support request that there is a 16 MB limit, imposed by the JSON2BSON and BSON2JSON functions. They note that MongoDB has the same limit.
They implied that if you implemented your own versions of JSON2BSON and BSON2JSON in C or Java, you could get around the limit. But they have no plans to increase the limit themselves, presumably because the values would not be cacheable by DB2.
For my specific purpose, I need to alter Overpass API's definition of an area to include all buildings, whether or not they have names (so is_in() will return these buildings when requested).
To achieve this, I've installed a local copy of the API with 3 specific countries and have modified the os3m script that generates areas to suite my definition -- so far, so good.
Obviously this script will require more memory than the default one as it is handling a lot more ways. The machine I'm running on has 16GB of RAM. If I specify 2GB or less for the script (i.e. element-limit="2073741824") then it will run out of memory, but specifying any more (even by 100MB) will always result in the error:
Dispatcher_Client::request_read_and_idx::timeout
after just a few seconds.
The question is, how I can tell the Overpass API/dispatchers that using more than 2GB is perfectly fine, and in fact, allow it to allocate up to ~15GB for this query?
You could try to increase both values for total_available_space in settings.cc (currently at 4GB) and recompile Overpass API from source.
AFAIK nobody has tried to process a huge number of buildings via areas before, so be prepared for further issues. The Overpass API developer list may be a good place to discuss this, also to get Roland's attention (Overpass API developer/maintainer), as he's not around on stackoverflow.
We're in the process of building an internal, Java-based RESTful web services application that exposes domain-specific data in XML format. We want to supplement the architecture and improve performance by leveraging a cache store. We expect to host the cache on separate but collocated servers, and since the web services are Java/Grails, a Java or HTTP API to the cache would be ideal.
As requests come in, unique URI's and their responses would be cached using a simple key/value convention, for example...
KEY VALUE
http://prod1/financials/reports/JAN/2007 --> XML response of 50Mb
http://prod1/legal/sow/9004 --> XML response of 250Kb
Response values for a single request can be quite large, perhaps up to 200Mb, but could be as small as 1Kb. And the number of requests per day is small; not more than 1000, but averaging 250; we don't have a large number of consumers; again, it's an internal app.
We started looking at MongoDB as a potential cache store, but given that MongoDB has a max document size of 8 or 16Mb, we did not feel it was the best fit.
Based on the limited details I provided, any suggestions on other types of stores that could be suitable in this situation?
The way I understand your question, you basically want to cache the files, i.e. you don't need to understand the files' contents, right?
In that case, you can use MongoDB's GridFS to cache the xml as a file. This way, you can smoothly stream the file in and out of the database. You could use the URI as a 'file name' and, well, that should do the job.
There are no (reasonable) file size limits and it is supported by most, if not all, of the drivers.
Twitter's engineering team just blogged about their SpiderDuck project that does something like what you're describing. They use Cassandra and Scribe+HDFS for their backends.
http://engineering.twitter.com/2011/11/spiderduck-twitters-real-time-url.html
The simplest solution here is just caching these pieces of data in a file system. You can use tmpfs to ensure everything is in the main memory or any normal file system if you want the size of your cache be larger than the memory you have. Don't worry, even in the latter case the OS kernel will efficiently cache everything that is used frequently in the main memory. Still you have to delete the old files via cron if you're using Linux.
It seems to be like an old school solution, but it could be simpler to implement and less error prone than many others.
I have an idea for a webapp for the iPhone but its unknown to me how much data can be stored in mobile Safari's SQLite db. I tried searching through the Apple docs but found nothing:
Safari Client-Side Storage and Offline Applications Programming Guide: Using the JavaScript Database
Most of these answers are totally wrong. Safari will not allow you to create SQLite databases over 50MB (or expand existing databases beyond that size).
This is a limit imposed by Safari - as other people have noted, SQLite itself supports much larger databases that you can use from native apps. But webapps are limited to 50MB.
It might be useful to note that this is per database - if you really need the extra space, you can create multiple databases, although this would obviously cause a lot of hassle.
It's as the other posters say. You're only limited by the drive space on the device.
You also need to consider your in memory footprint though. There is a finite amount of memory on the iphone, and in general it's quiet small, so the amount of data/hydrated objects you'll be able to have in memory is another potential limitation for your app.
There are a LOT of people answering that have clearly never tested it. I am on the latest version of iOS (4.3.3) and have set up a system to create multiple databases and keep them under 45 MB but found that the 50 MB cap is for the site as a whole. So, no matter how much you split the data up, it still restricts it to an aggregated cap of 50 MB.
The database size limit on safari mobile, is 50 mb per site not per database. i have tested this. even if you have an extra empty database you cannot add to it if the total size of all databases on a single site is 50 mb
whats worth noting as well is that characters are saved as double bytes on websql, that is 2 million characters will be 4 megabytes not 2 megabytes on disk.
You are only limited by the amount of free space on the device.
I'm not sure. If you were doing your own application you'd be limited by free space on the device and to some extent in memory footprint (as Bryan McLemore points out).
However since you're looking at using JavaScript inside of Safari there's no easy way to tell. According to the document you found it looks like it may be limited by site, but there's nothing telling you how much. I'd suggest writing a quick script to fill up the database and figure out how much it actually is. After that, I'd probably halve that value and assume I'd be always be able to use that much.
Be sure to report back so we'll all know!
It's most likely 32 terabytes... which is well over the available disk space.
I reached this number by multiplying the maximum page size by the maximum page count listed at the bottom of the SQLite limits page.
Limits In SQLite
"Limits" in the context of this article means sizes or quantities that can not be exceeded. We are concerned with things like the maximum number of bytes in a BLOB or the maximum number of columns in a table.
SQLite was originally designed with a policy of avoiding arbitrary limits. Of course, every program that runs on a machine with finite memory and disk space has limits of some kind. But in SQLite, those limits were not well defined. The policy was that if it would fit in memory and you could count it with a 32-bit integer, then it should work.
Unfortunately, the no-limits policy has been shown to create problems. Because the upper bounds were not well defined, they were not tested, and bugs (including possible security exploits) were often found when pushing SQLite to extremes. For this reason, newer versions of SQLite have well-defined limits and those limits are tested as part of the test suite.
As of version 3.6.19 (all statistics in the report are against that release of SQLite), the SQLite library consists of approximately 65.7 KSLOC of C code. (KSLOC means thousands of "Source Lines Of Code" or, in other words, lines of code excluding blank lines and comments.) By comparison, the project has 690 times as much test code and test scripts - 45409.7 KSLOC.
The default storage limit on iPhone seems to be 5mb
davibe has done some work to raise the limit up to 1GB with his PhoneGap plugin.
https://github.com/davibe/Phonegap-SQLitePlugin
The plugin calls the native sqlite3 API, with a wrapper on the Javascript side.
The relevant code extracted from sqlite.js are:
update origins set quota = '999999999999' where origin = 'file__0';
"update databases set estimatedSize = '999999999999' where name = '" + dbName + "';'";
Caution: my iphone is jailbroken! But I don't suspect that this changes anything.
The limit of 50MB is no longer correct.
On my iPhone 4S with iOS 6.1 I have a database of 58.66 MB (448496 records) for my webclip (website pinned to the springboard).
No special tricks, just standard HTML5 usage.
Maximum Database Size
Please refer Official Sqlite site
Every database consists of one or more "pages". Within a single database, every page is the same size, but different database can have page sizes that are powers of two between 512 and 65536, inclusive. The maximum size of a database file is 2147483646 pages. At the maximum page size of 65536 bytes, this translates into a maximum database size of approximately 1.4e+14 bytes (140 terabytes, or 128 tebibytes, or 140,000 gigabytes or 128,000 gibibytes).
This particular upper bound is untested since the developers do not have access to hardware capable of reaching this limit. However, tests do verify that SQLite behaves correctly and sanely when a database reaches the maximum file size of the underlying filesystem (which is usually much less than the maximum theoretical database size) and when a database is unable to grow due to disk space exhaustion.