What's the minimum memory needed to run a RAR decompression algorithm?
I want to port a RAR decompression algorithm to mobiles (iPhone, Android and BlackBerry) and want to know if there's a bare minimum of memory needed before starting. I've heard that RAR decompression requires much more memory than ZIP decompression.
Quite a lot. The maximum size of the dictionary are 4 MB, but at least the official unrar library (which is built from the same source as WinRAR) takes over 24 MB in some decompression algorithms.
(as to the last statement: note that the t is at least 1 MB (uint t=SASize << 20;), but can be more because SASize may be more than 1)
Can't give you a concrete number, but I remember using WinRAR back in 2001 on my PocketPC with only 64mb of RAM, about half of which was shared for storage -- so I'm pretty sure a modern phone should suffice.
There are plenty of Comic Viewers on the iPhone that support .cbr so I its doable.
Related
I was reading this article Why mmap is faster than system calls, where the main difference appeared to be mmap's ability to use vector instructions like AVX-2, something system calls can't.
I understand that the SIMD instructions used by GPUs tend to be much wider. A Nvidia warp of size 32 operating on float32 = 1024 bits (?) vs 256 bits of AVX-2. So potentially a 4x speedup. I guess this is not used in traditional discrete gpu settings as host-to-device (and back) copy would outweigh any benefit from wide registers.
However in APUs, GPU shares memory with CPU, eliminating the need for these expensive copies. I was wondering if those GPU instructions can therefore be used to accelerate mmap like vector operations further (numpy is another example). Has it already been done (in M1 mac or any CPUs with integrated graphics)? or can you please detail the architectural issues that prevent this?
You're kind of asking 2 separate questions: whether an OS (or user-space standard libraries?) can use GPGPU to speed up reading from the pagecache (into user-space memory with a read system call, or from an mmaped region). And separately whether GPGPU on normally-allocated process memory (and/or the pagecache) can avoid a copy to memory dedicated to the GPU.
For the 2nd part Apple has said the answer is yes for MacOS on M1 thanks to making the integrated GPU's memory accesses cache-coherent with the CPU. I think AMD made similar suggestions that copying could be avoided in graphics or GPGPU drivers on their APUs (Fusion IIRC?), but IDK if software ever took full advantage.
For the first part; doubtful. Large memory copies are bottlenecked by DRAM bandwidth, not CPU-core <-> L1d cache bandwidth (which scales with SIMD register width). On x86, an AVX2 loop on a single core can come pretty close to maxing out the DRAM bandwidth of an Intel "client" chip (quad-core or similar, not a big xeon with a higher-latency interconnect). Single-core bandwidth (to L3 or DRAM) tends to be limited by the number of outstanding cache misses that a core can track, not by doing the copy with fewer instructions. That mostly helps in terms of seeing farther with the same size out-of-order execution window, to start page walks sooner across page boundaries and stuff like that. See Why is std::fill(0) slower than std::fill(1)? for SSE (16-byte) vs. AVX (32-byte) vectors.
GPU offload would thus not help for large copies. It could only possibly help for small copies, and then it would not leave the copy result hot in L1d cache of the CPU. And/or not be able to take advantage of the source or destination already being hot in L1d cache of a CPU working with the data.
Also, setup overhead (to communicate with the GPU, going outside the current core) would dominate any faster copying for small copies.
I am currently facing a new batch of hyperspectral images in Matlab. Usually, I have no problems, but this time images of around 2000x2000x324 (double) have flooded me with "Out of memory" messages for all the image processing and statistical analyses I was a happily doing before.
My pc is relatively good ( Intel Core i7 and a total of available RAM of 15.8 GB but says available 7), so I am surprised.
I was wondering if you have any suggestions, strategy for dealing with such files or I am missing something here.
I have been suggested to switch to python for larger files, but with Matlab I have been having an intimate relationship for a while now.
Thank you very much for your support.
E
For example...
16GB USB flash drive's real capacity is almost 14.8~15.4 GiB.
these are different by each manufaturers or models.
In this situation....
How can I expect real minimum capacity?(GiB or Sectors)
Is there a standard? or de facto?
Back in the days, the size for hard-drives were always in gibibytes (1000 mibibytes) instead of gigabytes. That accounts for 7,4% size difference when comparing a gigabyte with a gibibyte.
I need to spec out a new computer. I'm only going to use this computer for developing Scala software. I'm going to be running Intellij, doing builds with Maven and SBT, and perhaps firing up a couple of Virtual Machines. I'm going to building a mixture of fairly large Play Framework and micro-services. What is a reasonable machine for doing this work?
The Scala compiler still has poor parrelisation. I doubt that's going to change before you'll be due an upgrade. For this reason I would suggest as a minimum using a Haswell 4670. Going up to an i7 will probably be of doubtful benefit. if you want to spend extra money over-clock a 4770K or a 4670k. If you've really got money to burn use an Ivybridge 4960x, but you won't see much benefit for that extra money. Intel beats AMD on core for core performance. Make sure you've got a 4 memory slot motherboard. 2 Eight Gig DDR3 1600 sticks are probably more than sufficient but allow for an upgrade to 32 Gig in a year or so's time when hopefully memory's come down in price.
As already stated a decent SSD. Run your operating System, your IDE and your projects off the SSD. You'll want a SATA drive for mass storage.
Anyway above $1500 or so for the Base unit diminishing returns set in rapidly. Unless you've really got money to burn.
You'll probably want a graphic cards to run multiple monitors. An AMD 7790 should do the job. I'm assuming that a budget of a 1000 to 1500 dollars for a base unit is not as issue. Personally I find 3 24" 1920 * 1200 monitors just right for civilised development.
You should turn your focus to these key components:
CPU
Since Scala (and the compiler) parallelize well: go for more cores, the more the better. Depending on your budget you can think of multi-CPU systems.
RAM
A lot of RAM helps a lot. I am pretty happy with 16GB, but depending on the size you are planning you might need more, but 16 is a decent amount. You can also think about using RAM to make a RAM-disk for faster compilation etc.
HDD
You definitely want to have a fast SSD. You should look for one with high IOPS, the transfer rate is for developing not that important. If you have a large budget, you can go for 2 SSDs in RAID-0. But be aware that some RAID controllers are not fast enough and will not give you the full possible performance of your SSD RAID.
What limits the size of a memory-mapped file? I know it can't be bigger than the largest continuous chunk of unallocated address space, and that there should be enough free disk space. But are there other limits?
You're being too conservative: A memory-mapped file can be larger than the address space. The view of the memory-mapped file is limited by OS memory constraints, but that's only the part of the file you're looking at at one time. (And I guess technically you could map multiple views of discontinuous parts of the file at once, so aside from overhead and page length constraints, it's only the total # of bytes you're looking at that poses a limit. You could look at bytes [0 to 1024] and bytes [240 to 240 + 1024] with two separate views.)
In MS Windows, look at the MapViewOfFile function. It effectively takes a 64-bit file offset and a 32-bit length.
This has been my experience when using memory-mapped files under Win32:
If your map the entire file into one segment, it normally taps out at around 750 MB, because it can't find a bigger contiguous block of memory. If you split it up into smaller segments, say 100MB each, you can get around 1500MB-1800MB depending on what else is running.
If you use the /3g switch you can get more than 2GB up to about 2700MB but OS performance is penalized.
I'm not sure about 64-bit, I've never tried it but I presume the max file size is then limited only by the amount of physical memory you have.
Under Windows: "The size of a file view is limited to the largest available contiguous block of unreserved virtual memory. This is at most 2 GB minus the virtual memory already reserved by the process. "
From MDSN.
I'm not sure about LINUX/OSX/Whatever Else, but it's probably also related to address space.
Yes, there are limits to memory-mapped files. Most shockingly is:
Memory-mapped files cannot be larger than 2GB on 32-bit systems.
When a memmap causes a file to be created or extended beyond its current size in the filesystem, the contents of the new part are unspecified. On systems with POSIX filesystem semantics, the extended part will be filled with zero bytes.
Even on my 64-bit, 32GB RAM system, I get the following error if I try to read in one big numpy memory-mapped file instead of taking portions of it using byte-offsets:
Overflow Error: memory mapped size must be positive
Big datasets are really a pain to work with.
The limit of virtual address space is >16 Terabyte on 64Bit Windows systems. The issue discussed here is most probably related to mixing DWORD with SIZE_T.
There should be no other limits. Aren't those enough? ;-)