Is there any difference between 8k block on 32bit file system vs 4k block on 64bit file system?
If there is, how big would the difference in the largest file size be? Would I calculate it the same way I would calculate 8k on 32bit and just change 4bytes on 32bit to 8bytes on 64bit?
Ex. 8k bytes w/ 32bit disk addresses
So 8192/4bytes = 2048*8KB = 16MB of file data (singly indirect)
and 2048*2048*8k = 32GB of file data (doubly indirect)
Ex. 4k bytes w/ 64bit disk addresses
4096/8 = 512*4k = 2MB of data (singly indirect)
and 512*512*4k = 10GB of file data (doubly indirect)
So if the example is true, there would be a big difference between the two (specially in triply indirect, which gives the max file size)?
If you have real-world filesystems in mind, you can check the maximum
file size on Wikipedia.
In particular, if it's ext4, the block size vs. number of bits
doesn't change the max size of an individual file.
If you're talking about a theoretical system, your calculations are
fine. Do keep in mind that inode-based filesystems typically
have direct blocks as well, ext4 defines 12 of them. So add 48k to
your 4k-block example, 96k to the 8k-block one. And don't forget to add the triply-indirect blocks!
Related
A 32-bit register can store 232 different values. The signed range of integer values that can be stored in 32 bits is -2,147,483,648 through 2,147,483,647 (unsigned: 0 through 4,294,967,295). Hence, a processor with 32-bit memory addresses can directly access 4 GiB of byte-addressable memory. so how this kind of processor deal with disk of size more than 4 gb?
Hence, a processor with 32-bit memory addresses can directly access 4 GiB of byte-addressable memory. so how this kind of processor deal with disk of size more than 4 gb?
For disks; typically they're not byte addressable and the smallest amount that can be read or written (the block size) is 512 bytes or larger (maybe 4096 bytes). Block numbers may also be larger than 32 bits (e.g. maybe 48 bit block numbers).
With 512-byte blocks and 48 bit block numbers (which was common in the late 1990s; for ATA and SATA, etc) you'd end up with a maximum disk size of 134217728 GiB.
Of course the CPU probably (see note) can't directly access any of the data on disk. Software (file system) has to ask a device driver to fetch the block/s it wants, and device driver asks hardware (disk controller) to copy data between disk and memory. Depending on OS; this software interface (used by file system to ask device driver to read or write blocks) most likely uses 64-bit block numbers (e.g. two 32-bit registers joined together).
Note: More recently, the possibility of using non-volatile RAM (e.g. https://en.wikipedia.org/wiki/3D_XPoint ) as storage changed things (it is byte addressable and does use physical addresses); but modern hardware is all "64-bit" (with physical addresses that may be 48 bits or larger in practice) so even though the theoretical limit is much smaller it's still large enough in practice (e.g. maybe 200000 GiB).
On x86-64, the main CPU architecture of most desktop computers, an interface is standardized in the AHCI (Advanced Host Controller Interface). The computer accesses the hard-disk using this interface which is, in practice, a PCI-Express device compliant with the PCI-Express specification.
With PCI, the CPU has a memory controller which will write to the PCI devices registers instead of RAM when writing to some portions of RAM. Software (the operating-system) will write in uncached RAM and it will instead write to the registers of the PCI device. This way it can tell the devices, including an AHCI, to do some operations like a DMA operation from the hard-disk to RAM and vice-versa.
I didn't read the specification in full but the specification probably holds 64 bits registers that can be written as 2 32 bits words. More often, software will write to the lower part of the register then to the higher part of the register. This allows any 32 bits computers to still be able to interact with AHCI. On a 64 bits computer, the registers will be written as one 64 bits write.
A 32 bits computer is thus still able to trigger DMA writes to some portions of the hard-disk which are much higher than 4GB.
a) Suppose there is a RAM of size 4GB installed in an intel Core i5 computer. If the system supports 64bit word then how many words can be stored in the RAM? Also how many bits will be required to address each Word uniquely?
There are a lot of variables to this question. How big are the words? How much RAM does Microsoft let Word use before it starts swapping data in a pagefile? Is any compression used?
If we assume that Word has access to all 4GB of RAM (the OS, and Word aren't using any in this situation), that there is no pagefile to dump to, and no compression going on in the background, we can do some simple math to find out. We know that typically a single ascii character takes up 1 byte of memory, and that there are 1024 bytes in a KB, 1024 KB in a MB, and so on. So, we just multiply until we get to 4GB worth of bytes. 1024 Bytes X 1024 KB X 1024 MB X 4 GB = 4,294,967,296 ascii characters.
Coincidentally, that's the maximum number of possible 32-bit addresses in 32bit systems, and why 32bit systems were unable to support more than 4GB of RAM without Physical Address Extension (PAE). With so little RAM, it doesn't matter that the program is 64bit, it just means it can handle addressing and integers larger than that if the RAM is available.
I hope that helps you in some way! I'm kind of concerned that you are having to worry about the maximum number of words that can be stored in 4GB.
For a 2GBytes memory, suppose its memory width is 8 bits….
what is the address space of the memory?
what is the address width of the memory?
I’m not looking for the answer to question, I’m just trying to understand the process of how to get there.
EDIT: all instances of Gb replaced with GB
The address-space is the same as the memory size. This was not true (for example) in 32 bit operating systems that had more than 2^32 bytes of memory installed. Since the number of bits used for addressing is not specified in your question, one can only assume it to be sufficient to address the installed memory. To contrast, while you could install more than 4GB in a 32 bit system, you couldn't access more than 4GB, since (2^32)-1 is the location of the last byte you could access. Bear in mind, this address-space must include video memory and any/all bioses in the system. This meant that in 32bit WinXP MS limited the amount of user-accessible memory to a figure considerably less than 4GB.
Since the memory width is 8 bits, each address will point to 1 byte. Since you've got 2GB, you need to use a number of bits for addressing that is equal-to or greater-than that which will allow you to point to any one of those bytes.
Spoiler:
Your address-space is 2GB and you need 31 bit wide addresses to use it all.
I do not know why Ubuntu becomes slow or even no response when I load a large MAT file (e.g., 1GB) in Matlab?
My Ubuntu is 14.04 and the PC has 8GB RAM.
Run memory at the command line to see what the capabilities of the system that freezes is, versus the one that doesn't. The important values are "Maximum possible array" and "Memory available for all arrays.", and are generally quite a bit lower than the available RAM.
These may vary depending on your version of MATLAB (particularly 32 vs. 64 bit - including 32 bit MATLAB running on a 64 bit system).
In addition, there is some compression in *.mat files (Version 7 onwards). Therefore, it may be that the file is 1GB but would require more than 1GB memory to fully read in. You can use matfile to examine the sizes of variables in the file.
What limits the size of a memory-mapped file? I know it can't be bigger than the largest continuous chunk of unallocated address space, and that there should be enough free disk space. But are there other limits?
You're being too conservative: A memory-mapped file can be larger than the address space. The view of the memory-mapped file is limited by OS memory constraints, but that's only the part of the file you're looking at at one time. (And I guess technically you could map multiple views of discontinuous parts of the file at once, so aside from overhead and page length constraints, it's only the total # of bytes you're looking at that poses a limit. You could look at bytes [0 to 1024] and bytes [240 to 240 + 1024] with two separate views.)
In MS Windows, look at the MapViewOfFile function. It effectively takes a 64-bit file offset and a 32-bit length.
This has been my experience when using memory-mapped files under Win32:
If your map the entire file into one segment, it normally taps out at around 750 MB, because it can't find a bigger contiguous block of memory. If you split it up into smaller segments, say 100MB each, you can get around 1500MB-1800MB depending on what else is running.
If you use the /3g switch you can get more than 2GB up to about 2700MB but OS performance is penalized.
I'm not sure about 64-bit, I've never tried it but I presume the max file size is then limited only by the amount of physical memory you have.
Under Windows: "The size of a file view is limited to the largest available contiguous block of unreserved virtual memory. This is at most 2 GB minus the virtual memory already reserved by the process. "
From MDSN.
I'm not sure about LINUX/OSX/Whatever Else, but it's probably also related to address space.
Yes, there are limits to memory-mapped files. Most shockingly is:
Memory-mapped files cannot be larger than 2GB on 32-bit systems.
When a memmap causes a file to be created or extended beyond its current size in the filesystem, the contents of the new part are unspecified. On systems with POSIX filesystem semantics, the extended part will be filled with zero bytes.
Even on my 64-bit, 32GB RAM system, I get the following error if I try to read in one big numpy memory-mapped file instead of taking portions of it using byte-offsets:
Overflow Error: memory mapped size must be positive
Big datasets are really a pain to work with.
The limit of virtual address space is >16 Terabyte on 64Bit Windows systems. The issue discussed here is most probably related to mixing DWORD with SIZE_T.
There should be no other limits. Aren't those enough? ;-)