I know that in Unix (specifically, Mac OS X) the superblock stores information about the layout of data on the disk, including the disk addresses at which the inodes begin and end. I want to scan the list of inodes in my program to look for deleted files. How can I find the disk address at which the inodes begin? I have looked at the statfs command but it does not provide this information.
Since you mention Mac OS X, let's assume you mean to do this for HFS+ only. The Wikipedia page provides some information about possible ways to start, for instance it says this about the on-disk layout:
Sectors 0 and 1 of the volume are HFS boot blocks. These are identical to the boot blocks in an HFS volume. They are part of the HFS wrapper.
Sector 2 contains the Volume Header equivalent to the Master Directory Block in an HFS volume. The Volume Header stores a wide variety of data about the volume itself, for example the size of allocation blocks, a timestamp that indicates when the volume was created or the location of other volume structures such as the Catalog File or Extent Overflow File. The Volume Header is always located in the same place.
The Allocation File which keeps track of which allocation blocks are free and which are in use. It is similar to the Volume Bitmap in HFS, each allocation block is represented by one bit. A zero means the block is free and a one means the block is in use. The main difference with the HFS Volume Bitmap, is that the Allocation File is stored as a regular file, it does not occupy a special reserved space near the beginning of the volume. The Allocation File can also change size and does not have to be stored contiguously within a volume.
It becomes more complicated, after that. Read up on B* trees, for instance.
I'm no Mac OS user, but it would surprise me if there weren't already tools written to scan for deleted files, perhaps some are open source and could provide a more concrete starting point?
You'll have quite some trouble to find deleted files because there's not much left on the disk to find when you delete a file.
If you delete a file on a FAT (or UDF) file system, its directory entry simply gets marked as "deleted", with most of the dir entry still intact.
On HFS volumes, due to their use of B-Trees, deleted edits must be removed from the directory or else searching for items wouldn't work any more efficiently (well, this argument may be a bit weak, but fact is that deleted entries get removed and overwritten).
So, unless the deletion took place by writing over a directory sector by accident, or by re-initializing the volume, you'll not find much.
Related
I had some doubts regarding an Inode vs a Vnode. As far as my understanding goes, inode is the representation of a file that is used by the Virtual File System. Whereas vnodes are file system specific. Is this correct?
Also, I am confused whether inode is a kernel data structure i.e whether it is an in-memory data structure or a data structure that exists on blocks in an actual disk?
To add something to this: The best explanation I could find for a vnode is here on the FreeBSD docs. For the more academically minded there is also the original paper that introduced the concept, [Vnodes: An Architecture for Multiple File System Types in Sun UNIX], which provides a much more in depth resource.
That said vnodes were originally created for FreeBSD because of the proliferation of different types of filesystems that needed to be used like UFS, NFS, etc. Vnodes are meant to provide an abstraction across all possible filesystems so that the OS can interface with them and so that kernel functions don't have to specifically support every filesystem under the sun; they only have to have knowledge of how to interact with the file's vnode.
Back to your original question vnodes, as #Allen Luce mentioned, are in memory abstractions and are not filesystem specific. They can be used interchangeably in UFS, ext4, and whatever else. In contrast, inodes are stored on disk and are specific to the exact filesystem being used. inodes contain metadata about the file like size, owner, pointers to the block addresses among other things. Vnodes contain some data about the file but only attributes that do not change over the file's lifetime so an inode would be the location to reference if you wanted the most information possible about a file. If you're more interested in inodes I would suggest you check out wikipedia which has a good article on them.
Typically (like for Linux and BSD mainstream filesystems), an inode is first an on-disk structure that describes the storage of a file in terms of that disk (usually in blocks). A vnode is an in-memory structure that abstracts much of what an inode is (an inode can be one of its data fields) but also captures things like operations on files, locks, etc. This lets it support non-inode based filesystems, in particular networked filesystems.
It depends on your OS. On a Linux system for instance there is no v-node, only a generic i-node struct inode which although is conceptually similar to a v-node is implemented differently.
For BSD-derived and UNIX kernels, the v-node points to an i-node structure specific to the filesystem, along with some additional information including pointers to functions that operate on the file and metadata not included in the inode. A major difference is the inode is files system while the vnode is not. (In Linux as mentioned above there is both a system-independent inode and a file system-dependent inode)
An inode is not a kernel data structure, the vnode/generic inode is however, being an in-kernel representation of the inode.
The concept of vnode differs to a degree depending on what system you're under, everyone just kind of took the name and ran with it.
Under Linux and Unix, you can consider the abstraction as follow.
Suppose there exists
f.tmp
You want f to stick around while your program is running, because you're accessing it, but if your program ends or crashes, you want to be sure it goes away.
You can do this by opening f, and then unlink() it. You will still retain a reference to f, even though its inode now has 0 directory entries, and so has been marked free. The operating system is still retaining where the file started and its allocation state, until your program ends. This "virtualization" of the inode that no longer exists is a vnode.
Another common situation for this is when you're reading a resource that disappears out from under you. Suppose you're watching a movie, while it is streaming to a temporary location. When the movie is completely downloaded, it will be relocated to another volume for storage. Somehow you can continue watching and scrubbing through the movie so long as it remains open. In this case even though there are again no links, since there is a vnode, this inode can't be cleaned up yet.
This depends on the operating system and the file system you are using or working on. For instance VXFS and ADVFS inode's are nothing but on-disk data structure called vnode's. In general both refer to file metadata.
Simply put, the in-memory data structure vnode is just an inode cache that stores information about
the file(typically inode stores in the disk) so that it can be accessed more quickly.
Given a .tar archive, Matlab allows one to extract the contained files to disk via UNTAR command. One can then manipulate the extracted files in the ordinary way.
Issue: When several files are stored in a tarball, they are stored contiguously on disk and, in principle, they can be accessed serially. When such files are extracted, this contiguity doesn't hold any more and the file access can become random, hence slow & inefficient.
This is especially critical when the considered files are many (thousands) and small.
My question: is there any way to access to the archived files avoiding the preliminary extraction (in a sort of HDF5 fashion)?
In other words, would it be possible to cache the .tar so to access the contained files from the memory rather than from the disk?
(In general, direct .tar manipulation is possible, e.g. is C# tar-cs, in python).
No, as far as i know.
If you're using Matlab on Linux, try to extract to tmpnam. This will extract to tmpfs whitch should be faster accessible (bad idea if we are takling about several GB).
Otherway you can use system('untar xf file.tar only/needed/file') or python to get a more flexible untar behavior.
After some time I finally worked out a solution which gave me unbelievable speedups (like 10x or so).
In a word: ramdisk (tested on Linux (Ubuntu & CentOs)).
Recap:
Since the problem has some generality, let me state it again in a more complete fashion.
Say that I have many small files stored on disk (txt,pict, order of millions) which I want to manipulate (e.g. via matlab).
Working on such files (i.e. loading them/transmitting them on network) when they are stored on disk is tremendously slow since the disk access is mostly random.
Hence, tarballing the files in archives (e.g. of fixed size) looked to me like a good way to keep the disk access sequential.
Problem:
If case the manipulation of the .tar requires a preliminary extraction to disk (as it happens with matlab's UNTAR), the speed up given by sequential disk access is mostly loss.
Workaround:
The tarball (provided it is reasonably small) can be extracted to memory and then processed from there. In matlab, as I stated in the question, .tar manipulation in memory is not possible, though.
What can be done (equivalently) is untarring to ramdisk.
In linux, e.g. Ubuntu, a default ramdisk drive is mounted in /run/shm (tempfs). Files can be untarred via matlab there, having then extremely fast access.
In other words, a possible workcycle is:
untar to /run/shm/mytemp
manipulate in memory
possibly tar again the output to disk
This allowed me to change the scale-time of my processing from 8hrs to 40min and full CPUs load.
Can a FAT based file system be modified to support multiple references to a file (i.e. aliases) by using the same FAT block sequence in directory table entries?
No because then when any reference was deleted, the file would be added to free space and possibly reused. This would result in two different files sharing space with any write to one corrupting the other.
This could work if the file system was immutable. For example if it was written to an unwritable medium.
Surely, you can have directory items points to same FAT records, but there are two things you should keep in mind:
1) never run any standard check disk utilities otherwise you get it wrong
2) you have to implement own delete operation to remove records from directory which points to the same item that you delete.
UPD: answer consider that question has 'can be modified' approach
The FAT File System stores all information about a file in a single structure inside a directory, except the addresses of disk blocks that contain file data. Disk block numbers of all files are kept in a File Allocation Table (FAT).
Since the link information and file container information are bound together in a single structure, FAT file system does not support multiple links to a single file. It does not support symbolic links either, though it could have. However, Windows supports shortcuts that are similar to symbolic links.
I was studying operating system concepts from galvin's sixth edition and i have some questions about the flow of execution of a program. A figure explains the processing of the user program as:
We get an executable binary file when we reach linkage editor point. As the book says, "The program must be brought into memory and placed within a process for it to be executed" Now some of my stupid questions are:
Before the program is loaded into the memory, the binary executable file generated by the linkage editor is stored in the hard disk. The address where the binary executable file is stored in the hard disk is the logical address as generated by the CPU ?
If the previous answer is yes, Why CPU has to generate the logical address ? I mean the executable file is stored somewhere in the hard disk which pertains to an address, why does CPU has to separately do the stuff ? CPU's main aim is processing after all!
Why does the executable file needs to be in the physical memory i.e ram and can not be executed in the hard disk? Is it due to speed issues ?
I know i am being stupid in asking these questions, but trust me, I can't find the answers! :|
1) The logical address where the binary file is stored in the hard disk is determined by the file system, the Operating System component that is aimed to manage files in the disks.
2) & 3) The Hard Disk is not a) fast enough b) does not support word addressing. The hard disks are addressed in sectors blocks. Usually the sector size is 512 bytes. The CPU need to be able to address each machine word in a program to execute it. So, the program is stored in the hard disk, that retains its content even being powered off (in contrast to the RAM that losts its content when it is powered off). Then the program is loaded into RAM to be executed. After program finished and possibly stored the result of its execution in the hard disk, the memory is freed for running another programs. The Compiler and the Linkage Editor in your sample are also programs. They are kept in the hard disk. The compiler get its input (the source text of your program) from the file in the hard disk. Then it stores the object file. The linkage editor, or linker for short does the same: it reads the object file and necessary library files and then produces a file with a binary representation of your program.
It's widely known that the most significant mmap() feature is that file mapping is shared between many processes. But it's not less widely known that every process has its own address space.
The question is where are memmapped files (more specifically, its data) truly kept, and how processes can get access to this memory?
I mean not *(pa+i) and other high-level stuff, but I mean the internals of the process.
This happens at the virtual memory management layer in the operating system. When you memory map a file, the memory manager basically treats the file as if it were swap space for the process. As you access pages in your virtual memory address space, the memory mapper has to interpret them and map them to physical memory. When you cross a page boundary, this may cause a page fault, at which time the OS must map a chunk of disk space to a chunk of physical memory and resolve the memory mapping. With mmap, it simply does so from your file instead of its own swap space.
If you want lots of details of how this happens, you'll have to tell us which operating system you're using, as implementation details vary.
This is very implementation-dependent, but the following is one possible implementation:
When a file is a first memory-mapped, the data isn't stored anywhere at first, it's still on disk. The virtual memory manager (VMM) allocates a range of virtual memory addresses to the process for the file, but those addresses aren't immediately added to the page table.
When the program first tries to read or write to one of those addresses, a page fault occurs. The OS catches the page fault, figures out that that address corresponds to a memory-mapped file, and reads the appropriate disk sector into an internal kernel buffer. Then, it maps the kernel buffer into the process's address space, and restarts the user instruction that caused the page fault. If the faulting instruction was a read, we're all done for now. If it was a write, the data is written to memory, and the page is marked as dirty. Subsequent reads or writes to data within the same page do not require reading/writing to/from disk, since the data is in memory.
When the file is flushed or closed, any pages which have been marked dirty are written back to disk.
Using memory-mapped files is advantageous for programs which read or write disk sectors in a very haphazard manner. You only read disk sectors which are actually used, instead of reading the entire file.
I'm not really sure what you are asking, but mmap() sets aside a chunk of virtual memory to hold the given amount of data (usually. It can be file-backed sometimes).
A process is an OS entity, and it gains access to memory mapped areas through the OS-proscribed method: calling mmap().
The kernel has internal buffers representing chunks of memory. Any given process is assigned a memory mapping in its own address space which refers to that buffer. A number of proccesses may have their own mappings, but they all end up resolving to the same chunk (via the kernel buffer).
This is a simple enough concept, but it can get a little tricky when processes write. To keep things simple in the read-only case there's usually a copy-on-write functionality that's only used as needed.
Any data will be in some form of memory or others, some cases in HDD, in embedded systems may be some flash memory or even the ram (initramfs), barring the last one, the data in the memory are frequently cached in the RAM, RAM is logical divided into pages and the kernel maintains a list of descriptors which uniquely identify an page.
So at best accessing data would be accessing the physical pages. Process gets there own process address space which consists of many vm_are_struct which identifies a mapped section in the address space. In a call to mmap, new vm_area_struct may be created or may be merged with an existing one if the addresses are adjacent.
A new virtual address is returned to the call to mmap. Also new page tables are created which consists the mapping of the newly created virtual addresses to the physical address where the real data resides. mapping can be done on a file, or anonymously like malloc. The process address space structure mm_struct uses the pointer of pgd_t (Page global directory) to reach the physical page and access the data.