I have a 32GB MicroSD in my smartphone (Motorola Atrix 4G). The phone is connected to my Windows XP (SP3) laptop and the micro sd is accessed in 'mass storage mode' i.e. it is assigned a drive letter and is accessible from the laptop. The MicroSD is formatted with FAT32. The MicroSD contains a virtual hard disk (vmdk) comprising 12 extents (2GB in size). My code merges a redo file on a local hard disk with the vmdk extents on the microsd. The code loops over all 'grain table' entries in the redo file and if the grain table entry is non-zero it reads the referenced 'grain' or contiguous group of grains (grain in vmware terminology is a 64k block in a redo file) from the redo file, seeks to the corresponding location in the vmdk and writes the grain to that location.
Problem is that in the middle of this 'commit' operation the MicroSD gets disconnected (not always exactly at the same point in time). I get a "no disk" error message. The event log logs a warning with id 51 - "An error was detected on device \Device\Harddisk2\D during a paging operation". After this error the files on the microsd are no longer 'visible' from the Windows system. After reconnecting the device all files are again visible from the laptop.
So far I have tried the following:
Tried different cables, one of them as short as 13'' -> no effect
Connected the microsd card (Samsung, class 10 32GB) to the host system through a card reader -> problem did not appear
Tried a different microsd (Lexar, class 10 32GB) -> same issue
Tried the Lexar microsd with a card reader that is directly attached -> problem did not appear
Tried different phones (Samsung Galaxy S2, Motorola Razr M) -> same problem
Ran the code on several systems running Windows 7, including the same system that shows the behavior for XP -> Problem did not appear (!!!!)
I have an identical 2nd XP system -> same problem
I throttled down the write ops inserting a Sleep(500) right after a 'grain' is written to disk -> problem did not appear
Unfortunately 8 is not a great workaround as it slows down the commit operation.
Any ideas anybody?
Thank you in advance
Related
First of all, I'm pretty new to IDA and don't understand all its features just yet.
I'd like to inspect the ROM of a Sega Master System game. The console is based on the Z80 microprocessor, with 8K of RAM (C000-DFFFh, shadowed at E000-FFFFh). Since the Z80 can only address 64K of memory and most games are larger than that (128K, 256K, and 512K are used), the console implements a bank switching mechanism with 16K chunks to make all of the memory available. The usual way this is accomplished is by keeping the first 32K (or two banks) permanently resident (0000-7FFFh) and bank switch the rest (all addressed 8000-BFFFh) by writing the bank number to a pseudo-register at FFFFh in RAM.
Now, is there a way to make IDA understand this scheme? For now, I can load all of the ROM in segments (one segment per bank) but doing it this way, jumps/calls after a bank switch are pointing to the wrong location because it doesn't know which one is currently selected.
My company uses the Raspberry Pi 3 as an embedded controller in a product. User's don't power it off gracefully, they just flip a switch. To avoid corruption, the /boot and /root file systems are read-only. This appears to be bulletproof - we've used test rig to "pull the plug" over and over (2000+ cycles) with no problems.
We are working on a new feature that requires local logging. To do so, we created an additional ext4 read/write partition on the SD card (we are currently using about 2GB on an 8GB card) for the log file. To minimize wear, the application buffers the log data and writes to the card only once every minute. The log file is closed between writes. Nothing else uses that partition. The log file is not written to when the application is in a state that likely indicates the user is about to shut down.
In testing this, we've found that in spite of the rather conservative approach we're using, the read/write partition is always marked as "dirty" after a reboot, frequently contains filesystem errors, and often has a damaged log file. We've also had a number of cards suffer unrecoverable errors which prevent the device from booting up.
Loss of the last set of log entries is not a problem.
Loss of the log file is undesireable but acceptable.
Damage to the /root and /boot filesystems is unacceptable, as is physical damage (other than standard NAND flash wear) to the card.
Short of adding a UPS to gracefully shut down the Pi, is there any approach that will safely allow for read/write operations?
Is there a configuration of the SD card partition "geometry" that would ensure that no two partitions overlap one flash erase block?
Just some points:
Dirty flag: I guess that you are not unmounting the filesystem, right? This is a possible reason to see dirty flag after each unclean reboot. Another (probably better way) is to switch filesystem to read-only mode after writing and make it read-write before writing the file.
BTW, ext4 defers writes to the disk. close() on file doesn't mean that the files are written to the disk, you need to call extra fsync() or sync (see Does Linux guarantee the contents of a file is flushed to disc after close()?). So it is better to ask system to really write the file.
I suggest to use UBIFS or JFFS2 or YAFFS2. Its best practics way. Also I heard about LogFS.
All time mount and writing without delay posible because this FS designed to work with hard shutdown.
Copy-Pasted oveview from https://superuser.com/questions/248078/choice-of-filesystem-for-gnu-linux-on-an-sd-card
JFFS2
Includes compression and elegant wear leveling protection.
YAFFS2
Single thing that makes the difference: short mount times, after successful umount.
Implements write once property: once data is written to one block, there is no need to rewrite it. This is important, as it reduces wear.
LogFS
Not very mature, but already included in Linux kernel tree.
Supports larger filesystems than JFFS2/YAFFS2 without problems.
UBIFS
More mature than LogFS
Write caching support
On scalability: [article][3]. On large disks, better performance than with JFFS2
ext4
If no driver or card (for example SSD drives do have internal wear leveling, at least usually) handle wear leveling, then ext4 is not the best idea, as it is not intended for raw flash usage.
I am having a bit of trouble understanding how applications and data are accessed by the CPU from RAM after the application has been loaded into RAM and a file opened (thus data for the file also stored in RAM).
By my understanding, a CPU just gets instructions from RAM as the program counter ticks or carries out tasks after an interrupt. How then does it access the application and data. Is it that it doesn't and still just gets instructions (for example to load a file on the hard drive to be opened in the application) and processes any requests made by the application which are stored in RAM as instructions thereafter (like saving a file). Or does the application and data relating to an opened file (for example) just stay in RAM and not get accessed by the CPU at all.
Similarly, after reading an article, it said that a copy of the operating system is stored in RAM. The CPU can then access the operating system. (I thought the CPU just worked with instructions from RAM). How does it then communicate with the operating system and how are interrupts sent to the CPU, from the copy of the OS in RAM or from the OS in the hard drive.
Sorry if this is really confusing, alot i didn't understand.
Root of your question: Lack of clear differentiation between Computer's Hardware and Computer's Software.
Components of a Computer System
Just so that we are clear about both of them and that we understand their nature, let me state as follows:
Hardware: It includes CPU, RAM, Disk, Register, Graphics Card, Network Card, Memory BUS and everything that you can touch and call to be the 'Computer'. It is the body.
Software: It includes Operating System, Program, CPU instruction, Compiler, Programming Language and almost everything intangible about the computer. It is the soul.
Firmware: It is that basic code which is absolutely essential for hardware's working. This is stored on a Read Only Memory installed in the hardware itself. This piece of software is vital for hardware therefore is considered in the mid of hardware and software and hence called Firmware.
We will start with understanding from the time when we say that the computer is up and running and is properly executing our instructions. But at that time you will say - How did I reach here? So I will mention a few points about the startup of the computer.
When the power button is pressed...
...the most primitive and basic input output system (therefore called BIOS), which is hard written on the computer hardware begins execution. This is written on Read Only Memory and this starts the process to get the machine to stand on its own. And it loads the software (Operating System) from one piece of hardware (disks) into another piece of hardware (RAM and CPU registers) enabling the software to work properly with hardware.
Now the body and soul are together and the individual (machine) can work.
Until now, OS is already in RAM and CPU. (Read When the power button is pressed if you doubt it.) Let's handle your question paragraph by paragraph now -
First Paragraph
I am having a bit of trouble understanding how applications and data
are accessed by the CPU from RAM after the application has been loaded
into RAM and a file opened (thus data for the file also stored in
RAM).
The explanation is as follows:
The exact issue here is your thinking that it is CPU and RAM that access the data. CPU and RAM are only executing units.
It is OS (software) that accesses the data by means of CPU and RAM (hardware). It is in the realm of OS where applications are executed.
This is why you can install Linux and Windows on same hardware but cannot execute .exe files in Linux because OS does the execution and not RAM/CPU.
Further, how do CPU and RAM and disk physically interact to bring in the data, execute it, save it back etc. is in the domain of hardware. That would require explanation which involves logic gates (AND, OR, NOT...), diodes, circuitry and a hell lot of other things which an Electronics guy can explain.
Second Paragraph
By my understanding, a CPU just gets instructions from RAM as the
program counter ticks or carries out tasks after an interrupt. How
then does it access the application and data. Is it that it doesn't
and still just gets instructions (for example to load a file on the
hard drive to be opened in the application) and processes any
requests made by the application which are stored in RAM as
instructions thereafter (like saving a file).
As you have guessed it - CPU doesn't get instructions, Operating System does it through CPU. Also, just the way brain doesn't directly instruct the hands and legs to move and instead uses nerves for interaction, the CPU doesn't tell the disks to give/take the data. CPU works with RAM and registers only. Multiple units of hardware work in conjunction to provide a path for data and instruction to travel. The important pieces of involved hardware are:
Processor (CPU and registers built in the CPU)
Cache
Memory (RAM)
Disk
Tape
I like the image provided in this answer. This image not only lists the hardware pieces but also illustrates the mammoth difference in the execution speed of these pieces.
Let's move on to the...
Third Paragraph
Similarly, after reading an article, it said that a copy of the
operating system is stored in RAM. The CPU can then access the
operating system. (I thought the CPU just worked with instructions
from RAM). How does it then communicate with the operating system and
how are interrupts sent to the CPU, from the copy of the OS in RAM or
from the OS in the hard drive.
By now you already know that indeed OS is present in RAM and CPU registers. That is where it lives. That is from where it tells the CPU how to work. If OS would be small enough (or if Registers and Caches would be big enough), the OS would live even closer to CPU.
The CPU does not communicate with the OS. It can't. It is the worker that is controlled by a boss. OS is that boss.
CPU cannot access Operating System. CPU is the body, OS is the soul. Soul tells the body what to do, not vice-versa.
CPU doesn't work with instructions from RAM. It merely executes the instructions given by the Operating System (which may be living in RAM). So even when there is an instruction to load some module of OS into the RAM, it is not RAM/CPU but OS itself that issues that instruction.
Interrupts are of two types - Hardware and Software - and your query is about the software interrupts. Since the executive part of OS is in the RAM, in simple words we can say that interrupts are sent to CPU from OS living in RAM.
Conclusions
The lack of distinction between hardware and software is the basic cause of your confusions. Take some course about Operating Systems on Coursera or Academic Earth for deeper understanding.
It is confusing indeed. Let me try to explain.
CPU and RAM
The CPU is hardwired to the RAM via the 'motherboard', and they work together. The CPU can perform many instructions, but it has to be told what to do by instructions in RAM. The CPU is basically in a loop: all it does it fetch the next instruction from RAM and execute it, over and over.
So how does this RAM get filled with instructions?
BIOS (basic input/output system)
When the computer first boots up, a portion of RAM is filled with data from a chip on the motherboard (the BIOS chip), and the CPU is turned on and starts processing. These are the factory settings.
The data from the BIOS chip that is copied to RAM consists of a library of instructions to access hardware devices (hard disks, CD/ROM, USB storage, network cards etc.),
and a program using that library to load what is called the bootsector, the first sector on the boot device, into RAM, and transfer control to it (with a jump instruction).
BOOTLOADER
The bootsector data that the BIOS program loaded from the boot device is very small - only 440 bytes - but with the help of the BIOS library, this is enough to be able to load more sectors and execute these. The bootsector and the data it loads is called the bootloader, which is in charge of loading the Operating System.
In effect, the bootloader is a more dynamic version of the BIOS: the BIOS program resides in flash memory, whereas the bootloader resides on hard disks, USB sticks, SSD drives etc., and thus can be larger and more complex.
OPERATING SYSTEM
In it's turn, The operating system (OS) is simply a more advanced version of the bootloader, as it can load and run multiple programs from multiple locations at the same time.
--
The BIOS knows about drives.
The Bootloader knows about drives and partitions.
The OS knows about drives, partitions, and file systems.
CPU,as you've noticed, reads the program from RAM, instruction by instruction. When an instruction is executed, it might refer to data stored in memory, which it either fetches explicitly to the registers (internal storage of the CPU, quite small - on x86_64 that's like several 64-bit registers + other stuff like segment registers, IP, SP etc) with a separate instruction, or the data read from the memory (we are talking about small amount of data). That's all it really does.
Loading a file from a disk would be done by asking the appropriate controller to fetch the data into a specific place in memory. CPU is connected to buses which will carry instructions to appropriate controllers.
As to interrupts these are special things - CPU has several interrupt lines which can be activated by various devices, for example your network card. When it receives such an interrupt, it is usually handled by an interrupt handler, which is just a program located in a well-known place in memory. They can be registered by, for example, operating system. Each interrupt line has its own interrupt handler. When interrupt happens, the CPU saves the current state of the program it happens to be executing, handles interrupt, restores the state and resumes the program.
You seem to be asking about addressing modes. At the risk of gross oversimplification (ignoring caching, segments, and logical memory), memory stored as a sequential array accessed by an integer address.
The CPU has a number of internal storage areas called registers. We will call them R0 to Rn. The processor assigns some registers dedicated purposes. One of those registers is the PC.
One common addressing mode is deferred. I indicate this mode as (Rn). An instruction like this:
MOV (R0), R1
uses the value contained in R0 as a memory address, fetches the value stored that memory location, and stores a copy of that value in R1.
An instruction sequence like this:
MOV (R0), R1
MOV (R2), R3
is stored in memory as data (ignoring protection), code, data, and variables all use the same type of memory. In other words, any memory location can be interpreted as code, data, or variable.
The CPU executes the next instruction located at (PC). After executing the instruction, the CPU automatically increments the PC to point to the next instruction.
Staff, this question is for anyone who believes in Debian linux, more precisely of Raspbian, which is a version to run on the board Raspberry Pi:
As all users of Raspberry Pi should know: The operating system is installed on an SD card. AND the problem is that the SD card is a Flash memory, and this type of memory supports only a limited quantity of write operations.
I would like to know if the Raspbian writes the SD card when it is idle. If this happens, how can I disable?
I found this:
Tips for running Linux on a flash device by David Härdeman
If you are running your NSLU2 on a USB flash key, there are a number
of things you might want to do in order to reduce the wear and tear on
the underlying flash device (as it only supports a limited number of
writes).
Note: this document currently describes Debian etch (4.0) and needs to
be updated to Debian squeeze (6.0) and Debian wheezy (7.0). Some of
the hints may still apply, but some may not.
The ext3 filesystem per default writes metadata changes every five
seconds to disk. This can be increased by mounting the root filesystem
with the commit=N parameter which tells the kernel to delay writes to
every N seconds.
The kernel writes a new atime for each file that has been read which
generates one write for each read. This can be disabled by mounting
the filesystem with the noatime option.
Both of the above can be done by adding e.g. noatime,commit=120,... to /etc/fstab. This can also be done on an
already mounted filesystem by running the command:
mount -o remount,noatime,commit=120 /
The system will run updatedb every day which creates a database of all
files on the system for use with the locate command. This will also
put some stress on the filesystem, so you might want to disable it by
adding
exit 0
early in the /etc/cron.daily/find script.
syslogd will in the default installation sync a lot of log files to
disk directly after logging some new information. You might want to
change /etc/syslog.conf so that every filename starts with a - (minus)
which means that writes are not synced immediately (which increases
the risk that some log messages are lost if your system crashes). For
example, a line such as:
kern.* /var/log/kern.log
would be changed to:
kern.* -/var/log/kern.log
You also might want to disable some classes of messages altogether by
logging them to /dev/null instead, see syslog.conf(5) for details.
In addition, syslogd likes to write -- MARK -- lines to log files
every 20 minutes to show that syslog is still running. This can be
disabled by changing SYSLOGD in /etc/default/syslogd so that it reads
SYSLOGD="-m 0"
After you've made any changes, you need to restart syslogd by running
/etc/init.d/syslogd restart
If you have a swap partition or swap file on the flash device, you
might want to move it to a different part of the disk every now and
then to make sure that different parts of the disk gets hit by the
frequent writes that it can generate. For a swap file this can be done
by creating a new swap file before you remove the old one.
If you have a swap partition or swap file stored on the flash device,
you can make sure that it is used as little as possible by setting
/proc/sys/vm/swappiness to zero.
The kernel also has a setting known as laptop_mode, which makes it
delay writes to disk (initially intended to allow laptop disks to spin
down while not in use, hence the name). A number of files under
/proc/sys/vm/ controls how this works:
/proc/sys/vm/laptop_mode: How many seconds after a read should a
writeout of changed files start (this is based on the assumption that
a read will cause an otherwise spun down disk to spin up again).
/proc/sys/vm/dirty_writeback_centisecs: How often the kernel should
check if there is "dirty" (changed) data to write out to disk (in
centiseconds).
/proc/sys/vm/dirty_expire_centisecs: How old "dirty" data should be
before the kernel considers it old enough to be written to disk. It is
in general a good idea to set this to the same value as
dirty_writeback_centisecs above.
/proc/sys/vm/dirty_ratio: The maximum amount of memory (in percent) to
be used to store dirty data before the process that generates the data
will be forced to write it out. Setting this to a high value should
not be a problem as writeouts will also occur if the system is low on
memory.
/proc/sys/vm/dirty_background_ratio: The lower amount of memory (in
percent) where a writeout of dirty data to disk is allowed to stop.
This should be quite a bit lower than the above dirty_ratio to allow
the kernel to write out chunks of dirty data in one go.
All of the above kernel parameters can be tuned by using a custom init
script, such as this example script. Store it to e.g.
/etc/init.d/kernel-params, make it executable with
chmod a+x /etc/init.d/kernel-params
and make sure it is executed by running
update-rc.d kernel-params defaults
Note: Most of these settings reduce the number of writes to disk by
increasing memory usage. This increases the risk for out of memory
situations (which can trigger the dreaded OOM killer in the kernel).
This can even happen when there is free memory available (for example
when the kernel needs to allocate more than one contiguous page and
there are only fragmented free pages available).
As with any tweaks, you are advised to keep a close eye on the amount
of free memory and adapt the tweaks (e.g. by using less aggressive
caching and increasing the swappiness) depending on your workload.
This article has been contributed by David Härdeman
Go back to the Debian on NSLU2 page.
http://www.cyrius.com/debian/nslu2/linux-on-flash/
Someone has some more tip?
I have been using various raspberry pi setups and haven't had SD card troubles to date (fingers crossed). That being said, there is a bit of evidence for SD card lifespan related issues
A quick google search does show a few more tips though:
Bigger is better - reduces the load on specific sections
Write to ram for temp
Only store the boot partition on SD card and leave the OS on USB drive
(http://www.makeuseof.com/tag/extend-life-raspberry-pis-sd-card/)
Anyway, it'll be interesting to hear from someone who has a raspberry cluster or some such on their SD card lifespans!
(https://resin.io/blog/what-would-you-do-with-a-120-raspberry-pi-cluster/)
You can put files in tmpfs after load and write them back before shutdown using script from http://www.observium.org/wiki/Persistent_RAM_disk_RRD_storage
But it can be detrimental:
Tmpfs will destroy all changes on power outage, you must use UPS;
Raspberry Pi RAM is far from big, don't waste it.
If your pi often writes small files this can work for you
my question is how exactly does operating system protect it's kernel part.
From what I've found there are basically 2 modes kernel and user. And there should be some bits in memory segments which tels if a memory segment is kernel or user space segment. But where is the origin of those bits? Is there some "switch" in compiler that marks programs as kernel programs? And for example if driver is in kernel mode how does OS manages its integration to system so there is not malicious software added as a driver?
If someone could enlighten me on this issue, I would be very grateful, thank you
The normal technique is by using a feature of the virtual memmory manager present in most modern cpus.
The way that piece of hardware works is that it keeps a list of fragments of memory in a cache, and a list of the addresses to which they correspond. When a program tries to read some memory that is not present in that cache, the MMU doesn't just go and fetch the memory from main ram, because the addresses in the cacher are only 'logical' addresses. Instead, it invokes another program that will interpret the address and fetch that memory from wherever it should be.
That program, called a pager, is supplied by the kernel, and special flags in the MMU prevent that program from being overridden.
If that program determines that the address corresponds to memory the process should get to use, it supplies the MMU with the physical address in main memory that corresponds to the logical address the user program asked for, the MMU fetches it into its cache, and resumes running the user program.
If that address is a 'special' address, like for a memory mapped file, then the kernel fetches the corresponding part of the file into the cache and lets the program run along with that.
If the address is in the range that belongs to the kernel, or that the program hasn't allocated that address to itself yet, the pager raises a SEGFAULT, killing the program.
Because the addresses are logical addresses, not physical addresses, different user programs may use the same logical addresses to mean different physical addresses, the kernel pager program and the MMU make this all transparent and automatic.
This level of protection is not available on older CPU's (like 80286 cpus) and some very low power devices (like ARM CortexM3 or Attiny CPUs) because there is no MMU, all addresses on these systems are physical addresses, with a 1 to 1 correspondence between ram and address space
The “switch” is actually in the processor itself. Some instructions are only available in kernel mode (a.k.a. ring 0 on i386). Switching from kernel mode to user mode is easy. However, there are not so many ways to switch back to kernel mode. You can either:
send an interrupt to the processor
make a system call.
In either case, the operation has the side effect of transferring the control to some trusted, kernel code.
When a computer boots up, it starts running code from some well known location. That code ultimately ends up loading some OS kernel to memory and passing control to it. The OS kernel then sets up the CPU memory map via some CPU specific method.
And for example if driver is in kernel mode how does OS manages its integration to system so there is not malicious software added as a driver?
It actually depends on the OS architecture. I will give you two examples:
Linux kernel: A driver code can be very powerful. The level of protections are following:
a) A driver is allowed to access limited number of symbols in the kernel, specified using EXPORT_SYMBOL. The exported symbols are generally functions. But nothing prevents a driver from trashing a kernel using wild pointers. And the security using EXPORT_SYMBOL is nominal.
b) A driver can only be loaded by the privileged user who has root permission on the box. So as long as root privileges are not breached system is safe.
Micro kernel like QNX: The operating system exports enough interface to the user so that a driver can be implemented as a user space program. Hence the driver at least cannot easily trash the system.