What would be the best practice for writing to the flash on a regular basis.
Considering the hardware I am working on is supposed to have 10 to 20 years longevity, what would be your recommandation? For example, is it ok I write some state variables every 15 minutes thru Preferences?
That depends on
number of erase cycles your Flash supports,
size of the NVS partition where you store data and
size and structure of the data that you store.
Erase cycles mean how many times a single sector of Flash can be erased before it's no longer guaranteed to work. The number is found in the datasheet of the Flash chip that you use. It's usually 10K or 100K.
Preferences library uses the ESP-IDF NVS library. This requires an NVS partition to store data, the size of which determines how many Flash sectors get reserved for this purpose. Every time you store a value, NVS writes the data together with its own overhead (total of 32 bytes for primitive data types like ints and floats, more for strings and blobs) into the current Flash sector. When the current sector is full, it erases the next sector and proceeds to write there; thereby using up sectors in a round robin fashion as write requests come in.
If we assume that your Flash has 100K erase cycles, size your NVS partition is 128 KiB and you store a set of 8 primitive values (any int or float) every 15 minutes:
Each store operation uses 8 * 32 = 256 bytes (32 B per data value).
You can repeat that operation 131072 / 256 = 512 times before you've written to every sector of your 128KiB NVS partition (i.e. erased every sector once)
You can repeat that cycle 100K times so you can do 512 * 100000 = 51200000 or roughly 5.1M store operations before you've erased every sector its permitted maximum number of times.
Considering the interval of 15 minutes creates 365 * 24 * 4 = 35040 operations per year, you'd have 51200000 / 35040 = 1461 years until Flash is dead.
Obviously, if your Flash chip is rated at 10K erase cycles, it drops to only 146 years.
There's probably some NVS overhead in there somewhere that I didn't account for, and the Flash erase cycle ratings are not 100% reliable so I'd cut it in half for good measure - I would expect 700 or 70 years in real life.
If you store non-primitive values (strings, blobs) then the estimate changes based on the length of that data. I don't know to calculate the exact Flash space used by those but I'd guess 32B plus length of your data multiplied by 10% of NVS overhead. Plug in the numbers, see for yourself.
I am trying to store some data on the device that I do not want to be overwritten by when the datalogger is full. And have run into some minor issues. I was looking for the "eeprom_logbook_app" but could not find it in firmware version 1.6.2 of the device-lib.
I have defined how much space I want for my persistent data and in App.cpp I have used the LOGBOOK_MEMORY_AREA(offset, size) macro.
where I have used the size of what I want to store as the offset and set the size to be
(2097152 + 1048576) - (size of data I want to store)
as this was what was returned when I asked the sensor for the eeprom size. (The eeprom is devided between 2 ICs one with 1MB capasity and one with 2MB capacity?)
Then I remembered that there was some talk about ExtflashChunkStorage::StorageHeader being stored as the first 256 bytes in this answer.
So my question is where the data will be offset from and what is the max size I can set as size so that I can subtract the correct amount to fit my data? I presume I at least need to take another 256 bytes off from the size to get my correct storage size.
As stated in my comment I got this working the only thing you need to do is using the LOGBOOK_MEMORY_AREA(offset, size) function.
let's say you want to set aside 256 bits for your own config you could then do this:
#define RESERVED_CONFIG 256
#define TOTAL_MEMMORY_SIZE (2097152 + 1048576)
static const uint32_t offset = RESERVED_CONFIG;
static const uint32_t size = TOTAL_MEMMORY_SIZE -offset;
LOGBOK_MEMORY_AREA(offset, size);
This will set aside 256 bytes on the start of the EEPROM memory and offset the logbook to accommodate this. As a result, the logbook header will also be moved to the start of the logbook's memory area.
"The address consists of 16 bits, and the page size is 8KB. The page table thus consists of eight entries that are kept in fast registers."
How do we get the total entries in the page table as 8?
According to the calculation it should be 1.
Total Entries in the Page Table= ((2^16)/(2^3*2^10*2^3))=1.
(The first 2^3 is for 8 in 8KB, the second one is for bytes to bits conversion and 2^10 is for "Kilo" in 8KB.)
Thanks
Memory is byte-addressable hence, you do not need to divide by 2^3 for bytes to bit conversion.
Explaining it further, 16-bits for address means that the processor will generate memory addresses of length 16 bits which will be used to address the byte or half-word or word present starting (or ending - depends on the endianess of the machine) at that 16-bit value.
Now, the page size is the total size of a page in bits which in this case is 2^16 bits. But as memory is byte addressable, hence number of processor addresses in one page will be 2^16/2^3 i.e 2^13 addresses.
Hence number of page table entries are 2^16/2^13 = 8.
I have a homework question during studying for a test:
You have a new device with logic addresses space of 32 bit and physical addresses space of 34 bit. Size of page is 8KB (2^13 B). Calculate size of page table (of single level).
There are 2^32 / 2^13 = 2^19 entries. And what is the size of entry, I have to assume that it's 4B or I have a way to calculate it?
As you have already mentioned no. of enteries will be 2^32/2^13, but the page size can be anything (predefined), also you can have operating systems supporting multiple page sizes, hence to best of my understanding you can't calculate the page size through this as it is predefined and virtual memory can be any big.
There are 2^19 entries now each of these entry will be 32 Bit wide so space occupied = (2^19) * (32) Bits
I found this example.
Consider a system with a 32-bit logical address space. If the page
size in such a system is 4 KB (2^12), then a page table may consist of
up to 1 million entries (2^32/2^12). Assuming that
each entry consists of 4 bytes, each process may need up to 4 MB of physical address space for the page table alone.
What is the meaning of each entry consists of 4 bytes and why each process may need up to 4 MB of physical address space for the page table?
A page table is a table of conversions from virtual to physical addresses that the OS uses to artificially increase the total amount of main memory available in a system.
Physical memory is the actual bits located at addresses in memory (DRAM), while virtual memory is where the OS "lies" to processes by telling them where it's at, in order to do things like allow for 2^64 bits of address space, despite the fact that 2^32 bits is the most RAM normally used. (2^32 bits is 4 gigabytes, so 2^64 is 16 gb.)
Most default page table sizes are 4096 kb for each process, but the number of page table entries can increase if the process needs more process space. Page table sizes can also initially be allocated smaller or larger amounts or memory, it's just that 4 kb is usually the best size for most processes.
Note that a page table is a table of page entries. Both can have different sizes, but page table sizes are most commonly 4096 kb or 4 mb and page table size is increased by adding more entries.
As for why a PTE(page table entry) is 4 bytes:
Several answers say it's because the address space is 32 bits and the PTE needs 32 bits to hold the address.
But a PTE doesn't contain the complete address of a byte, only the physical page number. The rest of the bits contain flags or are left unused. It need not be 4 bytes exactly.
1) Because 4 bytes (32 bits) is exactly the right amount of space to hold any address in a 32-bit address space.
2) Because 1 million entries of 4 bytes each makes 4MB.
Your first doubt is in the line, "Each entry in the Page Table Entry, also called PTE, consists of 4 bytes". To understand this, first let's discuss what does page table contain?", Answer will be PTEs. So,this 4 bytes is the size of each PTE which consist of virtual address, offset,( And maybe 1-2 other fields if are required/desired)
So, now you know what page table contains, you can easily calculate the memory space it will take, that is: Total no. of PTEs times the size of a PTE.
Which will be: 1m * 4 bytes= 4MB
Hope this clears your doubt. :)
The page table entry is the number number of bits required to get any frame number . for example if you have a physical memory with 2^32 frames , then you would need 32 bits to represent it. These 32 bits are stored in the page table in 4 bytes(32/8) .
Now, since the number of pages are 1 million i.e. so the total size of the page table =
page table entry*number of pages
=4b*1million
=4mb.
hence, 4mb would be required to store store the table in the main memory(physical memory).
So, the entry refers to page table entry (PTE). The data stored in each entry is the physical memory address (PFN). The underlying assumption here is the physical memory also uses a 32-bit address space. Therefore, PTE will be at least 4 bytes (4 * 8 = 32 bits).
In a 32-bit system with memory page size of 4KB (2^2 * 2^10 B), the maximum number of pages a process could have will be 2^(32-12) = 1M. Each process thinks it has access to all physical memory. In order to translate all 1M virtual memory addresses to physical memory addresses, a process may need to store 1 M PTEs, that is 4MB.
Honestly a bit new to this myself, but to keep things short it looks like 4MB comes from the fact that there are 1 million entries (each PTE stores a physical page number, assuming it exists); therefore, 1 million PTE's, which is 2^20 = 1MB. 1MB * 4 Bytes = 4MB, so each process will require that for their page tables.
size of a page table entry depends upon the number of frames in the physical memory, since this text is from "OPERATING SYSTEM CONCEPTS by GALVIN" it is assumed here that number of pages and frames are same, so assuming the same, we find the number of pages/frames which comes out to be 2^20, since page table only stores the frame number of the respective page, so each page table entry has to be of atleast 20 bits to map 2^20 frame numbers with pages, here 4 byte is taken i.e 32 bits, because they are using the upper limit, since page table not only stores the frame numbers, but it also stores additional bits for protection and security, for eg. valid and invalid bit is also stored in the page table, so to map pages with frames we need only 20 bits, the rest are extra bits to store protection and security information.