This question already has answers here:
Creating a bootable ISO image with custom bootloader
(2 answers)
Closed 2 years ago.
I have finished writing my simple Operating System and I want to test it on a real hardware(PC), not bochs or qemu. My OS has a custom bootloader and a kernel and I used cat to concat them into one single bin file. But I spent hours on finding out a way to convert the bin file to a bootable iso file but failed each time. According to OSDev.org, I think I need to use genisoimage(mkisofs) to do the convert, but I don't know exactly how this command works, I finally outputted a iso file but this one is not working.(I think I used the wrong command, can someone explain a little bit more to me?)
Other Approaches I tried:
Directly burn the bin file to a CD. Error: Missing Operating System.
Convert the bin file to ISO using winbin2iso and other windows platform software. Error: Could not boot. Not even in qemu.
Also, what is El-Torito?
When a CD is being booted, the firmware checks the CD for a "boot catalogue" in the CD's meta-data. This is a list of entries with (up to) one entry for each kind of computer; so that it's possible to create a single CD that works for 80x86 BIOS and 80x86 UEFI (and PowerPC and Sparc and ...).
For 80x86 there are 4 different types of entries:
UEFI. The rest of the entry tells the firmware the starting sector and size of a FAT file system image. UEFI figures out which file it wants from the FAT file system based on converting the architecture into a file name (e.g. for 80x86 it'll probably want the file \EFI\BOOT\BOOTX64.EFI).
"No emulation, 80x86 BIOS". The rest of the entry tells the firmware the starting sector and size of a boot loader. The boot loader can be any size you like (up to about 639 KiB); and the "device number" the BIOS tells you (in dl) will be for the CD drive itself, so you can load more stuff from the same disk using it.
"Hard disk emulation, 80x86 BIOS". The rest of the entry tells the firmware the starting sector and size of a disk image. The disk image should have an MBR with a BPB and partition/s, with an active partition pointing to where the operating system's boot loader is at the start of its partition. In this case the BIOS will create a fake "device 0x80" (from info in the BPB in the MBR) and mess up the device numbers for any real hard drives (e.g. the first hard drive which would've been "device 0x80" will become "device 0x81" instead, etc). Note that this is inefficient because sectors on a CD are 2048 bytes but the BIOS will emulate 512 byte sectors, so every time you try to read a 512 byte sector the BIOS will actually read 2048 bytes and throw the "wrong" 75% of the data away. It's also very annoying (trying to squeeze anything good in 512 bytes is impossible). It's mostly only used for obsolete junk (e.g. MS-DOS).
"Floppy disk emulation, 80x86 BIOS". The rest of the entry tells the firmware the starting sector and size of a disk image. The disk image should have a boot loader in the first sector with a BPB. In this case the BIOS will create a fake "device 0x00" (from info in the BPB in the MBR) and mess up the device numbers for any real floppy drives. Just like hard disk emulation, this is inefficient, even more "very annoying" (because it also limits the OS to the size of a floppy which is nowhere near enough space), and only intended for obsolete junk.
The best way to deal with CDs is to write 2 new boot loaders (one for "no emulation, 80x86 BIOS" and another for UEFI); then use any tool you like to create the appropriate structures on the CD (e.g. genisoimage with the -no-emul-boot option for your "no emulation, 80x86 BIOS" boot loader plus some other option for UEFI that doesn't seem to exist!?).
Note that it's easy to write your own utility that is "more clever". You'd want to create a FAT file system (for UEFI) and an ISO9660 file system (for the rest of the operating system's file - help/docs, drivers, etc), but most tools won't create the FAT file system for you, and it's possible for files in both file systems (FAT and ISO9660) to use the same sectors (so that the same files appear in both file systems without costing twice as much disk space). Something like this would probably only take you 1 week to write yourself (and you'll learn a lot about CDs and ISO9660 that you're going to have to learn eventually anyway). The relevant documentation (for booting from CD, ISO9660 file systems, FAT file systems, and UEFI) are all easily obtained online.
Also, what is El-Torito?
El-Torito is another name for the "Bootable CD-ROM Specification" that describes the structures needed on a CD to make it bootable.
Related
I have an STM32 application which uses two blocks of memory. In 0th block, I have a boot code (which runs just after power-on) and in 7th block, I have an application code (which may or may not run depending on the authorization decision given by the boot code).
Those two codes are developed hence generated by two separate projects. They are flashed on the specific blocks (boot code to 0th block and application code to 7th block) of STM32 NOR memory using openocd tool by giving an offset value to the openocd's write_image command.
What I would like to do basically in the boot code is that I want to calculate the hash of the application code and compare it with the reference digest. If they are equal, I will give the hand to the application code. For that, after I generate the executable (can be in elf, hex or bin format) of the application code, I want to:
Create another file (in any format listed above) which has 128K byte size
Copy the content of the executable file to the recently created file from its beginning (0 offset)
Write the hash of the executable to the last 32 bytes of the recently created file
Fill the gap with 0xFF
Finally flash this executable file (if it is still) to the 7th block of the memory
Do you think that it is doable and feasible? If so:
Which format should I use to generate the executable?
Do I have something that I need to give specific attention to achieve this?
Lastly, do you think that it makes sense to do that or is there any other more standard way for this purpose?
Thanks a lot in advance.
You just need to add an additional step to your building sequence. After the linking extract the binary file from elf
Then write a program in your favourite programming language which will calculate something and append the result to that bin file
I'm trying to analyze some mini crash dumps. I'm using Windows 10 Pro Build 1607 and WinDbg 10.0.14321.1024. I have my symbol file path set to
SRV*C:\SymCache*https://msdl.microsoft.com/download/symbols
Basically, whenever I load up a minidump (all < 1 MB .dmp files), it takes WinDbg forever to actually analyze them. I understand the first run can take long, but it took mine almost 12 hours before it would let me enter a command. I assumed that, since the symbols were cached, it wouldn't take long at all to re-open the same .dmp. This is not the case. It loads up, goes pretty much instantaneously to "Loading Kernel Symbols", then takes another 30 minutes before it prints the "BugCheck" line. It's been another 30 minutes, and I still can't enter commands into it.
My PC has a 512 GB SSD, 8 GB of RAM, and an i5-4590. I don't think it should be this slow.
What am I doing wrong?
These kind of complaints seem to occur more often lately and I can reproduce it on my PC. This is not your fault but some issue with the Internet or the symbol server on Microsoft side.
Monitoring the traffic with Wireshark and looking at my disk on how the symbol cache get populated, I can say:
only one file is being downloaded at a time.
the problem also occurs with older WinDbg versions (6.2.9200)
the problem occurs with HTTP and HTTPS
when symbols are found, the transfer speed is very slow, then increasing. The effective transfer rate is down at 11 kb/s to 20 kb/s (on a line which can handle 6500 kb/s)
there's quite a high number of packets out of order, duplicate packets etc., especially during the "lookup phase" where no file is downloaded yet. Such a lookup phase can easily take 8 minutes.
even if the file already exists on disk, the "lookup phase" is performed.
the HTTP roundtrip time (request to response) is 8 to 9 seconds
This is the symbol server being really slow. Other have noticed as well: https://twitter.com/BruceDawson0xB/status/772586358556667904
Your symbol path contains a local cache so it should load faster next time around, but it seems that the cache is not effective, I can't tell really why (I suspect the downloaded symbols are not a perfect match and they are being downloaded again, every time).
I would recommend modifying the _NT_SYMBOL_PATH (or whatever is the way your sympath is initialized) to SRV*C:\SymCache only, ie. do not attempt to automatically download, just use the symbols you already have cached locally. The image should open fairly fast. Only enable the symbols server if you discover missing symbols.
I ran into the same problem (extremely slow windbg), but loading/reloading/fixing/caching symbols did not help. By accident, I figured out that this problem persists when I try to print memory with address taken from a register, like
db rax
The rule of thumb is to always use # with the register name.
db #rax
Without this symbol, the debugger considers rax to be a symbol name, and looks for it some time (depending on the amount of symbols you have loaded) and fails to find it eventually, and falls back to treating it like a register name. Printing memory from register with # symbol works instantly, even if you have gigs of symbols loaded in memory. As you can see, this problem is also symbol-related, but in a different way.
I have a data file (6.3GB) that I'm attempting to work on in MATLAB, but I'm unable to get it to load, and I think it may be a memory issue. I've tried loading in a smaller "sample" file (39MB) and that seems to work, but my actual file won't load at all. Here's my code:
filename = 'C://Users/Andrew/Documents/filename.mat';
load(filename);
??? Error using ==> load
Can't read file C://Users/Andrew/Documents/filename.mat.
exist(filename);
EDU>> ans = 2
Well, at least the file exists. When I check the memory...
memory
Maximum possible array: 2046 MB (2.146e+009 bytes) *
Memory available for all arrays: 3442 MB (3.609e+009 bytes) **
Memory used by MATLAB: 296 MB (3.103e+008 bytes)
Physical Memory (RAM): 8175 MB (8.572e+009 bytes)
* Limited by contiguous virtual address space available.
** Limited by virtual address space available.
So since I have enough RAM, do I need to increase the maximum possible array size? If so, how can I do that without adding more RAM?
System specifics: I'm running 64-bit Windows, 8GB of RAM, MATLAB Version 7.10.0.499 (R2010a). I think I can't update to a newer version since I'm on a student license.
As the size might be the issue, you could try load('fileName.mat', 'var1'); load('fileName.mat', 'var2'); etc. For this, you'll have to know the variable names though.
An option would be to use the matfile object to load/index directly into the file instead of loading into ram.
doc matfile
But one limitation is that you can not index directly into a struct. So you would need to find a friend to convert the struct in your mat file and save it with the version option
save(filename, variables, '-v7.3')
May be you can load part by part your data to do your stuff using load part of variables from mat file. You must have matlab 7.3 or newer.
From your file path I can see you are using Windows. Matlab is only 32 bit for Windows and Linux (there is no 64 bit for these OSes at least for older releases, please see my edit), which means you are limited to <4GB ram total for a single application (no matter how much you have in your system), this is a 32 bit application issue so there is nothing you can do to remedy it. Interestingly the Mac version is 64 bit and you can use as much ram as you want (in my computer vision class we often used my mac to do our big video projects because windows machines would just say "out of memory")
As you can see from your memory output you can only have ~3.4GB total for matrix storage, this is far less than the 6.3GB file. You'll also notice, you can only use ~2GB for one particular matrix (that number changes as you use more memory).
Typically when working with large files you can read the file line by line, rather than loading the entire file into memory. But since this is a .mat file that likely wouldn't work. If the file contains multiple variables maybe separate them each into their own individual files that are small enough to load
The take home message here is you can't read the entire file at once unless you hop onto a Mac with enough RAM. Even then the size for a single matrix is still likely less than 6.3GB
EDIT
Current Matlab student versions can be purchased in 64 bit for all OSes as of 2014 see here so a newer release of Matlab might allow you to read the entire file at once. I should also add there has been a 64 bit version before 2014, but not for the student license
Is there a maximum amount of packets for pcap file (beyond OS limits?
Looking at the specification I don't see anything in the pcap file format that would limit the number of packets.
Perhaps will libpcap or tcpdump limit the number of packets written?
No, there is nothing in the pcap file format that limits the number of packets in a file. (The pcap-ng file format allows a maximum of 18446744073709551614 bytes in a section, if the Section Header Block must contain the number of bytes in a section, but, with more than 18446744073709551615 bytes in a file, even current 64-bit operating systems will have a problem with the file. :-))
Tcpdump just uses libpcap to write out files, so it would be limited only by libpcap's limitations.
libpcap 1.0 and later uses the AC_SYS_LARGEFILE macro in the configure script, so, on UN*X, if the underlying operating system supports files larger than 2GB (which they should all do for 64-bit machines, and which most if not all of them have done for several years on 32-bit machines), and if the configure script properly sets things up on 32-bit machines to allow large files (or if no such setup is necessary - it's not necessary on *BSD and OS X, for example), and if whoever built the library didn't somehow prevent that from working, libpcap should have no problem writing files larger than 2GB (as long as you have enough disk space, of course...).
I don't know whether the C support library that comes with whatever compiler was used to build WinPcap will support large files on Windows NT (NT 3.x, NT 4.0, 2000, XP, Vista, 7, 8, and various Server versions), but I suspect it would. I don't think Windows OT (95, 98, Me) supports large files.
I read some unix manual (http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html), and there was a mention about execution.
The new process image shall be constructed from a regular executable
file called the new process image file.
The expression process image caught my eyes.
I have been thought executable file is just a kind of sequence of command. Just as the word program means. But actually, I don't know the concept and structure of the executable file. And I felt executable file could be looks like an execution state image from the mention.
Could you explain me something about this? About the concept and structure of regular executable files in nowadays. In any OS.
Usually the executable file does not contain only instructions but also global data, readonly data and many more. I suggest you briefly look e.g. on the ELF format widely used in UNIX-like operating systems or PE format used in Windows.
The OS may also need for example to replace some addresses of functions (jump targets) with the real addresses of these functions in the memory, although this technique is probably not used anymore in common OSes. Anyway, there can be more work to do than just copy the file into memory and start executing from the first byte.