Related
Could you recommend sources/literature to prepare for Scientific Programmer/HPC programmer Interview?
Thanks!
"Introduction to High-Performance Scientific Computing" by Victor Eijkhout is a very good book to start with (and is relatively up to date). You can find it freely on his personal home page.
As this book mainly focus especially on quite theoretical/abstract concepts, you probably need to complete the book with practical HPC programming. MPI and OpenMP are two programming standard massively used in HPC applications. As a result, I strongly advise you to learn how to program with both, especially from practical exercises.
There is a lot of resource you can find on internet for both. If you don't now how to start, please look here for MPI and here for OpenMP.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I want to learn linux kernel device driver programming. So can anyone please post good tutorials pages or links here. I am new to linux kernel environment. I have searched for it but I don't know how to start and which one to read for easy understanding basics. Thanks in advance.
Depends on your current skills. If you're really new to Linux, perhaps you should start with user space system programming with Advanced Linux Programming. You'll get good knowledge of Unix system calls and other concepts such as signals, processes/threads and so on with this free resource. This is a must (understanding the user space API) if you're developing on the kernel side since the role of a kernel is providing services to users in a secure way.
Otherwise one often cited book is Linux Device Drivers, Third Edition (LDD3). Keep in mind that this edition was written at the time of Linux 2.6.10 and some things changed since then. This article shows the differences as 2.6 evolved (until 2.6.31, that is, so not very useful). I should mention martinezjavier/ldd3, which contains example drivers of LDD3 updated for more recent kernels (thanks to 42n4 for pointing that out).
Another interesting book that's not as often cited is Essential Linux Device Drivers. You won't find a free version of this one, but it still features an interesting approach. What I like about this one is it covers lots of different device types and is up-to-date as of 2.6.24, which is a bit better than LDD.
Finally, one great book about the kernel itself (not specifically for drivers) is Understanding the Linux Kernel, 3rd Edition. This covers in-depth kernel facilities and internal mechanisms. It's up-to-date as of 2.6.11.
As for online tutorials, I found this post on Pete's Blog is a really great example. Not only does it show how to create a character device (the most easy kernel driver type, i.e. the one you should start with), it uses modern Linux kernel features in an easy to understand fashion, including:
use of udev
use of a kernel data structure (FIFO)
use of kernel synchronization (mutex)
use of Sysfs with custom attributes
module options for insmod
Plus: it's aimed at Linux 3.0, which means it's more up-to-date compared to other resources.
You might also like this post about how to create Sysfs entries manually, although the Linux device model will take care of registering your device as a Sysfs entry if you don't need additional nodes or attributes.
Edit: I should add that the best way to learn real Linux device driver programming is to look at actual drivers. There are thousands of drivers in drivers. Start reading and understanding the concept of simple ones like drivers/leds and you will see how rewarding this is.
The site with very important links:
http://elinux.org/Device_drivers
Great tutorial with real examples (ends with an usbpen formatted with our own file system - search for author further articles):
http://www.linuxforu.com/tag/linux-device-drivers-series/
http://sysplay.in/index.php?pagefile=lfy_articles
Source code of famous LDD3 book ported to newest kernels:
https://github.com/martinezjavier/ldd3/
Videos:
https://www.youtube.com/playlist?list=PL16941B715F5507C5
Maybe other cources:
http://www.skilledup.com/courses?keyword=linux+kernel&price_filter=0..0
Some examples:
http://www.staerk.de/thorsten/index.php/My_Tutorials/Writing_Linux_kernel_modules
http://www.freesoftwaremagazine.com/articles/drivers_linux
http://techblog.aasisvinayak.com/kernel-module-programming-tutorial/
http://www.linuxforu.com/2009/05/a-voyage-to-the-kernel-day-11/
Books (without mentioned in this thread LDD3):
http://www.coopj.com/ with an updated source code http://www.coopj.com/LPD/
KernelHacking:
http://kernelnewbies.org/KernelHacking
Visual map of the linux kernel: http://www.makelinux.net/kernel_map/
Linux and kernel books for beginners:
http://www.kroah.com/lkn/
http://swift.siphos.be/linux_sea/
You might be interested in the newly released Linux Driver Templates. As the name suggests, it provides templates and demonstrates frequently used Linux facilities to get started quickly.
I understand it is a delayed response !!
You can pick any book, those are really great books suggested above.
But you need to really work practically. Try to be involved into Kernel as much as possible.
Mostly you need to look into kernel source code itself.
And the most interesting document you can find in Documentation folder under Kernel tree.
best book for learning device driver programming is
"LDD"
Professional Linux Kernel Architecture is a good read also.
The best source is the linux man pages but they are somewhat critical to understand for a beginner, Directly programming device drivers is not a easy task. I recommend you to go through pointers and structures through following books
Basic C Books 1. Programming C - Byron gottfried 2. The C Programming Language - Dennis Ritchie
Intermediate Books 1. Pointers on C Kenneth Reek 2. Expert C Programming Deep Secrets - Linden
Coming to device Drivers I have uploaded the Kernel Source documentation in pdf format https://drive.google.com/folderview?id=0B7iRyndFhHldR3hjOHpOZTdKTjA&usp=sharing Youcan download from this link.
Device Drivers Basics (User mode Programming)
Linux Programming interface - Michael Kerrisk
Beginning Linux Programming Wrox Publishers
Device Drivers (Kernel Deleopment) 1. Linux Kernel Development - Robert Love 2. Linux Kernel Internals - m beck
Device Drivers (Driver Programming) 1. Linux Device Drivers - Third Edition (Free Download is available for 2.6 Kernel) 2. Essential Linux Device Drivers - Venkateswaran
For Basic Driver Knowledge Follow this site http://www.tldp.org/LDP/khg/HyperNews/get/devices/devices.html
My Experience is First of all we must gain a sound knowledge on C Programming, mainly Structures and Pointers before going through Driver Programming otherwise it will be bit cumbersome to understand driver programming.
Comming to Startup :-
User space programming
Basics of gcc generating proprocessed, assembler and object files using gcc commands analysis of object files using different object tools, Basic programming in linux, threads(locks, synchronization techniques), signals, processes(fork, exec), timers(Posix). This will help you lot in understanding kernel concepts.
Kernel Programming
Understanding the kernel source tree, different branches and their importance, using ctags and cscope to browse kernel source code, this will help you in kernel source browsing.
Driver Programming
First of all Learn different driver classes, char, block etc.,
Next learn the device protocols, transmission and reception, the product manual is the best source for going through this.
Then start writing the simple char driver for example RTC (a port mapped device), IOAPIC (A memory mapped device), then go to USB, PCI, Network, I2C, SPI etc., if you want the help you can find the skeleton code for all this in kernel source tree.
I'm looking for any website(s) devoted and/or related to Operating System Development, and would love for you to kindly share any links you may have.
I have searched myself, but keep winding up at the same old sites, and am wondering if there are others that just don't get noticed by the major search engines or something.
I've put this as a community wiki, simply because I don't think it's very fair to judge an answer based on a single link.
Thank you.
I might suggest a book, rather than any website. There are a number of good ones:
Operating Systems: Design and Implementation (Tanenbaum)
Modern Operating Systems (Tanenbaum)
Operating System Concepts (Silberschatz & Galvin)
Unix Systems for Modern Architectures (Schimmel)
Once you understand the concepts, you can start to design and develop for your particular project.
How about a wiki? There's the OSDev Wiki.
I would recommend a couple of books:
Developing Your Own 32-Bit Operating System
Dissecting DOS: A Code-Level Look at the DOS Operating System
Reading these will give you an idea of how an operating system is built and the amount of work it takes to build one.
Here's one website with a tutorial walkthrough: http://www.brokenthorn.com/
(Googling 'operating system development tutorial' reveals several but most seem to be dead ends or links to Linux.)
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
I'm planning to write an operating system and I don't know very much about operating systems. Are there any good resources or books to read in order for me to learn? What are your recommendations?
We used Andrew Tannenbaum's Modern Operating Systems at the university I attended. I highly recommend it for it's clear explanations of the tradeoffs inherent in many of the design decisions that you'll run up against. This book is a little bit more "fair and balanced" than the Minix book.
I also recommend this book because, despite his net-famous flame war with Linus Torvalds, few of his biases come through in the book. Also, he's a pretty decent writer, and the book is actually entertaining.
Operating Systems Implementation Prentice Software
This book is written by Tanenbaum, the main guy behind Minix, which is what Linux was based on. It provides good overviews for basic OS concepts like memory management, file systems, processes, etc. The concepts in this book book are intimately tied to examples of the Minix OS, which is a good thing.
I think you should start by something like that.
Similar threads on this very site:
OS Development
Building a new operating system
How to get started in operating system development
Operating System Concepts is the book we used at University. It's quite ugly BUT the information inside are well explain (from basic memory management, to how to OS decide what to execute or how to avoid deadlock). Pretty wide.
Os dev website is rich in information if you want to start coding your own OS too,
While old, these books are very good:
Operating System Design with Xinu
http://ecx.images-amazon.com/images/I/51AVJFBS3EL._SL500_BO2,204,203,200_AA219_PIsitb-sticker-dp-arrow,TopRight,-24,-23_SH20_OU01_.jpg
Operating System Design-Internetworking With XINU, Vol. II
3: http://Operating System Design-Internetworking With XINU, Vol. II
Take a look at HelenOS, which is a from scratch microkernel based OS that aims to be a fully modern OS. Disclamer, I'm a contributor, I'm working on its shell from scratch.
HelenOS has been ported to ia32/64, SPARC, ARM and more, its very well designed and easy to read. Its still in its infancy but shows one possible design that really takes advantage of the microkernel design and solves many issues in a microkernel implementation (such as IPC).
It also includes scripts that automatically set up a proper tool chain needed for cross compiling. Its very easy to build and runs very well in most simulators (i.e. QEMU) or bare metal.
I would also study L4, Minix3 and the GNU HURD (based on Mach), the latter being an illustration of design pitfalls when trying to leverage a microkernel.
If you want to go the monolithic route, just study Linux.
I'd highly recommend taking a look at the MIT Operating Systems class. It's got lots of useful references, and a bunch of lab exercises which you can play around with (including automated grading scripts, so you don't have to be an MIT student to do them).
textbook http://ecx.images-amazon.com/images/I/411E3CQQYZL._SS500_.jpg
I used Operating Systems and Middleware: Supporting Controlled Interaction when I was in college. It is probably one of the best textbooks on the subject.
Operating Systems Design and Implementation
The design and implementation of the FreeBSD OS
Just off the top of my head.
Developing Your Own 32-Bit Operating System by Richard A. Burgess. Went into great details about boot loaders, setting up those strange memory and process management registers, etc. It was a great read back in 1996 when i thought i'd take a crack at writing a simple OS from scratch, but may be dated by now, dealing only with the first few generations of Pentium-class CPUs.
If I remember correctly, the Powerup to Bash Prompt HOWTO contained a lot of information that looked like it would be useful for this. So did older versions of the Linux From Scratch HOWTO, but in recent versions that has been removed.
You'll also find a lot of good information in Understanding the Linux Kernel.
I would recommend looking at embedded operating systems and building an embedded OS. It will deal with the core concepts without the overhead of a modern desktop CPU.
I wrote a multitasking embedded OS last spring as a final project, it's easier than you might think.
You should look into MINIX 3. This is an operating system that was written in, I believe, less than 10,000 lines. You can get a very good idea of how an OS works with the aid of one of Tanenbaum's books and understanding how MINIX 3 works. You could go straight to Linux, but I think this is a useful task and really helps you see how it really doesn't take that many lines to build a working OS.
http://www.minix3.org/
Apart from books, there are many sites that learn OS Development
BrokenThorn Entertainment is on of this website that learn OS dev from base
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 8 years ago.
Improve this question
One thing I've always wanted to do is develop my very own operating system (not necessarily fancy like Linux or Windows, but better than a simple boot loader which I've already done).
I'm having a hard time finding resources/guides that take you past writing a simple "Hello World" OS.
I know lots of people will probably recommend I look at Linux or BSD; but the code base for systems like that is (presumably) so big that I wouldn't know where to start.
Any suggestions?
Update: To make it easier for people who land on this post through Google here are some OS development resources:
Writing Your Own Operating System (Thanks Adam)
Linux From Scratch (Thanks John)
SharpOS (C# Operating System) (Thanks lomaxx)
Minix3 and Minix2 (Thanks Mike)
OS Dev Wiki and Forums (Thanks Steve)
BonaFide (Thanks Steve)
Bran (Thanks Steve)
Roll your own toy UNIX-clone OS (Thanks Steve)
Broken Thorn OS Development Series
Other resources:
I found a nice resource named MikeOS, "MikeOS is a learning tool to demonstrate how simple OSes work. It uses 16-bit real mode for BIOS access, so that it doesn't need complex drivers"
Updated 11/14/08
I found some resources at Freebyte's Guide to...Free and non-free Operating Systems that links to kits such as OSKit and ExOS library. These seem super useful in getting started in OS development.
Updated 2/23/09
Ric Tokyo recommended nanoos in this question. Nanoos is an OS written in C++.
Updated 3/9/09
Dinah provided some useful Stack Overflow discussion of aspiring OS developers: Roadblocks in creating a custom operating system discusses what pitfalls you might encounter while developing an OS
and OS Development is a more general discussion.
Updated 7/9/09
LB provided a link to the Pintos Project, an education OS designed for students learning OS development.
Updated 7/27/09 (Still going strong!)
I stumbled upon an online OS course from Berkley featuring 23 lectures.
TomOS is a fork of MikeOS that includes a little memory manager and mouse support. As MikeOS, it is designed to be an educational project. It is written in NASM assembler.
Updated 8/4/09
I found the slides and other materials to go along with the online Berkeley lectures listed above.
Updated 8/23/09
All questions tagged osdev on stackoverflow
OS/161 is an academic OS written in c that runs on a simulated hardware. This OS is similar in Nachos. Thanks Novelocrat!
tangurena recommends http://en.wikipedia.org/wiki/MicroC/OS-II, an OS designed for embedded systems. There is a companion book as well.
Linux Kernel Development by Robert Love is suggested by Anders. It is a "widely acclaimed insider's look at the Linux kernel."
Updated 9/18/2009
Thanks Tim S. Van Haren for telling us about Cosmos, an OS written entirely in c#.
tgiphil tells us about Managed Operating System Alliance (MOSA) Framework, "a set of tools, specifications and source code to foster development of managed operating systems based on the Common Intermediate Language."
Update 9/24/2009
Steve found a couple resources for development on windows using Visual Studio, check out BrokenThorn's guide setup with VS 2005 or OSDev's VS Section.
Updated 9/5/2012
kerneltrap.org is no longer available. The linux kernel v0.01 is available from kernel.org
Updated 12/21/2012
A basic OS development tutorial designed to be a semester's project. It guides you through to build an OS with basic components. Very good start for beginners. Related paper. Thanks Srujan!
Updated 11/15/2013
Writing a Simple Operating System From Scratch. Thanks James Moore!
Updated 12/8/2013
How to make a computer operating system Thanks ddtoni!
Updated 3/18/2014
ToAruOS an OS built mostly from scratch, including GUI
Updated Sept 12 2016
Writing your own Toy Operating System
Updated Dec 10 2016
Writing a Simple Operating System —from Scratch (thank you #Tyler C)
There are a lot of links after this brief overview of what is involved in writing an OS for the X86 platform.
The link that appears to be most promising (www.nondot.org/sabre/os/articles) is no longer available, so you'll need to poke through the Archive.org version to read it.
At the end of the day the bootloader takes the machine code of the kernel, puts it in memory, and jumps to it. You can put any machine code in the kernel that you want, but most C programs expect an OS so you'll need to tell your compiler that it won't have all that, or the bootloader has to create some of it.
The kernel then does all the heavy lifting, and I suspect it's the example kernel you want. But there's a long way to go between having a kernel that says, "Hello world" to having a kernel that loads a command interpretor, provides disk services, and loads and manages programs.
You might want to consider subscribing to ACM to get access to their older literature - there are lots of articles in the late 80's and early 90's in early computing magazines about how to create alternative OSs. There are likely books that are out of print from this era as well. You might be able to get the same information for free by looking up the indexes of those magazines (which are available on that site - click "index" near the magazine name) and then asking around for people with a copy.
Lastly, I know that usenet is dead (for so sayeth the prophets of internet doom) but you'll find that many of the craggy old experts from that era still live there. You should search google groups (they have dejanews's old repository) and I expect you'll find many people asking the same questions a decade or 1.5 ago that you're asking now. You may even run across Linus Torvalds' many queries for help as he was developing linux originally. If searches don't bring anything up, ask in the appropriate newsgroup (probably starts with comp.arch, but search for ones with OS in the name).
Minix is a lot smaller, and designed for learning purposes, and the book to go with it is a good one too.
Update: I guess Minix 3 is a bit of a different goal, but Minix 2 (and of course the first version) were for teaching purposes.
As someone who has written a real-time multi-tasking operating system from scratch...
keyboard debounce routine, keyboard driver, disk driver, video driver, file system, and finally a boot-loader - and that's just to launch it for the first time with nothing to do!
... I cannot emphasize enough how important it is to get familiar with the hardware! This is especially so if you really want to do it all yourself instead of just picking up a primitive system someone else has already laid out for you. For example, contact Intel and ask them for a CPU card for your type of CPU! This will lay it out for you - the "pin-outs", interrupts, opcodes, you name it!
Remember the hardware makes it all possible. Study the hardware. You won't regret it.
.
Write a microcontroller OS. I recommend an x86 based microcontroller. A modern OS is just huge. Learn the basics first.
I wish there was one place to get all of the info about developing your own OS. The closest to come to that is OS Dev Wiki and Forums. They offer a ton of good information regarding the setup, development, and device hardware information.
Also there are some great tutorials at BoneFide, I've used the getting started tutorial by Bran, and am now looking at a more recent one based on his called Roll your own toy UNIX-clone OS.
I second checking out: "Operating Systems : Design and Implementation"
And if you want to develop on Windows, check out jolson's blog post.
Edit:
For development on windows using Visual Studio, check out BrokenThorn's guide or OSDev's wiki.
An excellent resource is the material of the MIT course 6.828: Operating System Engineering.
XV6 - simple Unix-like teaching OS written in ANSI C for x86
http://pdos.csail.mit.edu/6.828/2012/xv6.html
XV6 source - as a printed booklet with line numbers
http://pdos.csail.mit.edu/6.828/2012/xv6/xv6-rev7.pdf
XV6 book - explains the main ideas of os design
http://pdos.csail.mit.edu/6.828/2012/xv6/book-rev7.pdf
The material is compact: 92 pages source and 96 pages commentary.
I like it more than the Minix book! It's a true gem!
you also might want to take a look at SharpOS which is an operating system that they're writing in c#.
There are good resources for operating system fundamentals in books. Since there isn't much call to create new OS's from scratch you won't find a ton of hobbyist type information on the internet.
I recommend the standard text book, "Modern Operating Systems" by Tanenbaum. You may also be able to find "Operating System Elements" by Calingaert useful - it's a thin overview of a book which give a rough sketch of what an OS is from a designer's standpoint.
If you have any interest in real time systems (and you should at least understand the differences and reasons for real time OS's) then I'd also recommend "MicroC/OS-II" by Labrosse.
Edit:
Can you specify what you mean by "more technical"? These books give pseudo code implementation details, but are you looking for an example OS, or code snippets for a particular machine/language?
-Adam
Just coming from another question. I'd like to mention Pintos... I remembered my OS course with Nachos and Pintos seems to be the same kind of thing that can run on x86.
I found Robert Love's Linux Kernel Development quite interesting. It tells you about how the different subsystems in the Linux kernel works in a very down-to-earth way. Since the source is available Linux is a prime candidate for something to hack on.
Here are some other Stack Overflow pages worth incorporating into this discussion:
Roadblocks in creating a custom operating system
Developing an operating system for the x86 architecture
My operating systems course in undergrad had us building a number of subsystems for OS/161, a simple, BSD-like kernel that provides some of the basics while leaving the freedom to explore various design space decisions in implementing higher-level services.
Start hacking away at Minix. It's a lot smaller than Linux (obviously) and it's designed with teaching purposes in mind (some, at least). Not Minix 3 though, that's a whole different story.
Already answer, but when I took Operating Systems in college we started with an early linux kernel and added simplistic modern features (basic file systems, virtual memory, multitasking, mutexes). Good fun. You get to skip some of the REALLY crazy low level assembly only stuff for mucking w/ protected mode and page tables while still learned some of the guts.
http://kerneltrap.org/node/14002
http://kerneltrap.org/files/linux-0.01.tar.bz2
I would like to include this repo How-to-Make-a-Computer-Operating-System by Samy Pesse. Is a work-in-progress. Very interesting.
You might want to look at linuxfromscratch.
Linux From Scratch (LFS) is a project
that provides you with step-by-step
instructions for building your own
custom Linux system, entirely from
source code.
A simple and basic OS development tutorial designed to be a semester's project. It guides you through to build an OS with basic components. Very good start for beginners. Related paper is here.
One reasonably simple OS to study would be µC/OS. The book has a floppy with the source on it.
http://en.wikipedia.org/wiki/MicroC/OS-II
Check out the Managed Operating System Alliance (MOSA) Project at www.mosa-project.org. They are designing an AOT/JIT compiler and fully managed operating system in C#. Some of the developers are from the inactive SharpOS project.
I've toyed with Cosmos, which is "an operating system project implemented completely in CIL compliant languages." It's written in C#, so that was right up my alley. For someone like myself who has never attempted to build an operating system, it was actually pretty cool to be able to get a "Hello World" operating system running in no time.
Check out this site: http://osix.net/modules/article/?id=359
As mentioned above, the OSDev Wiki is (by far) the best source for OS development. For those of you who speak German, the lowlevel.eu Wiki is also great. Something relatively unknown Incitatus OS, a simple kernel with a tiny set of userspace apps. It's great to use for getting into the complicated topic of OS development.
Movitz is a Lisp environment written in Common Lisp and running "on the metal". Unfortunately, some links on the Movitz main page deny access, but you can find instructions on how to download and compile the source code from the trac page. Also, a ready image can be found on the archive of this page.
IMHO this is utmost interesting, as it brings back the Lisp machine concept on the currently available hardware. It failed commercially, but this does not prove to me that the idea was bad.
The Unix haters handbook is a fun book that semi-seriously berates the concept of Unix and its derivatives. Many sections argument about how better the Lisp machine concept was.
Here's a paper called "Writing a Simple Operating System From Scratch". It covers writing a bootloader, entering x86-32 protected mode, and writing a basic kernel in C. It seems to do a good job at explaining everything in detail.
The x86 JS simulator and ARM simulator can also be very useful to understand how different pieces hardware works and make tests without exiting your favourite browser.
Intresting Question for the programmers. See it will take long long long time to build OS like Windows or Mac but if you want build a simple ones then you can try your best
You need to focus on Assembly Language,C and C++. You should be expert in these languages.
First read a good book on how OS works[Google it], then read all the info from Wiki OS
Search in youtube "How to create your own OS in Assembly Language" watch the video, Eg. Video
Download Linux OS source code and compile it yourself and try to modify the code yourself
Now you are an experienced OS editor now download Minix and QNX and start developing with them and get their docs from here Minix Doc and QNX Doc
Now you have gained the master degree(Not completely just a little more to go) in creating OS now distribute this knownledge to your freinds and with their help try to create an OS as powerful as Mac, Linux or Windows
When you have made a basic operating system it's actually hard to continue because there isn't many ressources on making GUIs or porting libraries. But i think taking a look at ToAruOS would help a lot!
The code under the surface of that OS is so damn simple! but at the same time he has ported things like cairo, python, (not yet but soon) sdl, made share memory and he has also made his own widget toolkit. It's all written in C.
Another interesting OS would be pedigreeOS. It's made by JamesM (the man behind jamesM's kernel tutorial. While it has more features than ToaruOS it's also bigger and more confusing.
But anyway these 2 OS will help you a lot especially ToAruOS.
When I started working on my basic operating systems I needed a basic guide like Stepping stones for a basic operating system. It helped me not loose my head.
That if you want to make it from absolutely nothing (pure assembly code)