how page directory is generated on Windows kernel ?
GDT setups memory limits for user memory layout.
But i want to learn how windows internally setups process page directory & VAD tree
Windows does not use the GDT to isolate user mode from kernel mode on the x86, it uses per-page protection in the page tables. Check the Windows Internals book for details.
Related
first of all hello to all hope you are good and safe.
well i have some questions about machine code and hardware and operating system.
1- i was search about how is pure machine code and i find somethings in here and net but not enough to answer my questions since im new to low level programming language. so how to write a pure machine code like open just my computer with 0,1 are machine code have any file extensions like assembly and .exe i wana write code just directly get area in ram and talk with processor and do what i writed for example open my computer or open a text file for example. so i wana know how to do it are pure machine code have a file extension like .exe or .asm
2- i know each cpu have it owen machine language somethigns is different on them it could not be a way to all cpu's undrestand our machine code. also i wana know for example we have 2 cpu both of them are x64 or x32 but 1 of them are windows other is linux are machine code of x64 windows will work also on x64 cpu linux?
thank you for give your time to me and read.
for now gathering information
An operating system provides the capability to run programs. So, one program, like the desktop or command line shell, can ask the operating system to run another program.
When that happens, the operating system creates an environment to run the program called a process, and then loads a program file from disc into the process, and directs the CPU to begin executing that program file starting at its beginning.
The operating system has a loader, whose job is to load the disc-based program file into memory of the process.
The loader knows about certain executable file formats. Different operating system have different loaders and most likely understand different executable formats.
Machine code is contained in these program files, stored on disc using those file formats. There are other ways to load machine code into memory, though a program file stored on disc loaded by the loader is the most common approach.
Asm, .asm, is a text file, human readable, for storing program code in assembly language. To use it, such text file is given as input to a build system, which converts that human readable program code into a program file containing equivalent machine code, for later loading into a process by the operating system.
Not only do different operating systems support different file formats for program files, they also support different ways to interact with the operating system, which goes to their programming model that is described by an Application Binary Interface aka ABI. All programs need to interact with the operating system for basic services like input, output, mouse, keyboard, etc.. Because ABIs differ between operating systems, the machine code in a program written for one operating system won't necessarily run on a different operating system, even if the processor is exactly the same.
Most disc-based file formats for executable program files contain indicators telling what processor the program will run on, so the same operating system on different processors requires different machine code, and hence usually different executable program files. (Some file formats support "fat" binaries meaning that the machine code for several different processors is in one program file.)
Operating systems also have features that allow execution of new machine code within an existing process. That machine code can be generated on the fly as with JTT compilers, or loaded more informally by an application program rather than the operating system loader. Further, most operating system loaders support dynamically loading additional program file content from executable program files.
So, there's lots of ways to get machine code into the memory of a process for execution — support for machine code is one of the fundamental features of operating systems.
Let's also note that no real program is pure machine code — programs use machine code & data together, so all executable file formats store both machine code and data (and metadata).
I'm doing a paper on NTFS vs FAT32 and showing a comparison between both file systems.
As far as my knowledge goes, I know that NTFS uses the MFT for holding all the files and directories whereas FAT32 only knows the following cluster for a specific file or directory. This means that FAT32 doesn't know a-priori the first block of a file if not found first by looking up in a specific directory.
My question is the following, if NTFS holds all information regarding the file system in a file does it mean it's going to be faster when doing a raw search for a filename "test.txt" within the system? From what I know, FAT will have to scan every directory in the hard drive and in each directory look if the filename exists whereas in NTFS, it only needs to scan the MFT file which is contiguous for a record that has name : "text.txt".
Am I right or I'm missing something?
I don't know, probably yes(40% bet), but to turn your problem into Stack Overflow coding on-topic here are some resources where you can find your answer and give us your self-answer:
probably, by reading "text":
NTFS.com: NTFS Basic
Microsoft TechNet: File Systems Technologies → How NTFS Works
for sure, by reading "code":
GitHub: /torvalds/linux/fs/ntfs - C source code of the NTFS file system driver used by Linux - "Linux is a clone of the operating system Unix, written from scratch by Linus Torvalds with assistance from a loosely-knit team of hackers across the Net. It aims towards POSIX and Single UNIX Specification compliance"
svn.reactos.org: /reactos/drivers/filesystems/ntfs - C source code of the NTFS file system driver used by ReactOS - "free open source operating system based on the best design principles found in the Windows NT® architecture (Windows versions such as Windows XP, Windows 7, Windows Server 2012 are built on Windows NT architecture). Written completely from scratch, ReactOS is not a Linux based system, and shares none of the UNIX architecture"
All of the above resources should agree on the concept and the two last resources show how the machines really treat the problem without some marketing talk (machines don't understand marketing and advertising very well)
I can build and manually execute an EFI application in a UEFI environment. But, I'd like to have my application executed automatically at boot time.
Is there a way to tell the bootloader to do this or do I need to turn my application in to a driver to have it automatically executed? Is there maybe some entry in an FDF, DEC, DSC, or INF file I'm missing?
You can add a startup.nsh script file to your UEFI Shell boot drive that calls you application.
The following link should serve as a handy reference for working with scripts in the shell: UEFI Shells and Scripting. Look at section 3 (EFI Shell Scripts) specifically.
However, if you are actually building your own firmware, you can also look at creating a bootable EFI image and set your default boot option to this binary. This is most useful if you are including the binary as a part of your ROM, but it might be a little involved to set up the filesystem so that it is seen as a normal boot option.
Also, if you have complete access to your firmware source, you could also just load and call your binary from your BDS driver. I don't recommend this route, unless you are specifically aiming to create a standalone device that will never boot any other image.
#NicholasEmbry answer contain almost all what is needed, but I would like to clarify some things and add recent improvements in that area.
Adding UEFI application image to boot option is actually best known method when you don't have source code of your firmware, what is typical situation. You don't have to create any special bootable image it should be simple UEFI application image, what means that your INF should contain:
MODULE_TYPE = UEFI_APPLICATION
This option is not just for Option ROMs this is for all UEFI readable medium. For example you can create FAT32 partition on your storage (no matter if it is USB, HDD, SSD, etc. it just have to be readable by firmware) and place application image on this partition. Then boot to UEFI Shell and use bcfg command to affect your boot order.
bcfg command can be blocked by you BIOS vendor. In that situation please follow procedure from this post. This is booting rEFInd using USB stick. rEFInd contain bcfg tool and give you ability to use it.
How to use bcfg ?
All is described in help help bcfg -b. -b is UEFI pager. For those who like shortcuts, this command will display all boot options in system:
bcfg boot dump -v
You can add your application using command:
bcfg boot add <boot_pos> <path_to_uefi_image> <boot_order_name>
<boot_pos> - position in boot order. Note you usually want to put your application before OS. If there is no free boot option number before your OS you can of course move your OS one option down bcfg boot mv <old_pos> <new_pos> and then add your application image in between.
<path_to_uefi_image> - path to your UEFI image. This is UEFI readable path ie. fs0:\foobar.efi
<boot_order_name> - this is how you application will be visible in boot BIOS menu
Two questions.
If I start task manager I can generate dump file either by right clicking on Applications Tab or right clicking on Process tab. Is there any difference in the dump file been generated from these tabs?
If I have a x86 application running on a x64 machine, can I use x64 task manager to generate process dump?
1) I don't know, but probably it's the same.
2) To dump 32-bit processes on 64 bits OS, 32-bit task manager must be used:
C:\Windows\SysWOW64\taskmgr.exe
Check the link Here
This question's answers are a community effort. Edit existing answers to improve this post. It is not currently accepting new answers or interactions.
Are there good tutorials for showing how to use WinDbg?
Basic Tutorials & Usage Demos
Installing and Configuring WinDbg (Windows Debug Tools)
Mike Taulty - A word for WinDbg
WinDbg Tutorials
Windows Debuggers: Part 1: A WinDbg Tutorial
Different Ways to "Start"/Attach WinDbg
Start Debugging with WinDbg (includes how to debug an .msi)
How to debug a Windows service
Setting up Windows Debugging
Debugging SQL Server...here, here, here, here
Workspaces (Understanding how they work)
Pimp up your debugger: Creating a custom workspace for WinDbg debugging
Uncovering How Workspaces Work in WinDbg
Cmdtree
Allows you to define a "menu" of debugger commands for easy access to frequently used commands without having to remember the terse command names. Don't have to put all the command definitions into the same cmdtree text file....you can keep them separate and load multiple ones (they then get their own window).
Amazing helper .cmdtree
How do I make a cmdtree window dock at startup in WinDbg
Making it easier to debug .NET dumps in WinDbg using .cmdtree
Microshaoft Cmdtree
Special Command—Execute Commands from a Customized User Interface with .cmdtree
Startup Script
You can use the -c option on the command line to automatically run a WinDbg script when you start WinDbg.
Gives opportunity to turn on DML (Debugger Markup Language) mode, load particular extensions, set .NET exception breakpoints, set kernel flags (e.g. when kernel debugging you might need to change the DbgPrint mask so you see tracing information....ed nt!Kd_DEFAULT_Mask 0xFFFFFFFF), load cmdtrees, etc.
http://yeilho.blogspot.co.uk/2012/10/windbg-init-script.html
Take Control of WinDbg
An example script:
$$ Include a directory to search for extensions
$$ (point to a source controlled or UNC common directory so that all developers get access)
.extpath+"c:\svn\DevTools\WinDBG\Extensions"
$$ When debugging a driver written with the Windows Driver Framework/KMDF
$$ load this extension that comes from the WinDDK.
!load C:\WinDDK\7600.16385.1\bin\x86\wdfkd.dll
!wdftmffile C:\WinDDK\7600.16385.1\tools\tracing\i386\wdf01009.tmf
$$ load some extensions
.load msec.dll
.load byakugan.dll
.load odbgext.dll
.load sosex
.load psscor4
$$ Make commands that support DML (Debugger Markup Language) use it
.prefer_dml 1
.dml_start
$$ Show NTSTATUS codes in hex by default
.enable_long_status 1
$$ Set default extension
.setdll psscor4
$$ Show all loaded extensions
.chain /D
$$ Load some command trees
.cmdtree c:\svn\DevTools\WinDBG\cmdtree\cmdtree1.txt
.cmdtree c:\svn\DevTools\WinDBG\cmdtree\cmdtree2.txt
$$ Show some help for the extensions
!wdfkd.help
!psscor4.help
.help /D
Command Cheat Sheets
Crash Dump Analysis Poster v3.0
SOS Cheat Sheet (.NET 2.0/3.0/3.5)
WinDbg cheat sheet (Art of Dev)
WinDbg Kernel-Mode Extension Commands Flashcards
Extensions (extend the range of commands/features supported)
AddSym- allows transfer of symbol names between IDA and WinDbg
bigLasagne (bldbgexts & blwdbgue)- assembly syntax highlighting and a driver mapping tool)
BigLib Number Reader
Byakugan- detect antidebugging methods, vista heap visualization/emulation, track buffers in memory
CmdHist- records every command you executed in your debug session so you can re-execute easily
Core Analyzer- check heap structures for corruption, detect objects shared by threads, etc.
dom WinDbg Extension- (!stlpvector, !idt, !unhex, !grep, etc.)
dumppe- dumps PE file from memory
Image Viewer Extension (Vladimir Vukicevic)
Intel UEFI Development Kit Debugger Tool- debug UEFI firmware
leaktrap- GDI/USER handle tracker to aid in leak detection
Mona (requires PyKD)- set of commands to aid in advanced analysis/find exploits
MSEC- provides automated crash analysis and security risk assessment
narly- lists info about loaded modules such as if using SafeSEH, ASLR, DEP, /GS (Buffer Security Checks)
netext (Rodney Viana)- (!wservice - list WCF service objects, !wconfig - show .config lines, !whttp - list HttpContexts, !wselect/!wfrom - support SQL like queries on arrays)
ODbgExt- open debugger extensions
OllyMigrate- pass debuggee to another debugger without restarting
Psscor2- a superset of SOS for assisting in debugging .NET 2.0 managed code
Psscor4- a superset of SOS for assisting in debugging .NET 4 managed code
PyDBGExt- allows Python scripting to be used
PyKD- allows Python to be used to script WinDbg
sdbgext (Nynaeve)-(!valloc, !vallocrwx, !heapalloc, !heapfree, !remotecall, !remotecall64, !loaddll, !unloaddll, !close, !killthread, !adjpriv, !ret)
SieExtPub-legacy extension...now built into WinDbg in ext.dll
SOSEX- more commands for helping to debug managed NET 2.0 or 4.0 code
SPT/SDBGExt2 (Steve Niemitz)- (!DumpHttpContext, !DumpASPNetRequests, !DumpSqlConnectionPools, !DumpThreadPool, etc.)
Uniqstack- source to a debugger extension (need an OSR Online account to access it)
viscope- code coverage graph
Wait Chain Traversal/wct.dll (CodePlex Debugging Extensions- display wait chains of application threads (helps find deadlocks)
windbgshark- integrates the Wireshark protocol analyser to enable VM traffic manipulation and analysis
WinDbg Extensions (Sasha Goldstein)- Tracer, WCT, heap_stat, bkb, traverse_map, traverse_vector)
WinDbg Highlight (ColorWindbg.dll) (Use Google Translate to translate link)- assembly language syntax highlighting
Write your own extension
Developing WinDbg ExtEngCpp Extension in C++
Tools of the Trade: Part IV - Developing WinDbg Extension DLLs
The Basics of Debugger Extensions: Short Term Effort, Long Term Gain
Debugging Managed Code
Breaking on an Exception
Breaking on specific CLR Exception
Debugging .NET framework source code within WinDbg
Debugging exceptions in managed code using WinDbg
Debugging managed code using WinDbg and SOS.dll
Debugging with WinDbg. Deadlocks in Applications.
MANAGED DEBUGGING with WinDbg. Introduction and Index
Setting .NET breakpoints in WinDbg for applications that crash on startup
Scripting (C#, PS, Python, and WinDbg)
KDAR (Kernel Debugger Anti Rootkit)- a collection of WinDbg scripts
Sysnative BSOD Scripts/Processing Apps
WinDbg Script library- a collection of WinDbg scripts
Scripting MDbg and DbgHostLib- allows managed code to script the Managed Debugger (MDBG) and the DbgEng
ExtCS- allows control of WinDbg via C# scripts
PowerDBG- allows control of WinDbg via PowerShell scripts
Pykd - allows control of WinDbg via Python scripts
windbglib - Python wrapper library around the pykd extension for WinDbg, mimicking immlib (so you can use scripts originally written for Immunity Debugger)
Debuggers/Tools that use the dbgeng.dll API/WinDbg Tools
A Simple Dbgeng Based User Mode Debugger
Acorns.Debugging NET Deadlock Detector (uses cdb.exe) (download)
CLR Managed Debugger (MDBG)
DbgHost - How to control a debugging engine
Debug Diagnostic Tool v1.2 (DebugDiag), Ver 2.0 + DebugDiag Blog
Dynamorio - dynamic binary instrumentation tool which can interact with WinDbg
IDA + WinDbg plugin
GUI WinDbg
LeakShell (find managed leaks)
mdbglib - Managed Debug API
PyDbgEng- Python wrapper for Windows Debugging Engine
SOSNET - a WinDbg Fork/alternative shell that concentrates on using the SOS extension and supports C# scripting
SOSNET O2 fork - fork of SOSNET that uses Roslyn for the C# REPL (read-eval-print-loop) scripting engine
VDB/Vivisect (kenshoto) - provides a cross-platform debugging API layered on WinDbg
WinAppDbg + Heappie-WinAppDbg
Writing a basic Windows debugger
Different Ways to Generate Crash Dump Files for Post-Mortem Analysis
DebugDiag 2.0
Dump Cheat Sheet- includes how to generate dump from Hyper-V, VMware ESX, and XenServer VMs.
Citrix SystemDump
Keyboard Keypress Combination
MiniDumpWriteDump- (via Win32 API call inside your application). (Example for C# applications)
NMI Switch(hardware based feature to generate an NMI...usually found on high-end servers e.g. HP or you can obtain an add-in PCI card "Universal PCI Dump Switch"). Microsoft NMI technology background.
Procdump
Menu System ? Advanced System Settings ? Startup and Recovery(registry info), (how to configure a Complete (Full) Memory Dump), (how to enable Complete Memory Dump), (how to enable Complete Memory Dump on Windows 7 when PC has lots of memory...normally not available when more than 2 GB of memory)
Task Manager "Create Dump File"
UserDump, instructions (very old tool)
UserModeProcessDumper, instructions
Visual Studio "Save Dump As…"
WER (Windows Error Reporting....local dumps)
WinDbg
Dump Analysis Tools
BlueScreenView - finds the minidump .dmp files saved by Windows after a BSOD, and extracts information about what caused the crash
Debug.Analyzer (can analyse dump files and plug-ins can be written in .NET)
SAD - Simple After Dump (postmortem analyzer)
Volatility - framework for analyzing "memory" recorded in dump files (cheat sheet)
Dump related Tools
Citrix dumpcheck - checks consistency of dump file (looks like it's been abandoned link + link)
dumpchk (part of Debugging Tools) - checks consistency of a Dump file
MoonSols Windows Memory Toolkit (formerly windd) - converts various raw memory dump files into WinDbg compatible dmp files
vm2dmp - Microsoft Hyper-V VM State to Memory Dump Converter
vmss2core - converts VMware snapshot file into a core dump file (download), (instructions)
Kernel Debugging Virtual Machines
VMKD - Virtual Machine KD Extensions
VirtualKD - (kernel debugger support for OS's hosted in VMware/VirtualBox)
Videos
.NET Cracking 101 #2 - WinDbg basics
.NET Debugging for the Production Environment (Channel9)
dotnetConf - Advanced Debugging with WinDbg and SOS
David Truxall "Debugging with WinDbg"
Mike Taulty Debugging Memory Leaks
oredev 2009 Session: Debugging .NET Applications with WinDbg
Pluralsight Advanced Windows Debugging(plus various other ones at Pluralsight)
Tess Ferrandez WinDbg (Channel9)
TiGa's Video Tutorial Series on IDA Pro
Blogs
Advanced .NET Debugging
All Your Base Are Belong To Us (Sasha Goldstein)
Analyze-v
ASP.NET Debugging
Cyberiafreak (threading and advanced Windows programming and debugging)
Debug Analyzer.NET
Debug and Beyond
Debugging Experts Magazine Online
Debugging Toolbox (WinDbg scripts, debugging and troubleshooting tools and techniques to help you isolate software problems.)
Decrypt my World
greggm's WebLog
Junfeng Zhang's Windows Programming Notes
Kristoffer's tidbits
Mark Russinovich's Blog
Mike Stalls .NET Debugging Blog
Naveen's Blog
Never Doubt Thy Debugger (Carlo)
Notes from a Dark Corner
Ntdebugging Blog (Microsoft Global Escalation Services team)
Nynaeve. Adventures in Windows debugging and reverse engineering
PFE Developer Notes for the Field
Visual Studio Debugger Team
WinDbg by Volker von Einem
Advanced Articles & Tutorial Resources
Advanced Debugging Techniques in WinDbg
Debugging Applications for MS.Net and Windows (PowerPoint Slides)
Debugging STL Containers with WinDbg
Debug Tutorials 1-7 (CodeProject-Toby Opferman)
Debugging.tv
Developmentor WinDbg Tagged articles
Dr Fu's Security Blog - Malware Analysis Tutorials - Reverse Engineering Approach
Exploit writing tutorial part 5 : How debugger modules & plugins can speed up basic exploit development
Hunting Rootkits
Remote Microsoft Windows Server OS Kernel Debugging Using Dell Windows Debugger Utility (DWDU) (DELL(TM) Windows(R) Debugger Utility 1.1 README)
Alternative Debuggers
Bokken - (Inguma) (GUI for radare)
BugDbg
Debug++ (not released yet)
Debuggy
Discoloured Ring 0 Debugger (download)
edb (Linux)
FDBG
GoBug
Hades (Ring 3 debugger with anti debugger detection strategy)
Hopper (Linux, OS X and Windows) (Windows debugging not currently implemented)
Hyperdbg
IDA Debugger
ImmunityDebugger
Nanomite
Obsidian (non-intrusive debugger)
OllyDBG
PEBrowse
RaceVB6 (VB6 P-Code debugger)
radare
radare2ui (GUI for radare)
Rasta Ring 0 Debugger (RR0D)
Syser Kernel Debugger
TRW 2000 (very old debugger circa W9x) + dions plugin archive
VisualDux Debugger
Wintruder (extendable debugger)
WKTVDebugger (a debugger for Visual Basic P-Code) (download)
x64_dbg
Zeta Debugger
Other Links
Collaborative RCE Tool Library- debugger and system level tools
cr4zyserb- plugins & other debugging tools
How to Write a Windows Debugger References (Devon Straw)- detailed information that you would need if you wanted to write your own debugger e.g. PDB file format, .DMP file formats, PE File structure, how to record stack traces, etc., etc.
Tuts4You- unpackers, IDA, OllyDBG, Immunity Debugger plugins, etc.