When debugging an application I am developing, using WinDbg, to my horror/surprise, I saw the following in the WinDbg window. I do have AMD graphics, am not a gamer, and have recently updated the graphics drivers.
In the following, I have changed the name of my exe file and the reported path to it, but nothing else. I know of no user named "Randal".
Can anyone tell me what is going on here? I have uninstalled Raptr. I think it was installed via the AMD update. But why would Raptr interfere with my own program in this way, and what else might it have been doing?
Thanks - extract from WinDbg log follows:
ModLoad: 65030000 65060000 C:\Windows\assembly\NativeImages_v4.0.30319_32\UIAutomationTypes\232495ea0368dada2d208c51f0e5349c\UIAutomationTypes.ni.dll
ModLoad: 62a70000 62afc000 C:\Windows\SysWOW64\UIAutomationCore.dll
ModLoad: 6b7c0000 6b7fc000 C:\Windows\SysWOW64\OLEACC.dll
ModLoad: 19310000 194ec000 C:\PROGRA~2\RAPTRI~1\Raptr\ltc_game32-111020.dll
[17:50:11.00848-110772] NotRealName.exe dbglog.cpp(204) DbgLog_F Log for module 'game' (111020) started with process 'NotRealName.exe' (x86, PID: 103208, CmdLine: C:\Users\[MyLegitimatePath]\NotRealName.exe) - Sun May 29 17:50:11 2016
[17:50:11.00848-110772] NotRealName.exe ltc_game.cpp(5564) DllMain Current process is an auto-hook candidate
[17:50:11.00850-110772] NotRealName.exe ltc_game.cpp(2315) StartAutoHookMonitoring Auto-hook monitoring started
[17:50:11.00851-111588] NotRealName.exe ltc_game.cpp(5345) MainThread MainThread executing...
[17:50:11.00851-109468] NotRealName.exe autohook.cpp(265) AutoHookManager::_WatchForQuitOrTimeoutThread Watching
[17:50:11.00851-111588] NotRealName.exe console.h(158) CConsole::Output CONSOLE: Console logging has been enabled.
[17:50:11.00851-111588] NotRealName.exe console.h(158) CConsole::Output CONSOLE: Added alias [sta:"StartRecorder"]
[17:50:11.00853-111588] NotRealName.exe console.h(158) CConsole::Output CONSOLE: Added alias [stp:"StopRecorder"]
[17:50:11.00853-111588] NotRealName.exe console.h(158) CConsole::Output CONSOLE: Added alias [slsk:"SetLiveStreamKey live_37135925_lD9UDVtrXfCT8yf0kHeWdVhaITG9oy"]
[17:50:11.00853-111588] NotRealName.exe console.h(158) CConsole::Output CONSOLE: Added alias [srl:"RecorderLog C:\Users\Randal\Desktop\ffmpeg.log"]
[17:50:11.00853-111588] NotRealName.exe console.h(158) CConsole::Output CONSOLE: Added alias [twitchtest2:"BroadcasterDebug 1;GetTwitchAuthToken ql7dn49ya6oalq5xux4s33a16n3e7sv RaptrRandal raptrtest"]
[17:50:11.00853-111588] NotRealName.exe console.h(158) CConsole::Output CONSOLE: Added alias [twitchtest:"BroadcasterDebug 1;SetTwitchAuthToken 1eduwz3hdp3yprib6vhpr5c64go2nv3;StartBroadcast"]
[17:50:11.00853-111588] NotRealName.exe console.h(158) CConsole::Output CONSOLE: Added alias [gfastfb:"GetFramebufferAsSizeFastBench 1"]
[17:50:11.00853-111588] NotRealName.exe console.h(158) CConsole::Output CONSOLE: Added alias [dt1:"D3D11FastDumpTest 1"]
[17:50:11.00853-111588] NotRealName.exe console.h(158) CConsole::Output CONSOLE: Console initialised.
[17:50:11.00867-111588] NotRealName.exe overlayui.cpp(1176) OverlayUI_InitializeConsole This game has no on-load flags.
[17:50:11.00868- 96800] NotRealName.exe ltc_game.cpp(2813) ModuleScanThread ModuleScanThread executing...
[17:50:11.00869- 96800] NotRealName.exe ltc_game.cpp(2820) ModuleScanThread Process File Name:NotRealName.exe
[17:50:11.00869- 96800] NotRealName.exe ltc_game.cpp(2790) LogRunningProcesses Processes at startup:
[17:50:11.00869-111588] NotRealName.exe ltc_game.cpp(4075) EnumWindowsPrep Window [0012270E] has been input-enabled.
[17:50:11.00869-111588] NotRealName.exe ltc_game.cpp(4075) EnumWindowsPrep Window [0007273E] has been input-enabled.
[17:50:11.00870-111588] NotRealName.exe ltc_game.cpp(4075) EnumWindowsPrep Window [00391916] has been input-enabled.
[17:50:11.00871-111588] NotRealName.exe ltc_game.cpp(4075) EnumWindowsPrep Window [0028094C] has been input-enabled.
[17:50:11.00872-111588] NotRealName.exe ltc_game.cpp(4075) EnumWindowsPrep Window [003D154E] has been input-enabled.
[17:50:11.00872-111588] NotRealName.exe ltc_game.cpp(4075) EnumWindowsPrep Window [003100D0] has been input-enabled.
[17:50:11.00873-111588] NotRealName.exe ltc_game.cpp(5397) MainThread Waiting for shutdown event (00000DC0) or unload event...
[17:50:11.00882- 96800] NotRealName.exe ltc_game.cpp(2804) LogRunningProcesses 001. [System Process] PID 0 Parent PID: 0 Usage: 0 Threads: 8 Priority base: 0 Flags: 0x00000000
[17:50:11.00882- 96800] NotRealName.exe ltc_game.cpp(2804) LogRunningProcesses 002. System PID 4 Parent PID: 0 Usage: 0 Threads: 186 Priority base: 8 Flags: 0x00000000
[17:50:11.00884- 96800] NotRealName.exe ltc_game.cpp(2804) LogRunningProcesses 003. smss.exe PID 384 Parent PID: 4 Usage: 0 Threads: 2 Priority base: 11 Flags: 0x00000000
[17:50:11.00884- 96800] NotRealName.exe ltc_game.cpp(2804) LogRunningProcesses 004. csrss.exe PID 600 Parent PID: 592 Usage: 0 Threads: 11 Priority base: 13 Flags: 0x00000000
[17:50:11.00884- 96800] NotRealName.exe ltc_game.cpp(2804) LogRunningProcesses 005. wininit.exe PID 684 Parent PID: 592 Usage: 0 Threads: 3 Priority base: 13 Flags: 0x00000000
etc etc etc
[17:50:11.00926- 96800] NotRealName.exe ltc_game.cpp(2887) ModuleScanThread 147. ltc_game32-111020.dll # 0x19310000 - 0x194EBFFF Size: 1949696
[17:50:11.00926- 96800] NotRealName.exe ltc_game.cpp(2909) ModuleScanThread Found Graphics Module:d3d9.dll bDelay:0
[17:50:11.00927- 96800] NotRealName.exe com_hooks.cpp(197) Direct3D9 Installing d3d9.dll hooks...
[17:50:13.00562-110704] NotRealName.exe ltc_game.cpp(2380) OnGameRender Process: 0x000000FF System: 0x000000FF
[17:50:13.00563-110704] NotRealName.exe ltc_game.cpp(2382) OnGameRender Ideal processor is set to 5 (not changing)
[17:51:11.00852-109468] NotRealName.exe autohook.cpp(257) AutoHookManager::_WatchForQuitOrTimeout Timed-out
[17:51:11.00852-109468] NotRealName.exe ltc_game.cpp(2298) OnAutoHooked PID: 103208 timed-out
[17:51:11.00879- 96800] NotRealName.exe ltc_game.cpp(2948) ModuleScanThread Ending module scan thread...
[17:51:11.00879-111588] NotRealName.exe ltc_game.cpp(5405) MainThread Unload game module event fired! Cleaning up...
[17:51:11.00880-111588] NotRealName.exe ltc_game.cpp(5457) MainThread Shutdown capture
[17:51:11.00881-111588] NotRealName.exe ltc_game.cpp(5463) MainThread Waiting for module scan thread...
[17:51:11.00885-111588] NotRealName.exe com_hooks.cpp(207) Direct3D9 Removing d3d9.dll hooks...
[17:51:12.00174-111588] NotRealName.exe winapi_hooks.cpp(977) DoHooking_WinAPI WARNING! Cannot unhook, not hooked!
[17:51:12.00175-111588] NotRealName.exe ltc_game.cpp(5528) MainThread Shut down.
[end]
Related
there is a .NET application that seems to have memory leak in the unmanaged heap. I found a promising blog that explains how to debug the unmanaged heap and trace the heap frames up to the managed functions that caused the memory allocation (https://www.deleaker.com/blog/2021/03/19/unmanaged-memory-leaks-in-dotnet/). As I'm a first time windbg user, I decided to repeat the example shown in the blog. I copypasted the code, downloaded the debug tool package and used the windbg as suggested. Here is whgere I get stuck:
Microsoft (R) Windows Debugger Version 10.0.22000.194 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
*** wait with pending attach
************* Path validation summary **************
Response Time (ms) Location
Deferred srv*
Symbol search path is: srv*
Executable search path is:
ModLoad: 00000000`00120000 00000000`00128000 C:\<MyPath>\WinDbgTest.exe
ModLoad: 00007ffe`c4c10000 00007ffe`c4e05000 C:\WINDOWS\SYSTEM32\ntdll.dll
ModLoad: 00000000`77800000 00000000`779a3000 ntdll.dll
ModLoad: 00007ffe`c3180000 00007ffe`c31d9000 C:\WINDOWS\System32\wow64.dll
ModLoad: 00007ffe`c30d0000 00007ffe`c3153000 C:\WINDOWS\System32\wow64win.dll
ModLoad: 00000000`777f0000 00000000`777fa000 C:\WINDOWS\System32\wow64cpu.dll
ModLoad: 00000000`74890000 00000000`748e2000 mscoree.dll
ModLoad: 00000000`75e50000 00000000`75f40000 KERNEL32.dll
ModLoad: 00000000`76f40000 00000000`77154000 KERNELBASE.dll
ModLoad: 00000000`756b0000 00000000`7572a000 ADVAPI32.dll
ModLoad: 00000000`75730000 00000000`757ef000 msvcrt.dll
ModLoad: 00000000`75be0000 00000000`75c55000 SECHOST.dll
ModLoad: 00000000`76610000 00000000`766d0000 RPCRT4.dll
ModLoad: 00000000`74760000 00000000`747ed000 mscoreei.dll
ModLoad: 00000000`766d0000 00000000`76715000 SHLWAPI.dll
ModLoad: 00000000`74750000 00000000`7475f000 AppCore.dll
ModLoad: 00000000`74e30000 00000000`74e38000 VERSION.dll
ModLoad: 00000000`707a0000 00000000`70f50000 clr.dll
ModLoad: 00000000`75890000 00000000`75a26000 USER32.dll
ModLoad: 00000000`774e0000 00000000`774f8000 win32u.dll
ModLoad: 00000000`705b0000 00000000`7065b000 ucrtbase_clr0400.dll
ModLoad: 00000000`70660000 00000000`70674000 VCRUNTIME140_CLR0400.dll
ModLoad: 00000000`76ec0000 00000000`76ee3000 GDI32.dll
ModLoad: 00000000`75c60000 00000000`75d3c000 gdi32full.dll
ModLoad: 00000000`76810000 00000000`7688b000 msvcp_win.dll
ModLoad: 00000000`75a30000 00000000`75b50000 ucrtbase.dll
ModLoad: 00000000`75f50000 00000000`75f75000 IMM32.dll
ModLoad: 00000000`6f180000 00000000`7058e000 mscorlib.ni.dll
ModLoad: 00000000`76060000 00000000`76143000 ole32.dll
ModLoad: 00000000`77560000 00000000`777e2000 combase.dll
ModLoad: 00000000`76e50000 00000000`76ead000 bcryptPrimitives.dll
ModLoad: 00000000`6f0f0000 00000000`6f17a000 clrjit.dll
ModLoad: 00000000`757f0000 00000000`75886000 OLEAUT32.dll
(745c.6ac0): Break instruction exception - code 80000003 (first chance)
ntdll!DbgBreakPoint:
00007ffe`c4cb0810 cc int 3
0:005> !heap -s
************************************************************************************************************************
NT HEAP STATS BELOW
************************************************************************************************************************
NtGlobalFlag enables following debugging aids for new heaps:
stack back traces
LFH Key : 0xca6fcf0f30c2eb6b
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
0000000001db0000 08000002 60 32 60 11 5 1 0 0
0000000000150000 08008000 64 4 64 2 1 1 0 0
-------------------------------------------------------------------------------------
0:005> !heap -stat -h 0000000001db0000
heap # 0000000001db0000
group-by: TOTSIZE max-display: 20
size #blocks total ( %) (percent of total busy bytes)
2094 1 - 2094 (48.69)
838 1 - 838 (12.28)
800 1 - 800 (11.96)
120 5 - 5a0 (8.41)
1d8 2 - 3b0 (5.51)
100 3 - 300 (4.48)
238 1 - 238 (3.32)
50 4 - 140 (1.87)
42 3 - c6 (1.16)
3c 2 - 78 (0.70)
62 1 - 62 (0.57)
48 1 - 48 (0.42)
30 1 - 30 (0.28)
28 1 - 28 (0.23)
10 1 - 10 (0.09)
4 1 - 4 (0.02)
0:005> !heap -flt s 2094
_HEAP # 1db0000
HEAP_ENTRY Size Prev Flags UserPtr UserSize - state
0000000001db10c0 020c 0000 [00] 0000000001db10f0 02094 - (busy)
unknown!printable
_HEAP # 150000
0:005> !heap -p -a 0000000001db10f0
address 0000000001db10f0 found in
_HEAP # 1db0000
HEAP_ENTRY Size Prev Flags UserPtr UserSize - state
0000000001db10c0 020c 0000 [00] 0000000001db10f0 02094 - (busy)
unknown!printable
7ffec4c3b49d ntdll!RtlpAllocateHeapInternal+0x0000000000000a7d
7ffec4c5dce1 ntdll!RtlpInitEnvironmentBlock+0x0000000000000049
7ffec4ce27c1 ntdll!LdrpInitializeProcess+0x0000000000000ba1
7ffec4c84ceb ntdll!LdrpInitialize+0x000000000000015f
7ffec4c84b73 ntdll!LdrpInitialize+0x000000000000003b
7ffec4c84b1e ntdll!LdrInitializeThunk+0x000000000000000e
0:005> .load C:\<DllPath>\WinDbgDlls\sos.dll
0:005> !ip2md 7ffec4c84b1e
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
2) the file mscordacwks.dll that matches your version of clr.dll is
in the version directory or on the symbol path
3) or, if you are debugging a dump file, verify that the file
mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
4) you are debugging on supported cross platform architecture as
the dump file. For example, an ARM dump file must be debugged
on an X86 or an ARM machine; an AMD64 dump file must be
debugged on an AMD64 machine.
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.
0:005> .cordll -ve -u -l
CLR DLL status: No load attempts
Everything seems to work until I call !ip2md, here comes a "Failed to load data access DLL". Following some google results I have put the clr.dll, SOS.dll and mscordacwks.dll in one folder and made sure that they all have the same bitness and the same version.
How to proceed?
Instead of loading a specific version of SOS located somewhere on your PC, let .NET decide which version to load.
Replace
.load C:\<DllPath>\WinDbgDlls\sos.dll
by
.loadby sos clr
This tells WinDbg to load the SOS extension from whatever place CLR was loaded from. This will make sure that the versions match and the DAC matches as well.
.loadby may depend on the .NET version
.loadby sos mscorwks ; *** .NET 2
.loadby sos clr ; *** .NET 4
.loadby sos coreclr ; *** Silverlight and .NET Core
I tried to start a CloudSQL proxy on supervisor, however I have no idea what is wrong with it. The documentation does not show any clues to this issue. Any ideas would be much appreciated.
I tried the setup on a clean Ubuntu 16 and then installed supervisor and downloaded cloud_sql_proxy. And I put files under /root and execute as root for debugging.
Here is my current setup:
/etc/supervisord.conf
[unix_http_server]
file=/tmp/supervisor.sock ; the path to the socket file
chmod=0766 ; socket file mode (default 0700)
[supervisord]
logfile=/tmp/supervisord.log ; main log file; default $CWD/supervisord.log
logfile_maxbytes=50MB ; max main logfile bytes b4 rotation; default 50MB
logfile_backups=10 ; # of main logfile backups; 0 means none, default 10
loglevel=info ; log level; default info; others: debug,warn,trace
pidfile=/tmp/supervisord.pid ; supervisord pidfile; default supervisord.pid
nodaemon=false ; start in foreground if true; default false
minfds=1024 ; min. avail startup file descriptors; default 1024
minprocs=200 ; min. avail process descriptors;default 200
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
[include]
files = /etc/supervisor/conf.d/*.conf
/etc/supervisor/conf.d/cloud_sql_proxy.conf
[program:cloud_sql_proxy]
command=/root/cloud_sql_proxy -dir=/cloudsql -instances="project_id:us-central1:instance-name" -credential_file="/root/service-account.json"
autostart=true
autorestart=true
startretries=1
startsecs=8
stdout_logfile=/var/log/cloud_sql_proxy-stdout.log
stderr_logfile=/var/log/cloud_sql_proxy-stderr.log
I got the following error after inspecting /tmp/supervisord.log:
2018-10-14 15:49:49,984 INFO spawned: 'cloud_sql_proxy' with pid 3569
2018-10-14 15:49:49,989 INFO exited: cloud_sql_proxy (exit status 0; not expected)
2018-10-14 15:49:50,991 INFO spawned: 'cloud_sql_proxy' with pid 3574
2018-10-14 15:49:50,996 INFO exited: cloud_sql_proxy (exit status 0; not expected)
2018-10-14 15:49:51,998 INFO gave up: cloud_sql_proxy entered FATAL state, too many start retries too quickly
2018-10-14 15:51:46,981 INFO spawned: 'cloud_sql_proxy' with pid 3591
2018-10-14 15:51:46,986 INFO exited: cloud_sql_proxy (exit status 0; not expected)
2018-10-14 15:51:47,989 INFO spawned: 'cloud_sql_proxy' with pid 3596
2018-10-14 15:51:47,998 INFO exited: cloud_sql_proxy (exit status 0; not expected)
2018-10-14 15:51:47,999 INFO gave up: cloud_sql_proxy entered FATAL state, too many start retries too quickly
Finally I managed to figure out a working solution, and here is it:
Create a new file /root/start_cloud_sql_proxy.sh:
#!/bin/bash
/root/cloud_sql_proxy -dir=/cloudsql -instances="project_id:us-central1:instance-name" -credential_file="/root/service-account.json"
Under /etc/supervisor/conf.d/cloud_sql_proxy.conf, change the command to execute a bash file:
command=/root/start_cloud_sql_proxy.sh
What could cause the error:
Bad symbols for NTDLL (error 3). Aborting.
While using !cs commands?
I'm analyzing a keyboard-forced complete dump and have set the symbol path to the MS symbol server.
lml does not list ntdll as loaded
.sym noisy and a reload shows:
3: kd> .reload ntdll
"ntdll" was not found in the image list.
Debugger will attempt to load "ntdll" at given base 00000000`00000000.
Please provide the full image name, including the extension (i.e. kernel32.dll)
for more reliable results.Base address and size overrides can be given as
.reload <image.ext>=<base>,<size>.
DBGENG: ntdll - Partial symbol image load missing image info
DBGHELP: No header for ntdll. Searching for dbg file
DBGHELP: .\ntdll.dbg - file not found
DBGHELP: ntdll missing debug info. Searching for pdb anyway
DBGHELP: Can't use symbol server for ntdll.pdb - no header information available
DBGHELP: ntdll.pdb - file not found
DBGHELP: ntdll - no symbols loaded
Unable to add module at 00000000`00000000
3: kd> .reload nt
DBGHELP: nt - public symbols
c:\symbols\ntkrnlmp.pdb\BF9E190359784C2D8796CF5537B238B42\ntkrnlmp.pdb
Should ntdll be listed in the lm commands?
Is there something special needed to load the symbols or run these commands?
Additionally I am also curious why I cannot dump all threads in forced/complete system dumps:
3: kd> ~* k
^ Syntax error in '~* k'
Have I simply failed to set up my windbg correctly or are these commands meant for other types of dumps?
It seems you have done user mode debugging before. Now you're in kernel mode, which you can see from the x: kd> prompt.
Kernel mode debugging is somewhat different to user mode debugging. Most important IMHO: not all application memory (virtual memory) is available, just the part that was in RAM at the time of the dump (physical memory) aka. Working Set (physical memory of a particular process).
You can search for your executable using !process 0 0 <exename>
0: kd> !process 0 0 NotMyFault.exe
PROCESS ff3b58f0 SessionId: 0 Cid: 05ac Peb: 7ffde000 ParentCid: 039c
DirBase: 018c02e0 ObjectTable: e165d728 HandleCount: 35.
Image: NotMyfault.exe
Unfortunately there is no wildcard search possible at this point. You can then switch to that process using .process <process>:
0: kd> .process ff3b58f0
Implicit process is now ff3b58f0
To see the threads use !process <process> 2:
0: kd> !process ff3b58f0 2
PROCESS ff3b58f0 SessionId: 0 Cid: 05ac Peb: 7ffde000 ParentCid: 039c
DirBase: 018c02e0 ObjectTable: e165d728 HandleCount: 35.
Image: NotMyfault.exe
THREAD ff1d4020 Cid 05ac.05b0 Teb: 7ffdd000 Win32Thread: e1c6e2b0 RUNNING on processor 0
Next, use !thread <thread> to get output similar to ~:
0: kd> !thread ff1d4020
THREAD ff1d4020 Cid 05ac.05b0 Teb: 7ffdd000 Win32Thread: e1c6e2b0 RUNNING on processor 0
IRP List:
81942f68: (0006,0094) Flags: 40000000 Mdl: 00000000
Not impersonating
DeviceMap e169ffd0
Owning Process 0 Image: <Unknown>
Attached Process ff3b58f0 Image: NotMyfault.exe
Wait Start TickCount 13575 Ticks: 0
Context Switch Count 653 IdealProcessor: 0 LargeStack
UserTime 00:00:00.000
KernelTime 00:00:00.078
Win32 Start Address NotMyfault (0x01002945)
Start Address kernel32!BaseProcessStartThunk (0x7c8106f5)
Stack Init f36f1000 Current f36f030c Base f36f1000 Limit f36ec000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0 DecrementCount 16
ChildEBP RetAddr Args to Child
f36f0ad0 8052036a 00000050 81617000 00000001 nt!KeBugCheckEx+0x1b (FPO: [Non-Fpo])
f36f0b38 80544578 00000001 81617000 00000000 nt!MmAccessFault+0x9a8 (FPO: [Non-Fpo])
f36f0b38 fca6161d 00000001 81617000 00000000 nt!KiTrap0E+0xd0 (FPO: [0,0] TrapFrame # f36f0b50)
f36f0bd8 fca61a24 81942f68 f36f0c1c fca61b26 myfault+0x61d
f36f0be4 fca61b26 80fa7350 00000001 00000000 myfault+0xa24
f36f0c1c 804ef18f 80f7b600 81942f68 806e6428 myfault+0xb26
...
If you have trouble loading symbols, also try .reload /user after using the .process <process>.
I'm injecting some code to hook apis in processes but I have some issues in some applications like chrome.exe
My test app launches a suspended process, do injection and api hooking and then resumes it.
CreateProcessW is hooked in order to be able to hook child processes. If CreateProcessW is called, it is forced to be created suspended, hook the child and resume it.
The injected code only depends on ntdll api's so, although hooked processes are not fully initialized yet, ntdll.dll is always present.
Code is injected using a helper thread using CreateRemoteThread or NtCreateThreadEx with the CREATE_SUSPENDED flag. (No matter which one, the issue still there)
After this intro, the problem is that in some processes like some chrome childs, CreateRemoteThread returns TRUE but when I resume the injector thread, it exits with code 0xC0000022 and the process exits too.
If I attach WinDbg to a chrome.exe child process that is suspended, before I do anything, it fails too and chrome.exe ends with the same behavior.
Seems O.S. code executed before RtlUserThreadStart, generates the error but I don't know how to debug it.
How can I debug code that runs before RtlUserThreadStart? Is there a debugger or a windbg option that allows me to do that?
EDIT:
Following the last post from here, I could retrieve this info:
0a88:0814 # 02688302 - LdrpInitializeProcess - INFO: Beginning execution of chrome.exe (c:\Program Files (x86)\Google\Chrome\Application\chrome.exe)
Current directory: C:\Windows
Search path: C:\Windows\SYSTEM32 0a88:0814 # 02688318 - LdrpInitializeProcess - ERROR: Initializing the current directory to "C:\Windows" failed with status 0xc0000022
0a88:0814 # 02688334 - LdrLoadDll - ENTER: DLL name: C:\Windows\SYSTEM32\wow64.dll DLL path: NULL 0a88:0814 # 02688349 - LdrpLoadDll - ENTER: DLL name: C:\Windows\SYSTEM32\wow64.dll DLL path: C:\Windows\SYSTEM32
0a88:0814 # 02688365 - LdrpLoadDll - INFO: Loading DLL C:\Windows\SYSTEM32\wow64.dll from path C:\Windows\SYSTEM32 0a88:0814 # 02688380 - LdrpFindOrMapDll - ENTER: DLL name: C:\Windows\SYSTEM32\wow64.dll DLL path: C:\Windows\SYSTEM32
0a88:0814 # 02688396 - LdrpSearchPath - ENTER: DLL name: C:\Windows\SYSTEM32\wow64.dll DLL path: C:\Windows\SYSTEM32
0a88:0814 # 02688412 - LdrpResolveFileName - ENTER: DLL name: C:\Windows\SYSTEM32\wow64.dll
0a88:0814 # 02688427 - LdrpResolveFileName - RETURN: Status: 0xc0000022
0a88:0814 # 02688443 - LdrpSearchPath - RETURN: Status: 0xc0000022
0a88:0814 # 02688458 - LdrpFindOrMapDll - RETURN: Status: 0xc0000022
0a88:0814 # 02688474 - LdrpLoadDll - RETURN: Status: 0xc0000022
0a88:0814 # 02688490 - LdrLoadDll - RETURN: Status: 0xc0000022
0a88:0814 # 02688505 - LdrpInitializeProcess - ERROR: Loading WOW64 image management DLL "C:\Windows\SYSTEM32\wow64.dll" failed with status 0xc0000022
0a88:0814 # 02688521 - _LdrpInitialize - ERROR: Process initialization failed with status 0xc0000022
0a88:0814 # 02688536 - LdrpInitializationFailure - ERROR: Process initialization failed with status 0xc0000022
The process is created with a restricted token, the main thread inherits it but my injector thread isn't restricted because it is created by my app.
I can assume ntdll's apis are not hooked yet by chrome (in this case) because injection takes place before CreateProcess returns to chrome.
May the non-restricted token in my thread conflicts with process token in some way?
Take a look at Debugging WinLogon in the windbg help (debugger.chm). Simply substitute "chrome.exe" for "winlogon.exe". This technique controls a user mode debugger (ntsd) from the kernel mode debugger. I believe this will allow you debug chrome.exe's process initialization much earlier than using a user mode debugger alone.
The issue in chrome was the following:
Chrome launches child processes with very limited privileges (because of the sandbox) but before resuming the main thread it impersonates the main thread with a token with more privileges in order to let the process initialize.
My injector thread was not impersonating so the limited process token raised the 0xC0000022 exit code when the LdrpInitializeProcess routine was executed.
My dotcloud setup (django-celery with rabbitmq) was working fine a week ago - the processes were starting up ok and the logs were clean. However, I recently repushed (without updating any of the code), and now the logs are saying that the processes fail to start even though they seem to be running.
Supervisord log
dotcloud#hack-default-www-0:/var/log/supervisor$ more supervisord.log
2012-06-03 10:51:51,836 CRIT Set uid to user 1000
2012-06-03 10:51:51,836 WARN Included extra file "/etc/supervisor/conf.d/uwsgi.c
onf" during parsing
2012-06-03 10:51:51,836 WARN Included extra file "/home/dotcloud/current/supervi
sord.conf" during parsing
2012-06-03 10:51:51,938 INFO RPC interface 'supervisor' initialized
2012-06-03 10:51:51,938 WARN cElementTree not installed, using slower XML parser
for XML-RPC
2012-06-03 10:51:51,938 CRIT Server 'unix_http_server' running without any HTTP
authentication checking
2012-06-03 10:51:51,946 INFO daemonizing the supervisord process
2012-06-03 10:51:51,947 INFO supervisord started with pid 144
2012-06-03 10:51:53,128 INFO spawned: 'celerycam' with pid 159
2012-06-03 10:51:53,133 INFO spawned: 'apnsd' with pid 161
2012-06-03 10:51:53,148 INFO spawned: 'djcelery' with pid 164
2012-06-03 10:51:53,168 INFO spawned: 'uwsgi' with pid 167
2012-06-03 10:51:53,245 INFO exited: djcelery (exit status 1; not expected)
2012-06-03 10:51:53,247 INFO exited: celerycam (exit status 1; not expected)
2012-06-03 10:51:54,698 INFO spawned: 'celerycam' with pid 176
2012-06-03 10:51:54,698 INFO success: apnsd entered RUNNING state, process has s
tayed up for > than 1 seconds (startsecs)
2012-06-03 10:51:54,705 INFO spawned: 'djcelery' with pid 177
2012-06-03 10:51:54,706 INFO success: uwsgi entered RUNNING state, process has s
tayed up for > than 1 seconds (startsecs)
2012-06-03 10:51:54,731 INFO exited: djcelery (exit status 1; not expected)
2012-06-03 10:51:54,754 INFO exited: celerycam (exit status 1; not expected)
2012-06-03 10:51:56,760 INFO spawned: 'celerycam' with pid 178
2012-06-03 10:51:56,765 INFO spawned: 'djcelery' with pid 179
2012-06-03 10:51:56,790 INFO exited: celerycam (exit status 1; not expected)
2012-06-03 10:51:56,791 INFO exited: djcelery (exit status 1; not expected)
2012-06-03 10:51:59,798 INFO spawned: 'celerycam' with pid 180
2012-06-03 10:52:00,538 INFO spawned: 'djcelery' with pid 181
2012-06-03 10:52:00,565 INFO exited: celerycam (exit status 1; not expected)
2012-06-03 10:52:00,571 INFO gave up: celerycam entered FATAL state, too many st
art retries too quickly
2012-06-03 10:52:00,573 INFO exited: djcelery (exit status 1; not expected)
2012-06-03 10:52:01,575 INFO gave up: djcelery entered FATAL state, too many sta
rt retries too quickly
dotcloud#hack-default-www-0:/var/log/supervisor$
The djerror log:
dotcloud#hack-default-www-0:/var/log/supervisor$ more djcelery_error.log
Traceback (most recent call last):
File "hack/manage.py", line 2, in
from django.core.management import execute_manager
ImportError: No module named django.core.management
Traceback (most recent call last):
File "hack/manage.py", line 2, in
from django.core.management import execute_manager
ImportError: No module named django.core.management
Traceback (most recent call last):
File "hack/manage.py", line 2, in
from django.core.management import execute_manager
ImportError: No module named django.core.management
Traceback (most recent call last):
File "hack/manage.py", line 2, in
from django.core.management import execute_manager
ImportError: No module named django.core.management
dotcloud#hack-default-www-0:/var/log/supervisor$
The statusctrl shows that some processes are running, but the pids are different. Also, the celery functionality seems to be working ok. Messages are processed, and I can see the messages being processed in the django admin interface (dj celery cam is running).
# supervisorctl status
apnsd RUNNING pid 225, uptime 0:00:44
celerycam RUNNING pid 224, uptime 0:00:44
djcelery RUNNING pid 226, uptime 0:00:44
Supervisord.conf file:
[program:djcelery]
directory = /home/dotcloud/current/
command = python hack/manage.py celeryd -E -l info -c 2
stderr_logfile = /var/log/supervisor/%(program_name)s_error.log
stdout_logfile = /var/log/supervisor/%(program_name)s.log
[program:celerycam]
directory = /home/dotcloud/current/
command = python hack/manage.py celerycam
stderr_logfile = /var/log/supervisor/%(program_name)s_error.log
stdout_logfile = /var/log/supervisor/%(program_name)s.log
http://jefurii.cafejosti.net/blog/2011/01/26/celery-in-virtualenv-with-supervisord/ says that the problem may be that the python being used is incorrect, so I've explicitly specified the python in the supervisord file. It now works, but it doesn't explain what I'm seeing above and why I've had to change my configuration when it was working fine last week.
Also, not all of the pids are lining up:
2012-06-03 11:19:03,045 CRIT Server 'unix_http_server' running without any HTTP
authentication checking
2012-06-03 11:19:03,051 INFO daemonizing the supervisord process
2012-06-03 11:19:03,052 INFO supervisord started with pid 144
2012-06-03 11:19:04,061 INFO spawned: 'celerycam' with pid 151
2012-06-03 11:19:04,066 INFO spawned: 'apnsd' with pid 153
2012-06-03 11:19:04,085 INFO spawned: 'djcelery' with pid 155
2012-06-03 11:19:04,104 INFO spawned: 'uwsgi' with pid 156
2012-06-03 11:19:05,271 INFO success: celerycam entered RUNNING state, process h
as stayed up for > than 1 seconds (startsecs)
2012-06-03 11:19:05,271 INFO success: apnsd entered RUNNING state, process has s
tayed up for > than 1 seconds (startsecs)
2012-06-03 11:19:05,271 INFO success: djcelery entered RUNNING state, process ha
s stayed up for > than 1 seconds (startsecs)
2012-06-03 11:19:05,271 INFO success: uwsgi entered RUNNING state, process has s
tayed up for > than 1 seconds (startsecs)
the status shows that the celery cam pids aren't lining up:
# supervisorctl status
apnsd RUNNING pid 153, uptime 0:06:17
celerycam RUNNING pid 150, uptime 0:06:17
djcelery RUNNING pid 155, uptime 0:06:17
My first guess is that your using the wrong python binary (system python, instead of virtualenv python), and it is causing this error (below) because that system python binary doesn't have that package installed.
Traceback (most recent call last):
File "hack/manage.py", line 2, in
from django.core.management import execute_manager
ImportError: No module named django.core.management
You should change your supervisord.conf to the following to make sure you are pointing to the correct python version.
[program:djcelery]
directory = /home/dotcloud/current/
command = /home/dotcloud/env/bin/python hack/manage.py celeryd -E -l info -c 2
stderr_logfile = /var/log/supervisor/%(program_name)s_error.log
stdout_logfile = /var/log/supervisor/%(program_name)s.log
[program:celerycam]
directory = /home/dotcloud/current/
command = /home/dotcloud/env/bin/python hack/manage.py celerycam
stderr_logfile = /var/log/supervisor/%(program_name)s_error.log
stdout_logfile = /var/log/supervisor/%(program_name)s.log
The python path went fromt python to /home/dotcloud/env/bin/python.
I'm not sure why supervisor is saying it is running when it is not, but hopefully this one little change will help clear up your errors, and get everything back to working again.