How to change the Console Ouptut Mode using SetConsoleMode in powershell? - powershell

Question

I am trying to change the Windows Console Mode for output (CONOUT$) using the Windows API and SetConsoleMode calls. So I modified a PowerShell script, based on ConinMode.ps1 (and which works for input), to do this. Reading works fine with both the ConoutMode.exe and my script, and returns:
# .\ConoutMode.exe
mode: 0x3
ENABLE_PROCESSED_OUTPUT 0x0001 ON
ENABLE_WRAP_AT_EOL_OUTPUT 0x0002 ON
ENABLE_VIRTUAL_TERMINAL_PROCESSING 0x0004 off <====
DISABLE_NEWLINE_AUTO_RETURN 0x0008 off
ENABLE_LVB_GRID_WORLDWIDE 0x0010 off
# .\ConoutMode.ps1
OK! GetConsoleWindow handle : 0x1F06D6
Console Input Mode (CONIN) : 0x21f
Console Output Mode (CONOUT) : 0x3
However, both my script and the exe fails in writing the mode. (Possibly because it thinks that my output handle is not pointing to a console?)
In PowerShell:
# .\ConoutMode.exe set 0x000f
error: SetConsoleMode failed (is stdout a console?)
# .\ConoutMode.ps1 -Mode 7
OK! GetConsoleWindow handle : 0x1F06D6
old (out) mode: 0x3
SetConsoleMode (out) failed (is stdout a console?)
At C:\cygwin64\home\emix\win_esc_test\ConoutMode.ps1:112 char:9
+ throw "SetConsoleMode (out) failed (is stdout a console?)"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (SetConsoleMode ...out a console?):String) [], RuntimeException
+ FullyQualifiedErrorId : SetConsoleMode (out) failed (is stdout a console?)
Here is my ConoutMode.ps1 script in it's entirety:
param (
[int] $Mode
)
$signature = #'
[DllImport("kernel32.dll", SetLastError=true)]
public static extern IntPtr GetConsoleWindow();
[DllImport("kernel32.dll", SetLastError = true)]
public static extern IntPtr GetStdHandle(int nStdHandle);
[DllImport("kernel32.dll", SetLastError = true)]
public static extern uint GetConsoleMode(IntPtr hConsoleHandle, out uint lpMode);
[DllImport("kernel32.dll", SetLastError = true)]
public static extern uint SetConsoleMode(IntPtr hConsoleHandle, uint dwMode);
public const int STD_INPUT_HANDLE = -10;
public const int STD_OUTPUT_HANDLE = -11;
public const int ENABLE_PROCESSED_OUTPUT = 0x0001;
public const int ENABLE_WRAP_AT_EOL_OUTPUT = 0x0002;
public const int ENABLE_VIRTUAL_TERMINAL_PROCESSING = 0x0004;
public const int DISABLE_NEWLINE_AUTO_RETURN = 0x0008;
public const int ENABLE_LVB_GRID_WORLDWIDE = 0x0010;
'#
#----------------------------------------------------------
$WinAPI = Add-Type -MemberDefinition $signature -Name WinAPI -Namespace ConoutMode -PassThru
#$WinAPI = Add-Type -MemberDefinition $signature -Name WinAPI -Namespace IdentifyConsoleWindow -PassThru
$hwnd = $WinAPI::GetConsoleWindow()
if ($hwnd -eq 0) {
throw "Error: GetConsoleWindow returned NULL."
}
echo "OK! GetConsoleWindow handle : 0x$($hwnd.ToString("X"))"
function GetConIn {
$ret = $WinAPI::GetStdHandle($WinAPI::STD_INPUT_HANDLE)
if ($ret -eq -1) {
throw "Error: cannot get stdin handle"
}
return $ret
}
function GetConOut {
$ret = $WinAPI::GetStdHandle($WinAPI::STD_OUTPUT_HANDLE)
if ($ret -eq -1) {
throw "Error: cannot get stdout handle"
}
return $ret
}
function GetConInMode { #GetCOnsoleMode
$conin = GetConIn
$mode = 0
$ret = $WinAPI::GetConsoleMode($conin, [ref]$mode)
if ($ret -eq 0) {
throw "GetConsoleMode (in) failed (is stdin a console?)"
}
return $mode
}
function GetConOutMode {
$conout = GetConOut
$mode = 0
$ret = $WinAPI::GetConsoleMode($conout, [ref]$mode)
if ($ret -eq 0) {
throw "GetConsoleMode (out) failed (is stdout a console?)"
}
return $mode
}
function SetConInMode($mode) {
$conin = GetConIn
$ret = $WinAPI::SetConsoleMode($conin, $mode)
if ($ret -eq 0) {
throw "SetConsoleMode (in) failed (is stdin a console?)"
}
}
function SetConOutMode($mode) {
#$conout = GetConOut
# Different tack:
$conout = $hwnd
$ret = $WinAPI::SetConsoleMode($conout, $mode)
if ($ret -eq 0) {
throw "SetConsoleMode (out) failed (is stdout a console?)"
}
}
#----------------------------------------------------------
# MAIN
#----------------------------------------------------------
$oldInMode = GetConInMode
$oldOutMode = GetConOutMode
$newMode = $oldOutMode
$doit = $false
if ($PSBoundParameters.ContainsKey("Mode")) {
$newMode = $Mode
$doit = $true
}
if ($doit) {
echo "old (out) mode: 0x$($oldOutMode.ToString("x"))"
SetConOutMode $newMode
$newMode = GetConOutMode
echo "new (out) mode: 0x$($newMode.ToString("x"))"
} else {
echo "Console Input Mode (CONIN) : 0x$($oldInMode.ToString("x"))"
echo "Console Output Mode (CONOUT) : 0x$($oldOutMode.ToString("x"))"
}
So at the end of the day, I want to enable ENABLE_VIRTUAL_TERMINAL_PROCESSING for the Console.
The system I am using for this is:
---------------------------------------------------------
PowerShell Version : 6.1.1
OS Name : Microsoft Windows 8.1 (64-bit)
OS Version : 6.3.9600 [2018-11-16 00:50:01]
OS BuildLabEx : 9600.19179
OS HAL : 6.3.9600.18969
OS Kernel : 6.3.9600.18217
OS UBR : 19182
-------------------------------------------------------
with Privilege : Administrator
-------------------------------------------------------
ExecutionPolicy :
MachinePolicy : Undefined
UserPolicy : Undefined
Process : Undefined
CurrentUser : Undefined
LocalMachine : Bypass
Console Settings:
Type : ConsoleHost
OutputEncoding : Unicode (UTF-8)
Color Capability : 23
Registry VT Level : 1
CodePage (input) : 437
CodePage (output) : 437
I have also enabled the registry item: HKCU:\Console\VirtualTerminalLevel, as instructed elsewhere.
Q: How can I enable this bit/flag using PowerShell?
Some things I can think of that may be wrong:
I surely have the wrong understanding of how this works...
I might have the wrong permissions to write to console? (How to set?)
I might not write to the correct output buffer? (How to find?)
Note that setting the output mode of one screen buffer does not affect the output mode of other screen buffers.
When executing a script, perhaps a new output buffer is created? (What to do?)
I might have to create a new buffer?
I might have to create a new Console?
Related Questions and Links:
ENABLE_VIRTUAL_TERMINAL_PROCESSING and DISABLE_NEWLINE_AUTO_RETURN failing
MSDN: Inside the Windows Console
MSDN: Introducing the Windows Pseudo Console - ConPty

Answer 1

The code is using "GetStdHandle" to get stdin or stdout. And those might be console handles - but it's dependent on how the process is created. If you're launched from a console, and the stdin/stdout aren't redirected, then you'd probably get a console handle. But it's not guaranteed.
Rather than asking for stdin/stdout, just open the console handles directly - you can "CreateFile" for CONIN$ and CONOUT$. The documentation for CreateFile has a whole section on Consoles.
Update: I made a Gist to demonstrate how to use CreateFile to open CONIN$ and CONOUT$ from PowerShell.

Answer 2

I have been experimenting with this myself for a few days and have come to realize a few things. Apparently PowerShell enables Virtual Terminal Processing when it starts up. It disables it again, however, when it launches an executable and then reenables it when the executable exits. This means that whatever executable needs Virtual Terminal Processing must enable it itself or be called through a wrapper program that enables it and then pipes stdin/stdout/stderr through to the terminal. As for how would even go about writing such a program, I don't know.