Output to command-line if started from command line - command-line

Question

I'm writing an application that can be started either as a standard WinForms app or in unattended mode from the command-line. The application was built using the VS 2k5 standard WinForms template.
When the application is executed from the command-line, I want it to output information that can be captured by the script executing the application. When I do this directly from Console.WriteLine(), the output does not appear, although it can be captured by piping to a file.
On the other hand, I can force the application to pop up a second console by doing a P/Invoke on AllocConsole() from kernel32. This is not what I want, though. I want the output to appear in the same window the application was called from.
This is the salient code that allows me to pop up a console from the command line:
<STAThread()> Public Shared Sub Main()
If My.Application.CommandLineArgs.Count = 0 Then
Dim frm As New ISECMMParamUtilForm()
frm.ShowDialog()
Else
Try
ConsoleControl.AllocConsole()
Dim exMan As New UnattendedExecutionManager(ConvertArgs())
IsInConsoleMode = True
OutputMessage("Application started.")
If Not exMan.SetSettings() Then
OutputMessage("Execution failed.")
End If
Catch ex As Exception
Console.WriteLine(ex.ToString())
Finally
ConsoleControl.FreeConsole()
End Try
End If
End Sub
Public Shared Sub OutputMessage(ByVal msg As String, Optional ByVal isError As Boolean = False)
Trace.WriteLine(msg)
If IsInConsoleMode Then
Console.WriteLine(msg)
End If
If isError Then
EventLog.WriteEntry("ISE CMM Param Util", msg, EventLogEntryType.Error)
Else
EventLog.WriteEntry("ISE CMM Param Util", msg, EventLogEntryType.Information)
End If
End Sub

Answer 1

Raymond Chen recently posted (a month after the question was posted here on SO) a short article about this:
How do I write a program that can be run either as a console or a GUI application?
You can't, but you can try to fake it.
Each PE application contains a field
in its header that specifies which
subsystem it was designed to run
under. You can say
IMAGE_SUBSYSTEM_WINDOWS_GUI to mark
yourself as a Windows GUI application,
or you can say
IMAGE_SUBSYSTEM_WINDOWS_CUI to say
that you are a console application. If
you are GUI application, then the
program will run without a console.
The subsystem determines how the
kernel prepares the execution
environment for the program. If the
program is marked as running in the
console subsystem, then the kernel
will connect the program's console to
the console of its parent, creating a
new console if the parent doesn't have
a console. (This is an incomplete
description, but the details aren't
relevant to the discussion.) On the
other hand, if the program is marked
as running as a GUI application, then
the kernel will run the program
without any console at all.
In that article he points to another by Junfeng Zhang that discusses how a couple of programs (Visual Studio and ildasm) implement this behavior:
How to make an application as both GUI and Console application?
In VisualStudio case, there are actually two binaries: devenv.com and devenv.exe. Devenv.com is a Console app. Devenv.exe is a GUI app. When you type devenv, because of the Win32 probing rule, devenv.com is executed. If there is no input, devenv.com launches devenv.exe, and exits itself. If there are inputs, devenv.com handles them as normal Console app.
In ildasm case, there is only one binary: ildasm.exe. It is first compiled as a GUI application. Later editbin.exe is used to mark it as console subsystem. In its main method it determines if it needs to be run as console mode or GUI mode. If need to run as GUI mode, it relaunches itself as a GUI app.
In the comments to Raymond Chen's article, laonianren has this to add to Junfeng Zhang's brief description of how Visual Studio works:
devenv.com is a general purpose console-mode stub application. When it runs it creates three pipes to redirect the console's stdin, stdout and stderr. It then finds its own name (usually devenv.com), replaces the ".com" with ".exe" and launches the new app (i.e. devenv.exe) using the read end of the stdin pipe and the write ends of the stdout and stderr pipes as the standard handles. Then it just sits and waits for devenv.exe to exit and copies data between the console and the pipes.
Thus even though devenv.exe is a gui app it can read and write the "parent" console using its standard handles.
And you could use devenv.com yourself for myapp.exe by renaming it to myapp.com. But you can't in practise because it belongs to MS.

Answer 2

Update 1:
As said in Michael Burr answer, Raymond Chen recently posted a short article about this. I am happy to see that my guess was not totally wrong.
Update 0:
Disclaimer: This "answer" is mostly speculation. I post it only because enough time has passed to establish that not many people have the answer to what look like a fundamental question.
I think that the "decision" if the application is gui or console is made at compile time and not at runtime. So if you compile your application as gui application, even if you don't display the gui, its still a gui application and doesn't have console. If you choose to compile it as console application then at minimum you will have a console windows flashing before moving to gui "mode". And I don't know if it is possible in managed code.
The problem is fundamental, I think, Because a console application has to take "control" of the calling console application. And it has to do so before the code of the child application is running.

Answer 3

If you want to check if your app is started from the command line in .NET, you can use Console.GetCursorPosition().
The reason that this works is that when you start it from the command line, the cursor moves away from the initial point ((0, 0)) because you typed something in the terminal (the name of the app).
You can do this with an equality check (code in C#):
class Program
{
public static void Main
{
if (Console.GetCursorPosition() == (0, 0))
{
//something GUI
}
else
{
//something not GUI
}
}
}
Note: You must set the output type to Console Application as other output types will make Console.GetCursorPosition() throw an exception.