C# and Powershell Runspaces creating massive unmanaged memory leak - powershell

I have a strange issue with a windows service that I'm developing in .Net 4. After an hour of running the service the memory usage is at 274MB.
I have tried to find the cause of the leak by removing the functions one by one to find out where the leak occurs and at the moment because of this all the service does is run a PowerShell command Get-WBSummary every 10 seconds and capture the output to a string.
I am trying to track down what is hogging all the memory using Red Gate .Net Profiler 10. When I look at the biggest hog of memory after an hour the only thing that sticks out is
Namespace Class name Live size (bytes) Unmanaged size (bytes)
System.Reflection RuntimeAssembly 1,400 1,462,189
Which cant be right as that comes out to 1.462189mb.
Here is the code I am using which runs every time the service timer hits 10 seconds (For testing I set it to a low value)
var Config = RunspaceConfiguration.Create();
PSSnapInException Warning;
if (Environment.OSVersion.Version.Major == 6 && Environment.OSVersion.Version.Minor == 1)
{
Config.AddPSSnapIn("Windows.ServerBackup", out Warning);
}
using (var Runspace = RunspaceFactory.CreateRunspace(Config))
{
Runspace.Open();
PowerShell Ps;
using (Ps = PowerShell.Create())
{
Ps.Runspace = Runspace;
Ps.AddCommand("Get-WBSummary");
var Output = Ps.Invoke();
var Sb = new StringBuilder();
foreach (var PsObject in Output)
{
foreach (var PsPropertyInfo in PsObject.Properties)
{
Sb.AppendLine(PsPropertyInfo.Name + " " + PsPropertyInfo.Value);
}
}
PowershellOutput = Sb.ToString();
}
Runspace.Close();
Runspace.Dispose();
Ps.Dispose();
Config = null;
PowerShellSession.Stop();
PowerShellSession.Dispose();
GC.Collect();
Can anyone suggest what I might be doing wrong?
Edit:
The problem here has nothing to do with the C# code but the way I was using PowerShell. The solution is instead of using one PowerShell runspace and entering the same command again and again in it (Which causes the PowerShell memory to increase to infinity) but to create a new PowerShell process every time and then close it when you are done
using (PowerShellProcessInstance ExternalPowerShellProccess = new PowerShellProcessInstance())
{
ExternalPowerShellProccess.Process.Start();
var Config = RunspaceConfiguration.Create();
if (Environment.OSVersion.Version.Major == 6 && Environment.OSVersion.Version.Minor == 1)
{
Config.AddPSSnapIn("Windows.ServerBackup", out PSSnapInException Warning);
}
Runspace Runspace = RunspaceFactory.CreateOutOfProcessRunspace(TypeTable.LoadDefaultTypeFiles(), ExternalPowerShellProccess);
Runspace.Open();
using (PowerShell Ps = PowerShell.Create())
{
Ps.Runspace = Runspace;
Ps.AddCommand("Get-WBSummary");
var Output = Ps.Invoke();
var Sb = new StringBuilder();
foreach (var PsObject in Output)
{
foreach (var PsPropertyInfo in PsObject.Properties)
{
Sb.AppendLine(PsPropertyInfo.Name + " " + PsPropertyInfo.Value);
}
}
PowershellOutput = Sb.ToString();
Ps.Dispose();
Runspace.Close();
Runspace.Dispose();
}

Related

Executing scripts with Powershell via C#

I need to execute the following script:
Get-MailboxDatabase -Status | select ServerName,Name,DatabaseSize
I tried a few solutions with Powershell and Command classes, but they doesn't work.
Error that I received:
Value parameters cannot be null.
I think this will do what you're looking for:
private string RunLocalExchangePowerShell(string script)
{
// create the runspace and load the snapin
RunspaceConfiguration rsConfig = RunspaceConfiguration.Create();
PSSnapInException snapInException = null;
Runspace runSpace = RunspaceFactory.CreateRunspace(rsConfig);
runSpace.Open();
rsConfig.AddPSSnapIn("Microsoft.Exchange.Management.PowerShell.E2010", out snapInException);
Pipeline pipeLine = runSpace.CreatePipeline();
// load the script and convert the output to a string
pipeLine.Commands.AddScript(script);
pipeLine.Commands.Add("out-string");
// get the results
Collection<PSObject> results;
results = pipeLine.Invoke();
// loop through the results and build the output
StringBuilder sb = new StringBuilder();
foreach (PSObject obj in results)
{
sb.AppendLine(obj.ToString());
}
// close the pipeline and runspace
pipeLine.Dispose();
runSpace.Close();
return sb.ToString();
}
Example usage:
Console.WriteLine(prog.RunLocalExchangePowerShell("Get-MailboxDatabase -Status | select ServerName,Name,DatabaseSize"));

Log Fiddler Requests to Database Real-time

Is there any way to log all requests ongoing to a database or can you only log snapshots to a database?
The following example relies upon OLEDB 4.0 which is not available for 64bit processes. You can either select another data provider (e.g. SQLServer) or you can force Fiddler to run in 32bit mode.
Add the following to the Rules file to create a new menu item.
// Log the currently selected sessions in the list to a database.
// Note: The DB must already exist and you must have permissions to write to it.
public static ToolsAction("Log Selected Sessions")
function DoLogSessions(oSessions: Fiddler.Session[]){
if (null == oSessions || oSessions.Length < 1){
MessageBox.Show("Please select some sessions first!");
return;
}
var strMDB = "C:\\log.mdb";
var cnn = null;
var sdr = null;
var cmd = null;
try
{
cnn = new OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + strMDB);
cnn.Open();
cmd = new OleDbCommand();
cmd.Connection = cnn;
for (var x = 0; x < oSessions.Length; x++){
var strSQL = "INSERT into tblSessions ([ResponseCode],[URL]) Values (" +
oSessions[x].responseCode + ", '" + oSessions[x].url + "')";
cmd.CommandText = strSQL;
cmd.ExecuteNonQuery();
}
}
catch (ex){
MessageBox.Show(ex);
}
finally
{
if (cnn != null ){
cnn.Close();
}
}
}
Note: To use the Database Objects in Fiddler 2.3.9 and below, you'll need to add system.data to the References list inside Tools | Fiddler Options | Extensions | Scripting. In 2.3.9.1 and later, this reference will occur automatically.
Then, list the new import at the top of your rules script:
import System.Data.OleDb;
see FiddlerScript CookBook

Programmatically move files after virus scan

Is it possible to move files programmatically based on virus scan status?
What I want to do is have a set of folders:
Incoming
Scanned
Scanned/Clean
Scanned/Infected
Not Scanned
Files would be dropped into the Incoming folder. At that point, I would like to kick off the antivirus and scan the files in the Incoming folder. Once complete, the files would then need to be moved to the appropriate folder, either Clean or Infected. If, for whatever reason, the file could not be scanned or had trouble scanning, it would be moved to the Not Scanned folder.
I was hoping there would be a way to script this out. Has anyone ever done anything like this before?
public void Scan()
{
string[] uploadPath = Directory.GetFiles(ConfigurationManager.AppSettings["UploadPath"]);
foreach(string filePath in uploadPath)
{
string fileName = Path.GetFileName(filePath);
string cleanPath = Path.Combine(ConfigurationManager.AppSettings["CleanPath"], fileName);
try
{
Process AV = new Process();
AV.StartInfo.UseShellExecute = false;
AV.StartInfo.RedirectStandardOutput = true;
AV.StartInfo.FileName = ConfigurationManager.AppSettings["VSApp"];
AV.StartInfo.Arguments = " -Scan -ScanType 3 -file " + ConfigurationManager.AppSettings["UploadPath"] + " -DisableRemediation";
AV.Start();
string output = AV.StandardOutput.ReadToEnd();
AV.WaitForExit();
if (AV.ExitCode == 0)
{
File.Move(filePath, cleanPath);
}
else if (AV.ExitCode == 2)
{
using (TextWriter tw = new StreamWriter(ConfigurationManager.AppSettings["FailedPath"] + fileName + ".txt"))
{
tw.WriteLine("2");
tw.Close();
}
using (TextWriter tw1 = new StreamWriter(ConfigurationManager.AppSettings["FailedFiles"] + fileName + ".txt"))
{
tw1.WriteLine(AV.StandardOutput);
tw1.Close();
}
File.Delete(filePath);
}
AV.Close();
}
catch (Exception ex)
{
if (ex.ToString().Contains("Could not find file"))
{
string failedFile = ConfigurationManager.AppSettings["FailedPath"] + fileName + ".txt";
string failedFileDesc = ConfigurationManager.AppSettings["FailedPath"] + fileName + "_ErrorDesc" + ".txt";
using (TextWriter tw = new StreamWriter(failedFile))
{
tw.WriteLine("2");
tw.Close();
}
using (TextWriter tw1 = new StreamWriter(failedFileDesc))
{
tw1.WriteLine(ex.ToString());
tw1.Close();
}
}
else
{
Thread.Sleep(2000);
if (runCounter == 0)
{
Scan();
}
runCounter++;
string errorFile = ConfigurationManager.AppSettings["ProcessErrorPath"] + fileName + ".txt";
using (TextWriter tw = new StreamWriter(errorFile))
{
tw.WriteLine(ex.ToString());
tw.Close();
}
}
}
}
}
I created this as a Windows Service. My OnStart method creates my FileSystemWatcher to watch the Upload Path. For On Created, I have a method that runs my Scan method and creates my counter and sets it to 0. My On Error event just logs. I had an issue where the FileSystemWatcher was trying to open the file before it had been uploaded, hence why I added the sleep.
Finally, I am using Microsoft Forefront's command line scanner. File path: C:\Program Files\Microsoft Security Client\mpcmdrun.exe.
Let me know if any questions.

Microsoft Robotics and Sql

I have an issue implementing CCR with SQL. It seems that when I step through my code the updates and inserts I am trying to execute work great. But when I run through my interface without any breakpoints, it seems to be working and it shows the inserts, updates, but at the end of the run, nothing got updated to the database.
I proceeded to add a pause to my code every time I pull anew thread from my pool and it works... but that defeats the purpose of async coding right? I want my interface to be faster, not slow it down...
Any suggestions... here is part of my code:
I use two helper classes to set my ports and get a response back...
/// <summary>
/// Gets the Reader, requires connection to be managed
/// </summary>
public static PortSet<Int32, Exception> GetReader(SqlCommand sqlCommand)
{
Port<Int32> portResponse = null;
Port<Exception> portException = null;
GetReaderResponse(sqlCommand, ref portResponse, ref portException);
return new PortSet<Int32, Exception>(portResponse, portException);
}
// Wrapper for SqlCommand's GetResponse
public static void GetReaderResponse(SqlCommand sqlCom,
ref Port<Int32> portResponse, ref Port<Exception> portException)
{
EnsurePortsExist(ref portResponse, ref portException);
sqlCom.BeginExecuteNonQuery(ApmResultToCcrResultFactory.Create(
portResponse, portException,
delegate(IAsyncResult ar) { return sqlCom.EndExecuteNonQuery(ar); }), null);
}
then I do something like this to queue up my calls...
DispatcherQueue queue = CreateDispatcher();
String[] commands = new String[2];
Int32 result = 0;
commands[0] = "exec someupdateStoredProcedure";
commands[1] = "exec someInsertStoredProcedure '" + Settings.Default.RunDate.ToString() + "'";
for (Int32 i = 0; i < commands.Length; i++)
{
using (SqlConnection connSP = new SqlConnection(Settings.Default.nbfConn + ";MultipleActiveResultSets=true;Async=true"))
using (SqlCommand cmdSP = new SqlCommand())
{
connSP.Open();
cmdSP.Connection = connSP;
cmdSP.CommandTimeout = 150;
cmdSP.CommandText = "set arithabort on; " + commands[i];
Arbiter.Activate(queue, Arbiter.Choice(ApmToCcrAdapters.GetReader(cmdSP),
delegate(Int32 reader) { result = reader; },
delegate(Exception e) { result = 0; throw new Exception(e.Message); }));
}
}
where ApmToCcrAdapters is the class name where my helper methods are...
The problem is when I pause my code right after the call to Arbiter.Activate and I check my database, everything looks fine... if I get rid of the pause ad run my code through, nothing happens to the database, and no exceptions are thrown either...
The problem here is that you are calling Arbiter.Activate in the scope of your two using blocks. Don't forget that the CCR task you create is queued and the current thread continues... right past the scope of the using blocks. You've created a race condition, because the Choice must execute before connSP and cmdSP are disposed and that's only going to happen when you're interfering with the thread timings, as you have observed when debugging.
If instead you were to deal with disposal manually in the handler delegates for the Choice, this problem would no longer occur, however this makes for brittle code where it's easy to overlook disposal.
I'd recommend implementing the CCR iterator pattern and collecting results with a MulitpleItemReceive so that you can keep your using statements. It makes for cleaner code. Off the top of my head it would look something like this:
private IEnumerator<ITask> QueryIterator(
string command,
PortSet<Int32,Exception> resultPort)
{
using (SqlConnection connSP =
new SqlConnection(Settings.Default.nbfConn
+ ";MultipleActiveResultSets=true;Async=true"))
using (SqlCommand cmdSP = new SqlCommand())
{
Int32 result = 0;
connSP.Open();
cmdSP.Connection = connSP;
cmdSP.CommandTimeout = 150;
cmdSP.CommandText = "set arithabort on; " + commands[i];
yield return Arbiter.Choice(ApmToCcrAdapters.GetReader(cmdSP),
delegate(Int32 reader) { resultPort.Post(reader); },
delegate(Exception e) { resultPort.Post(e); });
}
}
and you could use it something like this:
var resultPort=new PortSet<Int32,Exception>();
foreach(var command in commands)
{
Arbiter.Activate(queue,
Arbiter.FromIteratorHandler(()=>QueryIterator(command,resultPort))
);
}
Arbiter.Activate(queue,
Arbiter.MultipleItemReceive(
resultPort,
commands.Count(),
(results,exceptions)=>{
//everything is done and you've got 2
//collections here, results and exceptions
//to process as you want
}
)
);

How to troubleshoot a 'System.Management.Automation.CmdletInvocationException'

Does anyone know how best to determine the specific underlying cause of this exception?
Consider a WCF service that is supposed to use Powershell 2.0 remoting to execute MSBuild on remote machines. In both cases the scripting environments are being called in-process (via C# for Powershell and via Powershell for MSBuild), rather than 'shelling-out' - this was a specific design decision to avoid command-line hell as well as to enable passing actual objects into the Powershell script.
An abridged version of the Powershell script that calls MSBuild is shown below:
function Run-MSBuild
{
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Build.Engine")
$engine = New-Object Microsoft.Build.BuildEngine.Engine
$engine.BinPath = "C:\Windows\Microsoft.NET\Framework\v3.5"
$project = New-Object Microsoft.Build.BuildEngine.Project($engine, "3.5")
$project.Load("deploy.targets")
$project.InitialTargets = "DoStuff"
# Process the input object
while ($input.MoveNext())
{
# Set MSBuild Properties & Item
}
# Optionally setup some loggers (have also tried it without any loggers)
$consoleLogger = New-Object Microsoft.Build.BuildEngine.ConsoleLogger
$engine.RegisterLogger($consoleLogger)
$fileLogger = New-Object Microsoft.Build.BuildEngine.FileLogger
$fileLogger.Parameters = "verbosity=diagnostic"
$engine.RegisterLogger($fileLogger)
# Run the build - this is the line that throws a CmdletInvocationException
$result = $project.Build()
$engine.Shutdown()
}
When running the above script from a PS command prompt it all works fine. However, as soon as the script is executed from C# it fails with the above exception.
The C# code being used to call Powershell is shown below (remoting functionality removed for simplicity's sake):
// Build the DTO object that will be passed to Powershell
dto = SetupDTO()
RunspaceConfiguration runspaceConfig = RunspaceConfiguration.Create();
using (Runspace runspace = RunspaceFactory.CreateRunspace(runspaceConfig))
{
runspace.Open();
IList errors;
using (var scriptInvoker = new RunspaceInvoke(runspace))
{
// The Powershell script lives in a file that gets compiled as an embedded resource
TextReader tr = new StreamReader(Assembly.GetExecutingAssembly().GetManifestResourceStream("MyScriptResource"));
string script = tr.ReadToEnd();
// Load the script into the Runspace
scriptInvoker.Invoke(script);
// Call the function defined in the script, passing the DTO as an input object
var psResults = scriptInvoker.Invoke("$input | Run-MSBuild", dto, out errors);
}
}
NOTE: The overload of the Invoke() method allows you to pass in an IEnumerable object and it takes care of instantiating an enumerator to in the Powershell variable '$input' - this then gets passed into the script via the pipeline. Here are some supporting links:
http://msdn.microsoft.com/en-us/library/ms569104(VS.85).aspx
http://knicksmith.blogspot.com/2007/03/managing-exchange-2007-recipients-with.html (jump to the 'Passing an Input Object to the Runspace' section)
Assuming that the issue was related to MSBuild outputting something that the Powershell runspace can't cope with, I have also tried the following variations to the second .Invoke() call:
var psResults = scriptInvoker.Invoke("$input | Run-MSBuild | Out-String", dto, out errors);
var psResults = scriptInvoker.Invoke("$input | Run-MSBuild | Out-Null", dto, out errors);
var psResults = scriptInvoker.Invoke("Run-MSBuild | Out-String");
var psResults = scriptInvoker.Invoke("Run-MSBuild | Out-String");
var psResults = scriptInvoker.Invoke("Run-MSBuild | Out-Null");
var psResults = scriptInvoker.Invoke("Run-MSBuild");
Note how the underlying issue still occurs irrespective of whether an input object is used.
I've also looked at using a custom PSHost (based on this sample: http://blogs.msdn.com/daiken/archive/2007/06/22/hosting-windows-powershell-sample-code.aspx), but during debugging I was unable to see any 'interesting' calls to it being made.
Do the great and the good of Stackoverflow have any insight that might save my sanity?
I can get the following code to work but I get a warning that MSBUILD engine wants to be run on a STA thread. Unfortunately the thread created by the PowerShell engine to execute the script is MTA. That said, my little sample works:
using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Linq;
using System.Management.Automation;
using System.Collections;
namespace RunspaceInvokeExp
{
class Program
{
static void Main()
{
var script = #"
function Run-MSBuild
{
[System.Reflection.Assembly]::LoadWithPartialName(""Microsoft.Build.Engine"")
$engine = New-Object Microsoft.Build.BuildEngine.Engine
$engine.BinPath = ""C:\Windows\Microsoft.NET\Framework\v3.5""
$project = New-Object Microsoft.Build.BuildEngine.Project($engine, ""3.5"")
$project.Load(""deploy.targets"")
$project.InitialTargets = ""DoStuff""
# Process the input object
while ($input.MoveNext())
{
# Set MSBuild Properties & Item
}
# Optionally setup some loggers (have also tried it without any loggers)
$consoleLogger = New-Object Microsoft.Build.BuildEngine.ConsoleLogger
$engine.RegisterLogger($consoleLogger)
$fileLogger = New-Object Microsoft.Build.BuildEngine.FileLogger
$fileLogger.Parameters = ""verbosity=diagnostic""
$engine.RegisterLogger($fileLogger)
# Run the build - this is the line that throws a CmdletInvocationException
$result = $project.Build()
$engine.Shutdown()
}
";
using (var invoker = new RunspaceInvoke())
{
invoker.Invoke(script);
IList errors;
Collection<PSObject> results = invoker.Invoke(#"$input | Run-MSBuild", new[] {0}, out errors);
Array.ForEach<PSObject>(results.ToArray(), Console.WriteLine);
}
}
}
}
Which line of your C# code fails? Also, can you post some of the specifics from the exception. You can work around the MTA thread issue by doing something like this:
using System;
using System.Collections;
using System.Collections.ObjectModel;
using System.Linq;
using System.Management.Automation;
using System.Management.Automation.Runspaces;
namespace RunspaceInvokeExp
{
class Program
{
[STAThread]
static void Main()
{
var script = #"
function Run-MSBuild
{
[System.Reflection.Assembly]::LoadWithPartialName(""Microsoft.Build.Engine"")
$engine = New-Object Microsoft.Build.BuildEngine.Engine
$engine.BinPath = ""C:\Windows\Microsoft.NET\Framework\v3.5""
$project = New-Object Microsoft.Build.BuildEngine.Project($engine, ""3.5"")
$project.Load(""deploy.targets"")
$project.InitialTargets = ""DoStuff""
# Process the input object
while ($input.MoveNext())
{
# Set MSBuild Properties & Item
}
# Optionally setup some loggers (have also tried it without any loggers)
$consoleLogger = New-Object Microsoft.Build.BuildEngine.ConsoleLogger
$engine.RegisterLogger($consoleLogger)
$fileLogger = New-Object Microsoft.Build.BuildEngine.FileLogger
$fileLogger.Parameters = ""verbosity=diagnostic""
$engine.RegisterLogger($fileLogger)
# Run the build - this is the line that throws a CmdletInvocationException
$result = $project.Build()
$engine.Shutdown()
}
Run-MSBuild
";
Runspace runspace = RunspaceFactory.CreateRunspace();
Runspace.DefaultRunspace = runspace;
runspace.Open();
EngineIntrinsics engine = runspace.SessionStateProxy.
GetVariable("ExecutionContext") as EngineIntrinsics;
ScriptBlock scriptblock =
engine.InvokeCommand.NewScriptBlock(script);
Collection<PSObject> results = scriptblock.Invoke(new[] { 0 });
Array.ForEach<PSObject>(results.ToArray(), Console.WriteLine);
runspace.Close(); // Really should be in a finally block
}
}
}