Remotely running Get-EventLog wipes actual event log - powershell

I wrote a script to remotely fetch event logs in PowerShell, but I'm not a fan of the way the script makes its own event log entries.
Specifically this script grabs several types of event IDs including the logon/logoff events. Recently I was asked to run the script to fetch events for another user, and had to have this data fetched in a few hours. Normally I start the script and let it run for the majority of the day (because there is usually a LOT of data here), but this time to speed up the process, I spun up 4 instances of the script to fetch this data faster than usual. Each instance of the script was looking at a different time frame so that all 4 scripts combined were fetching in the time frame I had been asked for.
Over the course of 3 hours or so, I had over a million different login attempts for my user ID on this remote computer. I had so many logins that I ended up overwriting the event log data I was originally asked to fetch.
Lessons learned, I'm now researching how to make this faster, more efficient, and more reliable.
Here's the heart of my code, pretty plain and simple, and works for the most part.
$output = Get-EventLog `
-instanceID 4672,4647,4634,4624,4625,4648,4675,4800,4801,4802,4803 `
-logName Security `
-after (Get-Date $inStartDate) `
-before (Get-Date $inEndDate).addDays(1) `
-message ("*" + $InUserID + "*") `
-computerName $inPCID
I guess there are several questions that I haven't been able to figure out in my research thus far. Why would Get-EventLog need to make so many connections? Is it because the connection kept dropping or something?
What would be the fastest way to fetch this data - Using the native Get-EventLog command by specifying a -ComputerName, or should I be using something like Enter-PSSession or Invoke-Command.
Will Enter-PSSession and Invoke-Command both have the same problem I'm having with Get-EventLog?
I'd like to avoid using Enter-PSSession and Invoke-Command for the simple fact that I can't guarantee all machines in the company will have remote-execution enabled.

So, the problem was that Get-EventLog was ultimately wiping the remote event logs I was trying to fetch. Not sure why, but Get-EventLog made over a million "connections" that appeared as logon/logoff events, thus overwriting my logs.
Per #Richard's comment, I did a bit of research, and decided to test using Get-WinEvent, the updated and more powerful replacement for Get-EventLog. After testing around with this for a week, the worst case scenario that I ran into was my script creating 4 logon/logoff events. This is completely acceptable and much nicer than over a million events.
A side question I had related to this topic was performance. Because we're gathering many remote logs, we sometimes need this done as fast as possible. My question is whether or not Get-WinEvent would be fast enough to pull logs, when compared to an Enter-PSSession or an Invoke-Command.
For the scope of this question, Get-WinEvent satisfied both the speed requirements as well as the event requirements and relying on the cmdlet to perform my remoting worked great.
My code is pretty simple, but I'll post it for record reasons:
Get-WinEvent `
-computerName $inPCID `
-FilterHashtable #{LogName="Security";Id=4672,4647,4634,4624,4625,4648,4675,4800,4801,4802,4803; `
StartTime = $inStartDate; EndTime = $inEndDate} | Where-object {$_.message -like ("*" + $InUserID + "*")} `
| export-csv $fullOutputFile

Related

Active Directory transient error when bulk using ActiveDirectory cmdlet

I have a system which synchronizes users from a data source. The data source consists of user information. When a new user is synchronized, a PowerShell task is triggered which creates or updates the user. This is all fine but when the amount of new/updated users becomes too big, some of the tasks fail with some interesting errors such as:
"The server has returned the following error: invalid enumeration
context."
or
"A connection to the directory on which to process the request was
unavailable. This is likely a transient condition."
When troubleshooting it seems obvious that the reason these errors occur are a lack of resources. It is because all the simultaneously triggered tasks are importing the module on their own PS session.
So I went trying some different things and measuring Import-Module speed etc. So I've concluded that it is faster to run Import-Module and then Get-ADUser for instance, then just Get-ADUser (which would also import the module).
Measure-Command {Import-Module ActiveDirectory}
Average time 340 ms
Measure-Command {Get-ADUser -Filter *}
Average time 420 ms
Get-ADUser after the module is imported
Average time 10 ms
But these marginal differences are not going to do anything to the issue. So I had to look further. I find that disabling the drive might help speed up the process, so I've added the following before importing the module:
$Env:ADPS_LoadDefaultDrive = 0
Measure-Command {Import-Module ActiveDirectory}
Average time 85 ms
4 times faster! But still the error persists at high amounts of users at the same time (e.g. 50 tasks). So I thought about polling the availability in the script, or make a do..while loop. Or maybe the system which fires the separate tasks needs to be redesigned, to have some sort of queue.
Does anyone recognize this situation? Or have some thoughts they'd like to share on this subject?
It is because all the simultaneously triggered tasks are importing the module on their own PS session.
Then you need to make sure this doesn't happen, or at least not so much that you run out of resources. So you have two options:
Limit the number of tasks that get run at any one time (maybe 5 at a time).
Make it one task that can work on several accounts. This way the module only gets loaded once.
I think option 2 is the better solution. For example, rather than triggering the script right away, your synchronization job could just write the username to a file (or in memory even) and once it's done finding all the users, it triggers the PowerShell script and passes the list (or the script could read the file that was written to). You have options there - whatever works best.
Update: All of .NET is available in PowerShell, so another option is to change the whole script to use .NET's DirectoryEntry instead of the ActiveDirectory module, which would use much less memory. There are even shortcuts for it in PowerShell. For example, [ADSI]"LDAP://$distinguishedName" would create a DirectoryEntry object for a user. It is a substantial rewrite, but the performance is much better (speed and memory consumption).
Here are some examples of searching with DirectorySearcher (using the [ADSISearcher] shortcut): https://blogs.technet.microsoft.com/heyscriptingguy/2010/08/23/use-the-directorysearcher-net-class-and-powershell-to-search-active-directory/
And here's an example of creating accounts with DirectoryEntry: https://www.petri.com/creating-active-directory-user-accounts-adsi-powershell

Get-MessageTracking logs for multiple users

We've been given a task to get stats on whether 6000+ Contacts have been used in the last 30 days.
We're on Exchange 2013 CU9
So obviously
$ht = Get-TransportService *EXC*
$ht | Get-MessageTrackingLog -Recipients user#domain.com -EventId send
So I'm writing a function to allow us to put single addresses into the above and if we want also feed in a list or CSV. that's easy enough.
But with over 6000 to search on, and 32 Transport servers with 30 day so of logs this is going to take an incredibly long time.
Just wondering if anyone had seen this issue before and come up with a way of speeding things up?
Thanks in advance.
You would be better just to collect the raw logs and parse those with the Log Parser studio https://gallery.technet.microsoft.com/office/Log-Parser-Studio-cd458765 which will perform much better.

Azure Data Factory Portal Blades Status Cache Reset

I have a fairly complex Azure Data Factory (ADF) with many datasets covering a variety of time slices intervals etc. Some of these datasets throw errors from time to time and need to be fixed. Typically I'll deal with the error behind the scenes, in other Azure services or offline. Then once its resolved I simply set the errored/failed time slice status manually using PowerShell.
For example:
Set-AzureRmDataFactorySliceStatus `
-ResourceGroupName $ResourceGroup `
-DataFactoryName $ADFName.DataFactoryName `
-DatasetName $Dataset.DatasetName `
-StartDateTime 2017-02-23 `
-EndDateTime 2017-02-24 `
-Status "Waiting" `
-UpdateType "Individual"
However, in doing this the portal does not react very well when it comes to reporting the actual errors at the main ADF blade. Snippet below.
In the case of the above I certainly do not have 21 errored datasets! How dare it! :-)
So, my question is. Do we know of a way to clear the cache or reset these values in the ADF portal blades? They are inaccurate and it doesn't look great if the customer decides to check on things.
Ideally I'd scrap the portal UI altogether! PowerShell for the win :-)
Many thanks for your time and support.

Powershell launch at specific times

I am building a system to restart computers for patch purposes. Most of the skeleton is there and working, I use workflows and some functions to allow me to capture errors and reboot the systems in a number of ways in case of failures.
One thing I am not sure of is how to set up the timing. I am working on a web interface where people can schedule their reboots, either dynamic (one-time) or regularly scheduled (monthly). The server info and times for the boots is stored in a SQL database.
The part that I am missing is how to trigger the reboots when scheduled. All I can think of is allowing for whole hour increments, and run a script hourly checking to see if any servers in the db have a reboot time that "matches" the current time. This will likely work, but is somewhat inflexible.
Can anyone think of a better way? Some sort of daemon?
For instance, user X has 300 servers assigned to him. He wants 200 rebooted at 10 PM on each Friday, and 50 once a month on Saturday at 11 PM. There will be over a dozen users rebooting 3000-4000 computers, sometimes multiple times monthly.
Ok, let's say you have a script that takes a date and time as an argument that will look up what computers to reboot based on that specific date and time, or schedule, or whatever it is that you're storing in your sql db that specifies how often to reboot things. For the sake of me not really knowing sql that well, we'll pretend that this is functional (would require the PowerShell Community Extensions snapin for Invoke-AdoCommand cmdlet):
[cmdletbinding()]
Param([string]$RebootTime)
$ConStr = 'Data Source=SQLSrv01;Database=RebootTracking;Integrated Security=true;'
$Query = "Select * from Table1 Where Schedule = $RebootTime"
$Data = Invoke-AdoCommand -ProviderName SqlClient -ConnectionString $ConStr -CommandText $Query
$data | ForEach{Do things to shutdown $_.ServerName}
You said you already have things setup to reboot the servers so I didn't even really try there. Then all you have to do is setup a scheduledjob for whenever any server is supposed to be rebooted:
Register-ScheduledJob –FilePath \\Srv01\Scripts\RebootServers.ps1 –Trigger #{Frequency=Weekly; At="10:00PM"; DaysOfWeek="Saturday"; Interval=4} -ArgumentList #{'RebootTime'="Day=Saturday;Weeks=4;Time=22:00"}
That's an example, but you know, you could probably work with it to accomplish your needs. That will run it once every 4 Saturdays (about monthly). That one scheduled task will query the sql server to match up a string against a field, and really you could format that any way you want to make it as specific or general as desired for the match. That way one task could reboot those 200 servers, and another could reboot the other 50 all dependent on the user's desires.

How do I run two Powershell WMI queries simultaneously?

I would like run two commands simultaneously. Currently I am facing an issue with running the commands serially as the perfmon data is not matching the stats.
Below is the idea.
I am running wmi_processes to get the name from the command line. As we have the application name in the command this is the only way to know the name and IDprocess.
Then I am querying the formatted perfmon data to get the CPU usage. But there is a delay in this and the most of the times I miss the details.
So is there any way I can trigger both the queries at the same time and work with the datasets only?
Any other ideas are most welcome to get the same solution. Thank you for the help.
Update:
I am trying to get the processes IDs matched, but by the time i search in the below foreach loop the execution may be over. So i want them to be executed at once. Here is my code
$rec=Get-WmiObject -Query "select CommandLine,CreationDate,Handle,HandleCount,Name,ProcessId,VirtualSize,WorkingSetSize from Win32_Process where commandline like 'Dtexec%'"
foreach ($P in $rec)
{
$PsId=$P.ProcessId
$FormattedData=Get-WmiObject -Query "select ElapsedTime,IDProcess,PercentProcessorTime,PercentUserTime,PercentPrivilegedTime,IOReadByte sPerSec,IOWriteBytesPerSec,PrivateBytes,WorkingSet from Win32_PerfFormattedData_PerfProc_Process where IDProcess='$Psid'"
$Usage=$P.CreationDate+","+$PsId+","+$P.Handle+","+$P.HandleCount+","+$P.VirtualSiz e+","+$FormattedData.IDProcess+","+$FormattedData.PercentProcessorTime+","+$FormattedData.E lapsedTime+","+$FormattedData.PercentUserTime+","+$FormattedData.PercentPrivilegedTime+","+ $FormattedData.IOReadBytesPerSec+","+$FormattedData.IOWriteBytesPerSec+","+$FormattedData.P rivateBytes+","+$FormattedData.WorkingSet+","+$P.CommandLine
$Usage
}
So how can i do it?