store data by day and hour - powershell - powershell

Id like to be able to store, change, and retrieve data by day and hour using powershell.
For instance, I have a list of data from which I can extract the dayOfMonth, hour, max, active, average.
I would like to be able to stick the 3 latter values in an array indexed by day and hour.
The raw data might look like this: (the avg val wouldnt be there yet):
12/13/19,08:13:04,13,13
12/13/19,08:43:04,10,9
12/13/19,09:13:04,11,9
12/13/19,09:43:04,12,7
12/13/19,08:13:04,10,12
12/14/19,08:13:04,8,2
12/14/19,08:43:04,10,4
12/14/19,09:13:04,8,7
12/14/19,09:43:04,11,1
12/14/19,10:13:04,9,7
I wish to be able to store the maxChannel,maxActive, and average, for each hour of each day, so the data in the array would look like this:
day,hr,max,actv,avg
13,8,13,13,11
13,9,12,9,8
13,10,12,2,2
14,8,10,4,3
14,9,11,7,4
14,10,9,7,7
I tried to store it in a previously created array
$arrOut2 = #() #summary data
like this(which needless to say didnt work):
initialize each needed row
ForEach ($line in $arrFile)
{
$arrLine = $line.split(",") #create an array with each word in line its own element
$Date=$arrLine[0]
$Time=$arrLine[1]
$ChanM=$arrLine[2]
$ChanA=$arrLine[3]
$DateA=$Date.Split("/")
$Day=$DateA[1]
$TimeA=$Time.Split(":")
$Hour=[int]$TimeA[0]
$arrOut2 += ,#[$Day][$Hour](0,0,0)
}
ForEach ($line in $arrFile)
{
$arrLine = $line.split(",") #create an array with each word in line its own element
$Date=$arrLine[0]
$Time=$arrLine[1]
$ChanM=$arrLine[2]
$ChanA=$arrLine[3]
$DateA=$Date.Split("/")
$Day=$DateA[1]
$TimeA=$Time.Split(":")
$Hour=[int]$TimeA[0]
IF (($Hour -ge 8) -And ($Hour -lt 17)) {
$AvCnt=$AvCnt+1 #count Intervals for averaging
$AvTot=$AvTot+$ChanA #sum active channels for averaging
$AvgActChan=$AvTot/$AvCnt
IF($ChanC -GT $arrOut2[$Hour][0]){$arrOut2[$Hour][0]=$ChanC} #grab max chanC frm current hr
IF($ChanA -GT $arrOut2[$Hour][1]){$arrOut2[$Hour][0]=$ChanC} #grab max chan Actv frm curr hr
$ArrOut2[$Day][$Hour][2]=$AvgActChan #grab curnt chan Avg Active from current hour
}
}
As you can see in the first forloop, I am attempting to use day and hour as indices. Powershell just doesnt like that - at least in that format.
Does anyone have any ideas of how I can accomplish this? I know I could keep track of the current day and hour, then process it in separate variables, then when I see the day or hour change, write those to a separate array, but the data is quite a bit messier than Ive represented in this example and that would make the whole thing quite messy, when processing it within an array would be so much cleaner.
Appreciate any suggestions
thanks

I would sort it this way:
$data = import-csv data.txt -header date,time,a,b
$data | sort date,time

Related

Counting date entries from file. - Powershell

I have a file which have several lines with different timestamps, however I want to count the timestamps from the last 15 minutes.
Counting timestamps would be easy enough, but I cannot figure out how to get the timestamps from the last 15 minutes.
Example of timestamps
2020-07-20T18:00:00Z
I have tried several things, but I couldn't get them to work. Perhaps it's just me...
I'm not sure what the formatting in your original file looks like but this code will count the entries within the last 15 minutes.
$lastFifteenMinutesCount = 0
$comparisonDate = [datetime]::UtcNow
Get-Content C:\tmp\overflow\63003625\stuff.txt | % {
if ($_ -match '(\d{4}-\d{2}-\d{2}T\d{1,2}:\d{1,2}:\d{1,2}Z)') {
$sDate = [datetime]::Parse($Matches[1]).ToUniversalTime().AddMinutes(15)
if ($sDate -ge $comparisonDate) {
$lastFifteenMinutesCount++
}
}
}
$lastFifteenMinutesCount

How to count % of difference between two numbers in Powershell

i'm fairly new to powershell. I have a task to create a script that reads a log of a job run, counts the number of items by line, and then compares that count to a previous run and throws an Event log if the difference in percent exceeds 30 percent.
I'm having some difficulty finding out how to compare the job count run and compare difference in percent.
so far i have this code:
$TodayCount = $todayrun.Count
$YesterdayCount = $yesterdayrun.count
$DifferenceCount = ($TodayCount - YesterdayCount) +1
That counts the number of ints between each too, but i need to find a percentage different between the two runs so I can throw an IF statement that if difference in items is > 30 Write to event log .. for instance today count could equal 2000 lines and yesterday could equal 1800 lines or 2200 .. i just need to store the percentage difference.
Hope that makes sense, and please be gentle :)
To calculate the percentage increase: First, work out the difference (increase) between the two numbers you are comparing. Then divide the increase by the original number and multiply the answer by 100.
I've made an assumption that you would want to check if it's greater than or equal to +30% or less than or equal to -30%.
If you want greater than and less than then change -ge to -gtand -le to -lt respectively.
$TodayCount = $todayrun.Count
$YesterdayCount = $yesterdayrun.count
$DifferenceCount = $TodayCount - $YesterdayCount +1
$percentageDifference = $DifferenceCount / $TodayCount * 100
if(($percentageDifference -ge 30) -or ($percentageDifference -le -30))
{
Write-Output "Percentage difference is +/-30%"
}

Creating PowerShell if statement for True over time

I have CSV document. Each row of the CSV has a timestamp, along with various performance metrics as they were when recorded at that time.
What I want to be able to do is consume the csv and perform a logic test on it of this kind of nature:
If said metric is above a specified threshold for a specified amount of time = True
I can easily setup a logic statement comparing two numbers/metrics, but I wasn't sure how to add in the component as to whether that condition is true for a certain amount of time.
CSV might look something like this:
time,KB/sec,IOPS,Avg. Latency ms
8/6/2017 10:30,10616.79606,117.2243058,35.63250298
8/6/2017 10:20,11337.82872,149.4084982,28.55670254
8/6/2017 10:17,12276.47016,172.2729097,25.43457276
8/6/2017 10:14,10125.01863,106.2432985,37.38492431
8/6/2017 10:11,12185.1857,127.452136,32.40784617
So I might want to know if the latency is consistently over a certain amount for a couple of hours for instance
I could use a bit of direction and would greatly appreciate any help anyone can provide.
First you need to convert the values from your CSV from string to the proper type (DateTime and double respectively).
Import-Csv 'C:\path\to\your.csv' | ForEach-Object {
$_.time = [DateTime]$_.time
$_.'KB/sec' = [double]$_.'KB/sec'
$_.IOPS = [double]$_.IOPS
...
}
Check if the value in question is above the given threshold. You have 3 cases to cover:
The value exceeds the threshold for the first time: remember the current timestamp.
The value exceeds the threshold and also exceeded it in the previous iteration: calculate the difference between the current timestamp and the previously remembered timestamp. Return $true if the timespan is greater than the maximum duration you want to accept.
The value does not exceed the threshold: forget the previously remembered timestamp.
Well not knowing exactly what you're aiming for perhaps the below will help.
function Monitor-Something
{
Param
(
$Date = '8/6/2017',
$StartTime = '10:17',
$StopTime = '10:30',
$Threshold,
$CSVLogfile = 'C:\temp\test.csv'
)
Begin
{
}
Process
{
$csv = Import-Csv $CSVLogfile
[datetime]$monitorStart = $("$Date $StartTime")
[datetime]$monitorStop = $("$Date $StopTime")
foreach($item in $csv)
{
[datetime]$objDateTime = $item.time
if($objDateTime -gt $monitorStart -and $objDateTime -lt $monitorStop)
{
#do stuff here, compare against your threshold report things build an object for an output
$item.IOPS #This is here just to show you a successful result
}
}
}
End
{
#return some report or object that you can analyze
}
}

Carbon Difference in Time between two Dates in hh:mm:ss format

I'm trying to figure out how I can take two date time strings that are stored in our database and convert it to a difference in time format of hh:mm:ss.
I looked at diffForHumans, but that does give the format I'd like and returns things like after, ago, etc; which is useful, but not for what I'm trying to do.
The duration will never span days, only a max of a couple hours.
$startTime = Carbon::parse($this->start_time);
$finishTime = Carbon::parse($this->finish_time);
$totalDuration = $finishTime->diffForHumans($startTime);
dd($totalDuration);
// Have: "21 seconds after"
// Want: 00:00:21
I ended up grabbing the total seconds difference using Carbon:
$totalDuration = $finishTime->diffInSeconds($startTime);
// 21
Then used gmdate:
gmdate('H:i:s', $totalDuration);
// 00:00:21
If anyone has a better way I'd be interested. Otherwise this works.
$finishTime->diff($startTime)->format('%H:%I:%S');
// 00:00:21
$start = new Carbon('2018-10-04 15:00:03');
$end = new Carbon('2018-10-05 17:00:09');
You may use
$start->diff($end)->format('%H:%I:%S');
which gives the difference modulo 24h
02:00:06
If you want to have the difference with more than 24h, you may use :
$start->diffInHours($end) . ':' . $start->diff($end)->format('%I:%S');
which gives :
26:00:06
I know this is an old question, but it still tops Google results when searching for this sort of thing and Carbon has added a lot of flexibility on this, so wanted to drop a 2022 solution here as well.
TL;DR - check out the documentation for different/more verbose versions of the diffForHumans method, and greater control over options.
As an example, we needed to show the difference between two Carbon instances (start and end) in hours & minutes, but it's possible they're more than 24 hours apart—in which case the minutes become less valuable/important. We want to exclude the "ago" or Also wanted to join the strings with a comma.
We can accomplish all of that, with the $options passed into diffForHumans, like this:
use Carbon\CarbonInterface;
$options = [
'join' => ', ',
'parts' => 2,
'syntax' => CarbonInterface::DIFF_ABSOLUTE,
];
return $end->diffForHumans($start, $options);
Which will result in values like what's seen in the Duration column:
Hope that's helpful for someone!
You can do it using the Carbon package this way to get the time difference:
$start_time = new Carbon('14:53:00');
$end_time = new Carbon('15:00:00');
$time_difference_in_minutes = $end_time->diffInMinutes($start_time);//you also find difference in hours using diffInHours()

Are earlier historical metrics published to CloudWatch Delayed?

I've got a custom CloudWatch metric based on a historical value, so I'm publishing them with explicit timestamps. I've seen an issue using both the C# and PowerShell APIs where I don't see all my metrics right away.
In the example below, I'm creating sample points for every hour for the past two weeks, but I only see the last ~48 hours in my graph in the CloudWatch console.
Initialize-AWSDefaults
$cwNamespace = 'Historical CW Sanity Check'
# generate some data
$now = [DateTime]::Now.ToUniversalTime()
$startTime = $now - [TimeSpan]::FromDays(14)
$t = $startTime
$x = 0
while ($t -le $now)
{
$t += [TimeSpan]::FromHours(1)
$datum = New-Object -TypeName 'Amazon.CloudWatch.Model.MetricDatum'
$datum.Unit = [Amazon.CloudWatch.StandardUnit]::Count
$datum.Value = 2 * $x
$datum.Timestamp = $t
$datum.MetricName = 'Test 2'
$x +=1.0
Write-Host "($t, $x)"
Write-CWMetricData -Namespace $cwNamespace -MetricData $datum
}
From the docs, I thought it supported up to two weeks' worth of historical data. I'm wondering why I see the new points before the old points (or if the old points are going to show up at all). It looks like I'm only getting two days' worth.
From the help text on Write-CWMetricData:
Data that is timestamped 24 hours or more in the past may
take in excess of 48 hours to become available from submission time using
GetMetricStatistics.
I'm betting the same limit holds true for the graphs in the Console.