Converting bytes to megabytes on the fly - powershell

In the PS command below I'm getting the name, creationTimeStamp, Disk_Size and storageBytes of yesterday's snapshots in my gcp project and outputting the data to a csv file (which is later converted to HTML and emailed):
$csv = gcloud --project $gcpProject compute snapshots list --format="csv(name,creationTimestamp,diskSizeGb,storageBytes)" --filter="creationTimestamp.date('%Y-%m-%d')=$yesterday" | Out-File C:\data.csv
The result looks something like this (the number of snapshots displayed varies):
+---------------------------+-------------------------------+--------------+---------------+
| NAME | CREATION_TIMESTAMP | DISK_SIZE_GB | STORAGE_BYTES |
+---------------------------+-------------------------------+--------------+---------------+
| snapshot1-us-central1 | 2019-10-24T19:24:09.061-07:00 | 50 | 1250586048 |
| snapshot2-data-us-east1 | 2019-10-24T19:01:49.791-07:00 | 150 | 425018600 |
+---------------------------+-------------------------------+--------------+---------------+
This is good except that the STORAGE_BYTES data is all in bytes thus making it hard to read. How can I record this data in MB instead in the csv file (or just replacing this data in the csv file that's in bytes with MB)

$data=Get-Content C:\data.csv
$NewData=#($data[0..2])#Adding Header in a new Variable
$Data[3..$($data.Count - 2)]|Foreach{
$startRange=$_.LastIndexOf("| ")+1
$length=$_.LastIndexOf(" |") - $_.LastIndexOf("| ")
#Converting Bytes into MB and replacing in the line
$NewData+=$_.Replace(($_.Substring($startRange,$length)).Trim(),"$([math]::Round(($_.Substring($startRange,$length)/1MB),2)) ")
}
#Adding last line
$NewData+=$data[$data.Count - 1]
#Modifying exsiting file
$NewData |Set-Content C:\data.csv -Force

I think, it should be simple using GCP PowerShell cmdlets.
Get-GceSnapshot -Project $gcpProject
I never used it, but you can use calculated properties to convert the size in oneline, an example below
Get-Process | Select-Object -First 3 -Property Name,#{E={$_.STORAGE_BYTES/1024};L='Size'}
PS: assuming the size property name is STORAGE_BYTES
https://googlecloudplatform.github.io/google-cloud-powershell/#/google-compute-engine/GceSnapshot/Get-GceSnapshot

Related

Combine two CSV files in powershell without changing the order of columns

I have "a.csv" and "b.csv" . I tried to merge them with below commands
cd c:/users/mine/test
Get-Content a.csv, b.csv | Select-Object -Unique | Set-Content -Encoding ASCII joined.csv
But I got Output file like b.csv added by end of the row of a.csv. I wanted add by end of the column of a.csv then b.csv columns should begin
Vm Resource SID
mnvb vclkn vxjcb
vjc.v vnxc,m bvkxncb
Vm 123 456 789
mnvb apple banana orange
vjc.v lemon onion tomato
My expected output should be like below. Without changing the order
Vm Resource SID 123 456 789
mnvb vclkn vxjcb apple banana orange
vjc.v vnxc,m bvkxncb lemon onion tomato
From here, there are two ways to do it -
Join-Object custom function by RamblingCookieMonster. This is short and sweet. After you import the function in your current PoSh environment, you can use the below command to get your desired result -
Join-Object -Left $a -Right $b -LeftJoinProperty vm -RightJoinProperty vm | Export-Csv Joined.csv -NTI
The accepted answer from mklement which would work for you as below -
# Read the 2 CSV files into collections of custom objects.
# Note: This reads the entire files into memory.
$doc1 = Import-Csv a.csv
$doc2 = Import-Csv b.csv
$outFile = 'Joined.csv'
# Determine the column (property) names that are unique to document 2.
$doc2OnlyColNames = (
Compare-Object $doc1[0].psobject.properties.name $doc2[0].psobject.properties.name |
Where-Object SideIndicator -eq '=>'
).InputObject
# Initialize an ordered hashtable that will be used to temporarily store
# each document 2 row's unique values as key-value pairs, so that they
# can be appended as properties to each document-1 row.
$htUniqueRowD2Props = [ordered] #{}
# Process the corresponding rows one by one, construct a merged output object
# for each, and export the merged objects to a new CSV file.
$i = 0
$(foreach($rowD1 in $doc1) {
# Get the corresponding row from document 2.
$rowD2 = $doc2[$i++]
# Extract the values from the unique document-2 columns and store them in the ordered
# hashtable.
foreach($pname in $doc2OnlyColNames) { $htUniqueRowD2Props.$pname = $rowD2.$pname }
# Add the properties represented by the hashtable entries to the
# document-1 row at hand and output the augmented object (-PassThru).
$rowD1 | Add-Member -NotePropertyMembers $htUniqueRowD2Props -PassThru
}) | Export-Csv -NoTypeInformation -Encoding Utf8 $outFile

Join-Object two different csv files using PowerShell

The first .csv file is an monthly backup size in KB on based client name. The second .csv file is an next monthly backup size in KB on based client name.
It lists all the Client Name in column A. Column B has the corresponding policy name of client and last column backup size in KB (i.e. - 487402463).
If the difference between client size (1638838488 - 1238838488 = 0.37 in TB ) is greater than 0.10 TB , the results will be spit out in TB size to a csv file like below.
Also , a client may be related multiple policy name.
My question is : I want to add something too.
Backup size may decrease in the next month such as hostname15,Company_Policy_11.
Also , hostname55,Company_Policy_XXX may have different policy name.
hostnameXX,Company_Policy_XXX,0 and hostnameXX,Company_Policy_XXX,41806794 it may be duplicate client and policy name. if this does not exist in CSV2 then I want to display as negative (-0.14) like below. Or may be exist in CSV2 hostnameZZ,Company_Policy_XXX as well.
Lastly just it may be in CSV2 such as hostnameSS,Company_Policy_XXX.
I used the Join-Object module. https://github.com/ili101/Join-Object
Example CSVFile1.csv
Client Name,Policy Name,KB Size
hostname1,Company_Policy,487402463
hostname2,Company_Policy,227850336
hostname3,Company_Policy_11,8360960
hostname4,Company_Policy_11,1238838488
hostname15,Company_Policy_11,3238838488
hostname1,Company_Policy_55,521423110
hostname10,Company_Policy,28508975
hostname3,Company_Policy_66,295925
hostname5,Company_Policy_22,82001824
hostname2,Company_Policy_33,26176885
hostnameXX,Company_Policy_XXX,0
hostnameXX,Company_Policy_XXX,141806794
hostnameYY,Company_Policy_XXX,121806794
hostname55,Company_Policy_XXX,41806794
hostnameZZ,Company_Policy_XXX,0
hostnameZZ,Company_Policy_XXX,141806794
Example CSVFile2.csv
Client Name,Policy Name,KB Size
hostname1,Company_Policy,487402555
hostname2,Company_Policy,227850666
hostname3,Company_Policy_11,8361200
hostname4,Company_Policy_11,1638838488
hostname1,Company_Policy_55,621423110
hostname15,Company_Policy_11,1238838488
hostname10,Company_Policy,28908975
hostname3,Company_Policy_66,295928
hostname5,Company_Policy_22,92001824
hostname2,Company_Policy_33,36176885
hostname22,Company_Policy,291768854
hostname23,Company_Policy,291768854
hostname55,Company_Policy_BBB,191806794
hostnameZZ,Company_Policy_XXX,0
hostnameZZ,Company_Policy_XXX,291806794
hostnameSS,Company_Policy_XXX,0
hostnameSS,Company_Policy_XXX,291806794
Desired Output :
Client Name,Policy Name,TB Size
hostname4,Company_Policy_11,0.37
hostname22,Company_Policy,0.27
hostname23,Company_Policy,0.27
hostnameYY,Company_Policy_XXX,-0.12
hostnameXX,Company_Policy_XXX,-0.14
hostname15,Company_Policy_11,-2
hostname55,Company_Policy_BBB,0.15
hostnameZZ,Company_Policy_XXX,0.15
hostnameSS,Company_Policy_XXX,0.29
Here is my script so far :
$CSV2 | FullJoin $CSV1 `
-On 'Client Name','Policy Name' `
-Property 'Client Name',
'Policy Name',
#{'TB Size' = {[math]::Round(($Left.'KB Size' - $Right.'KB Size') * 1KB / 1TB, 2)}} |
Where-Object {[math]::Abs($_.'TB Size') -gt 0.10} | Export-Csv C:\Toolbox\DataReport.csv -NoTypeInformation
You could so something similar to the following. This assumes you want to subtract CSV1 values from CSV2 values.
# Read CSV files and make CSV1 sizes negative. Makes summing totals simpler.
$1 = Import-Csv CSVFile1.csv | Foreach-Object { $_.'KB Size' = -$_.'KB Size'; $_ }
$2 = Import-Csv CSVFile2.csv
# Calculated Properties to be used with Select-Object
$CalculatedProperties = #{n='Client Name';e={$_.Group.'Client Name' | Get-Unique}},
#{n='Policy Name';e={$_.Group.'Policy Name' | Get-Unique}},
#{n='TB Size';e={[math]::Round(($_.Group.'KB Size' | Measure -Sum).Sum*1KB/1TB,2)}}
# Grouping objects based on unique client and policy name combinations
$1 + $2 | Group-Object 'Client Name','Policy Name' |
Select-object $CalculatedProperties |
Where {[math]::Abs($_.'TB Size') -gt 0.10}

Collecting Unique Items from large data set over multiple text files

I am using PowerShell to collect lists of names from multiple text files. May of the names in these files are similar / repeating. I am trying to ensure that PowerShell returns a single text file with all of the unique items. In looking at the data it looks like the script is gathering 271/296 of the unique items. I'm guessing that some of the data is being flagged as duplicates when it shouldn't, any suggestions?
#Take content of each file (all names) and add unique values to text file
#for each unique value, create a row & check to see which txt files contain
function List {
$nofiles = Read-Host "How many files are we pulling from?"
$data = #()
for ($i = 0;$i -lt $nofiles; $i++)
{
$data += Read-Host "Give me the file name for file # $($i+1)"
}
return $data
}
function Aggregate ($array) {
Get-Content $array | Sort-Object -unique | Out-File newaggregate.txt
}
#SCRIPT BODY
$data = List
aggregate ($data)
I was expecting this code to catch everything, but it's missing some items that look very similar. List of missing names and their similar match:
CORPINZUTL16 MISSING FROM OUTFILE
CORPINZTRACE MISSING FROM OUTFILE
CORPINZADMIN Found In File
I have about 20 examples like this one. Apparently the Get-Content -Unique is not checking every character in a line. Can anyone recommend a better way of checking each line or possibly forcing the get-character to check full names?
Just for demonstration this line creates 3 txt files with numbers
for($i=1;$i -lt 4;$i++){set-content -path "$i.txt" -value ($i..$($i+7))}
1.txt | 2.txt | 3.txt | newaggregate.txt
1 | | | 1
2 | 2 | | 2
3 | 3 | 3 | 3
4 | 4 | 4 | 4
5 | 5 | 5 | 5
6 | 6 | 6 | 6
7 | 7 | 7 | 7
8 | 8 | 8 | 8
| 9 | 9 | 9
| | 10 | 10
Here using Get-Content with a range [1-3] of files
Get-Content [1-3].txt | Sort-Object {[int]$_} -Unique | Out-File newaggregate.txt
$All = Get-Content .\newaggregate.txt
foreach ($file in (Get-ChildItem [1-3].txt)){
Compare-Object $All (Get-Content $file.FullName) |
Select-Object #{n='File';e={$File}},
#{n="Missing";e={$_.InputObject}} -ExcludeProperty SideIndicator
}
File Missing
---- -------
Q:\Test\2019\05\07\1.txt 9
Q:\Test\2019\05\07\1.txt 10
Q:\Test\2019\05\07\2.txt 1
Q:\Test\2019\05\07\2.txt 10
Q:\Test\2019\05\07\3.txt 1
Q:\Test\2019\05\07\3.txt 2
there are two ways to achieve this one is using select-object -Unique which works when data is not sorted and can be used for small data or lists.
When dealing with large files we can use get-Unique command which works with sorted input, if input data is not sorted then it will give wrong results.
Get-ChildItem *.txt | Get-Content | measure -Line #225949
Get-ChildItem *.txt | Get-Content | sort | Get-Unique | measure -Line #119650
Here is my command for multiple files :
Get-ChildItem *.txt | Get-Content | sort | Get-Unique >> Unique.txt

Need to remove specific portion from rows in a csv using powershell

I have a csv file with two columns and multiple rows, which has the information of files with folder location and its corresponding size, like below
"Folder_Path","Size"
"C:\MSSQL\DATA\UsersData\FTP.txt","21345"
"C:\MSSQL\DATA\UsersData\Norman\abc.csv","78956"
"C:\MSSQL\DATA\UsersData\Market_Database\123.bak","1234456"
What i want do is remove the "C:\MSSQL\DATA\" part from every row in the csv and keep the rest of the folder path after starting from UsersData and all other data intact as this info is repetitive. So my csv should like this below.
"Folder_Path","Size"
"UsersData\FTP.txt","21345"
"UsersData\Norman\abc.csv","78956"
"UsersData\Market_Database\123.bak","1234456"
What i am running is as below
Import-Csv ".\abc.csv" |
Select-Object -Property #{n='Folder_Path';e={$_.'Folder_Path'.Split('C:\MSSQL\DATA\*')[0]}}, * |
Export-Csv '.\output.csv' -NTI
Any help is appreciated!
Seems like a job for a simple string replace:
Get-Content "abc.csv" | foreach { $_.replace("C:\MSSQL\DATA\", "") | Set-Content "output.csv"
or:
[System.IO.File]::WriteAllText("output.csv", [System.IO.File]::ReadAllText("abc.csv" ).Replace("C:\MSSQL\DATA\", ""))
This should work:
Import-Csv ".\abc.csv" |
Select-Object -Property #{n='Folder_Path';e={$_.'Folder_Path' -replace '^.*\\(.*\\.*)$', '$1'}}, Size |
Export-Csv '.\output.csv' -NoTypeInformation

Left padding a string to sort accordingly

I have to get the database with the max space free in a exchange 2010 however as this gonna be launched from a pipeline in c#, I'm trying to sort the results and then pick the first row.
If I try using the field AvailableNewMailboxSpace, it is sorted using the string values instead the double values:
Get-MailboxDatabase -Status | Select Name,AvailableNewMailboxSpace | Sort-Object DatabaseSize
Name AvailableNewMailboxSpace
---- ------------------------
DBMB03 123.1 MB (129,073,152 bytes)
DBMB04 114.1 MB (119,635,968 bytes)
DBMB02 115.6 MB (121,176,064 bytes)
DBMB10 224.4 MB (235,307,008 bytes)
DBMB01 81.47 MB (85,426,176 bytes)
I guess I must left pad zeros in the string to get the right order but I don't know how.
I have to achieve it in a single line because this gonna be launched using a pipeline command in c#.
This is my attempt:
Get-MailboxDatabase -Status |`
Sort (("0" * (10 - {$_.AvailableNewMailboxSpace.Substring(0, $_.AvailableNewMailboxSpace.IndexOf("MB") - 1)}.length)) + `
{$_.AvailableNewMailboxSpace.Substring(0, $_.AvailableNewMailboxSpace.IndexOf("MB") - 1)}) | Select Name,AvailableNewMailboxSpace
Try this:
| sort { [decimal]($_.AvailableNewMailboxSpace -replace "MB (\(.+\))") }
The scriptblock {} make sort evaluating only the value before MB as decimal.
For a more accurate sorting based on bytes value you can try this:
| sort { [int]($_.AvailableNewMailboxSpace -replace ".+\((.+)b.+", '$1') }
The best way would be to use the native methods: ToKB(),ToMB(),ToGB() etc:
Get-MailboxDatabase -Status | Select Name,#{n='AvailableNewMailboxSpaceMB';e={$_.AvailableNewMailboxSpace.Value.ToMB()} | Sort-Object DatabaseSize
or
Get-MailboxDatabase -Status | Select Name,AvailableNewMailboxSpace | Sort-Object {$_.AvailableNewMailboxSpace.Value.ToMB()}
If for some reason you can't do that then do string manipulation:
Get-MailboxDatabase -Status | Select Name,AvailableNewMailboxSpace | Sort-Object {[double]$_.AvailableNewMailboxSpace.Split()[0]}
Couldn't you cast while sorting? So in your pipeline you'd get something like
... blah ... | Sort-Object { $_.AvailableNewMailboxSpace -as [int] }
Cannot test that because I don't have snapins for Exchange, but this is just to give you one track.