I have been stuck on trying to export in the format needed for this script for a month now. I can't figure out how to get it to export these variables in two separate loops into a single .csv file. We are given a .csv that contains staging_input and staging_location fields that both contain file locations on the network. We need to compare these two to make sure they are the same file count and size. I created a ForEach-Object loop for each of these fields and this gives me the desired output but I am unable to export-csv at the end of each loop into a single .csv.
Desired final output
Current output
#User input CSV File and Workorder
$workorder = Read-Host -Prompt 'Please enter the Fq job code'
$pos = $workorder.IndexOf("_")
$Client = $workorder.Substring(0, $pos)
$Matter = $workorder.Substring(6, $pos)
$job = $workorder.Substring($pos+7)
$csvoutputpath = "\\ldthost.pvt\client\" + $Client + "\" + $Matter + "\Proc\WKP\" + $job
$outputfilename = $workorder + ".csv"
$csvinputpath = Read-Host -Prompt 'Please Input the directory of the CSV file containing Staging input and Staging location'
$staginginput = Import-Csv $csvinputpath | select -ExpandProperty staging_input
$staginginputpath = $staginginput.TrimStart("\")
#Get Child Item for each line
$staginginputpath | ForEach-Object{
# In here, we do whatever want to the 'cell' that's currently in the pipeline
# Get File Counts
$staginginputcount = (Get-ChildItem -Recurse -File -Force -LiteralPath \\?\UNC\$_ | Measure-Object).Count
#Get size
$staginginputsize = Get-ChildItem -Recurse -File -Force -LiteralPath \\?\UNC\$_ | Measure-Object -property length -sum
$staginginputsize = $staginginputsize.sum / 1KB
}
$staginglocation = Import-Csv $csvinputpath | select -ExpandProperty staging_location
$staginglocationpath = $staginglocation.TrimStart("\")
#Get Child Item for each line
$staginglocationpath | ForEach-Object{
# In here, we do whatever want to the 'cell' that's currently in the pipeline
# Get File Counts
$staginglocationcount = (Get-ChildItem -Recurse -File -Force -LiteralPath \\?\UNC\$_ | Measure-Object).Count
#Get size
$staginglocationsize = Get-ChildItem -Recurse -File -Force -LiteralPath \\?\UNC\$_ | Measure-Object -property length -sum
$staginglocationsize = $staginglocationsize.sum / 1KB
}
##Export Final Output
$data = #()
$row = New-Object PSObject
$row | Add-Member -MemberType NoteProperty -Name "staging_input" -Value $staginginput
$row | Add-Member -MemberType NoteProperty -Name "staging_location" -Value $staginglocation
$row | Add-Member -MemberType NoteProperty -Name "staging_input_File_Count" -Value $staginginputcount
$row | Add-Member -MemberType NoteProperty -Name "staging_location_File_Count" -Value $staginglocationcount
$row | Add-Member -MemberType NoteProperty -Name "staging_input_File_Size" -Value $staginginputsize
$row | Add-Member -MemberType NoteProperty -Name "staging_location_File_Size" -Value $staginglocationsize
$data += $row
$Finaloutput = $csvoutputpath + "\" + $outputfilename
$data | Export-Csv $Finaloutput -NoTypeInformation -Append
Use a single loop instead of two:
#User input CSV File and Workorder
$workorder = Read-Host -Prompt 'Please enter the Fq job code'
$pos = $workorder.IndexOf("_")
$Client = $workorder.Substring(0, $pos)
$Matter = $workorder.Substring(6, $pos)
$job = $workorder.Substring($pos + 7)
$csvoutputpath = "\\ldthost.pvt\client\" + $Client + "\" + $Matter + "\Proc\WKP\" + $job
$outputfilename = $workorder + ".csv"
$csvinputpath = Read-Host -Prompt 'Please Input the directory of the CSV file containing Staging input and Staging location'
$Finaloutput = $csvoutputpath + "\" + $outputfilename
Import-Csv $csvinputpath |ForEach-Object {
# Calculate derived property values from `staging_input`
$staginginputpath = $_.staging_input.TrimStart("\")
$staginginputcount = (Get-ChildItem -Recurse -File -Force -LiteralPath \\?\UNC\$staginginputpath | Measure-Object).Count
$staginginputsize = Get-ChildItem -Recurse -File -Force -LiteralPath \\?\UNC\$staginginputpath | Measure-Object -property length -sum
$staginginputsize = $staginginputsize.sum / 1KB
# Calculate derived property values from `staging_location`
$staginglocationpath = $_.staging_location.TrimStart("\")
$staginglocationcount = (Get-ChildItem -Recurse -File -Force -LiteralPath \\?\UNC\$staginglocationpath | Measure-Object).Count
$staginglocationsize = Get-ChildItem -Recurse -File -Force -LiteralPath \\?\UNC\$staginglocationpath | Measure-Object -property length -sum
$staginglocationsize = $staginglocationsize.sum / 1KB
# output new object with the property values calculated above
[pscustomobject]#{
staging_input = $_.staging_input
staging_location = $_.staging_location
staging_input_File_Count = $staginginputcount
staging_location_File_Count = $staginglocationcount
staging_input_File_Size = $staginginputsize
staging_location_File_Size = $staginglocationsize
}
} | Export-Csv $Finaloutput -NoTypeInformation
Related
I have a powershell script that performs and inventory of a fileshare. I want to know how to add ID to each line in the csv file and also a parent ID to line in the csv.
I am new to Powershell but worked out how to get the inventory script working.
Here is the code.
#Set-ExecutionPolicy Unrestricted
$SourcePath = "G:\My Drive"
$DestinationCSVPath = "e:\G Drive Inventory 20180611.csv" #Destination for Temp CSV File
$CSVColumnOrder = 'Path', 'IsDIR', 'Directory', 'FileCount', 'Parent', 'Name', 'CreationTime', 'LastAccessTime', 'LastWriteTime', 'Extension', 'BaseName', 'B'
#, 'Root', 'IsReadOnly', 'Attributes', 'Owner', 'AccessToString', 'Group' #, #'MD5', #'SHA1' #Order in which columns in CSV Output are ordered
#FOLDERS ONLY
#$SourcePathFileOutput = Get-ChildItem $SourcePath -Recurse | where {$_.PSIsContainer}
#FILES AND FOLDERS
$SourcePathFileOutput = Get-ChildItem $SourcePath -Recurse #| where {$_.PSIsContainer} #Uncomment for folders only
$HashOutput = ForEach ($file in $SourcePathFileOutput) {
Write-Output (New-Object -TypeName PSCustomObject -Property #{
Path = $file.FullName
IsDIR = $file.PSIsContainer
Directory = $File.DirectoryName
FileCount = (GCI $File.FullName -Recurse).Count
Parent = $file.Parent
Name = $File.Name
CreationTime = $File.CreationTime
LastAccessTime = $File.LastAccessTime
LastWriteTime = $File.LastWriteTime
Extension = $File.Extension
BaseName = $File.BaseName
B = $File.Length
#Root = $file.Root
#IsReadOnly = $file.IsReadOnly
#Attributes = $file.Attributes
#Owner = $acl.owner
#AccessToString = $acl.accesstostring
#Group = $acl.group
#MD5 = Get-FileHash $file.FullName -Algorithm MD5 | Select-Object -ExpandProperty Hash
#SHA1 = Get-FileHash $file.FullName -Algorithm SHA1 | Select-Object -ExpandProperty Hash
}) | Select-Object $CSVColumnOrder
}
$HashOutput | Export-Csv -NoTypeInformation -Path $DestinationCSVPath
I want to know how to add ID to each line in the csv file and also a parent ID to line in the csv.
try Something like this:
$ID=1
$SourcePath="c:\temp"
$SourcePathFileOutput=#()
#for have the parent dir
$SourcePathFileOutput += Get-Item $SourcePath | %{
$ParentPath=if ($_.PSIsContainer){$_.parent.FullName}else{$_.DirectoryName}
Add-Member -InputObject $_ -MemberType NoteProperty -Name "ID" -Value ($ID++)
Add-Member -InputObject $_ -MemberType NoteProperty -Name "PARENTPATH" -Value $ParentPath
$_
}
#for have all directory and file
$SourcePathFileOutput += Get-ChildItem $SourcePath -Recurse | %{
$ParentPath=if ($_.PSIsContainer){$_.parent.FullName}else{$_.DirectoryName}
Add-Member -InputObject $_ -MemberType NoteProperty -Name "ID" -Value ($ID++)
Add-Member -InputObject $_ -MemberType NoteProperty -Name "PARENTPATH" -Value $ParentPath
$_
}
#List dir for optimise
$DirOutput=$SourcePathFileOutput | where {$_.psiscontainer}
#for output result (add all properties you want)
$list=foreach ($Current in $SourcePathFileOutput)
{
$Result=[pscustomobject]#{
Path = $Current.FullName
PARENTPATH=$Current.PARENTPATH
ISDIR=$Current.psiscontainer
ID=$Current.ID
PARENTID=($DirOutput | where {$_.FullName -eq $Current.PARENTPATH}).ID
}
#Initialise parent root
if ($Result.PARENTID -eq $null) {$Result.PARENTID=0}
#send result on output
$Result
}
$list | Out-GridView
This should do what you want:
#Set-ExecutionPolicy Unrestricted
$SourcePath = "G:\My Drive"
$DestinationCSVPath = "e:\G Drive Inventory 20180611.csv" #Destination for Temp CSV File
$CSVColumnOrder = 'Path', 'IsDIR', 'Directory', 'FileCount', 'Parent', 'Name', 'CreationTime', 'LastAccessTime', 'LastWriteTime', 'Extension', 'BaseName', 'B','ID','ParentID'
#, 'Root', 'IsReadOnly', 'Attributes', 'Owner', 'AccessToString', 'Group' #, #'MD5', #'SHA1' #Order in which columns in CSV Output are ordered
#FOLDERS ONLY
#$SourcePathFileOutput = Get-ChildItem $SourcePath -Recurse | where {$_.PSIsContainer}
#FILES AND FOLDERS
$SourcePathFileOutput = Get-ChildItem $SourcePath -Recurse | Sort-Object Fullname #| where {$_.PSIsContainer} #Uncomment for folders only
$CurrentID = 1
$IDs = [ordered]#{}
$IDs.add(($SourcePathFileOutput[0].fullname | split-path),0)
$HashOutput = ForEach ($file in $SourcePathFileOutput) {
$IDs.add($file.fullname,$CurrentID)
Write-Output (New-Object -TypeName PSCustomObject -Property #{
ID = $CurrentID
ParentID = $IDs.$($file.fullname | split-path)
Path = $file.FullName
IsDIR = $file.PSIsContainer
Directory = $File.DirectoryName
FileCount = (GCI $File.FullName -Recurse).Count
Parent = $file.Parent
Name = $File.Name
CreationTime = $File.CreationTime
LastAccessTime = $File.LastAccessTime
LastWriteTime = $File.LastWriteTime
Extension = $File.Extension
BaseName = $File.BaseName
B = $File.Length
#Root = $file.Root
#IsReadOnly = $file.IsReadOnly
#Attributes = $file.Attributes
#Owner = $acl.owner
#AccessToString = $acl.accesstostring
#Group = $acl.group
#MD5 = Get-FileHash $file.FullName -Algorithm MD5 | Select-Object -ExpandProperty Hash
#SHA1 = Get-FileHash $file.FullName -Algorithm SHA1 | Select-Object -ExpandProperty Hash
}) | Select-Object $CSVColumnOrder
$CurrentID++
}
$HashOutput | Export-Csv -NoTypeInformation -Path $DestinationCSVPath
Explanation:
I piped $SourcePathFileOutput to Sort-Object because it is easiest to assign an ordered ID if things are sorted by their location in the directory tree.
$CurrentID starts at 1 because the first item in the loop is not the root parent. Your loop will increment this variable by 1 ($CurrentID++) at the end of each loop iteration so that the next file/directory will a new number. Since this code creates $CurrentID as an Int32 type, you will have issues if you have more than 2 billion files/directories. You will have to initialize it as a 64-bit type ($CurrentID = [long]1). However, I believe there can only be Int32 number of keys in a hash table, so a different strategy will need to be adopted if you have billions of files.
$IDs is a ordered hash table that tracks IDs for all files and directories. Each key in hash table is the path to the current item in the loop. The value is the ID assignment. This means you can access the ID using the syntax $IDs.path. After the initialization, I added the first entry with ID 0, which represents the root parent.
Inside of the loop, I created the ID property that just stores the current value of $CurrentID. I created 'ParentID', which looks up the parent directory inside of the $IDs hash table and returns that key's ID value.
I updated $CSVColumnOrder to have the ID and ParentID as columns.
You could have the ID scheme be slightly different if you do your initial sort differently. You don't have to increment by 1. You may have a requirement that you want directories to have smaller IDs than files and that will require more code (I did not see this requirement though).
I am trying to print the path of a file if string is found. Problem is if 1 file does not contain the string in a folder then I do not get any output. Basically I am looking to see if a certificate epoch-time is within 30 days of expiration. Below is my code:
$c = Get-Date (Get-Date).ToUniversalTime() -UFormat %s
$epochtimes=[math]::Round($c)
$d = get-childitem C:\scripts\PALO\* -recurse | Select-String -pattern
"expiry-epoch"
$e=$d -split "epoch"
$certtime=[double] $e[1]
$certexp = $certtime - 2592000
ForEach ($i in $certexp){
If ($certexp -le $epochtime) {
Write-Host $i
}
}
I've made a couple of assumptions since it isn't totally clear from your question what is going on. Importantly, I've assumed you have a directory tree containing some text files each with a line in it like this:
expiry-epoch 1526854766.33933
If this is the case, then the following should display some useful information about the files:
Get-ChildItem -Path "C:\test" -File -Recurse |
ForEach-Object {$threshold = [Math]::Round((Get-Date (Get-Date).ToUniversalTime() -UFormat %s)) + 2592000} {
$certEpochTime = ([double]($_ | Select-String -Pattern "^expiry-epoch (\d+\.\d+)$").Matches.Groups[1].Value)
$certExpiryTime = (Get-Date "1/1/1970").AddSeconds($certEpochTime)
New-Object -TypeName PsCustomObject|
Add-Member -MemberType NoteProperty -Name ExpiresSoon -Value ($certEpochTime -le $threshold) -PassThru |
Add-Member -MemberType NoteProperty -Name DaysUntilExpiry -Value ([Math]::Round(($certExpiryTime - (Get-Date)).TotalDays)) -PassThru |
Add-Member -MemberType NoteProperty -Name CertExpiryTime -Value $certExpiryTime -PassThru |
Add-Member -MemberType NoteProperty -Name CertEpochTime -Value $certEpochTime -PassThru |
Add-Member -MemberType NoteProperty -Name FilePath -Value $_.FullName -PassThru
} | Format-Table -AutoSize
EDIT:
If all you need is the filename of any files with an expiry-epoch within 30 days, then this simplified version will do that:
Get-ChildItem -Path "C:\test" -File -Recurse |
ForEach-Object {$threshold = [Math]::Round((Get-Date (Get-Date).ToUniversalTime() -UFormat %s)) + 2592000} {
$certEpochTime = ([double]($_ | Select-String -Pattern "^expiry-epoch (\d+\.\d+)$").Matches.Groups[1].Value)
if($certEpochTime -le $threshold)
{
$_.FullName
}
}
What I am attempting to do is read the file size from 4 files every minute and then output the results to a CSV file. I would like each file size written to their own column. I have attempted this with the code below but I am not getting the results I need. When the file sizes are written to the CSV they are added to a single column, each size on their own row.
If I expand my array ($Process) to two records it forces all of the file lengths in the first record to the first row, each in their own column (which is what I want).
The second record does the same.
Is there any way to force a single record to one line and their values in their own columns by making an adjustment to the code? I appreciate your help. Thank you.
$Csv = 'c:\users\rob\LogFileSize.csv'
$today = get-date -f "ddd"
$day = $today.ToLower()
$date = Get-Date
$numColstoExport=5
$a = dir C:\RTscada\bin\ErrorLogs\error1_Log_$day.txt
$a | Add-Member -MemberType AliasProperty -Name FileLength -Value Length
$b = dir C:\RTscada\bin\ErrorLogs\error2_Log_$day.txt
$b| Add-Member -MemberType AliasProperty -Name FileLength -Value Length
$c = dir C:\RTscada\bin\ErrorLogs\error3_Log_$day.txt
$c | Add-Member -MemberType AliasProperty -Name FileLength -Value Length
$d = dir C:\RTscada\bin\ErrorLogs\error4_Log_$day.txt
$d | Add-Member -MemberType AliasProperty -Name FileLength -Value Length
$error1 = $a.Length
$error2 = $b.Length
$error3 = $c.Length
$error4 = $d.Length
# Array of date and file lengths
$Process = #($date, $error1, $error2, $error3, $error4)
$holdarr=#()
$pNames=#("Date", "Error1", "Error2","Error2","Error4")
foreach ($row in $Process){
$obj = new-object PSObject
for ($i=0;$i -lt $numColstoExport; $i++){
$obj | Add-Member -MemberType NoteProperty -Name $pNames[$i] -Value $row[$i]
}
$holdarr+=$obj
$obj=$null
}
$holdarr | export-csv $Csv -NoTypeInformation -Append
This should do the trick:
function MonitorLogFiles {
Param(
$LogFiles,
$Csv
)
$FirstCsvLine = 'Date'
foreach ($LogFile in $LogFiles) {
$FirstCsvLine = $FirstCsvLine + ",$(Split-Path $LogFile -leaf)"
}
$FirstCsvLine | Out-File $Csv -Encoding UTF8
while ($True) {
$CurrentCsvLine = (Get-Date -format "dd-MMM-yyyy HH:mm").ToString()
foreach ($LogFile in $LogFiles) {
$Size = (Get-Item $LogFile).Length / 1KB
$CurrentCsvLine = $CurrentCsvLine + ",$Size KB"
}
$CurrentCsvLine | Out-File $Csv -append -Encoding UTF8
Start-Sleep -Seconds 60
}
}
MonitorLogFiles -LogFiles C:\test.txt,C:\Test2.txt -Csv c:\test.csv
You can do something like this:
$holdarr|Select-Object -Property Name, Value | Export-Csv $Csv -NoTypeInformation -Append
Note: Whatever desired columns you wish to select , take all of them in the select-object.
Hope it helps.
I am writing a script that will iterate through folders, grabbing substrings of the folder names as variable values, and then iterate through the log files in each of the folders and get some data out of the log files for output to a .csv file. Where I am running into an issue is with the use of Get-ChildItem with variables I have already set. When I run this line by itself, it does not provide any value:
#running this to see the value of $files
$files = Get-ChildItem $_.FullName $folder
$files does not then contain any value.
Here is the entire portion of the script, for reference and context:
#get all folders from the Logs directory
$folders = Get-ChildItem "C:\Temp\MPOS\Logs"
#iterate through each folder
foreach ($folder in $folders) {
#set substrings of the folder name to variables
$storeNumber = $folder.Name.Substring(2,3)
$date = $folder.Name.Substring(9,7)
#get all files from the current folder being evaluated
$files = Get-ChildItem $_.FullName $folder
#iterate through each file in the current folder
foreach ($file in $files) {
#set substring of the file name to a variable
$registerNumber = $file.Name.Substring(12,4)
#get content of the file
$logfileContent = Get-Content $file
#look for all matches of the string "TransactionNumber"
$transactions = Select-String -InputObject $logfileContent -Pattern "TransactionNumber" -AllMatches
#count number of matches from above
$transactionCount = $transactions.Matches.Count
#below info is creating the object for the .csv
$transObject = New-Object PSObject
$transObject | Add-Member -MemberType NoteProperty -Name "StoreNumber" -Value $storeNumber
$transObject | Add-Member -MemberType NoteProperty -Name "Sales Date" -Value $date
$transObject | Add-Member -MemberType NoteProperty -Name "RegisterNumber" -Value $registerNumber
$transObject | Add-Member -MemberType NoteProperty -Name "Transactions" -Value $transactionCount
$resultsArray += $transObject
}
}
$resultsArray | Export-Csv C:\Temp\MPOS\MPOSTransactions.csv -NoTypeInformation
Edited code below - changed to read $folder.FullName - working now!
$resultsArray = #()
$folders = Get-ChildItem "C:\Temp\MPOS\Logs"
foreach ($folder in $folders) {
$storeNumber = $folder.Name.Substring(2,3)
$date = $folder.Name.Substring(9,7)
$files = Get-ChildItem $folder.FullName
foreach ($file in $files) {
$registerNumber = $file.Name.Substring(12,4)
$logfileContent = Get-Content $file.FullName
$transactions = Select-String -InputObject $logfileContent -Pattern "TransactionNumber" -AllMatches
$transactionCount = $transactions.Matches.Count
$transObject = New-Object PSObject
$transObject | Add-Member -MemberType NoteProperty -Name "StoreNumber" -Value $storeNumber
$transObject | Add-Member -MemberType NoteProperty -Name "Sales Date" -Value $date
$transObject | Add-Member -MemberType NoteProperty -Name "RegisterNumber" -Value $registerNumber
$transObject | Add-Member -MemberType NoteProperty -Name "Transactions" -Value $transactionCount
$resultsArray += $transObject
}
}
$resultsArray | Export-Csv C:\Temp\MPOS\MPOSTransactions.csv -NoTypeInformation
I had this exact issue, but I found out in my case I was trying to load a variable by piping in a value that also was using Format-Table command.
Example: (this worked)
$GetMostRecentFile = $GetLatestFile = Get-ChildItem -Force -Recurse -File -ErrorAction SilentlyContinue | Where-Object { $.LastWriteTime.Date -lt (Get-Date).Date -and $.Name -ne 'Thumbs.db' -and $_.Name -ne '.DS_Store'} | Sort LastWriteTime -Descending | Select-Object -First 1 LastWriteTime,FullName,Length
(this didn't work)
$GetLatestFile = Get-ChildItem -Force -Recurse -File -ErrorAction SilentlyContinue | Where-Object { $.LastWriteTime.Date -lt (Get-Date).Date -and $.Name -ne 'Thumbs.db' -and $_.Name -ne '.DS_Store'} | Sort LastWriteTime -Descending | Select-Object -First 1 LastWriteTime,FullName,Length | Format-Table -Wrap
Because I'd been using Format-Table as part of my commands to explore this data and build my queries, I forgot that if you pipe this (with FT) to a variable, the variable won't have the details.
In my case above, I needed to pull $GetMostRecentFile.FullName and $GetMostRecentFile.LastWriteTime pulled into another array. But when I was piping with FT, there was not $GetMostRecentFile.FullName. When I remove
I'm trying to write a PowerShell script retrieving the directory size and owner on my Windows file systems. I've got two separate scripts that work independently, but I don’t know how to put them together as a single script file.
Directory Name and Size:
$startFolder = "C:\Test”
$colItems = (Get-ChildItem $startFolder | Where-Object {$_.PSIsContainer -eq $True} | Sort-Object)
foreach ($i in $colItems)
{
$subFolderItems = (Get-ChildItem $i.FullName | Measure-Object -property length -sum)
$i.FullName + " -- " + "{0:N2}" -f ($subFolderItems.sum / 1MB) + " MB"
}
Directory owner:
Get-ACL C:\Test
I'd like for the script to output to a CSV file.
So something along the lines of this then?
$start = "c:\temp"
$output = "C:\temp\output.csv"
Get-ChildItem $start | Where-Object{$_.PSIsContainer} | ForEach-Object{
$singleFolder = $_.FullName
$folderSize = Get-ChildItem $singleFolder -Recurse -Force | Where-Object{!$_.PSIsContainer} | Measure-Object -Property length -sum | Select-Object -ExpandProperty Sum
$folderSize = [math]::round($folderSize/1MB, 2)
$owner = Get-Acl $singleFolder | Select-Object -ExpandProperty Owner
$_ | Add-Member -MemberType NoteProperty -Name FolderSize -Value $folderSize -PassThru |
Add-Member -MemberType NoteProperty -Name Owner -Value $owner -PassThru
} | Select-Object FullName,FolderSize,Owner | Export-Csv $output -NoTypeInformation
Needed to add some extra logic in length calculation to keep folders out of the mix > Where-Object{!$_.PSIsContainer}. This should work with PowerShell 2.0 just fine.
It loops through each folder in the root of $start and for each $singleFolder calculates the size of the contents and folder owner.
The variables are then added to object through pipeline with Add-Member as the properties FolderSize and Owner. The final Select-Object is used to isolate the properties we want for the output file.
Sample
FullName FolderSize Owner
-------- ---------- -----
C:\temp\37 Place i.Bay domain\guy
C:\temp\adoc42 1103186357 domain\guy
C:\temp\Adoc72 958330505 domain\guy
Notice the empty folder. That output is sent to Export-CSV in the file $output