Fairly new to PowerShell and having challenges with appending data in the first available row for each headered column in a csv file.
I would like to utilize foreach for each column's type of data that will be independent of another column's data. The column headers are $headers = "Scope", "Drawing", "Submittal", "Database", "Estimate", "Sequence" with a foreach to locate and append their individual items to each column. The current problem that is happening is that because each category/column with its respective foreach will add it on a separate row because the previous row already had data appended from another category's/column's data creating a diagonal appended data.
The reason that a separate foreach is being used is for each category/column is because the category's are looking for and filtering files independently for each category.
Below is what is happening in the CSV file:
| Scope | Drawing | Submittal | Database | Estimate | Sequence |
| ------| ------- |---------- |--------- |--------- |--------- |
| DATA01| empty | empty | empty | empty | empty |
| empty | DATA11 | empty | empty | empty | empty |
| empty | empty | DATA21 | empty | empty | empty |
| empty | empty | empty | DATA31 | empty | empty |
| empty | empty | empty | empty | DATA41 | empty |
| empty | empty | empty | empty | empty | DATA51 |
This is what would be the desired result be for the CSV file:
| Scope | Drawing | Submittal | Database | Estimate | Sequence |
| ------| ------- |---------- |--------- |--------- |--------- |
| DATA01| DATA11 | DATA21 | DATA31 | DATA41 | DATA51 |
Here is part of the code that is being worked on:
# Creates the CSV if it does not already exist
$headers = "Scope", "Mechanical Drawing", "Controls Submittal", "Database", "Estimate", "Sequence of Operations"
$psObject = New-Object psobject
foreach($header in $headers)
{
Add-Member -InputObject $psobject -MemberType noteproperty -Name $header -Value ""
}
$psObject | Export-Csv $CsvFile -NoTypeInformation
foreach ($file in $ScopeList)
{
$hash=#{
"Scope" = $file.Fullname
}
$NewItem = New-Object PSObject -Property $hash
Export-Csv $CsvFile -inputobject $NewItem -append -Force
}
foreach ($file in $DrawingList)
{
$hash=#{
"Drawing" = $file.Fullname
}
$NewItem = New-Object PSObject -Property $hash
Export-Csv $CsvFile -inputobject $NewItem -append -Force
}
foreach ($file in $SubtmittalList)
{
$hash=#{
"Submittal" = $file.Fullname
}
$NewItem = New-Object PSObject -Property $hash
Export-Csv $CsvFile -inputobject $NewItem -append -Force
}
foreach ($file in $DatabaseList)
{
$hash=#{
"Database" = $file.Fullname
}
$NewItem = New-Object PSObject -Property $hash
Export-Csv $CsvFile -inputobject $NewItem -append -Force
}
foreach ($file in $EstimateList)
{
$hash=#{
"Estimate" = $file.Fullname
}
$NewItem = New-Object PSObject -Property $hash
Export-Csv $CsvFile -inputobject $NewItem -append -Force
}
foreach ($file in $SequenceList)
{
$hash=#{
"Sequence" = $file.Fullname
}
$NewItem = New-Object PSObject -Property $hash
Export-Csv $CsvFile -inputobject $NewItem -append -Force
}
The PowerShell version being used is 5.1. Windows 10 OS.
Could someone help me understand how to append on the same row but a different column without erasing another column's existing row of data? Would this be something that could be done with splatting or looking at each variable ${named}List?
If I understand the question properly, you have 6 arrays of '$file' items that need to be combined into a CSV file.
Then instead of using 6 foreach loops, just use one indexed loop and create Objects from the various lists
If all lists have the same number of items:
$result = for ($i = 0; $i -lt $ScopeList.Count; $i++) {
[PsCustomObject]#{
Scope = $ScopeList[$i].FullName
Drawing = $DrawingList[$i].FullName
Submittal = $SubtmittalList[$i].FullName
Database = $DatabaseList[$i].FullName
Estimate = $EstimateList[$i].FullName
Sequence = $SequenceList[$i].FullName
}
}
# now save the result as CSV file
$result | Export-Csv -Path 'X:\Path\to\TheResult.csv' -NoTypeInformation
If the lists are not all of the same length, you need to do some extra work:
# get the maximum number of items of your lists
$maxItems = (($ScopeList, $DrawingList, $SubtmittalList, $DatabaseList, $EstimateList, $SequenceList) |
Measure-Object -Property Count -Maximum).Maximum
# loop over the maximum number of items and check each list
# if the index $i does not exceed the max number of items for that list
$result = for ($i = 0; $i -lt $maxItems; $i++) {
[PsCustomObject]#{
Scope = if ($i -lt $ScopeList.Count) {$ScopeList[$i].FullName} else { $null}
Drawing = if ($i -lt $DrawingList.Count) {$DrawingList[$i].FullName} else { $null}
Submittal = if ($i -lt $SubtmittalList.Count) {$SubtmittalList[$i].FullName} else { $null}
Database = if ($i -lt $DatabaseList.Count) {$DatabaseList[$i].FullName} else { $null}
Estimate = if ($i -lt $EstimateList.Count) {$EstimateList[$i].FullName} else { $null}
Sequence = if ($i -lt $SequenceList.Count) {$SequenceList[$i].FullName} else { $null}
}
}
# now save the result as CSV file
$result | Export-Csv -Path 'X:\Path\to\TheResult.csv' -NoTypeInformation
Like Theo posted, this mostly worked or brought the intentions closer to the goal.
within each foreach loop that created their independent lists, another list was created for filtering the list and used in the below section from Theo.
# loop over the maximum number of items and check each list
# if the index $i does not exceed the max number of items for that list
$result = for ($i = 0; $i -lt $maxItems; $i++) {
[PsCustomObject]#{
Scope = if ($i -lt $ScopeList.Count) {$ScopeList[$i].FullName} else { $null}
Drawing = if ($i -lt $DrawingList.Count) {$DrawingList[$i].FullName} else { $null}
Submittal = if ($i -lt $SubtmittalList.Count) {$SubtmittalList[$i].FullName} else { $null}
Database = if ($i -lt $DatabaseList.Count) {$DatabaseList[$i].FullName} else { $null}
Estimate = if ($i -lt $EstimateList.Count) {$EstimateList[$i].FullName} else { $null}
Sequence = if ($i -lt $SequenceList.Count) {$SequenceList[$i].FullName} else { $null}
}
}
# now save the result as CSV file
$result | Export-Csv -Path 'X:\Path\to\TheResult.csv' -NoTypeInformation
To get the $maxItems, an alternative way was to have a default value of 1 unless a user input has a keyword "list" when they are prompted for other info, which is another challenge on how to limit values to greater than or equal to 1...
Related
I have this first CSV:
Server,Info
server1,item1
server1,item1
and this 2nd CSV:
Server,Info
server2,item2
server2,item2
And I am trying to get this output:
Server,Server,Info,Info
server1,server2,item1,item2
server1,server2,item1,item2
As you see, the problem is that the headers of the 2 CSV have the same names, which cause a problem if I parse them into objects and loop over keys.
So I am trying to merge them then reordering them as strings, but my mind can't figure how to do it in the last for loop:
$file1 = Get-Content ".\Powershell test\A.csv"
$file2 = Get-Content ".\Powershell test\B.csv"
$content = for ($i = 0; $i -lt $file1.Length; $i++) {
'{0},{1}' -f $file1[$i].Trim(), $file2[$i].Trim()
}
$content | Out-File ".\Powershell test\merged.csv"
$firstFileParsed = Import-Csv -Path ".\Powershell test\B.csv"
$secondFileParsed = Import-Csv -Path ".\Powershell test\B.csv"
$secondFilePath = ".\Powershell test\B.csv"
$contentOf2ndFile = Get-Content $secondFilePath
$csvColumnNames = (Get-Content '.\Powershell test\B.csv' |
Select-Object -First 1).Split(",")
$newColumns = #()
foreach($header in $csvColumnNames) {
$newColumns += $header
}
$newColumns = $newColumns -join ","
$contentOf2ndFile[0] = $newColumns
$contentOf2ndFile | Out-File ".\Powershell test\temp.csv"
$tempObject = Import-Csv -Path ".\Powershell test\temp.csv"
$tempFile = Get-Content ".\Powershell test\temp.csv"
$array = #()
$tempArr = #()
for ($i = 0; $i -lt $file1.Length; $i++) {
$tempArr1 = $file1[$i] -split ","
$tempArr2 = $tempFile[$i] -split ","
for ($j = 0; $j -lt $tempArr1.Length; $j++) {
$tempArr += $tempArr1[$j] + "," + $tempArr2[$j]
$tempArr
}
$array += $tempArr
}
$array | Out-File '.\Powershell test\merged.csv'
What you suggest is not very useful or even valid CSV. IMHO only two results would make sense:
This:
Server1,Info1,Server2,Info2
server1,item1,server2,item2
server1,item1,server2,item2
Or this:
Server,Info
server1,item1
server1,item1
server2,item2
server2,item2
First approach:
$csv1 = Import-Csv ".\Powershell test\A.csv"
$csv2 = Import-Csv ".\Powershell test\B.csv"
$merged = for($i = 0; $i -lt $csv1.Count; $i++) {
$new = new-object psobject
$entry1 = $csv1[$i]
$entry1 | Get-Member -Type NoteProperty | foreach {
Add-Member -InputObject $new -MemberType NoteProperty -Name ($_.Name + "1") -Value $entry1.($_.Name)
}
$entry2 = $csv2[$i]
$entry2 | Get-Member -Type NoteProperty | foreach {
Add-Member -InputObject $new -MemberType NoteProperty -Name ($_.Name + "2") -Value $entry2.($_.Name)
}
$new
}
$merged | Export-Csv ".\Powershell test\merged.csv"
Second approach:
$csv1 = Import-Csv ".\Powershell test\A.csv"
$csv2 = Import-Csv ".\Powershell test\B.csv"
$merged = $csv1 + $csv2
$merged | Export-Csv ".\Powershell test\merged.csv"
UPDATE
If you want exactly your output (and the files are certain to have the same headers and line count), you could use unique headers first, and then simply rename them later:
$csv1 = Import-Csv ".\Powershell test\A.csv"
$csv2 = Import-Csv ".\Powershell test\B.csv"
$merged = for($i = 0; $i -lt $csv1.Count; $i++) {
$new = New-Object PSObject
("Server", "Info") | foreach {
Add-Member -InputObject $new -MemberType NoteProperty -Name ($_ + "1") -Value $csv1[$i].$_
Add-Member -InputObject $new -MemberType NoteProperty -Name ($_ + "2") -Value $csv2[$i].$_
}
$new
}
$header = $true
$merged | ConvertTo-Csv -NoTypeInformation | foreach {
if ($header) {
$header = $false
# remove the numbers from the headers
$_ -replace "\d", ""
}
else { $_ }
} | Out-File ".\Powershell test\merged.csv"
Explanations:
Count is available in Powershell for all collections, and safer than Length which is a property of arrays only. But in this case, both should work.
In the loop, a new empty object is created (with New-Object) and then populated by adding the members of the parsed CSV objects (with Add-Member). A counter is added to the property names to make them unique.
The collection of these objects ($merged) is then converted to CSV, the numbers in the header line removed, and everything saved to file.
As it appears that there several used cases to discern unrelated property keys instead of merging them, I have added a new feature. The -Unify (formally/alias -Mergeparameter) to the Join-Object cmdlet, now accepts a one or two dynamic keys to distinguish unrelated column pairs in a join.
The -Unify (alias-Merge) parameter defines how to unify the left and
right object with respect to the unrelated common properties. The
common properties can discerned (<String>[,<String>]) or merged
(<ScriptBlock>). By default the unrelated common properties wil be
merged using the expression: {$LeftOrVoid.$_, $RightOrVoid.$_}
<String>[,<String>]
If the value is not a ScriptBlock, it is presumed
a string array with one or two items defining the left and right key
format. If the item includes an asterisks (*), the asterisks will be
replaced with the property name otherwise the item will be used to
prefix the property name.
Note: A consecutive number will be automatically added to a common
property name if is already used.
...
Example:
$Csv1 = ConvertFrom-Csv 'Server,Info
server1,item1
server1,item1'
$Csv2 = ConvertFrom-Csv 'Server,Info
server2,item2
server2,item2'
$Csv1 | Join $Csv2 -Unify *1, *2
Result:
Server1 Server2 Info1 Info2
------- ------- ----- -----
server1 server2 item1 item2
server1 server2 item1 item2
I have this CSV file that I kind of do a lot to. My most recent task is to add a summary sheet.
With that said I have a CSV file I pull from a website and send through lot of checks. Code Below:
$Dups = import-csv 'C:\Working\cylrpt.csv' | Group-Object -Property 'Device Name'| Where-Object {$_.count -ge 2} | ForEach-Object {$_.Group} | Select #{Name="Device Name"; Expression={$_."Device Name"}},#{Name="MAC"; Expression={$_."Mac Addresses"}},Zones,#{Name="Agent"; Expression={$_."Agent Version"}},#{Name="Status"; Expression={$_."Is online"}}
$Dups | Export-CSV $working\temp\01-Duplicates.csv -NoTypeInformation
$csvtmp = Import-CSV $working\cylrpt.csv | Select #{N='Device';E={$_."Device Name"}},#{N='OS';E={$_."OS Version"}},Zones,#{N='Agent';E={$_."Agent Version"}},#{N='Active';E={$_."Is Online"}},#{N='Checkin';E={[DateTime]$_."Online Date"}},#{N='Checked';E={[DateTime]$_."Offline Date"}},Policy
$csvtmp | %{
if ($_.Zones -eq ""){$_.Zones = "Unzoned"}
}
$csvtmp | Export-Csv $working\cy.csv -NoTypeInformation
import-csv $working\cy.csv | Select Device,policy,OS,Zones,Agent,Active,Checkin,Checked | % {
$_ | Export-CSV -path $working\temp\$($_.Zones).csv -notypeinformation -Append
}
The first check is for duplicates, I used separate lines of code for this because I wanted to create a CSV for duplicates.
The second check backfills all blank cells under the Zones column with "UnZoned"
The third thing is does is goes through the entire CSV file and creates a CSV file for each Zone
So this is my base. I need to add another CSV file for a Summary of the Zone information. The Zones are in the format of XXX-WS or XXX-SRV, where XXX can be between 3 and 17 letters.
I would like the Summary sheet to look like this
ABC ###
ABC-WS ##
ABC-SRV ##
DEF ###
DEF-WS ##
DEF-SRV ##
My thoughts are to either do the count from the original CSV file or to count the number of lines in each CSV file and subtract 1, for the header row.
Now the Zones are dynamic so I can't just say I want ZONE XYZ, because that zone may not exist.
So what I need is to be able to either count the like zone type in the original file and either output that to an array or file, that would be my preferred method to give the number of items with the same zone name. I just don't know how to write it to look for and count matching variables. Here is the code I'm trying to use to get the count:
import-csv C:\Working\cylrpt.csv | Group-Object -Property 'Zones'| ForEach-Object {$_.Group} | Select #{N='Device';E={$_."Device Name"}},Zones | % {
$Znum = ($_.Zones).Count
If ($Znum -eq $null) {
$Znum = 1
} else {
$Znum++
}
}
$Count = ($_.Zones),$Znum | Out-file C:\Working\Temp\test2.csv -Append
Here is the full code minus the report key:
$cylURL = "https://protect.cylance.com/Reports/ThreatDataReportV1/devices/"
$working = "C:\Working"
Remove-item -literalpath "\\?\C:\Working\Cylance Report.xlsx"
Invoke-WebRequest -Uri $cylURL -outfile $working\cylrpt.csv
$Dups = import-csv 'C:\Working\cylrpt.csv' | Group-Object -Property 'Device Name'| Where-Object {$_.count -ge 2} | ForEach-Object {$_.Group} | Select #{Name="Device Name"; Expression={$_."Device Name"}},#{Name="MAC"; Expression={$_."Mac Addresses"}},Zones,#{Name="Agent"; Expression={$_."Agent Version"}},#{Name="Status"; Expression={$_."Is online"}}
$Dups | Export-CSV $working\temp\01-Duplicates.csv -NoTypeInformation
$csvtmp = Import-CSV $working\cylrpt.csv | Select #{N='Device';E={$_."Device Name"}},#{N='OS';E={$_."OS Version"}},Zones,#{N='Agent';E={$_."Agent Version"}},#{N='Active';E={$_."Is Online"}},#{N='Checkin';E={[DateTime]$_."Online Date"}},#{N='Checked';E={[DateTime]$_."Offline Date"}},Policy
$csvtmp | %{
if ($_.Zones -eq ""){$_.Zones = "Unzoned"}
}
$csvtmp | Export-Csv $working\cy.csv -NoTypeInformation
import-csv $working\cy.csv | Select Device,policy,OS,Zones,Agent,Active,Checkin,Checked | % {
$_ | Export-CSV -path $working\temp\$($_.Zones).csv -notypeinformation -Append
}
cd $working\temp;
Rename-Item "Unzoned.csv" -NewName "02-Unzoned.csv"
Rename-Item "Systems-Removal.csv" -NewName "03-Systems-Removal.csv"
$CSVFiles = Get-ChildItem -path $working\temp -filter *.csv
$Excel = "$working\Cylance Report.xlsx"
$Num = $CSVFiles.Count
Write-Host "Found the following Files: ($Num)"
ForEach ($csv in $CSVFiles) {
Write-host "Merging $CSVFiles.Name"
}
$EXc1 = New-Object -ComObject Excel.Application
$Exc1.SheetsInNewWorkBook = $CSVFiles.Count
$XLS = $EXc1.Workbooks.Add()
$Sht = 1
ForEach ($csv in $CSVFiles) {
$Row = 1
$Column = 1
$WorkSHT = $XLS.WorkSheets.Item($Sht)
$WorkSHT.Name = $csv.Name -Replace ".csv",""
$File = (Get-Content $csv)
ForEach ($line in $File) {
$LineContents = $line -split ',(?!\s*\w+")'
ForEach ($Cell in $LineContents) {
$WorkSHT.Cells.Item($Row,$Column) = $Cell -Replace '"',''
$Column++
}
$Column = 1
$Row++
}
$Sht++
}
$Output = $Excel
$XLS.SaveAs($Output)
$EXc1.Quit()
Remove-Item *.csv
cd ..\
Found the solution
$Zcount = import-csv C:\Working\cylrpt.csv | where Zones -ne "$null" | select #{N='Device';E={$_."Device Name"}},Zones | group Zones | Select Name,Count
$Zcount | Export-Csv -path C:\Working\Temp\01-Summary.csv -NoTypeInformation
I am fairly new in powershell scripting and need help on the following output in a csv format. I am trying to select a column e.g. ACCOUNT.UPLOAD and make and if/else statement to output it in another csv file. May someone help please.
Output csv should look like below:
$results = Import-Csv 'C:\Users\test\Desktop\Test\customer.csv' |
Select-Object "ACCOUNT.UPLOAD"
ForEach ($row in $results)
{
If ($row.Type0 -ne 'CP000101', 'CP000102')
{
$row."ACCOUNT.UPLOAD" = "$($row.ACCOUNT.UPLOAD)"
Write-Host $row."ACCOUNT.UPLOAD"
}
}
$results | Export-Csv C:\Users\test\Desktop\Test\test.csv -NoTypeInformation
Thank you
This will get you what you need. Added comments to explain what I have done.
$results = Import-Csv "C:\Users\test\Desktop\Test\customer.csv" | Select-Object "ACCOUNT.UPLOAD"
# Created array to be able to add individual results from foreach
$TheCSV = #()
ForEach ($row in $results) {
# You can use a -ne in the right hand side, if done like this.
If (($row.'ACCOUNT.UPLOAD' -ne 'CP000101') -and $row.'ACCOUNT.UPLOAD' -ne 'CP000102') {
# Adds the ROW column to the entry and finds the index that it was in from $results.
# Did a +2 as it does not include the header and it starts at value 0. So to match it up with the actual excel row numbers, add 2.
$row | Add-Member -Name 'ROW' -type NoteProperty -Value "$([array]::IndexOf($results.'ACCOUNT.UPLOAD',$row.'ACCOUNT.UPLOAD')+2)"
$TheCSV += $row
}
}
$TheCSV | Export-Csv "C:\Users\test\Desktop\Test\test.csv" -NoTypeInformation
Do it PowerShell way:
param(
[Parameter(Position=0)]
$InputFile = 'D:\\Temp\\Data.csv',
[Parameter(Position=1)]
$OutputFile = 'D:\\Temp\\Output.csv'
)
Import-Csv $InputFile |
Select-Object "ACCOUNT.UPLOAD" |
%{
$lineno++
if ($_.'ACCOUNT.UPLOAD' -notin #('CP000101', 'CP000102')) {
$_ | Add-Member -Name 'ROW' -type NoteProperty -Value $lineno
$_ # Output to pipeline
}
} -Begin { $lineno = 1 } |
Export-Csv $OutputFile -NoTypeInformation
Using:
.\Script.ps1
.\Script.ps1 inputfilename.csv outputfilefname.csv
Just want to ask how to Export the data in existing CSV file in new column. I have this following code.
$Ex=Compare-Object $ImportWin7 $Importafipd1 -includeequal
$Ex | Select-Object SideIndicator | Export-Csv -Append -Force -NoTypeInformation "C:\NotBackedUp\EndpointAudit\Win7machinetest2.csv"
but it appears that the Data from $EX was appended not on the first row of the column.
You can't just add an additional column to existing csv document. You would want to recreate it again with number of columns you need. Like so:
$ExistingCSV = Import-Csv "C:\NotBackedUp\EndpointAudit\Win7machinetest2.csv"
$Ex=Compare-Object $ImportWin7 $Importafipd1 -includeequal | Select-Object SideIndicator
$obj = #()
$i = 0
foreach ($row in $ExistingCSV)
{
$item = New-Object PSObject -ArgumentList $row
$item | Add-Member -MemberType NoteProperty extra_column -Value $Ex[$i]
$obj += $item
$i++
}
$obj | Export-Csv -Append -Force -NoTypeInformation "C:\NotBackedUp\EndpointAudit\Win7machinetest2.csv"
I have written a script that tries to determine the max no. of character for each column. This is what I wrote:
$path = 'folder path'
$file = Get-ChildItem $path\*
$FileContent = foreach ($files in $file) {
$FileHeader = #( (Get-Content $files -First 1).Split($delimiter) )
$importcsv = #( Import-Csv $files -Delimiter "$delimiter" )
for ($i=0; $i -lt $FileHeader.Length; $i++) {
#select each column
$Column = #( $importcsv | select $FileHeader[$i] )
#find the max no. of character
$MaxChar = #(($Column[$i] |
Select -ExpandProperty $FileHeader[$i] |
Measure-Object -Maximum -Property Length).Maximum)
$output = New-Object PSObject
$output | Add-Member NoteProperty FullName ($files.FullName)
$output | Add-Member NoteProperty FileName ($files.Name)
$output | Add-Member NoteProperty Time (Get-Date -Format s)
$output | Add-Member NoteProperty FileHeader ($($FileHeader[$i]))
$output | Add-Member NoteProperty MaxCharacter ($($MaxChar[$i]))
Write-Output $output
}
}
The script above is just part of it, so $delimiter is already defined. And finally I will export the result as CSV.
The script runs without any error, but when I open the file it only gives me the first column/header the max no. of character, and the rest of column/header are missing.
The perfect result will be showing each column/header the max no. of character.
Is something wrong with my loop?
my boss is trying to create an automate process to finding all the information from the raw data and use those information to upload to the database, so part of the script that is missing is about determine the delimiter of the raw file, the $CleanHeader is clean version of $FileHeader (remove all special characters, turn capital letters to small letters), those cleanheaders will be use for headers in the table in the database. and he also want to know the maximum character in each column, so that info can use them in creating the size of the column in the table in the database (he knows this part can be done in sql), but he ask me whether it can be done in PowerShell or not.
This should work:
$ht = #{}
# import a CSV and iterate over its rows
Import-Csv $f.FullName -Delimiter "$delimiter" | ForEach-Object {
# iterate over the columns of each row
$_.PSObject.Properties | ForEach-Object {
# update the hashtable if the length of the current column value is greater
# than the value in the hashtable
if ($_.Value.Length -gt $ht[$_.Name]) {
$ht[$_.Name] = $_.Value.Length
}
}
}
# create an object for each key in the hashtable
$date = Get-Date -Format s
$ht.Keys | ForEach-Object {
New-Object -Type PSObject -Property #{
FullName = $f.FullName
Name = $f.Name
Time = $date
FileHeader = $_
MaxCharacter = $ht[$_]
}
}
FileHeader[$i] was returning the column name with quotes : "ColumnName" instead of ColumnName
To fix, just add a trim to the line where you pull the header :
$FileHeader = #( (Get-Content $files -First 1).Split($delimiter).trim('"') )