Simple Task. Just trying to loop through a folder of .csv files and import them. I'm a bit confused as this does work on other scripts, but I must be missing something very obvious.
$CSVFiles correctly finds 3 files in the folder.
However $csvOutput only contains data from the last file in the folder.
The rest of the script is working fine.
$CSVFolder = 'D:\XXXX\2021_MAY\'
$OutputFile = 'D:\XXXX\2021_MAY\merged_MAY.csv'
$CSVFiles = Get-ChildItem -Path $CSVFolder -Filter *.csv
foreach ($csv in $CSVFiles)
{$csvOutput = Import-Csv $csv.FullName
$group = $csvOutput | Select-Object
'Channel Reference',
'Received Date',
'Currency',
'Payment Method',
'Channel Payment Reference',
'Line Total Excluding Tax',
'Order Tax','Order Total' | Group-Object {$_.'Channel Reference'}
$group | Where Count -gt 1 | Foreach-Object
{
$_.Group[0].'Line Total Excluding Tax' = ($_.Group | Measure-Object 'Line Total Excluding Tax' -Sum).Sum
}
$group | Foreach-Object { $_.Group[0] } | Export-Csv $OutputFile -NoType
}
I've now fixed this
$CSVFolder = 'D:\XXXX\2021_MAY\'
$OutputFile = 'D:\XXXX\2021_MAY\merged_MAY.csv'
$CSVFiles = Get-ChildItem -Path $CSVFolder -Filter *.csv
foreach ($csv in $CSVFiles){
$group = Import-Csv $csv.FullName | Select-Object 'Channel Reference','Received Date','Currency','Payment Method','Channel Payment Reference','Line Total Excluding Tax','Order Tax','Order Total' | Group-Object {$_.'Channel Reference'} |
Foreach-Object {
if ($_.Count -gt 1){
$_.Group[0].'Line Total Excluding Tax' = ($_.Group | Measure-Object 'Line Total Excluding Tax' -Sum).Sum
}
$_.Group[0]
}
$group | Export-Csv $OutputFile -notype -append
}
Related
I am trying to import multiple csv files and output a total score, i don't want to create another csv for the output, below is how the csv is stored. below is csv 1
and this is csv 2
i want to group by Name and total the wins, please see code below that i have tried
get-item -Path "File Path" |
ForEach-Object {
import-csv $_|
Group-Object Name
Select-Object Name, #{ n='Wins'; e={ ($_.Group | Measure-Object Wins -Sum).Sum } }
}
i was hoping for an outcome like below
any help would be awesome
for some reason the current code is showing the below
Its looking better but still not grouping on Name
This will give you the output you are expecting, with the names and total wins for each player.
$csv1 = import-csv "File path of CSV 1"
$csv2 = import-csv "File path of CSV 2"
$allRecords = $csv1 + $csv2
$allRecords | Group-Object Name | Select-Object Name, #{ n='Wins'; e={ ($_.Group | Measure-Object Wins -Sum).Sum } }
the ouptut
Update
With multiple Csv Files
$allRecords = #()
$directory = "Path of the directory containing the CSV files"
$filePaths = Get-ChildItem -Path $directory -Filter "*.csv"
foreach ($filePath in $filePaths) {
$csv = import-csv $filePath
$allRecords += $csv
}
$allRecords | Group-Object Name | Select-Object Name, #{ n='Wins'; e={ ($_.Group | Measure-Object Wins -Sum).Sum } }
If you have a very high number of csv files, you'll find something like this much faster:
$CombinedRecords = Get-ChildItem -Filter *.csv -Path C:\temp | Select-Object -ExpandProperty FullName | Import-Csv
$CombinedRecords | Group-Object Name | Select-Object Name, #{ n='Wins'; e={ ($_.Group | Measure-Object Wins -Sum).Sum } }
It can even be a one-liner:
Get-ChildItem -Filter *.csv -Path C:\temp | Select-Object -ExpandProperty FullName | Import-Csv | Group-Object Name | Select-Object Name, #{ n='Wins'; e={ ($_.Group | Measure-Object Wins -Sum).Sum } }
I want to compare 2 csv files test1 and test2. I want to export a test3 file only if the Department or the country has changed with the changed details. I have the following code but this compares the 2 whole files not just the Country and Department:
Compare-Object -ReferenceObject (Get-Content -Path C:\scripts\test1.csv) -DifferenceObject (Get-Content -Path C:\scripts\test2.csv) -PassThru | Where-Object{ $_.SideIndicator -eq "=>" } | % { $_ -Replace ',', ";"} | Out-File -FilePath C:\scripts\test3.csv
This is test1:
This is test2:
So test3 should be the same as test2. Anyone knows how you can compare the 2 files with just the changes of the country or department
Without using Compare-Object, you can do this like below:
$csv1 = Import-Csv -Path 'C:\scripts\test1.csv'
$csv2 = Import-Csv -Path 'C:\scripts\test2.csv'
$csv3 = foreach ($item in $csv2) {
$compare = $csv1 | Where-Object { $_.email -eq $item.email }
if ($compare.Country -ne $item.Country -or $compare.Department -ne $item.Department) {
# output the object from $csv2 to be collected in the new $csv3
$item
}
}
$csv3 | Export-Csv -Path 'C:\scripts\test3.csv' -Delimiter ';' -NoTypeInformation
If you do want to use Compare-Object, this also works:
$csv1 = Import-Csv -Path 'C:\scripts\test1.csv'
$csv2 = Import-Csv -Path 'C:\scripts\test2.csv'
Compare-Object -ReferenceObject $csv1 -DifferenceObject $csv2 -Property Country,Department -PassThru |
Where-Object{ $_.SideIndicator -eq "=>" } | Select-Object * -ExcludeProperty SideIndicator |
Export-Csv -Path 'C:\scripts\test3.csv' -Delimiter ';' -NoTypeInformation
I have a .csv file which I'm grouping on two properties 'DN', 'SKU' and then performing a sum on another property 'DeliQty'
This works fine and the sum is reflected back to the group.
However I then need to re group just on 'DN' and write out to separate files.
I've tried Select-Object -Expand Group but this reverts to the original contents without the summed lines.
Is there a way to un group preserving the summed lines and then group again?
$CSVFiles = Get-ChildItem -Path C:\Scripts\INVENTORY\ASN\IMPORT\ -Filter *.csv
foreach ($csv in $CSVFiles) {
$group = Import-Csv $csv.FullName | Group-Object DN, SKU
$group | Where Count -gt 1 | ForEach-Object {
$_.Group[0].'DeliQty' = ($_.Group | Measure-Object DeliQty -Sum).Sum
}
}
You may do the following:
$CSVFiles = Get-ChildItem -Path C:\Scripts\INVENTORY\ASN\IMPORT\ -Filter *.csv
foreach ($csv in $CSVFiles) {
$group = Import-Csv $csv.FullName | Group-Object DN, SKU | Foreach-Object {
if ($_.Count -gt 1) {
$_.Group[0].DeliQty = ($_.Group | Measure-Object DeliQty -Sum).Sum
}
$_.Group[0]
}
# outputs summed and single group objects
$group
}
I've figured out how to compare single columns in two files, but I cant figure out how to compare two files, with one column in the first and multiple columns in the second file. Both containing emails.
First file.csv (contains single column with emails)
john#email.com
jack#email.com
jill#email.com
Second file.csv (contains multiple column with emails)
john#email.nl,john#email.eu,john#email.com
jill#email.se,jill#email.com,jill#email.us
By comparing I would like to output, the difference. This would result in.
Output.csv
jack#email.com
Anyone able to help me? :)
Single columns comparison and output difference
#Line extracts emails from list
$SubscribedMails = import-csv .\subscribed.csv | Select-Object -Property email
#Line extracts emails from list
$ValidEmails = import-csv .\users-emails.csv | Select-Object -Property email
$compare = Compare-Object $SubscribedMails $ValidEmails -property email -IncludeEqual | where-object {$_.SideIndicator -eq "<="} | Export-csv .\nonvalid-emails.csv –NoTypeInformation
(Get-Content .\nonvalid-emails.csv) | ForEach-Object { $_ -replace ',"<="' } > .\nonvalid-emails.csv
Since the first file already contains email addresses per column, you can import it right away.
Take the second file and split the strings containing several addresses.
A new array with seperate addresses will be generated.
Judging from your output, you only seek addresses that are within the first csv but not in the second.
Your code could look like this:
$firstFile = Get-Content 'FirstFile.csv'
$secondFile = (Get-Content 'SecondFile.csv').Split(',')
foreach ($item in $firstFile) {
if ($item -notin $secondFile) {
$item | Export-Csv output.csv -Append -NoTypeInformation
}
}
If you want to maintain your code, can you consider a script like:
#Line extracts emails from list
$SubscribedMails = import-csv .\subscribed.csv | Select-Object -Property email
Rename-Item .\users-emails.csv users-emails.csv.bk
Get-Content .\users-emails.csv.bk).replace(',', "`r`n") | Set-Content .\users-emails.csv
#Line extracts emails from list
$ValidEmails = import-csv .\users-emails.csv | Select-Object -Property email
$compare = Compare-Object $SubscribedMails $ValidEmails -property email -IncludeEqual | where-object {$_.SideIndicator -eq "<="} | Export-csv .\nonvalid-emails.csv –NoTypeInformation
(Get-Content .\nonvalid-emails.csv) | ForEach-Object { $_ -replace ',"<="' } > .\nonvalid-emails.csv
Remove-Item .\users-emails.csv
Rename-Item .\users-emails.csv.bk users-emails.csv
or, more simplest
#Line extracts emails from list
$SubscribedMails = import-csv .\subscribed.csv | Select-Object -Property email
Get-Content .\users-emails.csv).replace(',', "`r`n") | Set-Content .\users-emails.csv.bk
#Line extracts emails from list
$ValidEmails = import-csv .\users-emails.csv.bk | Select-Object -Property email
$compare = Compare-Object $SubscribedMails $ValidEmails -property email -IncludeEqual | where-object {$_.SideIndicator -eq "<="} | Export-csv .\nonvalid-emails.csv –NoTypeInformation
(Get-Content .\nonvalid-emails.csv) | ForEach-Object { $_ -replace ',"<="' } > .\nonvalid-emails.csv
Remove-Item .\users-emails.csv.bk
None of the suggestions so far works :(
Still hoping :)
Will delete comment when happy :p
Can you try this?
$One = (Get-Content .\FirstFile.csv).Split(',')
$Two = (Get-Content .\SecondFile.csv).Split(',')
$CsvPath = '.\Output.csv'
$Diff = #()
(Compare-Object ($One | Sort-Object) ($two | Sort-Object)| `
Where-Object {$_.SideIndicator -eq '<='}).inputobject | `
ForEach-Object {$Diff += New-Object PSObject -Property #{email=$_}}
$Diff | Export-Csv -Path $CsvPath -NoTypeInformation
Output.csv will contain entries that exist in FirstFile but not SecondFIle.
I have a CSV file (file1) that looks like: (User dirs and the size)
Initials,Size
User1,10
User2,100
User3,131
User4,140
I have another CSV file (file2) that looks like: (VIP users)
User2
User4
Now what I'm trying to do, is to update file1, so it looks like:
User1,10
User3,131
User2 and User4 is removed because they are in file2
I can get them removed, but at the same time I remove the size for all users, so my output is only the Users:
User1
User3
My code:
$SourcePath = "\\server1\info\SYSINFO\UsrSize"
$DestinationFile = "\\server1\info\SYSINFO\UsrSize\OverLimit\UsersOverLimit1.log"
$VIP_Exclusion_List = "\\server1\info\SYSINFO\UsrSize\OverLimit\_VIP_EXCLUSION_LIST.txt"
$Database = "\\server1\info\SYSINFO\UsrSize\OverLimit\_UsersOverLimitDATABASE.log"
$INT_SizeToLookFor = 100
dir $SourcePath -Filter usr*.txt | import-csv -delimiter "`t" |
Where-Object {[INT] $_."Size excl. Backup/Pst" -ge $INT_SizeToLookFor} |
Select-Object Initials,"Size excl. Backup/Pst" | convertto-csv -NoTypeInformation | % { $_ -replace '"', ""} | out-file $DestinationFile ;
$Userlist = import-csv $DestinationFile | Select-Object Initials |
convertto-csv -NoTypeInformation | % { $_ -replace '"', ""};
compare-object ($Userlist) (get-content $VIP_Exclusion_List) |
select-object inputObject | convertto-csv -NoTypeInformation |
% { $_ -replace '"', ""} | out-file "\\server1\info\SYSINFO\UsrSize\OverLimit\UsersOverLimitThisTime.log";
If the files are small-ish and you don't care too much about performance, then the following would be a trivial way:
$data = Import-Csv file1
$vips = Import-Csv file2
$data = $data | ?{ $vips -notcontains $_.Initials }
$data | Export-Csv file1_new -NoTypeInformation
A faster way would be to add the names to remove to a set, but given the things you're talking about here I doubt you'll get into the range of a few thousand or million users.
I solved it using this code:
$ArrayVIP = get-content $VIP_Exclusion_List
select-string $DestinationFile -pattern $ArrayVIP -notmatch |
select -expand line |
out-file $DestinationFile
Taken from here: Removing lines from a CSV