Getting only a repeating files from directory and subdirectories - powershell

I'm trying to do script for finding non-unique files.
The script should take one .csv file with data: name of files, LastWriteTime and Length. Then I try to make another .csv based on that one, which will contain only those objects whose combination of Name+Length+LastWriteTime is NON-unique.
I tried following script which uses $csvfile containing files list:
$csvdata = Import-Csv -Path $csvfile -Delimiter '|'
$csvdata |
Group-Object -Property Name, LastWriteTime, Length |
Where-Object -FilterScript { $_.Count -gt 1 } |
Select-Object -ExpandProperty Group -Unique |
Export-Csv $csvfile2 -Delimiter '|' -NoTypeInformation -Encoding Unicode
$csvfile was created by:
{
Get-ChildItem -Path $mainFolderPath -Recurse -File |
Sort-Object $sortMode |
Select-Object Name, LastWriteTime, Length, Directory |
Export-Csv $csvfile -Delimiter '|' -NoTypeInformation -Encoding Unicode
}
(Get-Content $csvfile) |
ForEach-Object { $_ -replace '"' } |
Out-File $csvfile -Encoding Unicode
But somehow in another $csvfile2 there is only the one (first) non-unique record. Does anyone have an idea how to improve it so it can list all non-unique records?

You need to use -Property * -Unique to get a list of unique objects. However, you cannot use -Property and -ExpandProperty at the same time here, because you want the latter parameter to apply to the input objects ($_) and the former parameter to apply to an already expanded property of those input objects ($_.Group).
Expand the property Group first, then select the unique objects:
... |
Select-Object -ExpandProperty Group |
Select-Object -Property * -Unique |
...

Related

Combining output to CSV from two Powershell commands

I am trying to take an existing CSV file and append it with the output from another command. Whenever I run the script, it sets all values in the column to the $owner.
I'm not sure what I'm missing, but it is not working properly.
`Connect-MicrosoftTeams
#get list of teams
$TeamsFile = Get-Team | Select DisplayName, GroupID, Description | Export-CSV -Path "C:\Scripts\Microsoft_Teams_List.csv" -Delimiter "," -NoTypeInformation
#read row in CSV, add owner to new column.
ForEach ($Team in $TeamsFile) {
$Owner = Get-TeamUser -GroupId $Team.GroupID -Role Owner | Select Name
Write-Host $Team.GroupID " Owner: " $Owner.name
$TeamsFile = Import-CSV -Path "C:\Scripts\Microsoft_Teams_List.csv"
$TeamsFile | Select-Object -Property *, #{label = 'Team Owner'; expression = {$Owner.name}}
}
$TeamsFile | Export-CSV -Path "C:\Scripts\Microsoft_Teams_List.csv" -Delimiter "," -NoTypeInformation
`
Sample of existint CSV
If you only need a merged output of two commands, you may make use of calculated properties:
Get-Team |
Select DisplayName,GroupID,Description,
#{n='Team Owner';e={(Get-TeamUser -GroupId $_.GroupID -Role Owner).Name}}
If you already have a CSV file of Team data and need just the owner, you may update your CSV row objects with a new property using a calculated property also:
(Import-CSV -LiteralPath 'C:\Scripts\Microsoft_Teams_List.csv' |
Select *,#{n='Team Owner';e={(Get-TeamUser -GroupId $_.GroupID -Role Owner).Name}}) |
Export-Csv -LiteralPath 'C:\Scripts\Microsoft_Teams_List.csv' -NoTypeInformation
In your attempt, the following code does not yield the results you think:
$TeamsFile = Get-Team | Select DisplayName, GroupID, Description | Export-CSV -Path "C:\Scripts\Microsoft_Teams_List.csv" -Delimiter "," -NoTypeInformation
$TeamsFile won't contain any output because all output was sent down the pipeline to Export-Csv. If you want to capture output before it is sent to Export-Csv, you can use the common parameter -OutVariable.
Get-Team | Select DisplayName, GroupID, Description -OutVariable TeamsFile |
Export-CSV -Path "C:\Scripts\Microsoft_Teams_List.csv" -Delimiter "," -NoTypeInformation
$TeamsFile # now contains your Teams
If headers are the same, then you can keep source csv files in a directory and use this command to generate a combined csv.
Get-ChildItem -Filter *.csv | Select-Object -ExpandProperty FullName | Import-Csv | Export-Csv .\combinedreport.csv -NoTypeInformation

export csv rows where duplicate values found in column

I have a csv file where I am trying to export rows into another csv file only where the values in the id column have duplicates.
I have the following csv file...
"id","blablah"
"valOne","valTwo"
"valOne","asdfdsa"
"valThree","valFour"
"valFive","valSix"
"valFive","qwreweq"
"valSeven","valEight"
I need the output csv file to look like the following...
"valOne","valTwo"
"valOne","asdfdsa"
"valFive","valSix"
"valFive","qwreweq"
Here is the code I have so far:
$inputCsv = Import-CSV './test.csv' -delimiter ","
#$output = #()
$inputCsv | Group-Object -prop id, blablah | Where-Object {$_.id -gt 1} |
Select-Object
##{n='id';e={$_.Group[0].id}},
##{n='blablah';e={$_.Group[0].blablah}}
#Export-Csv 'C:\scripts\powershell\output.csv' -NoTypeInformation
#Write-Host $output
#$output | Export-Csv 'C:\scripts\powershell\output.csv' -NoTypeInformation
I've searched multiple how-to's but can't seem to find the write syntax. Can anyone help with this?
Just group on the ID property and if there is more than 1 count in the group then expand those and export.
$inputCsv = Import-CSV './test.csv' -delimiter ","
$inputCsv |
Group-Object -Property ID |
Where-Object count -gt 1 |
Select-Object -ExpandProperty group |
Export-Csv output.csv -NoTypeInformation
output.csv will contain
"id","blablah"
"valOne","valTwo"
"valOne","asdfdsa"
"valFive","valSix"
"valFive","qwreweq"

Use the name of the original file for the output file

I have been able to get this script working but the only thing I cant figure out is to take the original file name and apply it to the output file name with _new added to it. Could you please push me in the write direction?
Import-Csv '\\DESKTOP-QC1GB24\Allpay DD\Processing\*.csv' -Header (1..5|%{"Column$_"}) |
Select-Object Column2,Column3,Column5 -SkipLast 1 |
ConvertTo-Csv -NoTypeInformation |
Select-Object -Skip 1 |
Set-Content -Path "\\DESKTOP-QC1GB24\Allpay DD\Completed\New.CSV"
You can get a list of .csv files before the import and loop through them:
Get-ChildItem -Filter '*.csv' -Path '\\DESKTOP-QC1GB24\Allpay DD\Processing\'|
% {Import-Csv $_.Fullname -Header (1..5|%{"Column$_"}) |
Select-Object Column2,Column3,Column5 -SkipLast 1 |
ConvertTo-Csv -NoTypeInformation |
Select-Object -Skip 1 |
Set-Content -Path "\\DESKTOP-QC1GB24\Allpay DD\Completed\$($_.BaseName)_new.CSV"}
This way $_.Fullname points to each Full Path of an file and $_.BaseName gives you the Name of each file without the extension (in this case .csv). This way you can add the "_new" string during the Set-Content.

Export-Csv adding unwanted header double quotes

I have got a source CSV file (without a header, all columns delimited by a comma) which I am trying split out into separate CSV files based upon the value in the first column and using that column value as the output file name.
Input file:
S00000009,2016,M04 01/07/2016,0.00,0.00,0.00,0.00,0.00,0.00,750.00,0.00,0.00
S00000009,2016,M05 01/08/2016,0.00,0.00,0.00,0.00,0.00,0.00,600.00,0.00,0.00
S00000009,2016,M06 01/09/2016,0.00,0.00,0.00,0.00,0.00,0.00,600.00,0.00,0.00
S00000010,2015,W28 05/10/2015,2275.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
S00000010,2015,W41 04/01/2016,0.00,0.00,0.00,0.00,0.00,0.00,568.75,0.00,0.00
S00000010,2015,W42 11/01/2016,0.00,0.00,0.00,0.00,0.00,0.00,568.75,0.00,0.00
S00000012,2015,W10 01/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
S00000012,2015,W11 08/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
S00000012,2015,W12 15/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
My PowerShell script looks like this:
Import-Csv INPUT_FILE.csv -Header service_id,year,period,cash_exp,cash_inc,cash_def,act_exp,act_inc,act_def,comm_exp,comm_inc,comm_def |
Group-Object -Property "service_id" |
Foreach-Object {
$path = $_.Name + ".csv";
$_.group | Export-Csv -Path $path -NoTypeInformation
}
Output files:
S00000009.csv:
"service_id","year","period","cash_exp","cash_inc","cash_def","act_exp","act_inc","act_def","comm_exp","comm_inc","comm_def"
"S00000009","2016","M04 01/07/2016","0.00","0.00","0.00","0.00","0.00","0.00","750.00","0.00","0.00"
"S00000009","2016","M05 01/08/2016","0.00","0.00","0.00","0.00","0.00","0.00","600.00","0.00","0.00"
"S00000009","2016","M06 01/09/2016","0.00","0.00","0.00","0.00","0.00","0.00","600.00","0.00","0.00"
S00000010.csv:
"service_id","year","period","cash_exp","cash_inc","cash_def","act_exp","act_inc","act_def","comm_exp","comm_inc","comm_def"
"S00000010","2015","W28 05/10/2015","2275.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00"
"S00000010","2015","W41 04/01/2016","0.00","0.00","0.00","0.00","0.00","0.00","568.75","0.00","0.00"
"S00000010","2015","W42 11/01/2016","0.00","0.00","0.00","0.00","0.00","0.00","568.75","0.00","0.00"
It is generating the new files using the header value in column 1 (service_id).
There are 2 problems.
The output CSV file contains a header row which I don't need.
The columns are enclosed with double quotes which I don't need.
First of all the .csv file needs headers and the quote marks as a csv file structure. But if you don't want them then you can go on with a text file or...
$temp = Import-Csv INPUT_FILE.csv -Header service_id,year,period,cash_exp,cash_inc,cash_def,act_exp,act_inc,act_def,comm_exp,comm_inc,comm_def | Group-Object -Property "service_id" |
Foreach-Object {
$path=$_.name+".csv"
$temp0 = $_.group | ConvertTo-Csv -NoTypeInformation | Select-Object -Skip 1
$temp1 = $temp0.replace("""","")
$temp1 > $path
}
But this output is not a "real" csv file.
Hope that helps.
For your particular scenario you could probably use a simpler approach. Read the input file as a plain text file, group the lines by splitting off the first field, then write the groups to output files named after the groups:
Get-Content 'INPUT_FILE.csv' |
Group-Object { $_.Split(',')[0] } |
ForEach-Object { $_.Group | Set-Content ($_.Name + '.csv') }
Another solution,
using no named headers but simply numbers (as they aren't wanted in output anyway)
avoiding unneccessary temporary files.
removing only field delimiting double quotes.
Import-Csv INPUT_FILE.csv -Header (1..12) |
Group-Object -Property "1" | Foreach-Object {
($_.Group | ConvertTo-Csv -NoType | Select-Object -Skip 1).Trim('"') -replace '","',',' |
Set-Content -Path ("{0}.csv" -f $_.Name)
}

Possible to combine .csv where-object filters?

I'm trying to filter a .csv file based on a location column. The column has various location entries and I only need information from the rows that contain certain locations, that information then gets exported out to a separate .csv file. I can get it to work by searching the .csv file multiple times with each location filter, but I haven't had any luck when trying to combine it into 1 search.
What I have now is:
$csv = Import-Csv "${filepath}\temp1.csv"
$csv | Where-Object location -like "co*" | select EmployeeNumber | Export-Csv "${filepath}\disablelist.csv" -NoTypeInformation
$csv | Where-Object location -like "cc*" | select EmployeeNumber | Export-Csv "${filepath}\disablelist.csv" -Append -NoTypeInformation
$csv | Where-Object location -like "dc*" | select EmployeeNumber | Export-Csv "${filepath}\disablelist.csv" -Append -NoTypeInformation
$csv | Where-Object location -like "mf*" | select EmployeeNumber | Export-Csv "${filepath}\disablelist.csv" -Append -NoTypeInformation
What I'd like to have is something like below. I don't get any errors with it, but all I get is a blank .csv file:
$locations = "co*","cc*","dc*","mf*"
$csv = Import-Csv "${filepath}\temp1.csv"
$csv | Where-Object location -like $locations | select EmployeeNumber | Export-Csv "${filepath}\disablelist.csv" -NoTypeInformation
I've been lurking here for a while and I'm usually able to frankenstein a script together from what I find, but I can't seem to find anything on this. Thanks for your help.
You can replace multiple -like tests with a single -match test using an alternating regex:
$csv = Import-Csv "${filepath}\temp1.csv"
$csv | Where-Object {$_.location -match '^(co|cc|dc|mf)'} |
select EmployeeNumber |
Export-Csv "${filepath}\disablelist.csv" -NoTypeInformation
You can build that regex from a string array:
$locations = 'co','cc','dc','mf'
$LocationMatch = '^({0})' -f ($locations -join '|')
$csv = Import-Csv "${filepath}\temp1.csv"
$csv | Where-Object { $_.location -match $LocationMatch } |
select EmployeeNumber |
Export-Csv "${filepath}\disablelist.csv" -NoTypeInformation