So I am looking at breaking up a CSV using Powershell. The CSV is delmited by | which isn't a problem, and I am looking to break it up into multiple smaller csvs while retaining the original. The breaks would occur based off of the value in a single column containing one of a list of values.
What I have done so far is to import the csv (delimited by |) and then
foreach($line in $csv) {
if($columnValue -like $target1) {
export-csv filename1.csv -Delimiter `| $line -append)}
elseif($columnValue -like $target2) {
export-csv filename2.csv -Delimiter `| $line -append)}
etc.
However I do not think it is exporting correctly, and I do not want there to be the quotes (and yes I know this is standard but I do not want them) Also I want the header from the original csv to be applied to the child csvs and its not being applied.
sorry if theres a better way to format the code still new here
Here is where I suggest the awesomeness of the Switch cmdlet. It compares something against multiple potential matches, and executes those matches where appropriate.
Switch($csv){
{$_.column -match $target1} {$_ | Export-CSV filename1.csv -append -delimiter '|'}
{$_.column -match $target2} {$_ | Export-CSV filename2.csv -append -delimiter '|'}
{$_.column -match $target3} {$_ | Export-CSV filename3.csv -append -delimiter '|'}
}
$data = import-csv $csvfile
$data | ?{$_.val -eq $criteria1} | export-csv -path "File1.csv"
$data | ?{$_.val -eq $criteria2} | export-csv -path "File2.csv"
Related
Below is one of the file data I have in text file
B97SW | CHANGED | rc=0 >>
Server Name";"SystemFolderPath";"IdenityReference";"FileSystemRights";"Vulnerable
B97SW;C:\Windows\system32;CREATOR OWNER;268435456;No
B97SW;C:\Windows\system32;NT AUTHORITY\SYSTEM;268435456;No
B97SW;C:\Windows\system32;NT AUTHORITY\SYSTEM;Modify, Synchronize;No
........
I am trying to replace ";" with "," and write to csv.
Below is the code I wrote but it is not writing the data in csv.
$FileList = Get-ChildItem -Path "C:\Files"
$props=[ordered]#{
ServerName=''
SystemFolderPath=''
IdenityReference=''
FileSystemRights=''
Vulnerable=''
}
New-Object PsObject -Property $props |
Export-Csv C:\2021.csv -NoTypeInformation
$FinalData = #()
foreach($n_file in $FileList)
{
$FileName = $n_file.FullName
$FileContent = Get-Content -Path $FileName | Select-Object -Skip 2
foreach($line in $FileContent)
{
$line = $line -replace(";",",")
$line | Export-Csv -Path C:\2021.csv -Append -NoTypeInformation -Force
}
}
output I am getting
"ServerName","SystemFolderPath","IdenityReference","FileSystemRights","Vulnerable"
"","","","",""
,,,,
,,,,
Please let me know what is wrong I am doing here.
$line | Export-Csv -Path C:\2021.csv -Append -NoTypeInformation -Force
This doesn't work because Export-Csv expects object(s) with properties, but $line is just a string. You need to parse it into an object first, using ConvertFrom-Csv.
Try this:
$FileList = Get-ChildItem -Path "C:\Files"
foreach($n_file in $FileList)
{
$FileName = $n_file.FullName
Get-Content -Path $FileName |
Select-Object -Skip 2 |
ConvertFrom-Csv -Delimiter ';' -Header ServerName, SystemFolderPath, IdenityReference, FileSystemRights, Vulnerable |
Export-Csv -Path C:\2021.csv -Append -NoTypeInformation -Force
}
As we have skipped the original headers, we have to supply these through the -Header parameter of ConvertFrom-Csv.
Your CSV file is goofed up in two ways. First, there is a line of garbage before the header line. Second, in the header line the semi-colons are surrounded by double quotes. The correct form would be to surround the header names with quotes instead.
Once these format errors are fixed, you can read the csv file with this:
Import-Csv myfile.csv -delimiter ";"
Or if you want to produce a comma delimited csv file, try this:
Import-Csv myfile.csv -delimiter ";" | Export-Csv newfile.csv
The result will be correct but it will have a lot of unnecessary double quotes.
I use powershell to automate extracting of selected data from a CSV file.
My $target_servers also contains two the same server name but it has different data in each rows.
Here is my code:
$target_servers = Get-Content -Path D:\Users\Tools\windows\target_prd_servers.txt
foreach($server in $target_servers) {
Import-Csv $path\Serverlist_Template.csv | Where-Object {$_.Hostname -Like $server} | Export-Csv -Path $path/windows_prd.csv -Append -NoTypeInformation
}
After executing the above code it extracts CSV data based on a TXT file, but my problem is some of the results are duplicated.
I am expecting around 28 results but it gave me around 49.
As commented, -Append is the culprit here and you should check if the newly added records are not already present in the output file:
# read the Hostname column of the target csv file as array to avoid duplicates
$existingHostsNames = #((Import-Csv -Path "$path/windows_prd.csv").Hostname)
$target_servers = Get-Content -Path D:\Users\Tools\windows\target_prd_servers.txt
foreach($server in $target_servers) {
Import-Csv "$path\Serverlist_Template.csv" |
Where-Object {($_.Hostname -eq $server) -and ($existingHostsNames -notcontains $_.HostName)} |
Export-Csv -Path "$path/windows_prd.csv" -Append -NoTypeInformation
}
You can convert your data to array of objects and then use select -Unique, like this:
$target_servers = Get-Content -Path D:\Users\Tools\windows\target_prd_servers.txt
$data = #()
foreach($server in $target_servers) {
$data += Import-Csv $path\Serverlist_Template.csv| Where-Object {$_.Hostname -Like $server}
}
$data | select -Unique | Export-Csv -Path $path/windows_prd.csv -Append -NoTypeInformation
It will work only if duplicated rows have same value in every column. If not, you can pass column names to select which are important for you. For ex.:
$data | select Hostname -Unique | Export-Csv -Path $path/windows_prd.csv -Append -NoTypeInformation
It will give you list of unique hostnames.
I have a CSV file (with headers) filled with assortment data. The file will be updated once every day. I need to find the differences in those files (the old and the new one) and extract them into a separate file.
For instance: in the old file there could be a price of "18,50" and now it's an updated one of "17,90". The script should now extract this row into a new file.
So far, I was able to import both CSV files (via Import-Csv) but my current solution is to compare each row by findstr.
The problems are:
In 9 of 10 cases the strings are too long to compare.
What if a new row will be inserted - I guess the comparison wouldn't work any longer if the row isn't inserted at the end of the file.
My current code is:
foreach ($oldData in (Import-Csv $PSScriptRoot\old.csv -Delimiter ";" -Encoding "default")) {
foreach ($newData in (Import-Csv $PSScriptRoot\new.csv -Delimiter ";" -Encoding "default")) {
findstr.exe /v /c:$oldData $newData > $PSScriptRoot\diff.txt
}
}
Read both files into separate variables and use Compare-Object for the comparison:
$fields = 'idArtikel', 'Preis', ...
$csv1 = Import-Csv $PSScriptRoot\old.csv -Delimiter ';'
$csv2 = Import-Csv $PSScriptRoot\new.csv -Delimiter ';'
Compare-Object -ReferenceObject $csv1 -DifferenceObject $csv2 -Property $fields -PassThru | Where-Object {
$_.SideIndicator -eq '=>'
} | Select-Object $fields | Export-Csv 'C:\path\to\diff.csv' -Delimiter ';'
$csv1 | Join $csv2 idArtikel -Merge {$Right.$_} | Export-CSV 'C:\path\to\diff.csv' -Delimiter ';'
For details on Join (Join-Object), see: https://stackoverflow.com/a/45483110/1701026
As a continuation of a script I'm running, working on the following.
I have a CSV file that has formatted information, example as follows:
File named Import.csv:
Name,email,x,y,z
\I\RS\T\Name1\c\x,email#jksjks,d,f
\I\RS\T\Name2\d\f,email#jsshjs,d,f
...
This file is large.
I also have another file called Note.txt.
Name1
Name2
Name3
...
With help from #mathias-r-jessen
$Dir = PathToFile
$import = Import-Csv $Dir\import.csv
$NoteFile = "$Dir\Note.txt"
$Note = GC $NoteFile
$Import |Where-Object {$Note -contains $_.Name.Split('\')[4]} |Export-Csv "$Dir\Result.csv" -NoTypeInformation -Append
This code quickly and effortlessly parses the big csv and extracts every line that contains any of the lines in the $note file.
My next question is how do i log any lines in the $note file that were not found in the csv file.
I tried the following:
$result = $Import |Where-Object {$Note -contains $_.Name.Split('\')[4]} |Export-Csv "$Dir\Result.csv" -NoTypeInformation -Append
$Note | Where-Object {$result.Name.Split('\')[4] -notcontains $Note} | out-file $dir\not-found.log -append
This seems to return every line in $note.
#mathias-r-jessen any help you can provide would be appreciated.
You could use a Switch to do that.
Switch($Import){
{$Note -contains $_.Name.Split('\')[4]} {$_ | Export-Csv "$Dir\Result.csv" -NoTypeInformation -Append; continue}
default {$_ | Export-csv "$Dir\Not-Found.csv" -NoType -Append}
}
The continue in the first option makes it so that if the first case is a match it performs the relevant action, and then continues to the next record. If the first case doesn't match it moves on to the default action, which outputs it to a different file.
I solved it by using the following:
$result = $Import |Where-Object {$Note -contains $_.Name.Split('\')[4]}
$result | Export-Csv "$Dir\Result.csv" -NoTypeInformation -Append
$matches = $note | where-object { $result.Name -match $_}
compare-object $note $matches |where-object {$_.SideIndicator -like "<=" | select -ExpandProperty InputObject | Out-file "$Dir\Not_found.txt" -Append
Starting with a 500,000 line CSV, I need to split the files by day and hour (the second and third columns). I've tried the modify the group to include the hour and while I see the hour get added to my filename, I get no results in the exported file.
The foreach doing the work:
foreach ($group in $data | Group Day,hour) {
$data | Where-Object { $_.Day -and $_.Hour -eq $group.Name }
ConvertTo-Csv -NoTypeInformation |
foreach {$_.Replace('"','')} |
Out-File "$Path\Testfile_$($group.name -replace $regexA, '').csv"
Sample Data:
Bob,1/27/2012,8:00,Basic,Operations
Charlie,2/3/2012,9:00,Advanced,Production
Bill,3/7/2012,10:00,Advanced,Production
You could import the CSV, determine the output filename on the fly, and append each record to the matchning file:
Import-Csv 'C:\path\to\input.csv' | ForEach-Object {
$filename = ('output_{0}_{1}.csv' -f $_.Day, $_.Hour) -replace '[/:]'
$_ | Export-Csv "C:\path\to\$filename" -Append -NoType
}
Note that Export-Csv -Append requires PowerShell v3 or newer.