How to seperate CSV values within a CSV into new rows in PowerShell - powershell

I'm receiving an automated report from a system that cannot be modified as a CSV. I am using PowerShell to split the CSV into multiple files and parse out the specific data needed. The CSV contains columns that may contain no data, 1 value, or multiple values that are comma separated within the CSV file itself.
Example(UPDATED FOR CLARITY):
"Group","Members"
"Event","362403"
"Risk","324542, 340668, 292196"
"Approval","AA-334454, 344366, 323570, 322827, 360225, 358850, 345935"
"ITS","345935, 358850"
"Services",""
I want the data to have one entry per line like this (UPDATED FOR CLARITY):
"Group","Members"
"Event","362403"
"Risk","324542"
"Risk","340668"
"Risk","292196"
#etc.
I've tried splitting the data and I just get an unknown number of columns at the end.
I tried a foreach loop, but can't seem to get it right (pseudocode below):
Import-CSV $Groups
ForEach ($line in $Groups){
If($_.'Members'.count -gt 1, add-content "$_.Group,$_.Members[2]",)}
I appreciate any help you can provide. I've searched all the stackexchange posts and used Google but haven't been able to find something that addresses this exact issue.

Import-Csv .\input.csv | ForEach-Object {
ForEach ($Member in ($_.Members -Split ',')) {
[PSCustomObject]#{Group = $_.Group; Member = $Member.Trim()}
}
} | Export-Csv .\output.csv -NoTypeInformation

# Get the raw text contents
$CsvContents = Get-Content "\path\to\file.csv"
# Convert it to a table object
$CsvData = ConvertFrom-CSV -InputObject $CsvContents
# Iterate through the records in the table
ForEach ($Record in $CsvData) {
# Create array from the members values at commas & trim whitespace
$Record.Members -Split "," | % {
$MemberCount = $_.Trim()
# Check if the count is greater than 1
if($MemberCount -gt 1) {
# Create our output string
$OutputString = "$($Record.Group), $MemberCount"
# Write our output string to a file
Add-Content -Path "\path\to\output.txt" -Value $OutputString
}
}
}
This should work, you had the right idea but I think you may have been encountering some syntax issues. Let me know if you have questions :)

Revised the code as per your updated question,
$List = Import-Csv "\path\to\input.csv"
foreach ($row in $List) {
$Group = $row.Group
$Members = $row.Members -split ","
# Process for each value in Members
foreach ($MemberValue in $Members) {
# PS v3 and above
$Group + "," + $MemberValue | Export-Csv "\path\to\output.csv" -NoTypeInformation -Append
# PS v2
# $Group + "," + $MemberValue | Out-File "\path\to\output.csv" -Append
}
}

Related

Changing multiple lines in a text file based on a psobject

I'm working on a script which will add some additional informations to a txt file. These informations are stored in a CSV file which looks like this (the data will differs each time the script will launch):
Number;A;B;ValueOfB
FP01340/05/20;0;1;GTU_01,GTU_03
FP01342/05/20;1;0;GTU01
The txt file looks like this (data inside will of course differ each time):
1|1|FP01340/05/20|2020-05-02|2020-05-02|2020-05-02|166,91|203,23|36,32|nothing interesting 18|33333|63-111 somewhere|||||
2|zwol|9,00|9,00|0,00
2|23|157,91|194,23|36,32
1|1|FP01341/05/20|2020-05-02|2020-05-02|2020-05-02|12,19|14,99|2,80|Some info |2222222|blabla|11-111 something||||
2|23|12,19|14,99|2,80
1|1|FP01342/05/20|2020-05-02|2020-05-02|2020-05-02|525,36|589,64|64,28|bla|222222|blba 36||62030|something||
2|5|213,93|224,63|10,70
2|8|120,34|129,97|9,63
2|23|191,09|235,04|43,95
What I need to do is to find a line which contains 'Number' and then add value 'A' and 'B' from a CSV in a form: |0|1 and then on the first line below, at the end, add 'ValueofB' in a form |AAA_01,AAA_03
So the first two lines should look like this at the end:
1|1|FP01340/05/20|2020-05-02|2020-05-02|2020-05-02|166,91|203,23|36,32|nothing interesting 18|33333|63-111 somewhere||||||0|1
2|zwol|9,00|9,00|0,00|AAA_01,AAA_03
2|23|157,91|194,23|36,32
Rest of lines should not be touched.
I made a script which uses select-string method with context to find what I need to - put that into an object and then add to previously found strings what I need to and put that in to an another object.
My script is as follws:
$csvFile = Import-Csv -Path Somepath\file.csv -Delimiter ";"
$file = "Somepath2\SomeName.txt"
$LinesToChange = #()
$script:LinesToChange = $LinesToChange
$LinesOriginal = #()
$script:LinesOriginal = $LinesOriginal
foreach ($line in $csvFile) {
Select-String -Path $file -Pattern "$($Line.number)" -Encoding default -Context 0, 1 | ForEach-Object {
$1 = $_.Line
$2 = $_.Context.PostContext
}
$ListOrg = [pscustomobject]#{
Line_org = $1
Line_GTU_org = $2
}
$LinesOriginal = $LinesOriginal + $ListOrg
$lineNew = $ListOrg.Line_org | foreach { $_ + "|$($line.A)|$($line.B)" }
$GTUNew = $ListOrg.Line_GTU_org | foreach { $_ + "|$($line.ValueofB)" }
$ListNew = [pscustomobject]#{
Line_new = $lineNew
Line_GTU_new = $GTUNew
Line_org = $ListOrg.Line_org
Line_GTU_org = $ListOrg.Line_GTU_org
}
$LinesToChange = $LinesToChange + $ListNew
}
The output is an object $LinesToChange which have original lines and lines after the change. The issue is I have no idea how to use that to change the txt file. I tried few methods and ended up with file which contains updated lines but all others are doubbled (I tried foreach) or PS is using whole RAM and couldn't finish the job :)
My latest idea is to use something like that:
(Get-Content -Path $file) | ForEach-Object {
$line = $_
$LinesToChange.GetEnumerator() | ForEach-Object {
if ($line -match "$($LinesToChange.Line_org)") {
$line = $line -replace "$($LinesToChange.Line_org)", "$($LinesToChange.Line_new)"
}
if ($line -match "$($LinesToChange.Line_GTU_org)") {
$line = $line -replace "$($LinesToChange.Line_GTU_org)", "$($LinesToChange.Line_GTU_new)"
}
}
} | Set-Content -Path Somehere\newfile.txt
It seemed promising at first, but the variable $line contains all lines and as such it can't find the match.
Also I need to be sure that the second line will be directly below the first one (it is unlikely but it can be a case that there will be two or more lines with the same data while the "number" from CSV file is unique) so preferably while changing the txt file it would be needed to find a match for a two-liner; in short:
find this two lines:
1|1|FP01340/05/20|2020-05-02|2020-05-02|2020-05-02|166,91|203,23|36,32|nothing interesting 18|33333|63-111 somewhere|||||
2|zwol|9,00|9,00|0,00
change them to:
1|1|FP01340/05/20|2020-05-02|2020-05-02|2020-05-02|166,91|203,23|36,32|nothing interesting 18|33333|63-111 somewhere||||||0|1
2|zwol|9,00|9,00|0,00|AAA_01,AAA_03
Do that for all lines in a $LinesToChange
Any help will be much appreciated!
Greetings!
Some strange text file you have there, but anyway, this should do it:
# read in the text file as string array
$txt = Get-Content -Path '<PathToTheTextFile>'
$csv = Import-Csv -Path '<PathToTheCSVFile>' -Delimiter ';'
# loop through the items (rows) in the CSV and find matching lines in the text array
foreach ($item in $csv) {
$match = $txt | Select-String -Pattern ('|{0}|' -f $item.Number) -SimpleMatch
if ($match) {
# update the matching text line (array indices count from 0, so we do -1)
$txt[$match.LineNumber -1] += ('|{0}|{1}' -f $item.A, $item.B)
# update the line following
$txt[$match.LineNumber] += ('|{0}' -f $item.ValueOfB)
}
}
# show updated text on screen
$txt
# save updated text to file
$txt | Set-Content -Path 'Somehere\newfile.txt'

Powershell - Export array to CSV in different columns

I am trying to automate below API calls from a csv file.
http_uri
/ModuleName/api/12345/moverequest/MoveRequestQueue?batchSize=200
/ModuleName/api/Portal/GetGarageLocations?email=Dummy#mail.com
/ModuleName/api/DeliveryDate/CommitEta?ref=H7J3M1EA4LF
/ModuleName/api/35345/moverequest/MoveRequestQueue?batchSize=500
The output should be like below in a csv file.
ScenarioName Parameter Value
MoveRequestQueue batchSize 200
GetGarageLocations email Dummy#mail.com
CommitEta ref H7J3M1EA4LF
MoveRequestQueue batchSize 500
I am using below code
$csv = Import-Csv C:\Powershell\Documents\Source.csv
$scenario = #()
ForEach ($row in $csv){
$httpuri = $($row.http_uri)
#Iterating through CSV rows and segregate values
if ($httpuri -match "="){
$equalarr = $httpuri -split '='
if ($equalarr[0] -match "\?"){
$questionarr = $equalarr[0] -split '\?'
$scenarionamearr = $questionarr[0] -split '/'
$totalelements = $scenarionamearr.Count
$scenarioname = $scenarionamearr[$totalelements-1]
$Scenario += $scenarioname
$Scenario += $questionarr[1]
$Scenario += $equalarr[1]
}
}
}
#Adding columns to csv
$columnName = '"Scenario","Parameter","Value"'
Add-Content -Path C:\Powershell\Documents\Output.csv -Value $columnName
#Writing values to CSV
$Scenario | foreach { Add-Content -Path C:\Powershell\Documents\Output.csv -Value $_ }
But Outout is generated like below
Scenario Parameter Value
DequeueMoveRequestQueue
batchSize
200
GetCarrierLocations
email
x-qldanxqldanx
Since i am a newbie, searched a lot to solve this issue but couldn't succeed. Please throw some light on this.
Thanks in advance....
If you store your scenarios in structured objects you can use Powershell's built in Export-Csv command to generate your csv.
So, instead of
$Scenario += $scenarioname
$Scenario += $questionarr[1]
$Scenario += $equalarr[1]
store an array of powershell objects:
$Scenario += [PSCustomObject]#{
"Scenario" = $scenarioname;
"Parameter" = $questionarr[1];
"Value" = $equalarr[1];}
Then, when creating the csv file, just use Export-Csv:
$Scenario | Export-Csv -NoTypeInformation -Path C:\Powershell\Documents\Output.csv
So the issue is that you make an empty array, then add strings to it one at a time, which just makes it an array of strings. Then when you output it to the file it just adds each string to the file on its own line. What you want to do is create an array of objects, then use the Export-Csv cmdlet to output it to a CSV file.
Creating an array, and then adding things to it one at a time is not a good way to do it. PowerShell has to recreate the array each time you add something the way you're doing it. Better would be to have a pipeline that outputs what you want (objects, rather than strings), and capture them all at once creating the array one time. Or even better, just output them to the CSV file and not recollect them in general.
$CSV = Import-Csv C:\Powershell\Documents\Source.csv
$CSV.http_uri -replace '^.*/(.*)$','$1'|ForEach-Object{
$Record = $_ -split '[=\?]'
[PSCustomObject]#{
ScenarioName = $Record[0]
Parameter = $Record[1]
Value = $Record[2]
}
} | Export-Csv -Path C:\Powershell\Documents\Output.csv -Append

How can I count the number of CSV columns when the file has multiline data and no header

My CSV files have no headers and multi line entries like this:
11;"multi line
col12";13;foobar;foobar
21;22;23;24;25
And I'd like to count the number of columns. So 5 in this example. How do I do that?
What I tried:
Import-CSV doesn't work without the header parameter due to duplicate entries on the first line.
(Import-Csv .\bad.csv -Delimiter ";" | get-member -type NoteProperty).count
Adding a header parameter skews the count.
(Import-Csv .\bad.csv -Delimiter ";" -Header (1..99) | get-member -type NoteProperty).count
I had to abort reading the file manually via Get-Content because of all the parsing I would have to handle manually. Escaping characters and multi line entries...
My version of PowerShell is 3 and I have to port my script to version 2 later on.
If you are willing to accept the caveat that this could miscount the number of columns if there are quoted delimiters in string this could be good enough for you.
$path = "c:\temp\test.txt"
$delimiter = ";"
$numberOfColumns = Get-Content $path |
ForEach-Object{($_.split($delimiter)).Count} |
Measure-Object -Maximum |
Select-Object -ExpandProperty Maximum
Import-Csv $path -Header (1..$numberOfColumns) -Delimiter $delimiter
Read in the file with Get-Content and isolate the maximum number of columns by
splitting each line on its delimiter and then using that value to import the CSV. If the file is large you can read in the file once with Get-Content and then use ConvertTo-CSV once you know your column count.
If all lines contain a line break on them the above logic would fail. Still we could temporarily scrub the data by removing the correct line breaks in order to get the accurate count.
$delimiter = ";"
$fileData = (Get-Content $path | Out-String)
$numberOfColumns = ((($fileData -replace "(`"[^;]+?)`r`n",'$1') -split "`r`n" | Select -First 1).split($delimiter)).Count
$fileData | ConvertFrom-Csv -Header (1..$numberOfColumns) -Delimiter $delimiter
What this will do is find lines that end where there is a double quote followed by data that does not contain the delimiter. We also match the newline that follows but drop that same new line in the replacement. If that is done we know that the first line is proper. Use that same line to split and count just like before.
Since Excel knows, let's ask him :
$path = "path\to\bad.csv"
$excel = New-Object -ComObject Excel.Application
$workbook = $excel.Workbooks.Open($path)
$sheet = $workbook.ActiveSheet
$columnIndex = 1
while($sheet.Cells.Item(1, $columnIndex).Text -ne "") {
$columnIndex++
}
"There are $($columnIndex - 1) columns in CSV file $path"
Start-Sleep -Seconds 1
Get-Process excel | Stop-Process -Force
As pointed out by Ansgar Wiechers in comments, there is a much shorter solution :
$path = "path\to\bad.csv"
$excel = New-Object -ComObject Excel.Application
$workbook = $excel.Workbooks.Open($path)
$sheet = $workbook.ActiveSheet
$columnCount = $sheet.UsedRange.Columns.Count
"There are $columnCount columns in CSV file $path"
Start-Sleep -Seconds 1
Get-Process excel | Stop-Process -Force
(I know my way of killing Excel is dirty, but iirc it takes too much code to do so)
I know this is very old, but I came across a similar situation (did not have have rows of varying columns) today and found my own solution so I thought I would share for anyone else coming into this situation. My solution was to use Get-Content for the first row of the CSV and -split on the delimiter (,) to create an array and then return the count of the array. As mentioned in replies above, this will not account for delimiters existing within quotations.
((Get-Content $PathToCsv)[0] -split ",").count
I had the same issue and went with AAgent suggestion.
$CommaCount = ((Get-Content $PathToCsv)[0] -split ",").count
$SemicolonCount = ((Get-Content $PathToCsv)[0] -split ";").count
if ($CommaCount -gt $SemicolonCount){
$CMSlist = Import-Csv ($PathToCsv) –Delimiter “,”
}
else{
$CMSlist = Import-Csv ($PathToCsv) –Delimiter “;”

Powershell - reading ahead and While

I have a text file in the following format:
.....
ENTRY,PartNumber1,,,
FIELD,IntCode,123456
...
FIELD,MFRPartNumber,ABC123,,,
...
FIELD,XPARTNUMBER,ABC123
...
FIELD,InternalPartNumber,3214567
...
ENTRY,PartNumber2,,,
...
...
the ... indicates there is other data between these fields. The ONLY thing I can be certain of is that the field starting with ENTRY is a new set of records. The rows starting with FIELD can be in any order, and not all of them may be present in each group of data.
I need to read in a chunk of data
Search for any field matching the
string ABC123
If ABC123 found, search for the existence of the
InternalPartNumber field & return that row of data.
I have not seen a way to use Get-Content that can read in a variable number of rows as a set & be able to search it.
Here is the code I currently have, which will read a file, searching for a string & replacing it with another. I hope this can be modified to be used in this case.
$ftype = "*.txt"
$fnames = gci -Path $filefolder1 -Filter $ftype -Recurse|% {$_.FullName}
$mfgPartlist = Import-Csv -Path "C:\test\mfrPartList.csv"
foreach ($file in $fnames) {
$contents = Get-Content -Path $file
foreach ($partnbr in $mfgPartlist) {
$oldString = $mfgPartlist.OldValue
$newString = $mfgPartlist.NewValue
if (Select-String -Path $file -SimpleMatch $oldString -Debug -Quiet) {
$stringData = $contents -imatch $oldString
$stringData = $stringData -replace "[\n\r]","|"
foreach ($dataline in $stringData) {
$file +"|"+$stringData+"|"+$oldString+"|"+$newString|Out-File "C:\test\Datachanges.txt" -Width 2000 -Append
}
$contents = $contents -replace $oldString $newString
Set-Content -Path $file -Value $contents
}
}
}
Is there a way to read & search a text file in "chunks" using Powershell? Or to do a Read-ahead & determine what to search?
Assuming your fine isn't too big to read into memory all at once:
$Text = Get-Content testfile.txt -Raw
($Text -split '(?ms)^(?=ENTRY)') |
foreach {
if ($_ -match '(?ms)^FIELD\S+ABC123')
{$_ -replace '(?ms).+(^Field\S+InternalPartNumber.+?$).+','$1'}
}
FIELD,InternalPartNumber,3214567
That reads the entire file in as a single multiline string, and then splits it at the beginning of any line that starts with 'ENTRY'. Then it tests each segment for a FIELD line that contains 'ABC123', and if it does, removes everything except the FIELD line for the InternalPartNumber.
This is not my best work as I have just got back from vacation. You could use a while loop reading the text and set an entry flag to gobble up the text in chunks. However if your files are not too big then you could just read up the text file at once and use regex to split up the chunks and then process accordingly.
$pattern = "ABC123"
$matchedRowToReturn = "InternalPartNumber"
$fileData = Get-Content "d:\temp\test.txt" | Where-Object{$_ -match '^(entry|field)'} | Out-String
$parts = $fileData | Select-String '(?smi)(^Entry).*?(?=^Entry|\Z)' -AllMatches | Select-Object -ExpandProperty Matches | Select-Object -ExpandProperty Value
$parts | Where-Object{$_ -match $pattern} | Select-String "$matchedRowToReturn.*$" | Select-Object -ExpandProperty Matches | Select-Object -ExpandProperty Value
What this will do is read in the text file, drop any lines that are not entry or field related, as one long string and split it up into chunks that start with lines that begin with the work "Entry".
Then we drop those "parts" that do not contain the $pattern. Of the remaining that match extract the InternalPartNumber line and present.

Imported CSV data back to CSV

I am new to Powershell and have a quick question.
The output I am playing with is from the sysinternals.com autorunsc tool in CSV format.
Purpose of the script is to search the CSV for certain binary names.
The import goes as expected but I cannot get it write my filtered info
$IOCFiles = "es.dll", "null.sys"
$data = import-csv "D:\autoruns.csv" -header("Entry_Location","Entry","Enabled","Category","Description","Publisher","Image_Path","Launch_String","MD5","SHA-1","SHA-256")
foreach ($K in $data)
{
foreach($element in $IOCFiles)
{
if ($k.Image_Path -match $element)
{
$hits = $k.Entry + " " + $K.Publisher + " " + $K.Image_Path
$hits | export-csv -Path c:\HITS.csv -NoTypeInformation
}
}
}
The above doesn't generate the actual data in the CSV but info on length of the values.
There are a number of good examples of import/exports but I didn't see one with the columns on import. I have to use the column import method due to something similar I am playing with.
Thank you!
Powershell makes this sort of thing pretty easy:
Import-csv "D:\autoruns.csv" -header("Entry_Location","Entry","Enabled","Category" |
Where-Object{($IOCFiles -match $_.Image_Path) -ne $null} |
Select-Object Entry, Publisher, Image_Path |
Export-CSV -Path c:\HITS.csv -NoTypeInformation
I haven't got the time to test that right now so there may be some issues with it, but that is the gist of it.