Append existing column in csv by matching values with array and with condition - powershell
I will do my best to break this down as simply as I can.
what I have so far that is working:
Currently I have two csv files...
test1.csv
test1ColumnN,test1ColumnI,test1ColumnD,selectDomainOne,selectDomainTwo,selectDomainThree
asdf,asdf,asdf,,,
nValue1,iValue1,dValue1,sValue1,,
qwer,asdf,zxcv,,,
nValue2,iValue2,dValue2,,,
qwer,zxcv,asdf,lkjh,,
nValue3,iValue3,dValue3,sValue3,,
zxcv,qwer,asdf,,poiu,
nValue1,iValue1,dValue1,,sValue1,
nValue4,iValue4,dValue4,,sValue4,
asdf,qwer,zxcv,fghj,mnbv,
nValue5,iValue5,dValue5,,,
asdf,cvbn,erty,,,uytr
nValue7,iValue7,dValue7,,,sValue7
nValue8,iValue8,dValue8,,,sValue8
nValue9,iValue9,dValue9,,,sValue9
qwer,asdf,zxcv,poiu,lkjh,mnbv
test2.csv
DomainCatagories,test2ColumnS,test2ColumnA,test2ColumnN,test2ColumnI,test2ColumnD
DomainOne,sValue1,aValue1,nValue1,,dValueN
DomainOne,sValue2,aValue2,,iValue2,dValue2
DomainOne,sValue3,aValue2,nValue3,iValue3,dValue3
DomainTwo,sValue1,aValue2,,iValue1,dValueN
DomainTwo,sValue4,aValue1,nValue4,,dValueN
DomainTwo,sValue5,aValue1,nValue5,iValue5,dValue5
DomainThree,sValue7,aValue2,nValue7,iValue7,dValue7
DomainThree,sValue8,aValue1,nValue8,iValue8,dValue8
DomainThree,sValue9,aValue2,nValue9,iValue9,dValue9
Now I want to add a column (inside test2.csv) to match the sValue# from both test1.csv and test2.csv with the condition of ($_.DomainCatagories='DomainOne' from test2.csv) and ($_.selectDomainOne from test1.csv)
To do this, I am using the following code...
#Create Column
$domainNameOne = #{}
$domainNameOne = Import-Csv 'C:\Scripts\Tests\test1.csv' | Where-Object {$_.selectDomainOne} | Select-Object -Expand 'selectDomainOne'
(Import-Csv 'C:\Scripts\Tests\test2.csv') |
Select-Object -Property *, #{n='Test1sValues';e={
if($_.DomainCatagories -eq 'DomainOne'){
if(($domainNameOne -contains $_.test2ColumnS) -and ($_.test2ColumnS)){
$_.test2ColumnS
} Else {
'Not found in test1'
}}}} | Export-Csv "C:\Scripts\Tests\test2-Temp" -NoType
Move-Item "C:\Scripts\Tests\test2-Temp" 'C:\Scripts\Tests\test2.csv' -Force
After the code is run, I get the following test2.csv (isCorrect)...
"DomainCatagories","test2ColumnS","test2ColumnA","test2ColumnN","test2ColumnI","test2ColumnD","Test1sValues"
"DomainOne","sValue1","aValue1","nValue1","","dValueN","sValue1"
"DomainOne","sValue2","aValue2","","iValue2","dValue2","Not found in test1"
"DomainOne","sValue3","aValue2","nValue3","iValue3","dValue3","sValue3"
"DomainTwo","sValue1","aValue2","","iValue1","dValueN",""
"DomainTwo","sValue4","aValue1","nValue4","","dValueN",""
"DomainTwo","sValue5","aValue1","nValue5","iValue5","dValue5",""
"DomainThree","sValue7","aValue2","nValue7","iValue7","dValue7",""
"DomainThree","sValue8","aValue1","nValue8","iValue8","dValue8",""
"DomainThree","sValue9","aValue2","nValue9","iValue9","dValue9",""
What I have that is not working:
Next I run the following code...
#Append Column
$domainNameThree = #{}
$domainNameThree = Import-Csv 'C:\Scripts\Tests\test1.csv' | Where-Object {$_.selectDomainThree} | Select-Object -Expand 'selectDomainThree'
(Import-Csv 'C:\Scripts\Tests\test2.csv') | % {
if($_.DomainCatagories -eq 'DomainThree'){
if(($domainNameThree -contains $_.test2ColumnS) -and ($_.test2ColumnS)){
$_.Test1sValues = $_.test2ColumnS
} Else {
$_.Test1sValues = 'Not found in test1'
}}} | Export-Csv "C:\Scripts\Tests\test2-Temp" -NoType
Move-Item "C:\Scripts\Tests\test2-Temp" 'C:\Scripts\Tests\test2.csv' -Force
Instead of adding the values in the correct rows, it completely blanks out the whole file and saves it as an empty file.
End Goal
What I want the code to produce, is this (notice values filled in on last 3 rows in the last column)...
"DomainCatagories","test2ColumnS","test2ColumnA","test2ColumnN","test2ColumnI","test2ColumnD","Test1sValues"
"DomainOne","sValue1","aValue1","nValue1","","dValueN","sValue1"
"DomainOne","sValue2","aValue2","","iValue2","dValue2","Not found in test1"
"DomainOne","sValue3","aValue2","nValue3","iValue3","dValue3","sValue3"
"DomainTwo","sValue1","aValue2","","iValue1","dValueN",""
"DomainTwo","sValue4","aValue1","nValue4","","dValueN",""
"DomainTwo","sValue5","aValue1","nValue5","iValue5","dValue5",""
"DomainThree","sValue7","aValue2","nValue7","iValue7","dValue7","sValue7"
"DomainThree","sValue8","aValue1","nValue8","iValue8","dValue8","sValue8"
"DomainThree","sValue9","aValue2","nValue9","iValue9","dValue9","sValue9"
What am I doing wrong in that 2nd code snippet?
The example you show from What I have that is not working: is missing a key portion. Export-Csv will take everything piped into it to populate the CSV but you are not providing any.
Problem is that you are not passing anything through the pipe. Merely just updating one property. The simplest thing to do is add $_ after the if statement. Or you could just use a calculated property which you have done before in another one of your questions. The example below from Compare dates with different formats in csv file even uses an if statement.
Import-Csv $csvFile | Select-Object *, #{n='MatchDates';e={ if((([datetime]$_.Date1).Date -eq $_.Date3) -and (([datetime]$_.Date2).Date -eq $_.Date3) -and (([datetime]$_.Date1).Date -eq $_.Date2)){ 'Match Found' }Else{ 'No Match Found' }}} |
Export-Csv "$csvFile-results.csv" -NoTypeInformation -Force
Related
Powershell CSV removing rows and then remove from whole file if A column matches
I've created the following small script to remove 2++ strings from a CSV. Each row is a log of a given person and a answer they give. The CSV has X columns. The column named FIRST identifies the person. What I need to do is when I delete a row matching the answer, I also need to delete the person from the whole CSV if it had one of the two strings. What I've made so far, removes the row of people having the answers but the person is still left in the overall CSV with other answers. I want to remove the person fully if the questions have been answered. Can somebody help me out with making the addition or changes to make this happen? INPUT File FIRST,LAST,ADDR,ADDR2,GENDER,HOME,WORK 1,N/A,N/A,N/A,N/A,BAF,N/A 10005,JAS,AA,N/A,,ZAV,N/A 10007,JADE,BB,N/A,OMA,N/A,N/A 10007,JADE,N/A,RAV,N/A,N/A,N/A 10011,KIAH,N/A,N/A,BALI,BB,N/A SCRIPT $CSVfile = "C:\Temp\Test\Test.csv" $CSVfile_filtered = "C:\Temp\Test\Test.csv" $regex001 = "AA" $regex002 = "BB" $filterArray = #($regex001,$regex002) Get-Content $CSVfile | Select-String -pattern $filterArray -notmatch | Set-Content $CSVfile_filtered The file should then remove 10005, 10011 and both lines of 10007. But my version only removes one of the 10007 since it only matches one of the two patterns.
Using more of PowerShell's built-in cmdlets can make this a little easier to manage.
# Assuming searching only properties ADDR and ADDR2
$filter = 'AA','BB'
# Grouping by First and Last values to easily remove duplicates
# -match uses regex so | is needed for an OR of multiple items
Import-Csv Test.csv | Group-Object First,Last |
Where {!($_.Group.ADDR,$_.Group.ADDR2 -match ($filter -join '|'))} |
Foreach-Object Group |
Export-Csv output.csv -NoType
You would think strictly using text manipulation would be simpler, but it adds other scenarios to consider:
You will need to track users that have duplicate entries and potentially back track to remove them (if not grouping). This could require reading the file contents twice.
Your header row could match the string you want to filter so you will need to add it to the output if filtering removes it.
Keeping the scenarios above in mind, you can still use a grouping concept:
$filter = 'AA','BB'
$file = Get-Content Test.csv
# $file[0] is the header row
# -split string uses regex and splits at the second comma
# -split results' [0] element is First,Last values
$file[0],($file |
Select-Object -Skip 1 |
Group-Object {($_ -split '(?<=^[^,]*,[^,]*),')[0]} |
where {!($_.Group -match ($filter -join '|'))} |
Foreach-Object Group) | Set-Content output.csv
If I got it right you could do something like this:
$SearchPattern = 'AA', 'BB'
$INPUTCSV = #'
FIRST,LAST,ADDR,ADDR2,GENDER,HOME,WORK
1,N/A,N/A,N/A,N/A,BAF,N/A
10005,JAS,AA,N/A,,ZAV,N/A
10007,JADE,BB,N/A,OMA,N/A,N/A
10007,JADE,N/A,RAV,N/A,N/A,N/A
10011,KIAH,N/A,N/A,BALI,BB,N/A
'# | ConvertFrom-Csv
$ActualSearchPattern =
$INPUTCSV |
Where-Object {
$_.LAST -in $SearchPattern -or
$_.ADDR -in $SearchPattern -or
$_.ADDR2 -in $SearchPattern -or
$_.GENDER -in $SearchPattern -or
$_.HOME -in $SearchPattern -or
$_.Work -in $SearchPattern
} |
Select-Object -ExpandProperty FIRST
$INPUTCSV |
Where-Object -Property FIRST -NotIn -Value $ActualSearchPattern |
Format-Table -AutoSize
There might be more sophisticated or more elegant ways but I cannot think about one at the moment. ;-)
There is a nice PowerShell module you can use to manipulate the content of a csv or xlsx file: ImportExcel This give you a lot of options to manipulate the sheets, columns etc.
Powershell: delete duplicate entry in arraylist
In my Powershellscript I read some data from a csv-File in an Arraylist.
In the second step I eliminate every line without the specific char: (.
At the third step I want to eliminate every double entries.
Example for my list:
Klein, Jürgen (Klein01); salesmanagement national
Klein, Jürgen (Klein01); salesmanagement national
Meyer, Gerlinde (Meyer02); accounting
Testuser
Admin1
Müller, Kai (Muell04); support international
I use the following script:
$Arrayusername = New-Object System.Collections.ArrayList
$NewArraylistuser = New-Object System.Collections.ArrayList
$Arrayusername = Get-Content -Path "C:\Temp\User\Userlist.csv"
for ($i=0; $i -le $Arrayusername.length; $i++)
{
if ($Arrayusername[$i] -like "*(*")
{
$NewArraylistuser.Add($Arrayusername_ads[$i])
}
$Array_sorted = $NewArraylistuser | sort
$Array_sorted | Get-Unique
}
But the variable $Array_sorted still has double entries.
I don´t find the mistake.
Some Ideas how you could change your code:
Use the existing Command to import .csv files with the Delimiter ;.
Filter the output with Where-Object to only include Names with (.
Select only unique objects with Select-Object, or if you want to sort the Object, use the Sort-Object with the same paramets.
Something like this should work:
Import-csv -Delimiter ';' -Header "Name","Position" -Path "C:\Temp\User\Userlist.csv" | Where-Object {$_.Name -like "*(*"} | Sort-Object -Unique -Property Name,Position
Compare 2 .csv files
I have two .csv files with many information in it. If at the end of the sentence is a "M", I have to look if this row is in the other file. When it's there I have to look if the code at the beggining of the row is the same, when not then I have to do nothing, but when it's the same I have to make a new file.
This is the information I have to look if it's in the other file:
You can see that the information is here:
I also have rows with a "B" at the end but this is unimportant:
Now, when the information is here, I have to export all rows that are same in both files.
I have to export the rows in a new file which have the same code at the beginning which is circeld in red:
I have tried different solutions that I looked up in the Internet, but nothing really works.
Perhaps something like this?
$datenbank = Import-Csv "C:\Users\information1.csv"
$zentral = Import-Csv "C:\Users\information2.csv"
$new = ""
foreach ($line in $datenbank) {
$Spalte = $line.Split(",")
foreach ($z in $Zentral) {
$found = $false
foreach ($d in $Datenbanktyp) {
if ($d.$Spalte[1] -eq $z.$Spalte[1]) {
$found = $true
}
}
if ($found -eq $true) {
$new += $z
}
}
}
Or can it work with a if..elseif..else loop?
Let's see if I got this right. You have one file where the second-last column contains a letter. If that letter is "M" you want to check if the value of the column before that (partially) matches a column from a second file. If it does, you then want to export all rows from the second file that have the same value in the first column as the matched row to a new file.
Since you didn't reveal the column names I'm going to dub the third- and second-last columns from the first file "Erin" and "Marty", the match column from the second file "Pat", and the first column from the second file "Gene".
$datenbank | Where-Object {
$_.Marty -ceq 'M'
} | Select-Object -Expand Erin -Unique | ForEach-Object {
$outfile = "export_${_}.csv" # adjust output filename as you see fit
$firstcol = $zentral |
Where { $_.Pat -like "*${_}*" } |
Select-Object -Expand Gene
$zentral | Where-Object {
$_.Gene -eq $firstcol
} | Export-Csv $outfile
}
Another approach would be to group your second file by the first column and then check if the groups contain a matching value.
$groups = $zentral | Group-Object Gene
$datenbank | Where-Object {
$_.Marty -ceq 'M'
} | Select-Object -Expand Erin -Unique | ForEach-Object {
$outfile = "export_${_}.csv" # adjust output filename as you see fit
$groups | Where-Object {
$_.Group.Pat -like "*${_}*"
} | Select-Object -Expand Group | Export-Csv $outfile
}
Replace "Erin", "Marty", "Pat" and "Gene" with the actual column titles from your CSV files. Should your files not contain column titles you need to specify them via the -Header parameter of Import-Csv, otherwise the cmdlet will interpret the first data row as the headers.
Replace One CSV Field With Another Through PowerShell
In PowerShell I'm importing a CSV SAMTemp2 which will have a field called SO. Sometimes SO will be populated with "NW" and in these cases I just want to pull the field called ProdProj from the same line and replace the data in SO with the data in ProdProj then export it the data in that condition.
$RepNW = Import-Csv $SAMTemp2
foreach($d in $data){
If($d.SO -eq "NW"){($d.SO).Replace($d.ProdProj)}}
$RepNW | Export-Csv $SAMTemp -NoTypeInformation
I don't get an error, but this doesn't seem to do anything, either. Can anyone assist me, please?
Update
Per Matt below, I tried:
$RepNW = Import-Csv $SAMTemp2
foreach($d in $RepNW){
If($d.SO -eq "NW"){$d.SO = ($d.SO).Replace($d.ProdProj)}}
$RepNW | Export-Csv $SAMTemp -NoTypeInformation
But I'm not seeing any change. Any assistance is appreciated.
As LotPings pointed out in this line foreach($d in $data){, you haven't defined $data and it seems that you mean it to be foreach($d in $RepNW){
Secondly, rather than using Replace() you can just set one property to be equal to the other.
Last, this probably easiest to do all in the pipeline with ForEach-Object
Import-Csv $SAMTemp2 | ForEach-Object {
If($_.SO -eq "NW"){
$_.SO = $_.ProdProj
}
$_
} | Export-Csv $SAMTemp -NoTypeInformation
Parse line of text and match with parse of CSV
As a continuation of a script I'm running, working on the following.
I have a CSV file that has formatted information, example as follows:
File named Import.csv:
Name,email,x,y,z
\I\RS\T\Name1\c\x,email#jksjks,d,f
\I\RS\T\Name2\d\f,email#jsshjs,d,f
...
This file is large.
I also have another file called Note.txt.
Name1
Name2
Name3
...
I'm trying to get the content of Import.csv and for each line in Note.txt if the line in Note.txt matches any line in Import.csv, then copy that line into a CSV with append. Continue adding every other line that is matched. Then this loops on each line of the CSV.
I need to find the best way to do it without having it import the CSV multiple times, since it is large.
What I got does the opposite though, I think:
$Dir = PathToFile
$import = Import-Csv $Dir\import.csv
$NoteFile = "$Dir\Note.txt"
$Note = GC $NoteFile
$Name = (($Import.Name).Split("\"))[4]
foreach ($j in $import) {
foreach ($i in $Note) {
$j | where {$Name -eq "$i"} | Export-Csv "$Dir\Result.csv" -NoTypeInfo -Append
}
}
This takes too long and I'm not getting the extraction I need.
This takes too long and I'm not getting the extraction I need.
That's because you only assign $name once, outside of the outer foreach loop, so you're basically performing the same X comparisons for each line in the CSV.
I would rewrite the nested loops as a single Where-Object filter, using the -contains operator:
$Import |Where-Object {$Note -contains $_.Name.Split('\')[4]} |Export-Csv "$Dir\Result.csv" -NoTypeInformation -Append
Group the imported data by your distinguishing feature, filter the groups by name, then expand the remaining groups and write the data to the output file:
Import-Csv "$Dir\import.csv" |
Group-Object { $_.Name.Split('\')[4] } |
Where-Object { $Note -contains $_.Name } |
Select-Object -Expand Group |
Export-Csv "$Dir\Result.csv" -NoType