Need help on a power shell script to filter the CSV on Columns and display the filtered data in new sheet - powershell

I am new to power shell but learning it fast. So far i have made a script which is fetching the data from URL and creating a csv on the desktop and then i remove the first row from the CSV and saving it to desktop as csv2. I want to filter the column and copy the filtered data in new sheet. I may have to declare the array values to be looked for filtering and i need help on that. So far i have made the below script:-
\This script is made to download the asset analysis servers list with their corresponding site address and country and will save the output to CSV on C:\Users\vtarwani\Desktop.
Invoke-WebRequest -Uri "Link to URL" -OutFile "C:\Users\tarwaniv\Desktop\file1.csv"
$import = get-content "C:\Users\tarwaniv\Desktop\file1.csv"
$import | select-Object -Skip 1 | Set-Content "C:\Users\tarwaniv\Desktop\file2.csv"
Import-csv -Path "C:\Users\tarwaniv\Desktop\file2.csv" -Header "#Active Servers", "Street Address" , "City", "Country" | Where-Object {$_.Country -eq "UNITED STATES"}

I agree with #Theo and #Olaf. This is difficult to follow. Hopefully you'll be able to build off an example like this:
$WebFile = "C:\Users\tarwaniv\Desktop\file1.csv"
$Out_Dir = "C:\Users\tarwaniv\Desktop\"
$Headers = "#Active Servers", "Street Address" , "City", "Country"
Invoke-WebRequest -Uri "Link to URL" -OutFile $WebFile
Import-csv -Path $WebFile -Header |
Select-Object -Skip 1 |
Group-Object -Property Country |
ForEach-Object{
$OutFile = $Out_Dir + $_.Name
$_.Group | Export-Csv -Path $OutFile -NoTypeInformation
}
Note: using -Skip parameter on Select object should skip the first record coming from Import-Csv.
Use Group-Object to group on the Country. That will output its own objects, but the name property is what you grouped on, in this case Country, and the Group property has the original objects in it. So we can use these 2 properties to name an output file then export the objects to the new Csv file.
If you want to only output a subset of the columns coming from the initial file. add the -Property parameter to the Select-Object command like:
Select-Object -Skip 1 -Property <PropertiesToSelect>
Note: You can declare an array of properties similar to $Headers above.
If you want top filter on additional columns, and assuming they were included in the Select-Object command just build on the Where-Object clause.
Please bear in mind I couldn't really test this, so consider it a starting point. That said let me know how it turns out. Thanks.

Related

Powershell - Compare CSV 1 to CSV 2 and then update CSV1

I am not looking for a writing service but please can someone point me in the right direction as I am completely at a lost as to how to proceed.
Overview
I have a CSV which contains a lot of data, some of which comes from a script and some in manually imputed. I can run the script and get new data which is good. What I would like to do is find a way to compare the orginal CSV 1 to the new CSV 2 and update CSV 1.
Code I currently have
$Vips_to_check = #{}
Import-Csv 'C:\Users\user\Documents\20221201\Netscaler VIPs per Cluster_edited - Raw Data.csv' |
Where-Object {$_.PRD -match "No PRD code from VIP IP and VIP has no backend IPs" -or
$_.PRD -match "No PRD code found from VIP or backend IPs" -or
$_.PRD -match "No PRD code found from backend IPs" -and
$_.ipv46 -notcontains "0.0.0.0"} |
$Results_from_PIM = Import-Csv 'C:\Users\user\Documents\20221201\VIP-Owners_edited.csv'
Both of the CSV's have the same headers and layout which is good. I assume!
CSV 1
Name IPV46 Port Curstate Suggested PRD Display Name tech Owner Slack Channel Support Email
name 1 1.2.3.4 8080 Down No No No No No No No
CSV 2
Name IPV46 Port Curstate Suggested PRD Display Name tech Owner Slack Channel Support Email
name 1 1.2.3.4 8080 Down No PRD123 TMOL Gary TMOL Support Support#email.com nsr.sys
I would guess at creating a hashtable but I just can't seem to get my head around the format of them. I tried
$ht = $Results_from_pim #{}
$_.Name = (cant figure out how to reference the cell)
$_.PRD =
$_.("Display Name")
$_.("Tech Owner")
Once I have the data in the hash table how do I overwrite the CSV 1 data?
Any points or guides would be great. I have tried reading up on https://learn.microsoft.com/en-gb/powershell/scripting/learn/deep-dives/everything-about-hashtable?view=powershell-7.3 and https://learn.microsoft.com/en-us/powershell/scripting/learn/deep-dives/everything-about-pscustomobject?view=powershell-7.3
But that left me even more confused.
At the moment the difference is only 4 or 5 entries and it would of been quicker for me to manually edit in excel but as this script gets larger I can see it being more time consuming to do manually.
As always thank you.
UPDATE
$ht = #{}
foreach ($item in $Results_from_PIM) {
"name = $($item.name)"
"prd = $($item.PRD)"
"Display Name = $($item.'Display Name')"
"Tech Owner = $($item.'Tech Owner')"
"Slack Channel = $($item.'Slack Channel')"
"Support Email = $($Item.'Support Email')"
}
I have created the hash table that I wanted from the CSV 2. Just got to get it to compare to CSV 1.
Update 2
Further to #theo request I have adjusted the question. Also to clarify When I want to merge the CSV it is based on matching the Name, IPV46 and Port on both CSV and then moving the updated data from CSV2 into CSV1.
You can do that with the code below (no extra module needed):
$csv1 = 'C:\Users\user\Documents\20221201\Netscaler VIPs per Cluster_edited - Raw Data.csv'
$csv2 = 'C:\Users\user\Documents\20221201\VIP-Owners_edited.csv'
$Results_from_PIM = Import-Csv -Path $csv2
$newData = Import-Csv -Path $csv1 | ForEach-Object {
$search = $_.Name + $_.IPV46 + $_.Port # combine these fields into a single string
$compare = $Results_from_PIM | Where-Object { ($_.Name + $_.IPV46 + $_.Port) -eq $search }
if ($compare) {
# output the result from csv2
$compare
}
else {
# output the original row from csv1
$_
}
}
# now you can save the updated data to a new file or overwrite csv1 if you like
$csv3 = 'C:\Users\user\Documents\20221201\VIP-Owners_Updated.csv'
$newData | Export-Csv -Path $csv3 -NoTypeInformation
P.S. Please read about Formatting
After being direct to In PowerShell, what's the best way to join two tables into one? by #jdweng. I performed the following which seems to have meet my requirements
Install-Module -Name JoinModule -Scope CurrentUser
$Vips_to_check = Import-Csv 'C:\Users\user\Documents\20221201\Netscaler VIPs per Cluster - Raw Data.csv'
$Results_from_PIM = Import-Csv 'C:\Users\user\Documents\20221201\VIP-Owners.csv'
$Vips_to_check | Update-Object $Results_from_PIM -On name, Ipv46, port | Export-Csv 'C:\Users\user\Documents\20221201\Final_data1.csv'
Going to do further testing with larger data sets but appears to work as required.

Using Powershell, how can I export and delete csv rows, where a particular value is *not found* in a *different* csv?

I have two files. One is called allper.csv
institutiongroup,studentid,iscomplete
institutionId=22343,123,FALSE
institutionId=22343,456,FALSE
institutionId=22343,789,FALSE
The other one is called actswithpersons.csv
abc,123;456
def,456
ghi,123
jkl,123;456
Note: The actswithpersons.csv does not have headers - they are going to be added in later via an excel power query so don't want them in there now. The actswithpersons csv columns are delimited with commas - there are only two columns, and the second one contains multiple personids - again Excel will deal with this later.
I want to remove all rows from allper.csv where the personid doesn't appear in actswithpersons.csv, and export them to another csv. So in the desired outcome, allper.csv would look like this
institutiongroup,studentid,iscomplete
institutionId=22343,123,FALSE
institutionId=22343,456,FALSE
and the export.csv would look like this
institutiongroup,studentid,iscomplete
institutionId=22343,789,FALSE
I've got as far as the below, which will put into the shell whether the personid is found in the actswithpersons.csv file.
$donestuff = (Get-Content .\ActsWithpersons.csv | ConvertFrom-Csv); $ids=(Import-Csv .\allper.csv);foreach($id in $ids.personid) {echo $id;if($donestuff -like "*$id*" )
{
echo 'Contains String'
}
else
{
echo 'Does not contain String'
}}
However, I'm not sure how to go the last step, and export & remove the unwanted rows from allper.csv
I've tried (among many things)
$donestuff = (Get-Content .\ActsWithpersons.csv | ConvertFrom-Csv);
Import-Csv .\allper.csv |
Where-Object {$donestuff -notlike $_.personid} |
Export-Csv -Path export.csv -NoTypeInformation
This took a really long time and left me with an empty csv. So, if you can give any guidance, please help.
Since your actswithpersons.csv doesn't have headers, in order for you to import as csv, you can specify the -Header parameter in either Import-Csv or ConvertFrom-Csv; with the former cmdlet being the better solution.
With that said, you can use any header name for those 2 columns then filter by the given column name (ID in this case) after your import of allper.csv using Where-Object:
$awp = (Import-Csv -Path '.\actswithpersons.csv' -Header 'blah','ID').ID.Split(';')
Import-Csv -Path '.\allper.csv' | Where-Object -Property 'Studentid' -notin $awp
This should give you:
institutiongroup studentid iscomplete
---------------- --------- ----------
institutionId=22343 789 FALSE
If you're looking to do it with Get-Content you can split by the delimiters of , and ;. This should give you just a single row of values which you can then compare the entirety of variable ($awp) using the same filter as above which will give you the same results:
$awp = (Get-Content -Path '.\actswithpersons.csv') -split ",|;"
Import-Csv -Path '.\allper.csv' | Where-Object -Property 'Studentid' -notin $awp

Select specific column based on data supplied using Powershell

I have a csv file that may have unknown headers, one of the columns will contain email addresses for example.
Is there a way to select only the column that contains the email addresses and save it as a list to a variable?
One csv could have the header say email, another could say emailaddresses, another could say email addresses another file might not even have the word email in the header. As you can see, the headers are different. So I want to be able to detect the correct column first and use that data further in the script. Once the column is identified based on the data it contains, select that column only.
I've tried the where-object and select-string cmdlets. With both, the output is the entire array and not just the data in the column I am wanting.
$CSV = import-csv file.csv
$CSV | Where {$_ -like "*#domain.com"}
This outputs the entire array as all rows will contain this data.
Sample Data for visualization
id,first_name,bagel,last_name
1,Base,bcruikshank0#homestead.com,Cruikshank
2,Regan,rbriamo1#ebay.co.uk,Briamo
3,Ryley,rsacase2#mysql.com,Sacase
4,Siobhan,sdonnett3#is.gd,Donnett
5,Patty,pesmonde4#diigo.com,Esmonde
Bagel is obviously what we are trying to find. And we will play pretend in that we have no knowledge of the columns name or position ahead of time.
Find column dynamically
# Import the CSV
$data = Import-CSV $path
# Take the first row and get its columns
$columns = $data[0].psobject.properties.name
# Cycle the columns to find the one that has an email address for a row value
# Use a VERY crude regex to validate an email address.
$emailColumn = $columns | Where-Object{$data[0].$_ -match ".*#*.\..*"}
# Example of using the found column(s) to display data.
$data | Select-Object $emailColumn
Basically read in the CSV like normal and use the first columns data to try and figure out where the email address column is. There is a caveat that if there is more than one column that matches it will get returned.
To enforce only 1 result a simple pipe to Select-Object -First 1 will handle that. Then you just have to hope the first one is the "right" one.
If you're using Import-Csv, the result is a PSCustomObject.
$CsvObject = Import-Csv -Path 'C:\Temp\Example.csv'
$Header = ($CsvObject | Get-Member | Where-Object { $_.Name -like '*email*' }).Name
$CsvObject.$Header
This filters for the header containing email, then selects that column from the object.
Edit for requirement:
$Str = #((Get-Content -Path 'C:\Temp\Example.csv') -like '*#domain.com*')
$Headers = #((Get-Content -Path 'C:\Temp\Example.csv' -TotalCount 1) -split ',')
$Str | ConvertFrom-Csv -Delimiter ',' -Header $Headers
Other method:
$PathFile="c:\temp\test.csv"
$columnName=$null
$content=Get-Content $PathFile
foreach ($item in $content)
{
$SplitRow= $item -split ','
$Cpt=0..($SplitRow.Count - 1) | where {$SplitRow[$_] -match ".*#*.\..*"} | select -first 1
if ($Cpt)
{
$columnName=($content[0] -split ',')[$Cpt]
break
}
}
if ($columnName)
{
import-csv "c:\temp\test.csv" | select $columnName
}
else
{
"No Email column founded"
}

Switch Names from one side to other

I have to set description to a list of sames provided in csv format.
I know I need samaccountnames so i am trying to pull up samaccount from named, unfortunately the names in csv are in reverse order with a header as name
example
Name
Snow, Jon
Starc,arya
lannister,jamie
In a nutshell, I tried
Import-Csv C:\list.csv |
foreach {
$_.Name = "{1}, {0}" -f ($_.Name -split ', ')
$_
No luck, any help is appreciated.
The names should come as -
Jon snow
Arya starc
Jamie lannister
so I can query AD for sam's
To have lastname and firstname you can do this:
$names = $string.split(",")
[array]::Reverse($names)
$names
So if I understood correctly, you want to skip the header (first line). Try changing the following:
Import-Csv C:\list.csv | Select-Object -Skip 1 | ConvertFrom-Csv -Header Name
(or increase '-Skip' amount according to how many lines you want to skip)
The question you have here to me reads a little vague, but I wanted to offer the below. If you specify the header of the CSV when you import it, you can output whatever Property you want, in whatever order:
$test = Import-Csv -Path "C:\temp\test.txt" -Header Last_Name,First_Name
$test | % {"$($_.First_Name) $($_.Last_Name)"}
It was
Import-Csv c:\list.csv |
foreach{
$last,$first = $_.name -split ","
new-object psobject -Property #{name = "$first,$last"}
}

Compare 2 csv files and match based on 1 column then export new file that contains fields from both

I have 2 csv files. Each have different headers and different number of columns, and have different number of entries.
Here are some examples of the first couple lines
CSV 1
ID,Last_Name,First_Name,Middle_Name,Email_Addr,Title,Gender
###1,smith,bill,p,smith#soso.com,boss,m
###2,smith2,billy,p,smith2#soso.com,someguy,m
CSV 2
ID,Name Id,Last Name,First Name,Middle Name,Gender
###2,ID1010,smith2,billy,p,M
I am trying to import them and compare the ID column. When a match is found I am wanting a new csv file with All info from CSV 1 and the matched Name ID from csv 2.
New CSV Example:
ID,Last_Name,First_Name,Middle_Name,Email_Addr,Title,Gender,Name Id
###1,smith,bill,p,smith#soso.com,boss,m,
###2,smith2,billy,p,smith2#soso.com,someguy,m,ID1010
Ive been looking and came across this Stackoverflow from about a year ago that seemed to be on the right track but I cant seem to get code modified for my needs. Here is what I have tried.
$csv1 = Import-Csv -Path C:\STAFF\test1sky.csv
$csv2 = Import-Csv -Path C:\STAFF\test1power.csv
ForEach($Record in $csv2){
$MatchedValue = (Compare-Object $csv1 $Record -Property "ID" -IncludeEqual -ExcludeDifferent -PassThru).value
$Record = Add-Member -InputObject $Record -Type NoteProperty -Name "Name Id" -Value $MatchedValue
}
$csv2|Export-Csv 'C:\STAFF\combined.csv' -NoTypeInformation
I get the correct header in the new file but I never get the Name ID values to come though.
Any idea where I went wrong? I maybe on the wrong path completely and there be a easier way, but I need to be able to do this nightly without user interaction. Any help is appreciated!!
Let's try to simplify this. Add the 'Name ID' field to all records in CSV1. Then loop through it, and get the matches, and update the field. Something like:
$CSV1 = C:\Path\To\File1.csv
$CSV2 = C:\Path\To\File2.csv
$CSV1|ForEach{$_|Add-Member 'Name ID' $Null}
ForEach($Record in $CSV1){
$Record.'Name ID' = $CSV2|Where{$_.ID -eq $Record.ID}|Select -Expand 'Name ID'
}
$CSV1 = import-csv C:\Path\To\File1.csv
$CSV2 = import-csv C:\Path\To\File2.csv
#adds a row named "Name ID" to the PS Object( the CSV Import)
$CSV1|ForEach{$_|Add-Member 'Name ID' $Null}
ForEach($Record in $CSV1){
#gets the value from CSV1 for comparing to CSV2
$NameValue=Record."Last_Name"
#gets the Power Shell Object from the CSV2 Import that matches the Name ID from $csv1
$Nameobject= $CSV2|Where-object "Last Name" -contains $Namevalue
#Sets the Field "Name ID" in the PS Object $CSV1 Record to the Name ID from $csv2
$record."Name ID" = $Nameobject."Name ID"
}
You can easily grab addtional fields by adding other references to the CSV1 File by manipulating the CSV2 PS Object.
$record."Middle Name" = $nameobject."Middle_Name"
Since you have the entire object in the for loop form $csv2 you can call any of its fields or manipulate them by using variables and " |select -Property "Value" Like this
$objlength = $nameobject |select "First_Name"
$objlength.length
but i prefer to call it directly from the object as the output looks cleaner like this
$nameobject."First_Name".length
The operation you are looking for is called a relational join. Sometimes it's called an inner join, and sometimes just a join. My knowledge of join comes from SQL, not from Powershell.
Here's a description of "Join-Object". It seems to be what you are looking for.
http://blogs.msdn.com/b/powershell/archive/2012/07/13/join-object.aspx