Powershell - Using ConvertFrom-csv - powershell

I'm brand new to Powershell. I have a variable that contains comma separated values. What I want to do is read each entry in the csv string variable, and assign it to a variable. I am using ConvertFrom-csv to separate the data with headers.
How can I assign each value to a variable, or even better, use ConvertTo-csv to create a new csv string which only has, for example, columns 2/3/6/7 in it?
I would ultimately want to write that data out to a new csv file.
Here is my test code:
#Setup the variable
$Data = "test1,test2,test3,1234,5678,1/1/2021,12/31/2021"
$Data | ConvertFrom-csv -Header Header1,Header2, Header3, Header4, Header5, Header6, Header7
# Verify that an object has been created.
$Data |
ConvertFrom-csv -Header Header1,Header2, Header3, Header4, Header5, Header6, Header7 |
Get-Member
#Show header1
Write-Host "--------Value from $Data----------------------------------------"
$Data[0] #doesn't work, only displays the first character of the string
Write-Host "-----------------------------------------------------------------"

Let me suggest a different approach. If you use ConvertFrom-Csv and assign the result of a variable ($data), this will be an array of Custom Objects. You can run this through a loop that steps through the elements of the array , one at a time, and then through an inner loop that steps through the properties of each object one at a time, setting a variable with the same name as the field header and the same value as the current record's value.
I don't have code that does exactly what you want. But I'm including code that I wrote a few years back that does something similar only using Import-Csv instead of ConverFrom-Csv.
Import-Csv $driver | % {
$_.psobject.properties | % {Set-variable -name $_.name -value $_.value}
Get-Content $template | % {$ExecutionContext.InvokeCommand.ExpandString($_)}
}
Focus on the first inner loop. Each property of the current object will have a name that came from the header and a value that came from the current record of the Csv file. You can ignore the line that says ExpandString. That's just what I choose to do with the variables once they have been defined.

How can I assign each value to a variable, or even better, use ConvertTo-Csv to create a new csv string which only has, for example, columns 2/3/6/7 in it?
This is one way of automating this:
# Define the CSV without headers
$Data = "test1,test2,test3,1234,5678,1/1/2021,12/31/2021"
# Set the number of headers needed
$headers = $Data.Split(',') | ForEach-Object -Begin { $i = 1 } -Process {
"Header$i"; $i++
}
# Set the desired columns we want
$desiredColumns = 2,3,6,7 | ForEach-Object { $_ - 1 } | ForEach-Object {
$headers[$_]
}
# Convert to CSV and filter by Desired Columns
$Data | ConvertFrom-Csv -Header $headers | Select-Object $desiredColumns
Result
Header2 Header3 Header6 Header7
------- ------- ------- -------
test2 test3 1/1/2021 12/31/2021
Result as CSV
$Data | ConvertFrom-Csv -Header $headers |
Select-Object $desiredColumns | ConvertTo-Csv -NoTypeInformation
"Header2","Header3","Header6","Header7"
"test2","test3","1/1/2021","12/31/2021"

Related

Powershell Compare-object IF different then ONLY list items from one file, not both

I have deleted my original question because I believe I have a more efficient way to run my script, thus I'm changing my question.
$scrubFileOneDelim = "|"
$scrubFileTwoDelim = "|"
$scrubFileOneBal = 2
$scrubFileTwoBal = 56
$scrubFileOneAcctNum = 0
$scrubFileTwoAcctNum = 0
$ColumnsF1 = Get-Content $scrubFileOne | ForEach-Object{($_.split($scrubFileOneDelim)).Count} | Measure-Object -Maximum | Select-Object -ExpandProperty Maximum
$ColumnsF2 = Get-Content $scrubFileTwo | ForEach-Object{($_.split($scrubFileTwoDelim)).Count} | Measure-Object -Maximum | Select-Object -ExpandProperty Maximum
$useColumnsF1 = $ColumnsF1-1;
$useColumnsF2 = $ColumnsF2-1;
$fileOne = import-csv "$scrubFileOne" -Delimiter "$scrubFileOneDelim" -Header (0..$useColumnsF1) | select -Property #{label="BALANCE";expression={$($_.$scrubFileOneBal)}},#{label="ACCTNUM";expression={$($_.$scrubFileOneAcctNum)}}
$fileTwo = import-csv "$scrubFileTwo" -Delimiter "$scrubFileTwoDelim" -Header (0..$useColumnsF2) | select -Property #{label="BALANCE";expression={$($_.$scrubFileTwoBal)}},#{label="ACCTNUM";expression={$($_.$scrubFileTwoAcctNum)}}
$hash = #{}
$hashTwo = #{}
$fileOne | foreach { $hash.add($_.ACCTNUM, $_.BALANCE) }
$fileTwo | foreach { $hashTwo.add($_.ACCTNUM, $_.BALANCE) }
In this script I'm doing the following, counting header's to return the count and use it in a range operator in order to dynamically insert headers for later manipulation. Then I'm importing 2 CSV files. I'm taking those CSV files and pushing them into their own hashtable.
Just for an idea of what I'm trying to do from here...
CSV1 (as a hashtable) looks like this:
Name Value
---- -----
000000000001 000000285+
000000000002 000031000+
000000000003 000004685+
000000000004 000025877+
000000000005 000000001+
000000000006 000031000+
000000000007 000018137+
000000000008 000000000+
CSV2 (as a hashtable) looks like this:
Name Value
---- -----
000000000001 000008411+
000000000003 000018137+
000000000007 000042865+
000000000008 000009761+
I would like to create a third hash table. It will have all the "NAME" items from CSV2, but I don't want the "VALUE" from CSV2, I want it to have the "VALUE"s that CSV1 has. So in the end result would look like this.
Name Value
---- -----
000000000001 000000285+
000000000003 000004685+
000000000007 000018137+
000000000008 000000000+
Ultimately I want this to be exported as a csv.
I have tried this with just doing a compare-object, not doing the hashtables with the following code, but I abandoned trying to do it this way because file 1 may have 100,000 "accounts" where file 2 only has 200, and the result I was getting listed close to the 100,000 accounts that I didn't want to be in the result. They had the right balances but I want a file that only has those balances for the accounts listed in file 2. This code below isn't really a part of my question, just showing something I've tried. I just think this is much easier and faster with a hash table now so I would like to go that route.
#Find and Rename the BALANCE and ACCOUNT NUMBER columns in both files.
$fileOne = import-csv "$scrubFileOne" -Delimiter "$scrubFileOneDelim" -Header (0..$useColumnsF1) | select -Property #{label="BALANCE";expression={$($_.$scrubFileOneBal)}},#{label="ACCT-NUM";expression={$($_.$scrubFileOneAcctNum)}}
$fileTwo = import-csv "$scrubFileTwo" -Delimiter "$scrubFileTwoDelim" -Header (0..$useColumnsF2) | select -Property #{label="BALANCE";expression={$($_.$scrubFileTwoBal)}},#{label="ACCT-NUM";expression={$($_.$scrubFileTwoAcctNum)}}
Compare-Object $fileOne $fileTwo -Property 'BALANCE','ACCTNUM' -IncludeEqual -PassThru | Where-Object{$_.sideIndicator -eq "<="} | select * -Exclude SideIndicator | export-csv -notype "C:\test\f1.txt"
What you are after is filtering the Compare-Object function. This will show only one side of the result. YOu will need to place this before you exclude that property for it to work.
| Where-Object{$_.sideIndicator -eq "<="} |
Assuming that you have the following hash tables:
$hash = #{
'000000000001' = '000000285+';
'000000000002' = '000031000+';
'000000000003' = '000004685+';
'000000000004' = '000025877+';
'000000000005' = '000000001+';
'000000000006' = '000031000+';
'000000000007' = '000018137+';
'000000000008' = '000000000+';
}
$hashTwo = #{
'000000000001' = '000008411+';
'000000000003' = '000018137+';
'000000000007' = '000042865+';
'000000000008' = '000009761+';
}
you can create the third hash table by iterating over the keys from the second hash table and then assigning those keys to the value from the first hash table.
$hashThree = #{}
ForEach ($key In $hashTwo.Keys) {
$hashThree["$key"] = $hash["$key"]
}
$hashThree
The output of $hashThree is:
Name Value
---- -----
000000000007 000018137+
000000000001 000000285+
000000000008 000000000+
000000000003 000004685+
If you want the order of the data maintained (and you are using PowerShell 6 Core), you can use [ordered]#{} when creating the hash tables.

Select specific column based on data supplied using Powershell

I have a csv file that may have unknown headers, one of the columns will contain email addresses for example.
Is there a way to select only the column that contains the email addresses and save it as a list to a variable?
One csv could have the header say email, another could say emailaddresses, another could say email addresses another file might not even have the word email in the header. As you can see, the headers are different. So I want to be able to detect the correct column first and use that data further in the script. Once the column is identified based on the data it contains, select that column only.
I've tried the where-object and select-string cmdlets. With both, the output is the entire array and not just the data in the column I am wanting.
$CSV = import-csv file.csv
$CSV | Where {$_ -like "*#domain.com"}
This outputs the entire array as all rows will contain this data.
Sample Data for visualization
id,first_name,bagel,last_name
1,Base,bcruikshank0#homestead.com,Cruikshank
2,Regan,rbriamo1#ebay.co.uk,Briamo
3,Ryley,rsacase2#mysql.com,Sacase
4,Siobhan,sdonnett3#is.gd,Donnett
5,Patty,pesmonde4#diigo.com,Esmonde
Bagel is obviously what we are trying to find. And we will play pretend in that we have no knowledge of the columns name or position ahead of time.
Find column dynamically
# Import the CSV
$data = Import-CSV $path
# Take the first row and get its columns
$columns = $data[0].psobject.properties.name
# Cycle the columns to find the one that has an email address for a row value
# Use a VERY crude regex to validate an email address.
$emailColumn = $columns | Where-Object{$data[0].$_ -match ".*#*.\..*"}
# Example of using the found column(s) to display data.
$data | Select-Object $emailColumn
Basically read in the CSV like normal and use the first columns data to try and figure out where the email address column is. There is a caveat that if there is more than one column that matches it will get returned.
To enforce only 1 result a simple pipe to Select-Object -First 1 will handle that. Then you just have to hope the first one is the "right" one.
If you're using Import-Csv, the result is a PSCustomObject.
$CsvObject = Import-Csv -Path 'C:\Temp\Example.csv'
$Header = ($CsvObject | Get-Member | Where-Object { $_.Name -like '*email*' }).Name
$CsvObject.$Header
This filters for the header containing email, then selects that column from the object.
Edit for requirement:
$Str = #((Get-Content -Path 'C:\Temp\Example.csv') -like '*#domain.com*')
$Headers = #((Get-Content -Path 'C:\Temp\Example.csv' -TotalCount 1) -split ',')
$Str | ConvertFrom-Csv -Delimiter ',' -Header $Headers
Other method:
$PathFile="c:\temp\test.csv"
$columnName=$null
$content=Get-Content $PathFile
foreach ($item in $content)
{
$SplitRow= $item -split ','
$Cpt=0..($SplitRow.Count - 1) | where {$SplitRow[$_] -match ".*#*.\..*"} | select -first 1
if ($Cpt)
{
$columnName=($content[0] -split ',')[$Cpt]
break
}
}
if ($columnName)
{
import-csv "c:\temp\test.csv" | select $columnName
}
else
{
"No Email column founded"
}

How to detect and remove CSV columns based on common text in header names?

I am working on a CSV File which I recently created. The CSV file contains columns with headers and corresponding rows.
I need to remove entire columns (including its data) that have specific text common to their headers. For e.g column 1 has header named intID, column 2 has header named boolID, column 3 has header named charID and so on ('ID' being the common text). There are some columns that don't have 'ID' as text in their headers, so we need to retain those.
The csv file is getting generated dynamically, so there may be more/less columns based on what data we select for the csv. But we need these columns with their headers having some common text to be removed.
How can we achieve this?
Would something like that do the trick?
$yourfile = "<path to your csv>"
# Import the CSV
$csv = Import-Csv -Path $yourfile
# Find all columns that do not end with "ID"
$colsToKeep = $csv | Get-Member -MemberType NoteProperty |?{$_.name -notmatch "^.+ID$"} | Select-Object -ExpandProperty name
# Filter out all unwanted columns
$newCsv = $csv | Select-Object -Property $colsToKeep
# Export CSV to new file
$newCsv | Export-Csv -Path "<path to new csv>"
Assuming the following:
the ID part is not a plain text "ID" but a dynamic arbitrary text
headers of interest start with int, char, bool
Let's count occurrences of ID part and build a list of headers used just once, then export the CSV.
$csv = Import-Csv 1.csv
$prefix = '^(int|char|bool)' # or '^([a-z])' for any lowercase text
$headers = $csv[0].PSObject.Properties.Name
$uniqueIDs = $headers -creplace $prefix, '' | group | ? Count -eq 1 | select -expand Name
$uniqueHeaders = $headers | ?{ $_ -creplace $prefix, '' -in $uniqueIDs }
$csv | select $uniqueHeaders | Export-Csv 2.csv -NoTypeInformation
Note: in the old PowerShell 2.0 instead of ? Count -eq 1 use ?{ $_.Count -eq 1 }

Use Import-Csv to read changable column Titles by location

I'm trying to see if there is a way to read the column values in a csv file based on the column location. The reason for this is the file I'm being handed always has it's titles being changed...
For example, lets say csv file column A (via excel) looks like the following:
ColumnOne
ValueOne
ValueTwo
ValueThree
Now the user changes the title:
Column 1
ValueOne
ValueTwo
ValueThree
Now I want to create an array of the first column. Normally what I do is the following:
$arrayFirstColumn = Import-Csv 'C:\test\test1.csv' | where-object {$_.ColumnOne} | select-object -expand 'ColumnOne'
However, as we can see if ColumnOne is changed to Column 1, it breaks this code. How can I create this array to allow an interchangeable column title, but the column location will always be the same?
You can specify headers of your own on import:
Import-Csv 'C:\path\to\your.csv' -Header 'MyHeaderA','MyHeaderB',...
As long as you don't export the data back to a CSV (or don't require the original headers to be in the output CSV as well) you can use whatever names you like. You can also specify as many header names as you like. If their number is less than the number of the columns in the CSV the additional columns will be omitted, if it's greater then the columns for the additional headers will be empty.
If you need to preserve the original headers you could get the header name(s) you need to work with in variable(s) like this:
$csv = Import-Csv 'C:\test\test1.csv'
$firstCol = $csv | Select-Object -First 1 | ForEach-Object {
$_.PSObject.Properties | Select-Object -First 1 -Expand Name
}
$arrayFirstColumn = $csv | Where-Object {$_.$firstCol} |
Select-Object -Expand $firstCol
Or you could simply read the first line from the CSV and split it to get an array with the headers:
$headers = (Get-Content 'C:\test\test1.csv' -TotalCount 1) -split ','
$firstCol = $headers[0]
One option:
$ImportFile = 'C:\test\test1.csv'
$FirstColumn = ((Get-Content $ImportFile -TotalCount 2 | ConvertFrom-Csv).psobject.properties.name)[0]
$FirstColumn
$arrayFirstColumn = Import-Csv $ImportFile | where-object {$_.$FirstColumn} | select-object -expand $FirstColumn
If you are using PowerShell v2.0 then the expression for $FirstColumn in $mjolinor's answer would be:
$FirstColumn = ((Get-Content $ImportFile -TotalCount 2 | ConvertFrom-Csv).psobject.properties | ForEach-Object {$_.name})[0]
(Apologies for starting a new answer; I do not yet have enough reputation to add a comment to mjolinor's post)

PowerShell Import-Csv Issue - Why is my output being treated as a single column and not a CSV?

So I have a CSV file which I need to manipulate a bit, select the data I need and export to another CSV file.
The code I have is:
$rawCSV = "C:\Files\raw.csv"
$outputCSV = "C:\Files\output.csv"
Import-Csv -Header #("a","b","c","d") -Path $rawCSV |
select -Skip 7 |
Where-Object { $_.b.length -gt 1 } |
ft b,a,c,d |
Out-File $outputCSV
So this code uses the Import-Csv command to allow me to select just the columns I need, add some headers in the order I want and then I am simply putting the output in to a CSV file called $outputCSV. The contents of this output file look something like this:
b a c d
- - - -
john smith 29 England
mary poopins 79 Walton
I am not sure what the delimiter is in this output and rather than these columns being treated as individuals, they are treated as just one column. I have gone on further to replace all the spaces with a comma using the code:
$b = foreach ($line in $a)
{
$fields = $line -split '`n'
foreach ($field in $fields)
{
$field -replace " +",","
}
}
Which produces a file that looks like this:
b,a,c,d
john,smith,29,England
mary,poppins,79,Walton
But these are all still treated as one column instead of four separate columns as I need.
* UPDATE *
Using the answer given by #, I now get a file looking like this:
Don't use ft to reorder your columns - it's intended to format output for the screen, not really suitable for CSV.
"Manual" solution:
$rawCSV = "C:\Files\raw.csv"
$outputCSV = "C:\Files\output.csv"
# Import and filter your raw data
$RawData = Import-Csv -Header #("a","b","c","d") -Path $rawCSV
$Data = $RawData | Select -Skip 7 | Where-Object { $_.b.length -gt 1 }
# Write your headers to the output file
"b","a","c","d" -join ',' | Out-File $outputCSV -Force
$ReorderedData = foreach($Row in $Data){
# Reorder the columns in each row
'{0},{1},{2},{3}' -f $Row.b , $Row.a , $Row.c, $Row.d
}
# Write the reordered rows to the output file
$ReorderedData | Out-File $outputCSV -Append
Using Export-Csv:
As of PowerShell 3.0, you could also push the rows into a [pscustomobject] and pipe that to Export-Csv (pscustomobject preserves the order in which you supply the properties):
$rawCSV = "C:\Files\raw.csv"
$outputCSV = "C:\Files\output.csv"
# Import and filter your raw data
$RawData = Import-Csv -Header #("a","b","c","d") -Path $rawCSV
$Data = $RawData | Select -Skip 7 | Where-Object { $_.b.length -gt 1 }
# Take the columns you're interested in, put them into new custom objects and export to CSV
$Data | ForEach-Object {
[pscustomobject]#{ "b" = $_.b; "a" = $_.a; "c" = $_.c; "d" = $_.d }
} | Export-Csv -NoTypeInformation $outputCSV
Export-Csv will take care of enclosing strings in quotes to escape ',' properly (one thing less for you to worry about)
First of all, what your raw CSV file looks like? If it's already like this
john,smith,29,England
mary,poppins,79,Walton
then import-csv will give you an array of objects which you can easily manipulate (and objects are the main reason to use PowerShell ;). For example, to check what you have after import:
$r = Import-Csv -Path $rawCSV -Header #("b","a","c","d")
$r.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Object[] System.Array
$r[0] | get-member
TypeName: System.Management.Automation.PSCustomObject
Name MemberType Definition
---- ---------- ----------
Equals Method bool Equals(System.Object obj)
GetHashCode Method int GetHashCode()
GetType Method type GetType()
ToString Method string ToString()
a NoteProperty System.String a=smith
b NoteProperty System.String b=john
c NoteProperty System.String c=29
d NoteProperty System.String d=England
For now you have array of objects with properties named "a","b","c","d". To manipulate objects you have select-object cmdlet:
$r | Select-Object a,b,c,d
a b c d
- - - -
smith john 29 England
poppins mary 79 Walton
And after all use export-csv to set the output file:
$r | where { $_.b.length -gt 1 } |
select a,b,c,d |
Export-Csv -NoTypeInformation -Encoding utf8 -path $outputCSV
I could think of two possible reasons why your data teated as one column:
consuming application expect different encoding and can't find
delimiters
delimiters are not commas but something else