Insert blank columns into csv with Powershell - powershell

In my script, I am building a custom Powershell object which will be sent to Export-Csv. The receiving party has required that I include some blank columns (no data and no header) and I have no idea how to do that.
If the object looks like this:
$obj = [PSCustomObject][ordered]#{
EMPLOYER_EIN = '123456'
ACTION_CODE = 1
LAST_NAME = Smith
FIRST_NAME = John
MIDDLE_INITIAL = $null
EMPLOYEE_SSN = '111-11-1111'
}
How can I have the resulting .csv file's first row look like this:
EMPLOYER_EIN,ACTION_CODE,,LAST_NAME,FIRST_NAME,MIDDLE_INITIAL,,EMPLOYEE_SSN
Put another way, after I run Export-Csv, I want the file to look like this when opened in Excel:
EMPLOYER_EIN
ACTION_CODE
LAST_NAME
FIRST_NAME
MIDDLE_INITIAL
EMPLOYEE_SSN
123456
1
Smith
John
111-11-1111
Note the extra columns between action_code/last_name and middle_initial/employee_ssn. I am using PS 5.1 but could use 7 if necessary.

As a test, I created a CSV test.csv with fields A,B, and C, and put a couple of lines of values:
"A","B","C"
1,2,3
4,5,6
I then executed the sequence of commands
Import-CSV -path Test.csv | Select-Object -Prop A," ",B,C | Export-CSV -Path test2.csv
and looked at the resultant test2.csv, which contained
#TYPE Selected.System.Management.Automation.PSCustomObject
"A"," ","B","C"
"1",,"2","3"
"4",,"5","6"
I believe that this is going to be the closest you'll get without manually processing the CSV as a text file.
This is essentially what Santiago Squarzon was suggesting in the comments.
If you need multiple "blank" columns, each one will have to have a header with a different non-zero number of spaces.

I suggest:
constructing the object with blank dummy properties with a shared name prefix, such as BLANK_, followed by a sequence number (the property names must be unique)
initially piping to ConvertTo-Csv, which allows use of a -replace operation to replace the dummy property names with empty strings in the first output line (the header line).
the result - which already is in CSV format - can then be saved to a CSV file with Set-Content.
$obj = [PSCustomObject] #{
EMPLOYER_EIN = '123456'
ACTION_CODE = 1
BLANK_1 = $null # first dummy property
LAST_NAME = 'Smith'
FIRST_NAME = 'John'
MIDDLE_INITIAL = $null
BLANK_2 = $null # second dummy property
EMPLOYEE_SSN = '111-11-1111'
}
$first = $true
$obj |
ConvertTo-Csv |
ForEach-Object {
if ($first) { # header row: replace dummy property names with empty string
$first = $false
$_ -replace '\bBLANK_\d+'
}
else { # data row: pass through
$_
}
} # pipe to Set-Content as needed.
Output (note the blank column names after ACTION CODE and MIDDLE_INITIAL):
"EMPLOYER_EIN","ACTION_CODE","","LAST_NAME","FIRST_NAME","MIDDLE_INITIAL","","EMPLOYEE_SSN"
"123456","1",,"Smith","John",,,"111-11-1111"

Related

Need help understanding difference between Get-Content and Import-Csv in Powershell

I'm having trouble understanding when to use Get-Content and when to use Import-Csv for modifying a CSV file. Here is an example of my file, which is an output of a hash table:
Name Key Value
AOI1\\ABC1 AOI1\\ABC1 TRUE
AOI2\\DEF2 AOI2\\DEF2 TRUE
\#AOI3\#\\GHI3 \#AOI3\#\\GHI3 FALSE
I need to do the following to the file:
Remove '\\' and everything after it in the Name column
Remove '\\' and everything before it in the Key column
Replace '\#' with '#' in the last row
Rename the column headers
The result should look like this:
Loc FName Result
AOI1 ABC1 TRUE
AOI2 DEF2 TRUE
#AOI3# GHI3 FALSE
For removing the '\\' and everything after in the Name column, I came up with this script:
Import-Csv c:\test.csv |
% {$_.Name.split('\\\\')[0]}
This outputs the following:
AOI1
AOI2
However, I want to be able to write everything back to the same Csv file, so I tried changing it to:
Get-Content c:\test.csv |
% {$_.Name.split('\\\\')[0]}
However, I ended up with a "You cannot call a method on a null-valued expression" error message. If I keep the Import-Csv and modify the script to also do item 4, I get the same error message. Here's that script:
Import-Csv U:\To_Delete\Layer_search\results_STACK_layers.csv |
Select-Object #{n='AOI';e={$_.'Name'}}, #{n='LAYER';e={$_.'Key'}}, #{n='IN MAP';e={$_.'Value'}} |
% {$_.Name.split('\\\\')[0]}
So, how do I modify my CSV and output it to the same CSV afterwards? What could be causing that error message?
Sorry for the long-winded post, but I wanted to provide enough examples.
Thanks!
presuming you cannot use the advice from Bill_Stewart about fixing the source, here is one way to do the conversion. [grin] part of your problem was your use of the string split method instead of the string split operator. the 1st treats every character as a split target. the 2nd uses regex to split on the split string instead of on the characters in the split string.
# fake reading in a CSV file
# in real life, use Import-CSV
$InStuff = #'
Name, Key, Value
AOI1\\ABC1, AOI1\\ABC1, TRUE
AOI2\\DEF2, AOI2\\DEF2, TRUE
\#AOI3\#\\GHI3, \#AOI3\#\\GHI3, FALSE
'# | ConvertFrom-Csv
$Results = foreach ($IS_Item in $InStuff)
{
[PSCustomObject]#{
# new version removes the unwanted "\" from the ends of the "Loc" value
#Loc = ($IS_Item.Name -split [regex]::Escape('\\'))[0]
Loc = ($IS_Item.Name -split [regex]::Escape('\\'))[0].Replace('\')
FName = ($IS_Item.Key -split [regex]::Escape('\\'))[1]
Result = [convert]::ToBoolean($IS_Item.Value)
}
}
# on screen
$Results
# send to a CSV file
$Results |
Export-Csv -LiteralPath "$env:TEMP\Heather_RearrangedCSV.csv" -NoTypeInformation
on screen ...
Loc FName Result
--- ----- ------
AOI1 ABC1 True
AOI2 DEF2 True
#AOI3# GHI3 False
CSV file content ...
"Loc","FName","Result"
"AOI1","ABC1","True"
"AOI2","DEF2","True"
"#AOI3#","GHI3","False"

Extract content from log file from share location and send email in tabular format along with new headings to email

I am trying to automate few of the daily health check tasks using PowerShell.
I would like to achieve the following (Step by Step), though I have a partial success in few,
Extract content of text (Log) file located at shared location (I have succeeded) by defining $Path = Set-Location ... etc.,
Send email (succeeded) to mail box, by defining
Real help I need is here,
I want Headings to be appended in Email, along with original extracted text from Step 1,
For ex..
Original Text looks like this (extracted from text file at shared location):
01-01-2018 Number of Successful object - 1
I would like to add the header for this on email, like
date Description Number of Objects
01-01-2018 Successful objects 1
Assuming the content from the log file gathered in Step 1 is a string array of log entries where every string has a format similar to '01-01-2018 Number of Successful object - 1 '
In this example I call that array $logEntries
# create an array to store the results objects in
$result = #()
# loop through this array of log entries
$logEntries | ForEach-Object {
# Here every text line is represented by the special variable $_
# From your question I gather they all have this format:
# '01-01-2018 Number of Successful object - 1 '
if ($_ -match '(?<date>\d{2}-\d{2}-\d{4})\s+(?<description>[^\-\d]+)[\s\-]+(?<number>\d+)\s*$') {
# Try to get the 'Succesful' or 'Failed' (??) text part out of the description
$description = ($matches['description'] -replace 'Number of|object[s]?', '').Trim() + ' object'
if ([int]$matches['number'] -ne 1) { $description += 's' }
# add to the result array
$result += New-Object -TypeName PSObject -Property ([ordered]#{
'Date' = $matches['date']
'Description' = $description
'Number of Objects' = $matches['number']
})
}
}
# now decide on the format of this result
# 1) as plain text in tabular form.
# This looks best when used with a MonoSpaced font like Courier or Consolas.
$result | Format-Table -AutoSize | Out-String
# or 2) as HTML table. You will have to style this table in your email.
# You may include a stylesheet using the '-CssUri' parameter, but inserting
# it in a nicely drawn-up HTML template will give you more creative freedom.
$result | ConvertTo-Html -As Table -Fragment
p.s. because the PSObject has [ordered] properties, this needs PowerShell version 3.0 or better.

How to read a CSV file but exclude certain columns containing blanks using Get-Content

I want to read a CSV file and exclude rows where dynamically selected columns contain blanks but not all rows of those dynamically selected columns contain blanks.
Trying to use the where clause in the statement below (but not working):
Get-Content $Source -ReadCount 1000 |
Where {
ForEach($NotEqualBlankCol in $BlankColumns)
{
$NotEqualBlankCol -ne $null -and $NotEqualBlankCol -ne ''}
} |
ConvertFrom-Csv |
Sort-Object -Property $SortByColNames.Replace('"', '') -Unique |
.
.
.
| Out-File $Destination
$BlankColumns is my dynamic object string array which I would like to loop through containing the column names of the CSV that are blank. it can be 1 column or more. When more then all of the selected columns need to be blank to qualify as a row that does not need to be included in the final CSV file output.
How do I do it using Get-Content? Any help would be appreciated.
Using Get-Content
Ok. So what this will do it read in the contents of a file X lines at a time. It will parse each line into its indiviual columns. Then it will check the specified columns for blanks. If any of the flagged columns contains a black then it will be filtered out. Consider the test data I used for this
id,first_name,last_name,email,gender,ip_address
1,Christina,Tucker,ctucker0#bbc.co.uk,Female,91.33.192.187
2,Jacqueline,Torres,jtorres1#shop-pro.jp,Female,205.70.183.107
3,Kathy,Perez,kperez2#hugedomains.com,Female,35.175.154.127
4,"",Holmes,eholmes3#canalblog.com,,
5,Ernest,Walker,ewalker4#marketwatch.com,Male,140.110.129.21
6,,Garza,cgarza5#jugem.jp,,
7,,Cunningham,jcunningham6#ox.ac.uk,Female,
8,,Clark,lclark7#posterous.com,,
9,,Ortiz,lortiz8#shareasale.com,,
Notice that the first_name and gender are blank for some of these folks. id 1,2,3,5,10 have complete data. The rest should be filtered.
$BlankColumns = "first_name","gender"
$headers = (Get-Content $path -TotalCount 1).Split(",")
$potentialBlankHeaderIndecies = 0..($headers.Count - 1) | Where-Object{$BlankColumns -contains $headers[$_]}
$potentialBlankHeaderIndecies
Get-Content $path -ReadCount 3 | Foreach-Object{
# Check to see if any of the indexes from a split are empty
$_ | Where-Object{
[bool[]](($_.Split(","))[$potentialBlankHeaderIndecies] | ForEach-Object{
![string]::IsNullOrEmpty($_.Trim('"'))
}) -notcontains $false
}
}
The output of this code is the file, as string, with the removed entries. You can just pipe this into a variable, file or what even you need.
To go into a little more detail we take the header names we want to check and this read in the first line of the csv file. That should contain the column names. Using that we determine the column indexes that we want to scrutinize. The we read in the whole file and parse it line by line. For each line we split on the comma and check the elements matching the identified headers. Check each of those elements if they are blank or null. We trim quotes in case it is a string "" which I will assume you would count as blank. Of all the elements we evaluate as a Boolean whether or not it is empty. If at least one is then it fails the where-object clause and gets ommited.
Using Import-CSV
$BlankColumns = "first_name","gender"
Import-CSV $path | Where-Object{
$line = $_
($BlankColumns | ForEach-Object{
![string]::IsNullOrEmpty(($line.$_.Trim('"')))
}) -notcontains $false
}
Very similar approach just a lot less overhead since we are dealing with objects now instead of strings.
Now you could use Export-CSV or ConvertFrom-CSV depending on your needs in the rest of the project.
Changing the filter criteria.
Both examples above filter columns where any of the columns contain blanks. If you want to omit only where all are blank change the line }) -notcontains $false to }) -contains $true

How To Get The Value Of Header In CSV

In PowerShell I want to pass the name of a header in a CSV file into another function in the PowerShell script.
How can I retrieve the value-text of a header name into a variable in CSV?
e.g. if I have the following CSV data:
ID Name Country
-- ---- -------
1 John United States
2 Beatrice Germany
3 Jouni Finland
4 Marcel France
In the above example how can I retrieve the Country column value text as "Country" text into a variable in my script?
(Note: I am familiar with the notation $_.Country to retrieve the value of, for example, "Germany" from a row by importing the CSV in Powershell)
My specific issue is that currently I have the following function in my script:
function GetItemIdFromTitle([string]$LookupTitle, [ref]$LookupId)
{
$LookupField = $LookupList.Fields["DEPTCATEGORY"]
$LookupItem = $LookupList.Items | where {$_['DEPTCATEGORY'] -like "*$LookupTitle*"}
$LookupId.Value = $LookupItem.ID
}
This currently takes a string value -> $LookupTitle and uses that to find an item in a SharePoint list. As you can see in the script I am hard-coding in the column name as "DEPTCATEGORY". This is the column name that will be looked up to in the SharePoint list.
Instead of hard-coding the column name I want to pass in the name of the column for the corresponding $LookupTitle value and replace the hard-coded "DEPTCATEGORY".
I am calling the above function as follows:
#GET THE LOOKUP COLUMN ID
GetItemIdFromTitle $_.DEPTCAT ([ref]$LookupIdValue)
( $_.DEPTCAT is the value from the row in the CSV column. )
Can I do something like
$myCountryColumnName = $_.Country.Property.Title
or
$myCategoryColumnName = $_.DEPTCAT.Property.Name
to get the column name from the CSV?
If you have an object in $obj, you could list all the property headers like this:
$obj | Get-member -MemberType 'NoteProperty' | Select-Object -ExpandProperty 'Name'
This is an array, so you can reference them individually like this:
($obj | Get-member -MemberType 'NoteProperty' | Select-Object -ExpandProperty 'Name')[0]
This would just give you the name of the first property for instance.
Assuming that you have already read in the CSV using Import-CSV you can examine the property of the resulting object. Similar to this answer but maintains column order in the resulting array
Examining the first row / element
$data = import-csv $path
$data[0].psobject.properties.name
So the second line will return a string array of the properties.
To get the column name from the csv, first, put the names of the column headers into an array (this will also allow you to loop through each column, if needed) ...
$inFilePath = "C:\path\to\file.csv"
$csvColumnNames = (Get-Content $inFilePath | Select-Object -First 1).Split(",")
... , secondly, index into the array by column position (index starts at 0). Given your original example it would be;
$myCountryColumnName = $csvColumnNames[2]
This is more of a general comment than an answer.
I needed to pull the first column header name from CSVs and I started with selecting the NoteProperty fields from Get-Member. This doesn't work because the order of the NoteProperty column header names might not match the order of the column headers in the CSV file.
futureSPQR's method will work every time because the text won't get reordered on you. Below is my one-liner version of his method to get the first column header name.
((Get-Content filename.csv)[0] -split(','))[0]
$f = Import-Csv "...csv"
Get-Member -InputObject $f[0] |
Where-Object {$_.MemberType -eq "NoteProperty"} |
select-object Name
If you're looking to check if a header name exists in the array, use below code
$Array | Get-member -MemberType 'NoteProperty'| where {$_.Name -eq "ColumnName"}

Find and replace inside one csv column with powershell

I have a large CSV file that looks like this named student.export.text
Student Number,Last Name,Middle Name,First Name,Schoolid,Grade Level,Dob
I'm trying to build an automated task that will run nightly so that another piece of software can understand the CSV correctly.
Here is my code, but I'm missing something that is causing an error. I am new to Powershell and I am hoping for some advice.
Any help will be greatly appreciated!
$Replacements = #{
"5" = "AE";
"7" = "ER";
"10" = "FM";
"12" = "HC";
"14" = "JH";
"18" = "LE";
#...]
}
Import-Csv .\student.export.text | ForEach-Object {
$_.Schoolid = $Replacements[$_.Schoolid]
$_
} | Export-Csv -NoTypeInformation .\new.csv
Here's one approach that can work.
# declare hash table with School ID to School Name mapping
$schoolIdsToNames = #{
"3" = "SchoolA";
"4" = "SchoolB"
}
# import the CSV file
$csv = Import-Csv "C:\input.csv";
# for each row, replace the School ID field with the School Name
foreach($row in $csv)
{
$row.Schoolid = $schoolIdsToNames[$row.Schoolid];
}
# export the modified CSV
$csv | Export-Csv "C:\replaced.csv" -NoTypeInformation;
In the first step, we set up a PowerShell hashtable (a sort of key-value pair list), then import the CSV file using Import-Csv and store it in the $csv variable. This cmdlet will create an object from every row of the CSV that we can manipulate easily. For each row, we simply replace the Schoolid field with the value assigned to the ID key in the $schoolIdsToNames hashtable. Finally, we export the CSV to another file.
Another, more PowerShell-ly approach would be something like this:
Import-Csv "C:\test\school.csv" | Select-Object *, #{ Name = "SchoolName"; Expression = { $schoolIdsToNames[$_.Schoolid] } } | Export-Csv "C:\test\replaced2.csv" -NoTypeInformation
This one-liner imports the CSV and sends it down the pipeline. For each row, we select all properties of the row using Select-Object and add a new property called SchoolName, setting its value using the same hash table-based technique as above. Finally, we export the object list to CSV.