PowerShell -- Import csv, reformat date values, export csv - powershell

I have a large volume of CSV records that have a date column that needs to be reformatted. The format in the source files is "MM/dd/yyyy hh:mm". The format I need to change to is "yyyy-MM-dd". I've tried a few different things, and I've been able to get the two pieces to work (import and transform, or export), but not both together.
Sample data
Date Created,ID,Email Address,First Name,Last Name
3/15/2019 14:56,some-string,email#address.com,first,last
3/15/2019 13:56,some-string,email#address.com,first,last
Import and transform (w/ Write-Host to confirm reformat)
Import-Csv .\foo.csv |
ForEach-Object {
$_."Date Created" = [datetime]::Parse($_."Date Created").ToString('yyyy-MM-dd')
Write-Host $_."Date Created"
}
Import and Export (w/ no transform)
Import-Csv foo.csv |
Select-Object "Date Created", ID, "Email Address", "First Name", "Last Name" |
Export-Csv -NoTypeInformation foo-new.csv
I've tried this to combine the two pieces, but I end up with an empty CSV file
Import-Csv foo.csv |
ForEach-Object {
$_."Date Created" = [datetime]::Parse($_."Date Created").ToString('yyyy-MM-dd')
} |
Select-Object "Date Created", ID, "Email Address", "First Name", "Last Name" |
Export-Csv -NoTypeInformation foo-new.csv
Any help to smash these together, so I have my reformat and my successful export, would be greatly appreciated.

You need to use a calculated property with Select-Object
Try
Import-Csv foo.csv |
Select-Object #{Name = 'Date Created'; Expression = { '{0:yyyy-MM-dd}' -f [datetime]$_.'Date Created' }},
* -ExcludeProperty 'Date Created' |
Export-Csv -NoTypeInformation foo-new.csv

Easiest way to go about this is using a calculated property:
Import-Csv -Path C:\users\Abraham\Desktop\csvDate.csv |
Select-Object -Property #{
Name = "DateCreated"
Expression = {
([datetime]$_."Date Created").ToString("yyyy-MM-d")
}
}, ID, "Email Address", "First Name", "Last Name" # | Export-Csv ...
As for what you tried. . .you're basically de-selecting all your properties when piped to Foreach-Object then trying to select what's not already there; hence why you get an empty csv.

Related

Trouble importing CSV and then exporting with some manipulation

I have the following file:
2018|CIB_C21_A_1_1_FromRangeAmount|.0000
As you can see, it has no headers. The first column can be ignored. I need to take the 2nd column and then export but I also need to add (2) additional columns in the export.
The export should look like this:
Account,Parent,Data Storage
CIB_C21_A_1_1|CIB_C21_A|Never Share
As you can see, I need to take the first column and also use the first 9 chracters as a value for another column. Then I need to add another column with the same value at all times.
I have been trying to play around with this some of this logic but can't quote any anything to work...
Import-Csv -Path "C:\TEMPCT\Revenue_RebatesDataForPlanning.csv" -Header zz,Account -Delim "|" |
sort "Account" –Unique |
Select "Account", "Parent" |
Export-Csv -Path "C:\TEMPCT\Revenue_RebatesDataForPlanningzz.csv" -Force -NoTypeInformation
Could anyone recommend some suggestions?
You can use calculated properties with Select-Object to create the desired new columns. To get the 9 first characters of the value from "Account" column you can use the String.Substring Method.
# This would be:
# Import-Csv path/to/csv.csv -Header Ignore, Account -Delimiter '|' | ...
#'
2018|CIB_C21_A_1_1_FromRangeAmount|.0000
2018|CIB_C22_A_1_1_FromRangeAmount|.0000
2018|CIB_C23_A_1_1_FromRangeAmount|.0000
2018|CIB_C24_A_1_1_FromRangeAmount|.0000
2018|CIB_C25_A_1_1_FromRangeAmount|.0000
'# | ConvertFrom-Csv -Header Ignore, Account -Delimiter '|' |
Select-Object Account, #{
Name = 'Parent'
Expression = { $_.Account.Substring(0,9) }
}, #{
Name = 'Data Storage'
Expression = { 'Never Share' }
} | ConvertTo-Csv -NoTypeInformation
How your code should look:
Import-Csv path/to/csv.csv -Header Ignore, Account -Delimiter '|' |
Select-Object Account, #{
Name = 'Parent'
Expression = { $_.Account.Substring(0,9) }
}, #{
Name = 'Data Storage'
Expression = { 'Never Share' }
} | Export-Csv path/to/newCsv.csv -NoTypeInformation

Break csv data based on server name

I am having below data in my csv file
"Server Name","Control Group","Tape Number","Status"
"BR502","QCLDUSYSW","Q23028","ACTIVE"
"BR502","QCLDUSYSW","Q32521","ACTIVE"
"BR502","QCLDUSYSW","Q05599","ACTIVE"
"BR503","QCLDUIPLW","Q25582","ACTIVE"
"BR503","QCLDUIPLW","Q15529","ACTIVE"
"BR503","QCLDUIPLW","Q05109","ACTIVE"
"BR504","QCLDUSYSW","Q14445","ACTIVE"
"BR504","QCLDUSYSW","Q27785","ACTIVE"
"BR504","QCLDUUSRD","Q26071","ACTIVE"
"BR505","QCLDUGRPD","Q27657","ACTIVE"
"BR505","QCLDUIPLW","Q17404","ACTIVE"
Please let me know how to break it based on server name like below
"Server Name","Control Group","Tape Number","Status"
"BR502","QCLDUSYSW","Q23028","ACTIVE"
"BR502","QCLDUSYSW","Q32521","ACTIVE"
"BR502","QCLDUSYSW","Q05599","ACTIVE"
"Server Name","Control Group","Tape Number","Status"
"BR503","QCLDUIPLW","Q25582","ACTIVE"
"BR503","QCLDUIPLW","Q15529","ACTIVE"
"BR503","QCLDUIPLW","Q05109","ACTIVE"
so on for rest of the servers.
Use the Group-Object cmdlet to group the records together based on a particular property:
Import-Csv .\input.csv |Group-Object -Property 'Server Name' |ForEach-Object {
# for each group, output a new CSV file with just the records pertaining to that server, named after the server
$_.Group |Export-Csv .\$($_.Name).csv -NoTypeInformation
}
Something like this might do what you want:
import-csv tst.csv | where 'Server Name' -eq 'BR502'
If you want to examine each set you would use a Group-Object to bunch it on Server Name like this:
Import-Csv tst.csv | Group-Object 'Server Name'
This gives an object that has two properties Name, Count and Group which contain the Value you groped by, the number of items in the group and the list of items in the group. This can be used for calculations on each group. For example if you want to know how many in Control Groups in each server end with D you could use this:
Import-Csv tst.csv | Group-Object 'Server Name' | Foreach-Object { $name=$_.Name; Write-Output $_.Group | Where-Object 'Control Group' -LIKE '*D' | Measure-Object | Select-Object #{Label='Name';Expression={$name}},Count }
If you use aliases that command can be shortened to:
ipcsv tst.csv | Group 'Server Name' | % { $name=$_.Name; $_.Group | ? 'Control Group' -LIKE '*D' | Measure | Select #{L='Name';E={$name}},Count }

Powershell: How to import csv find a value in a row, list of the column data and over-write some coumns?

I think it's easiest to show what I have and what I'm trying to get as an output instead of me trying to verbalize what I'm doing and what I've tried.
Input is a csv that contains:
Full Name
Color
Access Type
Access ID
John Smith
Blue
Full_Acccess
jsmith
John Smith
Blue
Partial_Access
John Smith
Red
No_Access
Bill Bob
Red
Full_Access
Bill Bob
Pink
Access_1
Lisa Smith
Green
Access_2
I will call this script each time and look for "Full Name". Then if "Access ID" is empty, I want to populate Access ID (from a variable) and write out the "Output" below to a variable and send that as the body of an email.
So run script, look for "Full Name". In this case, it will look for and finds "Bill Bob" and since Access ID is empty, it will write "Rob Bob" (a variable value) to "Access ID" but also send email body of output below
OutPut:
Full Name: Bill Bob
Color: Red, Pink
Date: 03-21-2021
Access_Type: Full_Access, Access_1
Access ID: Rob Bob
So it's showing a unique Full Name (not all three rows), but also all values of Color and Access Type for "Full Name"
I've got this working using:
Import-CSV |Group-Object -Propert 'Full Name' | Select-Object #{Name = "Full Name"; Expression = {(_.Group.'Full Name' | Select-Object -Unique)}},
#{Name = "Access Type"; Expression = {($_.Group.'Access Type' | Select-Object -Unique) -join ","}},
#{Name = "Color"; Expression = {($_.Group.'Color' | Select-Object -Unique)}},
#{Name = "New Name"; Expression = {($_.Group.'New Name' | Select-Object -Unique)}} |
Where-Object {$_.'Full Name' -eq $PersonName}
Where I'm stuck:
Script will be called and imports CSV and searches for "Bill Bob". Then I want to list all the columns for Bill AND write the new Access_ID value back into the CSV.
Then I call the script again and look for John Smith, and then again for Lisa Smith, etc..
Thanks all!
I think I got what you were going for, here's what I would use:
Function Get-UserAccess ([string]$fullname,[string]$csvpath){
# Export your output
$csv = Import-CSV $csvpath
$filtered = $csv | Where-Object {$_.'Full Name' -eq $fullname}
$output = [pscustomobject]#{
'Full Name' = $fullname
'Access Type'=($filtered.'Access Type' | Select -Unique) -join ','
'Color' =($filtered.'Color' | Select -Unique) -join ','
'Access ID' =($filtered.'Access ID' | Select -Unique) -join ','
}
# check if Access ID is empty on Full_Access users
if (-not $output.'Access ID' -and
($output.'Access Type' -split ',') -contains 'Full_Access') {
# Set your AccessID here
$AccessID = Read-Host "Enter a new access ID for $fullname"
# Update the Access ID in the csv:
$entry = ([Collections.Generic.List[Object]]$csv).findIndex({
$args[0].'Full Name' -eq $fullname -and
$args[0].'Access Type'-eq 'Full_Access'})
$csv[$entry].'Access ID' = $AccessID
Write-Host "Updating $csvpath with new access ID for $fullname"
$csv | Export-Csv $csvpath -Force -NoTypeInformation
# Return output with Access ID
$output.'Access ID' = $AccessID
return $output
} Else { Return $output }
}
And some example usage with the data from your question:
PS C:\> Get-UserAccess 'Bill Bob' 'C:\temp\temp.csv' | fl
Enter a new access ID for Bill Bob: Rob Bob
Updating C:\temp\temp.csv with new access ID for Bill Bob
Full Name : Bill Bob
Access Type : Full_Access,Access_1
Color : Red,Pink
Access ID : Rob Bob
PS C:\> Get-UserAccess 'John Smith' 'C:\temp\temp.csv' | fl
Full Name : John Smith
Access Type : Full_Access,Partial_Access,No_Access
Color : Blue,Red
Access ID : jsmith,

Compare and merge 2 csv files based on 2 first columns with possible duplicate values

I have 2 csv files I'm asked to merge where the values from the first column match. Both files can have the possibility of having duplicate values, and if they do, a new row should be created to support those values. If no match is found, then print the value no match.
Except for looking for duplicate values, I am using the following code...
Function GetFirstColumnNameFromFile
{
Param ($CsvFileWithPath)
$FirstFileFirstColumnTitle = ((Get-Content $CsvFileWithPath -TotalCount 2 | ConvertFrom-Csv).psobject.properties | ForEach-Object {$_.name})[0]
Write-Output $FirstFileFirstColumnTitle
}
Function CreateMergedFileWithCsv2ColumnOneColumn
{
Param ($firstColumnFirstFile, $FirstFileFirstColumnTitle, $firstFile, $secondFile, $resultsFile)
Write-Host "Creating hash table with columns values `"Csv2ColumnOne`" `"Csv2ColumnTwo`" From $secondFile"
$hashColumnOneColumnTwo2ndFile = #{}
Import-Csv $secondFile | Where-Object {$firstColumnFirstFile -contains $_.'Csv2ColumnOne'} | ForEach-Object {$hashColumnOneColumnTwo2ndFile[$_.'Csv2ColumnOne'] = $_.Csv2ColumnTwo}
Write-Host "Complete."
Write-Host "Creating Merge file with file $firstFile
and column `"Csv2ColumnTwo`" from file $secondFile"
Import-Csv $firstFile | Select-Object *, #{n='Csv2ColumnOne'; e={
if ($hashColumnOneColumnTwo2ndFile.ContainsKey($_.$FirstFileFirstColumnTitle)) {
$hashColumnOneColumnTwo2ndFile[$_.$FirstFileFirstColumnTitle]
} Else {
'Not Found'
}}} | Export-Csv $resultsFile -NoType -Force
Write-Host "Complete."
}
Function MatchFirstTwoColumnsTwoFilesAndCombineOtherColumnsOneFile
{
Param ($firstFile, $secondFile, $resultsFile)
[string]$FirstFileFirstColumnTitle = GetFirstColumnNameFromFile $firstFile
$FirstFileFirstColumn = Import-Csv $firstFile | Where-Object {$_.$FirstFileFirstColumnTitle} | Select-Object -ExpandProperty $FirstFileFirstColumnTitle
CreateMergedFileWithCsv2ColumnOneColumn $FirstFileFirstColumn $FirstFileFirstColumnTitle $firstFile $secondFile $resultsFile
}
Function Main
{
$firstFile = 'C:\Scripts\Tests\test1.csv'
$secondFile = 'C:\Scripts\Tests\test2.csv'
$resultsFile = 'C:\Scripts\Tests\testResults.csv'
MatchFirstTwoColumnsTwoFilesAndCombineOtherColumnsOneFile $firstFile $secondFile $resultsFile
}
Main
The contents of the first csv file is:
firstName,secondName
1234,Value1
2345,Value1
3456,Value1
4567,Value1
7645,Value3
The contents of the second csv file is:
Csv2ColumnOne,Csv2ColumnTwo,Csv2ColumnThree
1234,abc,Value1
1234,asd,Value1
3456,qwe,Value1
4567,mnb,Value1
The results is:
"firstName","secondName","Csv2ColumnOne"
"1234","Value1","asd"
"2345","Value1","Not Found"
"3456","Value1","qwe"
"4567","Value1","mnb"
"7645","Value3","Not Found"
Since the second file has a duplicate value of 1234 the result file should be:
"firstName","secondName","Csv2ColumnOne"
"1234","Value1","abc"
"1234","Value1","asd"
"2345","Value1","Not Found"
"3456","Value1","qwe"
"4567","Value1","mnb"
"7645","Value3","Not Found"
Is there a way I can do this?

Export CSV field formatting

I have the following script which identifies an account number based on the file name, and adds the list of accounts to a CSV file next to the date it was created.
$ReconFolder = "C:\Users\tenba1\Documents\QlikView\Account Recons"
Get-ChildItem $ReconFolder -Filter *.xls | Where-Object {$_.BaseName -match '^Recon_\d{16,20}_\d+$'} | ForEach-Object{
$id,$date = $_.BaseName.Split('_')[1..2]
New-Object PSObject -Property #{
"Account Number" = $id
"Date Requested" = $date.Insert(4,'/').Insert(7,'/')
}
} | sort-object DateCreated -Descending | Export-Csv $ReconFolder\itemList.CSV - NoTypeInformation
There are 2 problems with it:
The sorting isn't working for some reason. I can get around this by re-importing the file, sorting, then exporting again, but would like to do it properly.
The Account Number can be between 16 and 20 digits - when sending this to CSV the format should be text. Currently an account number like this 10201314050019817277 ends up in the file like this: 10201314050019800000.
The sorting isn't working because the pipeline contains your PSObject which doesn't contain DateCreated, to include this add it to your object:
New-Object PSObject -Property #{
"Account Number" = $id
"Date Requested" = $date.Insert(4,'/').Insert(7,'/')
"DateCreated" = $_.CreationTime
}
With regards to the formatting of the number, check the csv in a text editor, I suspect that you are viewing the data in Excel and that the csv contains the correct data.