How to add text qualifiers to a PowerShell CSV export? - powershell

Is there a way to include text qualifiers i.e. "Column 1"|"Column 2" when using PowerShell to export to a csv file?
In the script below I remove the text qualifiers that were already transferred in the file because the first column didn't have any for some odd reason.
Get-MsolUser -all |
Where-Object {($_.IsLicensed -eq "True") -and ($_.Title -notlike "Volunteer") } |
Select-Object Firstname, LastName, Department, Title |
ConvertTo-Csv -NoTypeInformation -Delimiter "|"|
% { $_ -replace '","', "|"} | % { $_ -replace '"', ""} | out-file "$path\$file_name" -fo -en ascii

I think the output may be as you expect but you may be viewing the output file in Excel rather than with a text editor (I used Notepad++). When I tried to reproduce this I used the following code and viewed the output file in a text editor and each column and row of the output has text qualifiers as expected. However, when viewing the same file in Excel the first column is displayed by Excel as if it does not have text qualifiers. Here's the code I ran to test:
"[a known UPN from my tenant]","[a known UPN from my tenant]"|
foreach-object {Get-MsolUser -UserPrincipalName $_} |
Where-Object {($_.IsLicensed -eq "True") -and ($_.Title -notlike "Volunteer")} |
Select-Object Firstname, LastName, Department, Title |
ConvertTo-Csv -NoTypeInformation -Delimiter "|" |
out-file $path\$file_name" -fo -en ascii

The issue with text qualifiers missing in the first column first occurred 'before' I specified the -delimiter parameter, my code above is a bit of a fix and mashup of two methods of fixing that issue, and a bit redundant.
I'm able to generate the intended results with the script below:
$MyData = import-csv \\TestPath\licenses_v2.csv -delimiter '|' |
select-object FirstName,LastName,Department,Title,#{expression={'Office 365 AD'}};
Write-DbaDataTable -sqlinstance $server -database $database -table $table -inputobject $MyData -KeepNulls;

Related

Export-Csv adding unwanted header double quotes

I have got a source CSV file (without a header, all columns delimited by a comma) which I am trying split out into separate CSV files based upon the value in the first column and using that column value as the output file name.
Input file:
S00000009,2016,M04 01/07/2016,0.00,0.00,0.00,0.00,0.00,0.00,750.00,0.00,0.00
S00000009,2016,M05 01/08/2016,0.00,0.00,0.00,0.00,0.00,0.00,600.00,0.00,0.00
S00000009,2016,M06 01/09/2016,0.00,0.00,0.00,0.00,0.00,0.00,600.00,0.00,0.00
S00000010,2015,W28 05/10/2015,2275.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
S00000010,2015,W41 04/01/2016,0.00,0.00,0.00,0.00,0.00,0.00,568.75,0.00,0.00
S00000010,2015,W42 11/01/2016,0.00,0.00,0.00,0.00,0.00,0.00,568.75,0.00,0.00
S00000012,2015,W10 01/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
S00000012,2015,W11 08/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
S00000012,2015,W12 15/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
My PowerShell script looks like this:
Import-Csv INPUT_FILE.csv -Header service_id,year,period,cash_exp,cash_inc,cash_def,act_exp,act_inc,act_def,comm_exp,comm_inc,comm_def |
Group-Object -Property "service_id" |
Foreach-Object {
$path = $_.Name + ".csv";
$_.group | Export-Csv -Path $path -NoTypeInformation
}
Output files:
S00000009.csv:
"service_id","year","period","cash_exp","cash_inc","cash_def","act_exp","act_inc","act_def","comm_exp","comm_inc","comm_def"
"S00000009","2016","M04 01/07/2016","0.00","0.00","0.00","0.00","0.00","0.00","750.00","0.00","0.00"
"S00000009","2016","M05 01/08/2016","0.00","0.00","0.00","0.00","0.00","0.00","600.00","0.00","0.00"
"S00000009","2016","M06 01/09/2016","0.00","0.00","0.00","0.00","0.00","0.00","600.00","0.00","0.00"
S00000010.csv:
"service_id","year","period","cash_exp","cash_inc","cash_def","act_exp","act_inc","act_def","comm_exp","comm_inc","comm_def"
"S00000010","2015","W28 05/10/2015","2275.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00"
"S00000010","2015","W41 04/01/2016","0.00","0.00","0.00","0.00","0.00","0.00","568.75","0.00","0.00"
"S00000010","2015","W42 11/01/2016","0.00","0.00","0.00","0.00","0.00","0.00","568.75","0.00","0.00"
It is generating the new files using the header value in column 1 (service_id).
There are 2 problems.
The output CSV file contains a header row which I don't need.
The columns are enclosed with double quotes which I don't need.
First of all the .csv file needs headers and the quote marks as a csv file structure. But if you don't want them then you can go on with a text file or...
$temp = Import-Csv INPUT_FILE.csv -Header service_id,year,period,cash_exp,cash_inc,cash_def,act_exp,act_inc,act_def,comm_exp,comm_inc,comm_def | Group-Object -Property "service_id" |
Foreach-Object {
$path=$_.name+".csv"
$temp0 = $_.group | ConvertTo-Csv -NoTypeInformation | Select-Object -Skip 1
$temp1 = $temp0.replace("""","")
$temp1 > $path
}
But this output is not a "real" csv file.
Hope that helps.
For your particular scenario you could probably use a simpler approach. Read the input file as a plain text file, group the lines by splitting off the first field, then write the groups to output files named after the groups:
Get-Content 'INPUT_FILE.csv' |
Group-Object { $_.Split(',')[0] } |
ForEach-Object { $_.Group | Set-Content ($_.Name + '.csv') }
Another solution,
using no named headers but simply numbers (as they aren't wanted in output anyway)
avoiding unneccessary temporary files.
removing only field delimiting double quotes.
Import-Csv INPUT_FILE.csv -Header (1..12) |
Group-Object -Property "1" | Foreach-Object {
($_.Group | ConvertTo-Csv -NoType | Select-Object -Skip 1).Trim('"') -replace '","',',' |
Set-Content -Path ("{0}.csv" -f $_.Name)
}

Filter CSV using .txt list of values

I have about 100 CSVs that I need to filter through, with the desired ouptut being all rows where the 'First Name' and 'Last Name' fields match a list that I have. Unfortunately, since the list I'm searching from was made separately and by hand there is no unique identifier or key beyond the combination of first and last name (I have street addresses but they are written in a different format than the CSVs).
What I've tried:
$myinput = Get-Content 'C:\folder\names.txt'
Import-Csv 'C:\folder\mycsv1.csv' | Where {
$myimput -match $_.'Last Name' -and
$myinput -match $_.'First_Name'
} | Out-File 'C:\folder\results.txt'
I get a blank .txt. When I tried again without the Out-File I got no output in PowerShell.
Also, If there is a way to automate this with all 100 CSVs, that would be VERY helpful. Any recommendations?
you have an error into imput, try Something like this for search into all csv (if all csv have same columns) :
$ListNames=get-content "c:\temp2\names.txt"
$dircsv="c:\temp2\"
$PathResultFile="c:\temp\result.csv"
gci $dircsv -file -filter "*.csv" | %{
import-csv $_.FullName | where {$csvrow=$_; ($ListNames | where {$_ -match $csvrow.'Last Name' -and $_ -match $csvrow.'First_Name'}).count -gt 0 }
} | export-csv $PathResultFile -NoType

Export-Csv emits Length and not values

I want to read a CSV file and output a CSV file with only one (1) field. I have tried to create a concise example.
PS C:\src\powershell> Get-Content .\t.csv
field1,field2,field3
1,2,3
4,55,6
7,888,9
PS C:\src\powershell> Import-Csv -Path .\t.csv | `
>> ForEach-Object {
>> $_.field2 `
>> } | `
>> Export-Csv -Path .\x.csv -NoTypeInformation
>>
The problem is that the Length of field2 is written to the exported CSV file. I want the field header to be "field2" and the values to be the value from the original CSV file. Also, I only want quotes where they are required; not everywhere.
I have read Export-CSV exports length but not name and Export to CSV only returning string length. But these do not seem to address producing an actual CSV file with a header and one field value.
PS C:\src\powershell> get-content .\x.csv
"Length"
"1"
"2"
"3"
CSV object uses note properties in each row to store its fields so we'll need to filter each row object and leave just the field(s) we want using Select-Object cmdlet (alias: select), which processes the entire CSV object at once:
Import-Csv 1.csv | select field2 | Export-Csv 2.csv -NoTypeInformation
Note, there's no need to escape the end of line if it ends with |, {, (, or ,.
It's possible to specify several fields: select field2, field3.
To strip unneeded doublequotes, general multi-field case:
Import-Csv 1.csv |
select field2 |
%{
$_.PSObject.Properties | %{ $_.value = $_.value -replace '"', [char]1 }
$_
} |
ConvertTo-Csv -NoTypeInformation |
%{ $_ -replace '"(\S*?)"', '$1' -replace '\x01', '""' } |
Out-File 2.csv -Encoding ascii
Simplified one-field case:
Import-Csv 1.csv |
select field2 |
%{
$_.field2 = $_.field2 -replace '"', [char]1
$_
} |
ConvertTo-Csv -NoTypeInformation |
%{ $_ -replace '"(\S*?)"', '$1' -replace '\x01', '""' } |
Out-File 2.csv -Encoding ascii
A tricky case of embedded quotes inside a field was solved by temporary replacing them with a control character code 01 (there are just a few that can be used in a typical non-broken text file: 09/tab, 0A/line feed, 0D/carriage return).
As per WOxxOm's response, Select-Object is best way to select only field from an input and pipe to output.
Regarding the quote marks, this is a known (and frustrating) issue with PowerShell. Specifying , as the delimiter did not help.
I have gotten round it by using ConvertTo-Csv and Foreach-Object replacements. THe replacements will need to be more complex if your data contains quote marks.
Import-Csv .\1.csv |
Select-Object field2 |
ConvertTo-Csv -NoTypeInformation |
ForEach-Object {$_ -replace '"',''} |
Out-File .\2.csv

New Line in Export-CSV not working

I want to insert a blank new line when exporting to CSV, but what I get is "1" inserted instead of blank new line. Here is the code I am using:
foreach($Group in $Groups) {
Get-QADGroupMember $Group | select  Name | Export-CSV C:\Users\Nikhil.Tamhankar\Desktop\temp\GroupsInfo.CSV -NoTypeInformation -Append -Encoding UTF8 -UseCulture
$newLine = "`n"
$newLine | Export-CSV C:\Users\Nikhil.Tamhankar\Desktop\temp\GroupsInfo.CSV -NoTypeInformation -Append -Encoding UTF8 -UseCulture
}
Please suggest how I can get a blank new line inserted after every entry made.
It looks like your code is just outputting a single column of group names. If that is the case then I would not even bother with CSV output at that point and just use Set-Content. Adding in the newline then would be fairly easy.
$Groups | ForEach-Object{
(Get-QADGroupMember $_ | Select-Object -ExpandProperty Name) + "`n"
} | Set-Content C:\Users\Nikhil.Tamhankar\Desktop\temp\GroupsInfo.CSV -Encoding UTF8
I used a more pipeline friendly foreach-object loop so you don't have to keep using append.
Export-Csv tries to "discover" the properties of the object, in this case a string - which only has one property, the Length - which happens to be 1.
Use Add-Content or Out-File -Append -NoNewline (version 5 only) to append to an existing file:
"" |Add-Content -Path C:\Users\Nikhil.Tamhankar\Desktop\temp\GroupsInfo.CSV

Powershell Append CSV

I am trying to append a CSV file. Here are the lines I am using. I wasn't able to find an append option for export-csv unfortunately. Any ideas would be helpful to get this to work.
Get-ADGroupMember "Domain Admins" | select name, samaccountname | Export-Csv c:\bin\DomainAdmins.csv
$admins = Import-Csv C:\bin\DomainAdmins.csv
foreach ($i in $admins) {Get-ADUser $i.samaccountname -properties * | select name, lastlogondate | Export-Csv c:\bin\dalogon.csv}
The documentation suggests that there is an -append flag. The example given ends with
| export-csv –append –path \\Archive01\Scripts\Scripts.csv
Have you tried that? It works fine for me. I'm on version 3, if that matters.
-Append was introduced with PowerShell v3, it's not available in PowerShell v2 and earlier. You can work around it like this, though:
... |
ConvertTo-Csv -NoTypeInformation |
Select-Object -Skip 1 |
Out-File -Append "c:\bin\dalogon.csv"
I ran into this issue also a few days ago. There are a really two solutions I know in Powershell 2. The first would be to save all the data in an array and then use the export-csv commandlet. That did not work for me unless I rewrote my script. I needed to create a CSV and append line by line to build the file. So, I solved it with out-file -append and changing the encoding to ascii.
I basically created a string with my data in it and then piped it to out-file. Here is an example:
$myCSV = "C:\_PSScripts\data\myCSV.csv"
$firstOutputLine = "Column-1,Column-3,Column-3"
$firstOutputLine | out-file $myCSV -Encoding ascii -Force -Append
It's good! but it's only 1 column in CSV file! how to put to many columns as the code below:
Import-Module -Name 'C:\Program Files\Quest Software\Management Shell for AD\Quest.ActiveRoles.ArsPowerShellSnapIn.dll'
$group="HO-Internet","Internet1","Internet2","Internet3"
$group |ForEach-Object {Echo "--------------------Group Name $_ ----------------"; Get-QADGroupMember $_ | Select-Object Email,LogonName,ParentContainer,LastLogon,AccountIsDisabled |ConvertTo-Csv -NoTypeInformation `
| select -Skip 1 `
| Out-File -Append "D:\test.csv"}