Using duplicate headers in Powershell .csv file - powershell

I have a .csv file and I want to import it into powershell then iterate through the file changing certain values. I then want the output to append to the original .csv file, so that the values have been updated.
My issue is that the .csv file has headers which aren't unique, and can't be changed as then it won't work in another program. Originally I defined my own headers in the powershell to get around this but then the output file has these new headers when it needs to have the old ones.
I have also tried ConvertFrom-Csv which means I can no longer access the columns I need to, so lots of runtime errors.
What would be ideal is to be able to use the defined column headers and then convert back to the original column headers. My current code is below:
$csvfile = Import-Csv C:\test.csv| Where-Object {$_.'3' -eq $classID} | ConvertFrom-Csv
foreach($record in $csvfile){
*do something*}
$csvfile | Export-Csv -path C:\test.csv -NoTypeInformation -Append
I've searched the web now for some hours and tried everything I've come across, to no avail.
Thanks in advance.

This is a somewhat hackish implementation but should work.
Remove all the headers as a single line and save it somewhere
Parse the new result-set (with the headers removed)
Add the line at the top when you are finished
A CSV is a comma delimited file, you don't have to treat it like structured data. Feel free to splice and dice as you want.

Since you know beforehand how many columns are in the input CSV file, you can import without the header and process internally. Example:
$columns = 78
Import-Csv "inputfile.csv" -Header (0..$($columns - 1)) | Select-Object -Skip 1 | ForEach-Object {
$row = $_
$outputObject = New-Object PSObject
0..$($columns- 1) | ForEach-Object {
$outputObject | Add-Member NoteProperty "Col$_" $row.$_
}
$outputObject
} | Export-Csv "outputfile.csv" -NoTypeInformation
This example generates new PSObjects and then outputs a new CSV file with generic column names (Col0, Col1, etc.).

Related

From a CSV file get the file header and a portion of the file based on starting and ending line number parameters using PowerShell

So I have a very huge CSV file, the first line has the column headers. I want to keep the first line as a header and add a portion of the file from the file's mid-section or perhaps the end. I'm also trying to select only a few of the columns from the file. And finally, it would be great if the solution also changed the file delimiter from a comma to a tab.
I'm aiming for a solution that's a one-liner or perhaps 2?
Non-working Code version 30 ...
Get-Content -Tail 100 filename.csv | Export-Csv -Delimiter "`t" -NoTypeInformation -Path .\filename_out.csv
I'm trying to get a better grip on PowerShell. So far, so good but I'm not quite there yet. But trying to solve such challenges are helping me (and hopefully others) build a good collection of coding idioms. (FYI - the boss is trying PowerShell due to our efforts so.)
OK thanks to iRon tip. Import-CSV defaults to comma separated, the Select-Object -Property get the columns I want, the select -Last gets the last 200 rows, and the Export-CSV changes the delimiter to a tab:
Import-Csv iarf.csv |
Select-Object -Property Id,Name,RecordTypeId,CreatedDate |
select -Last 200 |
Export-Csv -Delimiter "`t" -NoTypeInformation -Path .\iarf100props6.csv
iRon provided the crucial pointer: Using Import-Csv rather than Get-Content allows you to retrieve arbitrary ranges from the original file as objects, if selected via Select-Object, and exporting these objects again via Export-Csv automatically includes a header line whose column names are the input objects' property names, as initially derived from the input file's header line.
In order to select an arbitrary range of rows, combine Select-Object's -Skip and -First parameters:
To only get rows from the beginning, use just -First $count:
To only get rows from the end, use just -Last $count
To get rows in a given range, use just -Skip $startRowMinus1 -First $rangeRowCount
For instance, the following command extracts rows 10 through 30:
Import-Csv iarf.csv |
Select-Object -Property Id,Name,RecordTypeId,CreatedDate -Skip 9 -First 20 |
Export-Csv -Delimiter "`t" -NoTypeInformation -Path .\iarf100props6.csv

Filtering data from CSV file with PowerShell

I have huge csv file where first line contains headers of the data. Because the file size I can't open it with excel or similar. I need to filter rows what I only need. I would want to create new csv file which contains only data where Header3 = "TextHere". Everything else is filtered away.
I have tried in PowerShell Get-Content Select-String | Out-File 'newfile.csv' but it lost header row and also messed up with the data putting data in to wrong fields. There is included empty fields in the data and I believe that is messing it. When I tried Get-Content -First or -Last data seemed to be in order.
I have no experience handling big data files or powershell before. Also other options besides PowerShell is also possible if it is free to use as "non-commercial use"
try like this (modify your delimiter if necessary):
import-csv "c:\temp\yourfile.csv" -delimiter ";" | where Header3 -eq "TextHere" | export-csv "c:\temp\result.csv" -delimiter ";" -notype

Powershell - Querying CSV column names and writing them to a new CSV

So I'm trying to look at column names in a CSV, write them to an array and then spit the data back out into a new CSV with a new column attached. I don't really care what the current data table looks like so long as I can add a column to the headers. This seems like a fairly basic thing to be able to do but I can't seem to find any situations where anyone has done this. For example:
"Name","Location","Phone"
"John Smith","Toronto","555.555.5555"
"Jane Doe","Dallas","555.555.5554"
I just want to keep the Name, Location and Phone column names and put them into another CSV with one extra column. The catch is, in reality, there are more columns and they aren't always the same so the script needs to be able to be column name-agnostic. I should be able to feed it any CSV with any number of columns with different names and be able to get an output file with just the column names. I've tried at least a dozen different ways to do this and keep bumping into different issues.
Example:
$validatedfilepath = "validfile.csv"
$csvData = New-Object PSObject
$csvData = Import-CSV "file.csv" | Add-Member #{ID="$null"} -PassThru
$csvdata = $csvdata | get-member -Name * -MemberType NoteProperty
$csvData = #($csvData.Name)
$csvData
That will show me the exact list of column names that I want in my new validfile but I have no idea how to export it into a CSV as the column names. Each time I've tried doing export-csv, it either gives me the character length count in a new row for each or some other goofy stuff.
Thanks
The Export-CSV reads the column names from the members of the input object. So you are on the right track already, you just need to simplify a bit, like this:
$csvData = Import-CSV "file.csv" | Add-Member #{ID="$null"} -PassThru
# You can now modify/enrich/filter the data somehow, for example like $csvdata[0].ID = 56387, then just export it:
$csvData | Export-CSV -NoTypeInformation -Path "newfile.csv"
The error you made is overwriting the imported data in $csvdata with the Get-Member result. And then you overwrite it again with the #($csvData.Name) expression.

How can I alternate column headers in a tab delimited file?

I have a tab delimited txt file and i need to switch first and second column names (without switching columns data). In other words I need to rename A(Id) to B(ExternalId) and B(ExternalId) to A(Id). Other columns in the file (other data) should stay unchanged. I'm very new in PowerShell, please advice. As I understand I need to use import/export csv cmdlet.
I tryed this, but it's not working the right way...
Import-Csv 'C:\original_users.txt' |
Select-Object Id, #{Name="ExternalId";Expression={$_."Id"}}; Select-Object ExternalId, #{Name="Id";Expression={$_."ExternalId"}} |
Export-Csv 'C:\changed_users.txt'
The Import-CSV and Export-CSV cmdlets have their strengths but this might not be one of them. The latter cmdlet would introduce quoting that might not be in your original file and that might not be desired.
Either way why not just do some text manipulation on the first line! Lets read in the file and and output the first lined, edited, and the remainder of the file. This sample uses a new location but you could easily write it back to the same file.
# Get the full file into a variable
$fullFile = Get-Content "c:\temp\mockdata.csv"
# Parse the first line into a column array
$columns = $fullFile[0].Split("`t")
# Rebuild the header by switching the columns order as desired.
$newHeader = ($columns[1],$columns[0] + ($columns | Select-Object -Skip 2)) -join "`t"
# Write the header back to file then the rest of the data.
$outputPath = "C:\somepath.txt"
$newHeader | Set-Content $outputPath
$fullFile | Select-Object -Skip 1 | Add-Content $outputPath
This also preserves the presence of other columns and their data.

How to export array to csv in powershell?

$x1 = (1,22,333,4444)
$x1 | export-csv 'd:\123.csv' -Force
Then I get this:
How do I Get a table like this?:
CSV's don't just accept arbitrary data properly, you can use | Out-File x.csv to dump them out on individual lines, and then read it back in with Import-Csv specifying headers, but a proper CSV needs headers when it is saved.
if you want to save it out properly you need to convert it into an object where the numbers are actually "named" within an object so powershell can create a valid CSV.
1,22,333,4444 | ForEach {
[PSCustomObject]#{Number = $_}
} | Export-Csv C:\++\123.csv -NoTypeInformation
-NoTypeInformation removes the #TYPE header.
that being said, Out-File is the only way it will match your 'expected' output table, you don't seem to be looking for a CSV here.
This will create a proper csv file with a header:
ConvertFrom-Csv (1,22,333,4444) -Header Number|Export-Csv .\123.csv -NoType
Loaded in Excel cell A1 will be Number
This will create a fake Csv accepted by Excel and returning your sample table.
(1,22,333,4444)|Set-Content .\234.csv