I have a CSV file with no usable header row (the first row is information about the file, i.e. creation date).
There are set record types in the CSV file which are in the first column. i.e. column 1 could be PRA, ASA or POA.
Depending on the value of column 1 will determine what's in the rest of the fields. From this file I need to blank out all data that I'm not going to require for security before the file is sent out to a third-party. As we have different record types I can't do a simple loop and block out columns 3, 4 and 6 for example.
My plan was to go through the CSV line by line, look at the first column then output each record type to a separate file so they could be processed individually.
Import-Csv -Delimiter ~ -Encoding UTF8 -Path tinysample.dat | Foreach-Object {
foreach ($property in $_.PSObject.Properties) {
if ($property.Name -eq 'HDR') {
if ($($property.Value) -eq 'PRA' -OR $($property.Value) -eq 'POA') {
Export-Csv -InputObject $_ -Append -Delimiter ~ -Encoding UTF8 -LiteralPath "$($property.Value).dat"
}
}
}
}
At the moment, the records are being output along with the header row which I don't want. I wanted to set my own header in each respective file which I can then use easily to determine which columns should be hidden.
Also some of the records are being truncated in the new CSV.
I was able to achieve this with bash using grep and awk and was hoping I'd be able to do the same with Powershell.
Related
I have a csv with columns A, B, C. I would like to import that to powershell and only on column C, remove any rows that have the word "Unknown" listed. If Column A or B has "Unknown", they stay, however, if Column C has it, the entire row gets deleted. Per the picture below, Row 4 would be deleted.
Can someone please provide a sample script to do this?
Thanks!
So, you have 3 problems you need to solve:
Import the data from the CSV file
Filter it based on the value of column C
Export the filtered data to a file again
To import, use the aptly named Import-Csv cmdlet:
$data = Import-Csv .\path\to\file.csv
Import-Csv will parse the CSV file and for each row it reads, it will output 1 object with properties corresponding to the column names in the header row.
To filter these objects based on the value of their C property, use the Where-Object cmdlet:
$filteredData = $data |Where-Object C -ne 'Unknown'
Where-Object will test whether the C property on each object does not have the value 'Unknown' (-ne = not equals), and discard any object for which that's not the case.
To re-export the filtered data, use the Export-Csv cmdlet:
$filteredData |Export-Csv .\path\to\output.csv -NoTypeInformation
You can also combine all three statements into a single pipeline expression:
Import-Csv .\path\to\file.csv |Where-Object C -ne 'Unknown' |Export-Csv .\path\to\output.csv -NoTypeInformation
This "one-liner" approach might be preferable if you're working on large CSV files (> hundreds of thousands of records), as it doesn't require reading the entire CSV file into memory at once.
$Data = Get-Content "C:\file.csv" | ConvertFrom-Csv
$Data | Where-Object {$_.C-ne 'Unknown'} | Export-Csv "C:\file_New.csv"
I have a tab delimited txt file and i need to switch first and second column names (without switching columns data). In other words I need to rename A(Id) to B(ExternalId) and B(ExternalId) to A(Id). Other columns in the file (other data) should stay unchanged. I'm very new in PowerShell, please advice. As I understand I need to use import/export csv cmdlet.
I tryed this, but it's not working the right way...
Import-Csv 'C:\original_users.txt' |
Select-Object Id, #{Name="ExternalId";Expression={$_."Id"}}; Select-Object ExternalId, #{Name="Id";Expression={$_."ExternalId"}} |
Export-Csv 'C:\changed_users.txt'
The Import-CSV and Export-CSV cmdlets have their strengths but this might not be one of them. The latter cmdlet would introduce quoting that might not be in your original file and that might not be desired.
Either way why not just do some text manipulation on the first line! Lets read in the file and and output the first lined, edited, and the remainder of the file. This sample uses a new location but you could easily write it back to the same file.
# Get the full file into a variable
$fullFile = Get-Content "c:\temp\mockdata.csv"
# Parse the first line into a column array
$columns = $fullFile[0].Split("`t")
# Rebuild the header by switching the columns order as desired.
$newHeader = ($columns[1],$columns[0] + ($columns | Select-Object -Skip 2)) -join "`t"
# Write the header back to file then the rest of the data.
$outputPath = "C:\somepath.txt"
$newHeader | Set-Content $outputPath
$fullFile | Select-Object -Skip 1 | Add-Content $outputPath
This also preserves the presence of other columns and their data.
$x1 = (1,22,333,4444)
$x1 | export-csv 'd:\123.csv' -Force
Then I get this:
How do I Get a table like this?:
CSV's don't just accept arbitrary data properly, you can use | Out-File x.csv to dump them out on individual lines, and then read it back in with Import-Csv specifying headers, but a proper CSV needs headers when it is saved.
if you want to save it out properly you need to convert it into an object where the numbers are actually "named" within an object so powershell can create a valid CSV.
1,22,333,4444 | ForEach {
[PSCustomObject]#{Number = $_}
} | Export-Csv C:\++\123.csv -NoTypeInformation
-NoTypeInformation removes the #TYPE header.
that being said, Out-File is the only way it will match your 'expected' output table, you don't seem to be looking for a CSV here.
This will create a proper csv file with a header:
ConvertFrom-Csv (1,22,333,4444) -Header Number|Export-Csv .\123.csv -NoType
Loaded in Excel cell A1 will be Number
This will create a fake Csv accepted by Excel and returning your sample table.
(1,22,333,4444)|Set-Content .\234.csv
I have a .csv file and I want to import it into powershell then iterate through the file changing certain values. I then want the output to append to the original .csv file, so that the values have been updated.
My issue is that the .csv file has headers which aren't unique, and can't be changed as then it won't work in another program. Originally I defined my own headers in the powershell to get around this but then the output file has these new headers when it needs to have the old ones.
I have also tried ConvertFrom-Csv which means I can no longer access the columns I need to, so lots of runtime errors.
What would be ideal is to be able to use the defined column headers and then convert back to the original column headers. My current code is below:
$csvfile = Import-Csv C:\test.csv| Where-Object {$_.'3' -eq $classID} | ConvertFrom-Csv
foreach($record in $csvfile){
*do something*}
$csvfile | Export-Csv -path C:\test.csv -NoTypeInformation -Append
I've searched the web now for some hours and tried everything I've come across, to no avail.
Thanks in advance.
This is a somewhat hackish implementation but should work.
Remove all the headers as a single line and save it somewhere
Parse the new result-set (with the headers removed)
Add the line at the top when you are finished
A CSV is a comma delimited file, you don't have to treat it like structured data. Feel free to splice and dice as you want.
Since you know beforehand how many columns are in the input CSV file, you can import without the header and process internally. Example:
$columns = 78
Import-Csv "inputfile.csv" -Header (0..$($columns - 1)) | Select-Object -Skip 1 | ForEach-Object {
$row = $_
$outputObject = New-Object PSObject
0..$($columns- 1) | ForEach-Object {
$outputObject | Add-Member NoteProperty "Col$_" $row.$_
}
$outputObject
} | Export-Csv "outputfile.csv" -NoTypeInformation
This example generates new PSObjects and then outputs a new CSV file with generic column names (Col0, Col1, etc.).
I have powershell script which connects to database & exports result in csv file.
However there is one column of date which size needs to be manually increased after opening csv file.
Do we have some command/property which will make columns AutoFit?
export-csv $OutFile -NoTypeInformation
I can't export excel instead CSV, cause I don't have excell installed on my machine.
This is what I have tried latest.
$objTable | Select Store,RegNo,Date,#{L="Amount";E={($_.Amount).PadLeft(50," ")}},TranCount
$objTable | export-csv $OutFile -NoTypeInformation
But even after adding PadLeft() output is same, Date column is short in width (showing ###, need to increase value manually)
When you say you need to increase one of your column sizes all the comments were right about how everything is formatted based on the object content. If you really need the cells to be a certain length you need to change the data before it is exported. Using the string methods .PadLeft() and .PadRight() I think you will get what you need.
Take this example using output from Get-ChildItem which uses a calculated property to pad the "column" so that all the data takes up at least 20 characters.
$path = "C:\temp"
$results = Get-ChildItem $path
$results | Select LastWriteTime,#{L="Name";E={($_.Name).PadLeft(20," ")}} | Export-CSV C:\temp\file.csv -NoTypeInformation
If that was then exported the output file would look like this (Notice the whitespace):
"LastWriteTime","Name"
"2/23/2015 7:33:55 PM"," folder1"
"2/23/2015 7:48:02 PM"," folder2"
"2/23/2015 7:48:02 PM"," Folder3"
"1/8/2015 10:37:45 PM"," logoutput"