Edit
I'll conclude that Import-Csv is not ideal for incorrect formatted CSV and will use Get-Content and split. Thanks for all the answers.
Example CSV:
"SessionID","ObjectName","DatabaseName",,,,,,,,
"144","","AC"
Using Import-Csv none of the empty fields at the end will be counted - it will simply stop after "DatabaseName".
Is there any way to include the empty fields?
Edit:
I simply need to count the fields and make sure there are less than X amount of them. It is not only the header that might contain empty fields but also the content. These files are often manually made and not properly formatted. Since the files also can get very large, I would prefer to not also use Get-Content and split since I'm already using Import-Csv and its properties.
Looks like it's missing its headers. If you would add some, it would work fine.
You could do something like
Get-Content My.CSV | Select -skip 1 | ConvertFrom-Csv -Header "SessionID","ObjectName","DatabaseName",'Whatnot1', 'Whatnot2', 'Whatnot3'
As dbso suggested split and Length will help you. I was on the way to code a header routine which now is obsolete. Nevertheless here it is:
$FileIn = "Q:\test\2017-01\06\SO_41505840.csv"
$Header= (Get-Content $FileIn|select -first 1)-split(",")
"Fieldcount for $FileIn is $($Header.Length)"
for($i=0; $i -lt $Header.Length; $i++){if ($Header[$i] -eq ""){$Header[$i]="`"Column$($i+1)`""}}
$Header -Join(",")
Returning this output
Fieldcount for Q:\test\2017-01\06\SO_41505840.csv is 11
"SessionID","ObjectName","DatabaseName","Column4","Column5","Column6","Column7","Column8","Column9","Column10","Column11"
Related
I found an answer to a previous question incredibly helpful, but I can't quite figure out how Get-Content is able able to store the 'line number' from the input.
Basically I'm wondering if PSObjects store information such as line number or row number. In the example below, it is basically like using Get-Content is able to store the line number as a variable you can use later. In the pipeline, the variable would be $_.psobject.Properties.value[5]
A bit of that seems redundant to me since $_ is an object (I think), but still it is very cool that .value[5] seems to be the line number or row number. The same is not true of Import-CSV and while I'm looking for a similar option with Import-CSV; I'd like to better understand why this works the way it does.
https://stackoverflow.com/a/23119235/15243610
Get-Content $colCnt | ?{$_} | Select -Skip 1 | %{if(!($_.split("|").Count -eq 210)){"Process stopped at line number $($_.psobject.Properties.value[5]), incorrect column count of: $($_.split("|").Count).";break}}
The answer in the other question works because Get-Content does indeed include the line number when it reads in the strings. When you run Get-Content each line will have a $_.ReadCount property as the 6th property on the object, which in my old answer I referenced in the PSObject for it as $_.psobject.Properties.value[5] (it was 7 years ago and I didn't know better yet, sorry). Mind you, if you use the -ReadCount parameter it will send that many lines through at a time, so Get-Content $file -readcount 5 | Select -first 1 | ForEach-Object{ $_.ReadCount } will come out as 5. Also -Raw sends everything through at once so it won't work with that.
Honestly, this isn't that hard to adapt to Import-Csv, we just increment a variable defined in the ForEach-Object loop.
Import-Csv C:\Path\To\SomeFile.csv | ForEach-Object -Begin {$x=1} -Process {
If($_.Something -eq $SomethingElse){
Write-Warning "Somethin' bad happened on line $x!"
break
}else{$_}
$x++
}
I have a log file which has data separated with "|" symbol. Like
"Username|servername|access|password|group"
"Username|servername|access|password|group"
I need to validate the data. And, If the group column(record) is missing information or empty. I need to write only that row into another file. Please help me. Thanks in Advance.
If you're just checking for missing data, you can run a quick check using a regex of '(\S+\|){4}\S+'. Use Get-Content with the -ReadCount parameter, and you can work in batches of a few thousand records at a time, minimizing disk i/o and memory usage without going through them one record at a time.
Get-Content $inputfile -ReadCount 2000 |
foreach {
$_ -notmatch '(\S+\|){4}\S+' |
Add-Content $outputfile
}
You could use 'Import-CSV with -Delimiter '|'. If your file doesn't have a header line, you would also need to use -Header to define it. You could then use Where to filter for the empty Group lines and Export-CSV with -Delimiter again to create a new file of just those lines.
For example:
Import-CSV 'YourLog.log' -Delimiter '|' -Header 'Username','Servername','Access','Password','Group' |
Where {$_.'Group' -eq ''} |
Export-CSV 'EmptyGroupLines.log' -Delimiter '|'
If your group column is always in the same place, which it looks like it is, you could use the split method. You can certainly neaten the code up. I have used the below as an example as to how you could use split.
The foreach statement is to iterate through each line in your file.
if (!$($groupstring.Split('|')[4])) checks if it is null.
$groupstring = 'Username|servername|access|password|group'
$groupstring.Split('|')[4]
foreach ($item in $collection)
{
if (!$($groupstring.Split('|')[4]))
{
Write-Host "variable is null"
}
}
Hope this helps.
Thanks, Tim.
I'm far away from being an expert in PowerShell, so I'll be my best to explain here.
I was able to add a column, but now I want to add stuff in a column (already there) using a separate script.
For example, the following CSV file:
WhenToBuyThings,ThingsToBuy
BuyNow ,Bed
BuyNow ,Desk
BuyNow ,Computer
BuyNow ,Food
BuyLater ,
BuyLater ,
BuyLater ,
I have the array:
$BuyStuffLater = "Books","Toys","ShinnyStuff"
So the end result of the file should look like this
BuyNow ,Bed
BuyNow ,Desk
BuyNow ,Computer
BuyNow ,Food
BuyLater ,Books
BuyLater ,Toys
BuyLater ,ShinnyStuff
Any help with how to do this in code would be much appreciated. Also, we can't use delimiter ",". Because in the real script some values will have commas.
I got it after a few hours of fiddling...
$myArray = "Books","Toys","ShinnyStuff"
$i = 0
Import-Csv "C:\Temp\test.csv" |
ForEach-Object {if($_.WhenToBuyThings -eq "BuyLater"){$_.ThingsToBuy = $myArray[$i];$i++}return $_} |
Export-Csv C:\Temp\testtemp.csv -NoTypeInformation
All is well now...
I am new to powershell, too. Here's what I found. This searches and returns all lines that fit. I'm not sure it can pipe.
$BuyStuffLater = "Books","Toys","ShinnyStuff"
$x = 0
Get-Content -Path .\mydata.txt | select-string -pattern "BuyLater" #searches and displays
# Im not sure about this piping. (| foreach {$_ + $BuyStuffLater[$x];$x++} | Set-Content .\outputfile.csv)
This filter will work, though I still have to work on the piping. The other answer might be better.
I don't see a point to iterating through each object to see if it is a WhenToBuyThings is "BuyLater". If anything what you are doing could be harmful if you run multiple passes adding to the list. It could remove previous things you wanted to by. If "Kidney Dialysis Machine" was listed as a "BuyLater" under WhenToBuyThings then you would overwrite it with dire consequences.
What we can do is build two lists and merge into new csv file. First list is your original file minus any entry where a "BuyLater" has a blank ThingsToBuy. The second list is an object array built from your $BuyStuffLater. Add these lists together and export.
Also there is zero need to worry about using a comma delimiter when using Export-CSV. The data is quoted so commas in data do not affect the data structure. If this was still a concern you could use -Delimiter ";". I noticed in your answer that you did not attempt to account for commas either (not that it matters based on what I just said).
$path = "C:\Temp\test.csv"
$ListOfItemsToBuy = "Books","Toys","ShinnyStuff: Big, ShinyStuff"
$BuyStuffLater = $ListOfItemsToBuy | ForEach-Object{
[pscustomobject]#{
WhenToBuyThings = "BuyLater"
ThingsToBuy = $_
}
}
$CurrentList = Import-Csv $path | Where-Object{$_.WhenToBuyThings -ne "BuyLater" -and ![string]::IsNullOrWhiteSpace($_.ThingsToBuy)}
$CurrentList + $BuyStuffLater | Export-Csv $path -NoTypeInformation
Since you have 3.0 we can use [pscustomobject]#{} to build the new object very easily. Combine both arrays simply by adding them together and export back to the original file.
You should notice I used slightly different input data. One includes a comma. I did that so you can see what the output file looks like.
"BuyLater","Books"
"BuyLater","Toys"
"BuyLater","ShinnyStuff: Big, ShinyStuff"
Question
How do I reformat an unknown CSV column name according to a formula or subroutine (e.g. rename column " Arbitrary Column Name " to "Arbitrary Column Name" by running a trim or regex or something) while maintaining data?
Goal
I'm trying to more or less sanitize columns (the names) in a hand-produced (or at least hand-edited) csv file that needs to be processed by an existing PowerShell script. In this specific case, the columns have spaces that would be removed by a call to [String]::Trim(), or which could be ignored with an appropriate regex, but I can't figure a way to call or use those techniques when importing or processing a CSV.
Short Background
Most files and columns have historically been entered into the CSV properly, but recently a few columns were being dropped during processing; I determined it was because the files contained a space (e.g., Select-Object was being told to get "RFC", but Import-CSV retrieved "RFC ", so no matchy-matchy). Telling the customer to enter it correctly by hand (though preferred and much simpler) is not an option in this case.
Options considered
I could manually process the text of the file, but that is a messy and error prone way to re-invent the wheel. I wonder if there's a syntax with Select-Object that would allow a softer match for column names, but I can't find that info.
The closest I have come conceptually is using a calculated property in the call to Select-Object to rename the column, but I can only find ways to rename a known column to another known column. So, this would require enumerating the columns and matching them exactly (preferred) or a softer match (like comparing after trimming or matching via regex as a fallback) with expected column names, then creating a collection of name mappings to use in constructing calculated properties from that information to select into a new object.
That seems like it would work, but more it's work than I'd prefer, and I can't help but hope that there's a simpler way I haven't been able to find via Google. Maybe I should try Bing?
Sample File
Let's say you have a file.csv like this:
" RFC "
"1"
"2"
"3"
Code
Now try to run the following:
$CSV = Get-Content file.csv -First 2 | ConvertFrom-Csv
$FixedHeaders = $CSV.PSObject.Properties.Name.Trim(' ')
Import-Csv file.csv -Header $FixedHeaders |
Select-Object -Skip 1 -Property RFC
Output
You will get this output:
RFC
---
1
2
3
Explanation
First we use Get-Content with parameter -First 2 to get the first two lines. Piping to ConvertFrom-Csv will allow us to access the headers with PSObject.Properties.Name. Use Import-Csv with the -Header parameter to use the trimmed headers. Pipe to Select-Object and use -Skip 1 to skip the original headers.
I'm not sure about comparisons in terms of efficiency, but I think this is a little more hardened, and imports the CSV only once. You might be able to use #lahell's approach and Get-Content -raw, but this was done and it works, so I'm gonna leave it to the community to determine which is better...
#import the CSV
$rawCSV = Import-Csv $Path
#get actual header names and map to their reformatted versions
$CSVColumns = #{}
$rawCSV |
Get-Member |
Where-Object {$_.MemberType -eq "NoteProperty"} |
Select-Object -ExpandProperty Name |
Foreach-Object {
#add a mapping to the original from a trimmed and whitespace-reduced version of the original
$CSVColumns.Add(($_.Trim() -replace '(\s)\s+', '$1'), "$_")
}
#Create the array of names and calculated properties to pass to Select-Object
$SelectColumns = #()
$CSVColumns.GetEnumerator() |
Foreach-Object {
$SelectColumns += {
if ($CSVColumns.values -contains $_.key) {$_.key}
else { #{Name = $_.key; Expression = $CSVColumns[$_.key]} }
}
}
$FormattedCSV = $rawCSV |
Select-Object $SelectColumns
This was hand-copied to a computer where I don't have the rights to run it, so there might be an error - I tried to copy it correctly
You can use gocsv https://github.com/DataFoxCo/gocsv to see the headers of the csv, you can then rename the headers, behead the file, swap columns, join, merge, any number of transformations you want
I have a pipe-delimited text file with a header row. (I said CSV in the question to make it a a bit more immediately understandable ... I imagine most solutions would be applicable to either format.)
The file looks like this:
COLUMN1|COLUMN2|COLUMN3|COLUMN4|...|
Field1|Field2|Field3|Field4|...|
...
I need to obscure the data in (for example) columns 3 and 9, without affecting any of the other entries in the file.
I want to do this using a hashing algorithm like SHA1 or MD5, so that the same strings will resove to the same hash values anywhere they are encountered.
EDIT - Why I want to do this
I need to send some data to a third party, and certain columns contain sensitive information (e.g. customer names). I need the file to be complete, and where a string is replaced, I need it to be done in the same way every time it is encountered (so that any mapping or grouping remains). It does not need military encryption, just to be difficult to reverse. As I need to to this intermittently, a scripted solution would be ideal.
/EDIT
What is the easiest way to achieve this using a command line tool or script?
By preference, I would like a batch script or PowerShell script, since that does not require any additional software to achieve...
Try
(Import-Csv .\my.csv -delimiter '|' ) | ForEach-Object{
$_.column3 = $_.column3.gethashcode()
$_.column4 = $_.column4.gethashcode()
$_
} | Export-Csv .\myobfuscated.csv -NoTypeInformation -delimiter '|'
$md5 = new-object -TypeName Security.Cryptography.MD5CryptoServiceProvider
$utf8 = new-object -TypeName Text.UTF8Encoding
import-csv original.csv -delimiter '|' |
foreach {
$_.Column3 = [BitConverter]::ToString($md5.ComputeHash($utf8.GetBytes($_.Column3)))
$_.Column9 = [BitConverter]::ToString($md5.ComputeHash($utf8.GetBytes($_.Column9)))
$_
} |
export-csv encrypted.csv -delimiter '|' -noTypeInformation