I have a text file. Each line record an event and the fields are delimited by '|'.
Is it possible using some cmdlet like "get-content event.log | export-csv event.csv" to convert text file to csv file?
xxx(1365)|2016-09-29 06:00:00.0|2016-09-29 06:30:00.0|
bbb(110)|2016-09-29 06:30:00.0|2016-09-29 07:00:00.0|
ccc(5243)|2016-09-29 07:00:00.0|2016-09-29 07:30:00.0|
ddd(1950)|2016-09-29 07:30:00.0|2016-09-29 08:00:00.0|
eee(10)|2016-09-29 08:00:00.0|2016-09-29 09:00:00.0|
fff(464)|2016-09-29 09:00:00.0|2016-09-29 10:00:00.0|
dddd(874)|2016-09-29 10:00:00.0|2016-09-29 11:00:00.0|
ggggg(6)|2016-09-29 11:00:00.0|2016-09-29 12:00:00.0|
kkkk(272)|2016-09-29 12:00:00.0|2016-09-29 12:30:00.0|
The Import-Csv cmdlet allows you to specify a delimiter
$file = Import-Csv .\event.log -Delimiter '|'
so in your case, it can be as simple as
Import-Csv .\event.log -Delimiter "|" | Export-Csv .\event.csv -NoTypeInformation
If the file isn't too big (under a few megabytes) you could do it with a straight-forward String.Replace operation:
$content = ( $fileName | Get-Content )
$content = $content -replace '|' ','
$content | Out-File $fileName
Or more succinctly:
( $fileName | Get-Content ) -replace '|' ',' | Out-File $fileName
This won't work very well for large files (more than a few megabytes) because the entire file is loaded into memory as a System.String instance, then the Replace operation will then create a new instance (thus doubling the memory requirement).
A faster version might read from the input file line-by-line and perform the -replace operation for each line - or even character-by-character.
Related
I need to convert strings in a csv file to strings with quotation marks around it.
My csv file looks like this:
Description;AllowHosts;SPNs;Owner
Description1;server1$, server2$, server3$;MSSQLSvc/PD01.dom1.com:1521,MSSQLSvc/PD01.dom1;Owner JDOE
Description2;server4$, server5$, server6$;MSSQLSvc/PD02.dom2.com:1521,MSSQLSvc/PD02.dom2;Owner JDOE
Description3;server7$, server8$, server9$;MSSQLSvc/PD03.dom1.com:1521,MSSQLSvc/PD03.dom1;Owner JDOE
I tried to search for header "AllowHosts" and replace with quotation mark in start and end,
$csv = #(
Import-Csv -Path $New -Delimiter ';' -Encoding UTF8
)
$data = ConvertFrom-Csv $csv
$Data[0].AllowHosts = '"'
$Data | where AllowHosts -Like '*$' | foreach {
$_.AllowHosts = '*$"'
}
$Data | where AllowHosts -Like 'SF' | foreach {
$_.AllowHosts = '"SF*'
}
$Data | ConvertTo-Csv -NoTypeInformation
but it did not work as expected....
I would like to have quotation mark around each string
in column "AllowHosts" (servernames)
in column "SPNs"
I am hoping for a result like this:
Description;AllowHosts;SPNs;Owner
Description1;"server1$", "server2$", "server3$";"MSSQLSvc/PD01.dom1.com:1521","MSSQLSvc/PD01.dom1";Owner JDOE
Description2;"server4$", "server5$", "server6$";"MSSQLSvc/PD02.dom2.com:1521","MSSQLSvc/PD02.dom2";Owner JDOE
Description3;"server7$", "server8$", "server9$";"MSSQLSvc/PD03.dom1.com:1521","MSSQLSvc/PD03.dom1";Owner JDOE
But how?
I have a powershell script that imports csv-file and creates json-files. My problem is that this line
" ""PrincipalsAllowedToRetrieveManagedPassword"""+": [" | Out-File $filepath1 -Append
gives this result
"PrincipalsAllowedToRetrieveManagedPassword": [ "server1$, server2$, server3$" ],
instead of
"PrincipalsAllowedToRetrieveManagedPassword": [ "server1$", "server2$", "server3$" ],
Use the -replace operator to add "'s around each "word" in the string:
# read data into memory
$csv = Import-Csv -Path $New -Delimiter ';' -Encoding UTF8
# modify all `AllowHosts` and `SPN` cells
$csv |ForEach-Object {
$_.AllowHosts = $_.AllowHosts -replace '([^\s,]+)','"$1"'
$_.SPNs = $_.SPNs -replace '([^\s,]+)','"$1"'
}
# re-export
$csv |Export-Csv -Path path\to\export.csv -NoTypeInformation
The pattern ([^\s,]+) matches (and captures) any consecutive sequence of characters not containing , or whitespace, and the substitution string "$1" expands to "".
Beware that this introduces ambiguity, as "'s are also used as value qualifiers in CSVs - so Export-Csv will escape the quotation marks you've added to retain them, and the resulting file will look like this:
"Description","AllowHosts","SPNs","Owner"
"Description1","""server1$"", ""server2$"", ""server3$""","""MSSQLSvc/PD01.dom1.com:1521"",""MSSQLSvc/PD01.dom1""","Owner JDOE"
"Description2","""server4$"", ""server5$"", ""server6$""","""MSSQLSvc/PD02.dom2.com:1521"",""MSSQLSvc/PD02.dom2""","Owner JDOE"
"Description3","""server7$"", ""server8$"", ""server9$""","""MSSQLSvc/PD03.dom1.com:1521"",""MSSQLSvc/PD03.dom1""","Owner JDOE"
I have the following code which removes all the commas in my csv file but there is an issue:
I only want to remove commas on Column P and and the remaining data should be untouched. Currently it appends data underneath the csv data.
$inform = Get-Content C:\Users\bmd\Desktop\report.csv
$inform.ToString()
$y=$inform -replace ',' -replace ''
$y | Out-file C:\Users\bmd\Desktop\report.csv -Append
Using Import-Csv and Export-Csv is usually going to be easier than trying to manipulate strings in a CSV. With no sample of the CSV file, we can only make assumptions. Assuming it contains true and qualified as needed CSV data, you can try the following:
$csv = Import-Csv C:\Users\bmd\Desktop\report.csv
foreach ($row in $csv) {
$row.UNSTRUCTURED_VARS = $row.UNSTRUCTURED_VARS -replace ','
}
$csv | Export-Csv C:\Users\bmd\Desktop\report.csv -NoType
Note that if you are on PowerShell Core or PowerShell 7, you can simply use Export-Csv with -UseQuotes AsNeeded and only the required fields will be qualified.
Good day,
with the script below I would like to use the following input txt from my output txt.
Input:
Klaus;Müller;Straße;PLZ;Ort;;;;;DE12345;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE12345678;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE999999;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE7777777;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE7777779;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE777777987;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE7777779765;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE77777797634;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE7777779763465;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE77777797623435435;
Output:
Klaus;Müller;Straße;PLZ;Ort;;;;;DE12345;DE12345678;DE999999;DE7777777;DE7777779;DE777777987;DE7777779765;DE77777797634;DE7777779763465;DE77777797623435435;
The script takes the last value from the following lines and appends them to the first line at the end and adds semicolons:
Import-Csv input.txt -delimiter ";" -Header (1..20)
1..9 | %{$data[0].($_+10) = $data[$_].10}
($data[0] | convertto-csv -delimiter ";" -NoType | select -skip 1) -replace '"' | out-file output.txt
gc test_neu.txt
if i save this into a .ps1 file it doesn´t work. anyone could say me why?
You don't assign Import-Csv to anything. The first line should be: $data = Import-Csv input.txt -delimiter ";" -Header (1..20) Your last line should be gc output.txt. And use the dot notation to location the input.txt file in the current directory. With these fixes, your script works:
$data = Import-Csv .\input.txt -delimiter ";" -Header (1..20)
1..9 | %{$data[0].($_+10) = $data[$_].10}
($data[0] | convertto-csv -delimiter ";" -NoType | select -skip 1) -replace '"' | out-file output.txt
gc output.txt
this seems to do what you want. [grin] it expects that the source lines are all to be combined.
i presume you can handle saving things to a file, so i leave that to you.
what it does ...
fakes reading in a text file
when ready to work with real data, replace the entire #region/#endregion block with a call to Get-Content.
iterates thru the collection by index number
if the line is the 1st, set $NewString to that entire value
else, add the last data item of the line to the existing $NewString value with a trailing ;
the .Where({$_}) filters out any blank items.
display the string
the code ...
#region >>> fake reading in a text file
# in real life, use Get-Content
$InStuff = #'
Klaus;Müller;Straße;PLZ;Ort;;;;;DE12345;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE12345678;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE999999;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE7777777;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE7777779;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE777777987;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE7777779765;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE77777797634;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE7777779763465;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE77777797623435435;
'# -split [System.Environment]::NewLine
#endregion >>> fake reading in a text file
foreach ($Index in 0..$InStuff.GetUpperBound(0))
{
if ($Index -eq 0)
{
$NewString = $InStuff[$Index]
}
else
{
$NewString += $InStuff[$Index].Split(';').Where({$_})[-1] + ';'
}
}
$NewString
output ...
Klaus;Müller;Straße;PLZ;Ort;;;;;DE12345;DE12345678;DE999999;DE7777777;DE7777779;DE777777987;DE7777779765;DE77777797634;DE7777779763465;DE77777797623435435;
Just in case you don't know how many lines there are going to be on the input file:
$fmt='$1$2'
gc .\input.txt | %{$_ -replace '(^.*;)(.*;$)',$fmt;$fmt='$2'} | sc output.txt -NoNewline
gc output.txt
I have a CSV file where I need to change the delimiter from comma to semicolon.
I fixed this issue with following command
(Import-CSV $File) | Export-Csv $File -delimiter ';' -notypeinformation
But when the imported CSV file contains only one line which I assume is being handled as the headers. It deletes this line in the output file and it becomes an empty file.
However when I have more than one line and the csv contains the header and some data the exported file is being executed correctly with the headers and the data.
Code that I used to set the header file in the beginning of my script
#set csv headers
Set-Content -Path "$FolderNameCiTypeDelta" -Value "name,type,location,size"
I've tried adding the code
(Import-CSV $File -Header "name,type,location,size") | Export-Csv $File -delimiter ';' -notypeinformation
But this made just an extra line with the headers I specified.
[EDIT]
The 2 csv files that I test with contain the following lines
The file that becomes empty with the import --> export command contains
name,type,location,size
The file that performs what I need with the import --> export command contains
name,type,location,size
test,extra,line
You can do the following:
$file = 't.txt'
$contents = Get-Content $file -totalcount 2
if ($contents.Count -gt 1) {
(Import-Csv $file) | Export-Csv $file -Delimiter ';' -NoType
}
if ($contents.Count -eq 1) {
$contents -replace ',',';' | Set-Content $file
}
The idea here is to read two lines of your file. If it only contains one line, it will use -replace to fix your delimiters. If it has more than two lines, it will use the *-Csv commands.
The -replace section only works if you don't have commas in your field values. If you do have commas in your fields, then you should have text qualifiers (double quotes most likely) around your fields. In that case, you can simply use -replace '","','";"'.
In Powershell Script, how do I convert a | (pipe) delimited CSV file to a , (comma) delimited CSV file?
When we use the following command in Windows Powershell Encoding 'UTF8' -NoType to convert from | (pipe delimiter) to , (comma delimiter), the file is converted with , delimited but the string was surrounded by " " (double quotes). Like given below:
Source file data:
ABC|1234|CDE|567|
Converted file data:
"ABC","1234","CDE","567",
I want to generate the following:
ABC,1234,CDE,567,
What command can I use to convert the delimiter from | to ,?
I would use:
(Get-Content -Path $file).Replace('|',',') | Set-Content -Path $file
You must escape the pipe, so:
(get-content "d:\makej\test.txt" ) -replace "\|","," | set-content "d:\makej\test.csv"
Seems easy enough:
(get-content $file) -replace '|',',' | set-content $file
In general, you should use the commands Import-Csv and Export-Csv which properly handle delimiters embedded in the field values, such as Field,1|Field2. The Get-Content based solutions would turn this into 3(!) fields Field,1,Field2, while the output actually should be quoted like "Field,1",Field2 or "Field,1","Field2".
Import-Csv input.csv -Delimiter '|' | Export-Csv output.csv -Delimiter ','
This always quotes fields in "output.csv". Since PowerShell (Core) 7+, the new Export-Csv parameters -UseQuotes and -QuoteFields allow us to control the quoting of the output file.
E. g. to quote only if necessary (when a field value contains the delimiter or quotation marks):
Import-Csv input.csv -Delimiter '|' | Export-Csv output.csv -Delimiter ',' -UseQuotes AsNeeded
Be careful with -UseQuotes Never, because it can render the output file unreadable, if a field value contains embedded delimiter or quotation marks.
Here is a function to convert to unquoted CSV for PowerShell 5.x (possibly supports older versions as well). This is like -UseQuotes Never, so make sure your data doesn't contain the delimiter. Additionally you may omit the header by passing the -NoHeader switch.
Function ConvertTo-CsvUnquoted {
[CmdletBinding()]
param (
[Parameter(Mandatory, ValueFromPipeline)] $InputObject,
[string] $Delimiter = ',',
[switch] $NoHeader
)
process {
if( -not $NoHeader ) {
$_.PSObject.Properties.Name -join $Delimiter
$NoHeader = $true
}
$_.PSObject.Properties.Value -join $Delimiter
}
}
Usage example:
Import-Csv input.csv | ConvertTo-CsvUnquoted -Delimiter '|' | Set-Content output.csv
Sorry this may need some tweaking on your part, but it does the job. Note that this also changes the file type from .txt to .csv which I dont think you wanted.
$path = "<Path>"
$outPath = $path -replace ".txt",".csv"
Get-Content -path $path |
ForEach-Object {$_ -replace "|","," } |
Out-File -filepath $outPath
I view the suggested answers as a little risky, because you are getting the entire contents of the existing file into memory, and therefore won't scale well, and risks using a lot of memory. My suggestion would be to use the string replace as the previous posts suggested, but to use streams instead for both reading and writing. That way you only need memory for each line in the file rather than the entire thing.
Have a look here at one of my other answers here:
https://stackoverflow.com/a/32337282/380016
And in my sample code you'd just change the string replace to:
$s = $line -replace '|', ','
And also adjust your input and output filenames accordingly.