I am writing a script to prepare a csv for uploading it into SQL.
I have two almost identical commands (1 and 3) but the first doesn't work, the third one does. There is no error, it just doesn't do what I expect it to do.
I do 3 steps, first I get rid of all the pipes in the source file (this is what does not work), then I read and write the CSV (this gets rid of the different handling of using quotes and not using quotes and it changes the delimiter to pipes). Lastly I get rid of all quotations, since there should be no more pipes other than the delimiters.
# ---------------------------------------------------------
# 1. Get rid of all pipes
#----------------------------------------------------------
get-content ($csvfile + ".csv") | ForEach-Object { $_ -replace "|",""} | Set-Content ($csvfile + "2.csv")
# ---------------------------------------------------------
# 2. Make standard CSV but use Pipes as delimiter
#----------------------------------------------------------
Import-csv -path ($csvfile + "2.csv") -Delimiter ',' | Export-CSV -path ($csvfile + " 3.csv") -Delimiter '|'
# ---------------------------------------------------------
# 3. Get rid of all Quotes
#----------------------------------------------------------
get-content ($csvfile + "3.csv") | ForEach-Object { $_ -replace '"',""} | Set-Content ($csvfile + "4.csv")
The getting rid-parts are the same. The second one works, it gets rid of all the quotes but the first one does not work, the pipes are still in. I tried different characters but for some reason none works at this position.
What am I missing?
Thank You!
Just escape the | symbol with \ like \| as below and it will work fine; cause | is a special character (command pipe symbol).
get-content ($csvfile + ".csv") | ForEach-Object { $_ -replace "\|",""} |
Set-Content ($csvfile + "2.csv")
So if I have input like dadad|xcvxvv|sdffgfg then after the command the output would look like dadadxcvxvvsdffgfg
Related
Good day,
with the script below I would like to use the following input txt from my output txt.
Input:
Klaus;Müller;Straße;PLZ;Ort;;;;;DE12345;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE12345678;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE999999;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE7777777;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE7777779;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE777777987;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE7777779765;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE77777797634;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE7777779763465;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE77777797623435435;
Output:
Klaus;Müller;Straße;PLZ;Ort;;;;;DE12345;DE12345678;DE999999;DE7777777;DE7777779;DE777777987;DE7777779765;DE77777797634;DE7777779763465;DE77777797623435435;
The script takes the last value from the following lines and appends them to the first line at the end and adds semicolons:
Import-Csv input.txt -delimiter ";" -Header (1..20)
1..9 | %{$data[0].($_+10) = $data[$_].10}
($data[0] | convertto-csv -delimiter ";" -NoType | select -skip 1) -replace '"' | out-file output.txt
gc test_neu.txt
if i save this into a .ps1 file it doesn´t work. anyone could say me why?
You don't assign Import-Csv to anything. The first line should be: $data = Import-Csv input.txt -delimiter ";" -Header (1..20) Your last line should be gc output.txt. And use the dot notation to location the input.txt file in the current directory. With these fixes, your script works:
$data = Import-Csv .\input.txt -delimiter ";" -Header (1..20)
1..9 | %{$data[0].($_+10) = $data[$_].10}
($data[0] | convertto-csv -delimiter ";" -NoType | select -skip 1) -replace '"' | out-file output.txt
gc output.txt
this seems to do what you want. [grin] it expects that the source lines are all to be combined.
i presume you can handle saving things to a file, so i leave that to you.
what it does ...
fakes reading in a text file
when ready to work with real data, replace the entire #region/#endregion block with a call to Get-Content.
iterates thru the collection by index number
if the line is the 1st, set $NewString to that entire value
else, add the last data item of the line to the existing $NewString value with a trailing ;
the .Where({$_}) filters out any blank items.
display the string
the code ...
#region >>> fake reading in a text file
# in real life, use Get-Content
$InStuff = #'
Klaus;Müller;Straße;PLZ;Ort;;;;;DE12345;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE12345678;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE999999;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE7777777;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE7777779;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE777777987;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE7777779765;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE77777797634;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE7777779763465;
Klaus;Müller;Straße;PLZ;Ort;;;;;DE77777797623435435;
'# -split [System.Environment]::NewLine
#endregion >>> fake reading in a text file
foreach ($Index in 0..$InStuff.GetUpperBound(0))
{
if ($Index -eq 0)
{
$NewString = $InStuff[$Index]
}
else
{
$NewString += $InStuff[$Index].Split(';').Where({$_})[-1] + ';'
}
}
$NewString
output ...
Klaus;Müller;Straße;PLZ;Ort;;;;;DE12345;DE12345678;DE999999;DE7777777;DE7777779;DE777777987;DE7777779765;DE77777797634;DE7777779763465;DE77777797623435435;
Just in case you don't know how many lines there are going to be on the input file:
$fmt='$1$2'
gc .\input.txt | %{$_ -replace '(^.*;)(.*;$)',$fmt;$fmt='$2'} | sc output.txt -NoNewline
gc output.txt
I've been trying to adapt the answer to this question: powershell - concatenate N text files and prepend filename to each line
My desired output based on an example of 2 .txt files:
First.txt
lines of
data
Second.txt
more lines
of
data
Output.txt
First1 lines of
First2 data
Second1 more lines
Second2 of
Second3 data
Based on the most similar question I could find the following answer:
Select-String '^' *.txt >output.txt
Would give:
C:\A\Filepath\First.txt:1:lines of
C:\A\Filepath\First.txt:2:data
C:\A\Filepath\Second.txt:1:more lines
C:\A\Filepath\Second.txt:2:of
C:\A\Filepath\Second.txt:3:data
So I was hoping to use -replace to remove the filepath, keep the file name (but remove .txt:), keep the line number (but replace the final : with a space) and keep the text from the line.
Any help would be appreciated reaching the desired output.txt. Thanks
Not beautiful but this is one approach.
Get-ChildItem *.txt |
%{$FILENAME=$_.BaseName;$COUNT=1;get-content $_ |
%{"$FILENAME"+"$COUNT"+" " + "$_";$COUNT++}}|
Out-File Output.txt
The select-string approach is very interesting. The way I would go about it is to use Get-Content. The advantage there is that each line has a readcount property that represents the line number.
Get-ChildItem "C:\temp\*.file" | ForEach-Object{
$fileName = $_.BaseName
Get-content $_ | ForEach-Object{
"{0}{1} {2}" -f $fileName,$_.ReadCount,$_
}
} | Add-Content "C:\temp\output.txt"
Take each file and use Get-Content. With each line we process we send to the output stream a formatted line matching your desired output. No need to count the lines as $_.ReadCount already knows.
Select-String still works
You just need to manipulate the output to match what you want. Using Get-Member we can check the properties of what select-string returns to get our desired output.
Select-String '^' "c:\temp\*.txt" | ForEach-Object{
"{0}{1} {2}" -f $_.Filename,$_.LineNumber,$_.Line
} | Add-Content "C:\temp\output.txt"
In Powershell Script, how do I convert a | (pipe) delimited CSV file to a , (comma) delimited CSV file?
When we use the following command in Windows Powershell Encoding 'UTF8' -NoType to convert from | (pipe delimiter) to , (comma delimiter), the file is converted with , delimited but the string was surrounded by " " (double quotes). Like given below:
Source file data:
ABC|1234|CDE|567|
Converted file data:
"ABC","1234","CDE","567",
I want to generate the following:
ABC,1234,CDE,567,
What command can I use to convert the delimiter from | to ,?
I would use:
(Get-Content -Path $file).Replace('|',',') | Set-Content -Path $file
You must escape the pipe, so:
(get-content "d:\makej\test.txt" ) -replace "\|","," | set-content "d:\makej\test.csv"
Seems easy enough:
(get-content $file) -replace '|',',' | set-content $file
In general, you should use the commands Import-Csv and Export-Csv which properly handle delimiters embedded in the field values, such as Field,1|Field2. The Get-Content based solutions would turn this into 3(!) fields Field,1,Field2, while the output actually should be quoted like "Field,1",Field2 or "Field,1","Field2".
Import-Csv input.csv -Delimiter '|' | Export-Csv output.csv -Delimiter ','
This always quotes fields in "output.csv". Since PowerShell (Core) 7+, the new Export-Csv parameters -UseQuotes and -QuoteFields allow us to control the quoting of the output file.
E. g. to quote only if necessary (when a field value contains the delimiter or quotation marks):
Import-Csv input.csv -Delimiter '|' | Export-Csv output.csv -Delimiter ',' -UseQuotes AsNeeded
Be careful with -UseQuotes Never, because it can render the output file unreadable, if a field value contains embedded delimiter or quotation marks.
Here is a function to convert to unquoted CSV for PowerShell 5.x (possibly supports older versions as well). This is like -UseQuotes Never, so make sure your data doesn't contain the delimiter. Additionally you may omit the header by passing the -NoHeader switch.
Function ConvertTo-CsvUnquoted {
[CmdletBinding()]
param (
[Parameter(Mandatory, ValueFromPipeline)] $InputObject,
[string] $Delimiter = ',',
[switch] $NoHeader
)
process {
if( -not $NoHeader ) {
$_.PSObject.Properties.Name -join $Delimiter
$NoHeader = $true
}
$_.PSObject.Properties.Value -join $Delimiter
}
}
Usage example:
Import-Csv input.csv | ConvertTo-CsvUnquoted -Delimiter '|' | Set-Content output.csv
Sorry this may need some tweaking on your part, but it does the job. Note that this also changes the file type from .txt to .csv which I dont think you wanted.
$path = "<Path>"
$outPath = $path -replace ".txt",".csv"
Get-Content -path $path |
ForEach-Object {$_ -replace "|","," } |
Out-File -filepath $outPath
I view the suggested answers as a little risky, because you are getting the entire contents of the existing file into memory, and therefore won't scale well, and risks using a lot of memory. My suggestion would be to use the string replace as the previous posts suggested, but to use streams instead for both reading and writing. That way you only need memory for each line in the file rather than the entire thing.
Have a look here at one of my other answers here:
https://stackoverflow.com/a/32337282/380016
And in my sample code you'd just change the string replace to:
$s = $line -replace '|', ','
And also adjust your input and output filenames accordingly.
I have a file cointaining a lot of lines in this format:
firstname ; lastname ; age ;
(it's a bit more complex but that's basically the file)
so the fields are of a fixed length, padded with spaces and with a semicolon in between the fields.
I would like to have it so:
firstname, lastname, age,
(commas and no fixed width)
I have replaced the commas with regexp but I would like to also trim the end of the strings. But I don't know how to do this.
The following is my start, but I can't manage to get a ".TrimEnd()" in there. I have also thought of trying a "-replace(" ", " ") but I can't integrate it in this expression:
Get-Content .\Bestand.txt | %{$data= [regex]::split($_, ';'); [string]:: join(',', $data)}
Can I get some information on how to achieve this?
I suggest you replace each occurrence of 'space;space' with a comma (assuming the replaced characters do not appear within a valid value), so the end result will look like:
firstname,lastname,age
Keeping it like the following is not a good idea cause now some of your headers (property names) start with a space:
"firstname, lastname, age,"
Give this a try (work on a copy of the file):
(Get-Content .\Bestand.txt) |
foreach {$_ -replace ' ; ',','} |
out-file .\Bestand.txt
Now it's easy to import and process the file with Import-Csv cmdlet.
The -replace operator takes a regular expression, which you can use to remove all leading and trailing spaces:
Get-Content .\Bestand.txt |
Foreach-Object { $_ -replace ' *; *',',' } |
Out-File .\Bestand.csv -Encoding OEM
Since you already create something CSV-ish, I'd go all the way and create proper CSV:
$cols = "firstname","lastname","age","rest"
Import-Csv "C:\input.txt" -Delimiter ";" -Header $cols | % {
foreach ($property in $_.PsObject.Properties) {
$property.Value = ([string]$property.Value).Trim()
}
$_
} | Export-Csv "C:\output.csv" -NoTypeInformation
I'm looking for a script, doesn't have to be in PS but must run under Windows, that converts a one column text file like below
abc
def
ghi
into
'abc',
'def',
'ghi'
I'm currently making this change in Excel using =concatenate, but a script would be better.
Use can use a regular expression to insert characters at beginning and end.
get-content ./myonlinecolumn.txt | foreach {$_ -replace "^","'" -replace "`$","',"}
Or you could use the format operator -f:
get-content ./myonlinecolumn.txt | foreach {"'{0}'," -f $_ }
Its a bit more work to remove the last trailing comma, but this also possible
$a = get-content ./myonlinecolumn.txt
get-content ./myonlinecolumn.txt | foreach { if ($_.readcount -lt $a.count) {"'{0}'," -f $_ } else {"'{0}'" -f $_ }}
My first idea was similar to what Chad already wrote, that is a check on the line number. So I've tried a different solution. Not very nice but I post it too :)
((gc c:\before.txt | % {"'"+$_+"'"} ) -join ",*").split("*") | out-file c:\after.txt
You can just use
(gc myfile | %{"'$_'"}) -join ',
'
or, if you love escapes:
(gc myfile | %{"'$_'"}) -join ",`n"
This loads the file into an array of strings (Get-Content), then processes each string by putting it into single quotes. (Use `"'$($_.Trim())'" if you need to trim whitespace, too). Then the lines are joined with a comma and line break (those can be embedded directly into strings).
If your values can contain single quotes (which need to be escaped) it's trivial to stick that in there, too:
(gc myfile | %{"'$($_.Trim() -replace "'","''")'"}) -join ",`n"