I know a little bit of Bash scripting, but I am very new to PowerShell. When I execute below code using bash, everything is fine. But, when I use PowerShell, each entry per echo is saved only in a single cell in Excel. Why is it like this? How can I accomplish my objective in the simplest way?
echo "1,2,3" > file.csv
echo "A,B,C" >> file.csv
UNDESIRED:
DESIRED:
I tried to Google it. But, in my understanding, they are converting the string type variables to something like PS Object and convert to CSV format. I tried it and it worked. But I had to force include a header.
New-Object -Type PSObject -Property #{
'X' = $A
'Y' = $B
'Z' = $C
} | Export-Csv 'C:\Temp\test.csv' -NoType
When I also opened the csv file using notepad, every word has double quotation marks (which I don't prefer to have)
You see, that is way more complicated compared to Linux Scripting. Can someone teach me the simplest way to do what I want? Thank you very much!
If in your system locale the ListSeparator character is NOT the comma, double-clicking a comma-delimited csv file will open Excel with all values in the same column.
I believe this is what happens here.
You can check by typing
[cultureinfo]::CurrentCulture.TextInfo.ListSeparator
in PowerShell
To have Excel 'understand' a CSV when you double-click it, add -UseCulture switch to the cmdlet:
Export-Csv 'C:\Temp\test.csv' -UseCulture -NoTypeInformation
As for the quotes around the values:
They are not always necessary, but sometimes essential, for instance if the value has leading or trailing space characters, or if the value contains the delimiter character itself.
Just leave them as-is, Excel knows how to handle that.
If you really can't resist on having a csv without quotes, please first have a look at the answers given here about that subject.
Edit
If you are absolutely sure all fields can do without quoting, you can do this:
$sep = [cultureinfo]::CurrentCulture.TextInfo.ListSeparator
"1,2,3", "A,B,C" -replace ',', $sep | Out-File -FilePath 'D:\Test\file.csv' -Encoding utf8
Related
I found a nifty command here - http://www.stackoverflow.com/questions/27892957/merging-multiple-csv-files-into-one-using-powershell that I am using to merge CSV files -
Get-ChildItem -Filter *.csv | Select-Object -ExpandProperty FullName | Import-Csv | Export-Csv .\merged\merged.csv -NoTypeInformation -Append
Now this does what it says on the tin and works great for the most part. I have 2 issues with it however, and I am wondering if there is a way they can be overcome:
Firstly, the merged csv file has CRLF line endings, and I am wondering how I can make the line endings just LF, as the file is being generated?
Also, it looks like there are some shenanigans with quote marks being added/moved around. As an example:
Sample row from initial CSV:
"2021-10-05"|"00:00"|"1212"|"160477"|"1.00"|"3.49"LF
Same row in the merged CSV:
"2021-10-05|""00:00""|""1212""|""160477""|""1.00""|""3.49"""CRLF
So see that the first row has lost its trailing quotes, other fields have doubled quotes, and the end of the row has an additional quote. I'm not quite sure what is going on here, so any help would be much appreciated!
For dealing with the quotes, the cause of the “problem” is that your CSV does not use the default field delimiter that Import-CSV assumes - the C in CSV stands for comma, and you’re using the vertical bar. Add the parameter -Delimiter "|" to both the Import-CSV and Export-CSV cmdlets.
I don’t think you can do anything about the line-end characters (CRLF vs LF); that’s almost certainly operating-system dependent.
Jeff Zeitlin's helpful answer explains the quote-related part of your problem well.
As for your line-ending problem:
As of PowerShell 7.2, there are no PowerShell-native features that allow you to control the newline format of file-writing cmdlets such as Export-Csv.
However, if you use plain-text processing, you can use multi-line strings built with the newline format of interest and save / append them with Set-Content and its -NoNewLine switch, which writes the input strings as-is, without a (newline) separator.
In fact, to significantly speed up processing in your case, plain-text handling is preferable, since in essence your operation amounts to concatenating text files, the only twist being that the header lines of all but the first file should be skipped; using plain-text handling also bypasses your quote problem:
$tokenCount = 1
Get-ChildItem -Filter *.csv |
Get-Content -Raw |
ForEach-Object {
# Get the file content and replace CRLF with LF.
# Include the first line (the header) only for the first file.
$content = ($_ -split '\r?\n', $tokenCount)[-1].Replace("`r`n", "`n")
$tokenCount = 2 # Subsequent files should have their header ignored.
# Make sure that each file content ends in a LF
if (-not $content.EndsWith("`n")) { $content += "`n" }
# Output the modified content.
$content
} |
Set-Content -NoNewLine ./merged/merged.csv # add -Encoding as needed.
I have a CSV column with alpha numerical combinations in a column.
I am later going to use this csv file in a PowerShell script by importing the data.
Examples: 1A01, 1C66, 1E53.
Now before putting these values in, I made sure to format the column as text.
Now at first it works. I input the data, save. I test in PowerShell to import it and
all data shows up valid including 1E53. But lets say I edit the file again later to add data and then save and close. I re-import into PowerShell and 1E53 comes in as 1.00E+53. How can I prevent this permanently? Note that the column is filled with codes and there are lots of #E##.
Your issue is not with PowerShell, its with Excel. For a demonstration, take 1E53 and enter it into Excel and then save that excel file as a CSV file. You will see that the value is now changed to 1.00E+53.
How to fix this?
There are a few ways of disabling scientific notation:
https://superuser.com/questions/452832/turn-off-scientific-notation-in-excel
https://www.logicbroker.com/excel-scientific-notation-disable-prevent/
I hope some of them work for you.
I think you can rename the file to .txt instead of .csv and excel may treat it differently.
Good Luck
As commented:
You will probably load the csv from file:
$csv = Import-Csv -Path 'X:\original.csv' -UseCulture
The code below uses a dummy csv in a Here-String here:
$csv = #'
"Column1","Column2","ValueThatMatters"
"Something","SomethingElse","1E53"
"AnotherItem","Whatever","4E12"
'# | ConvertFrom-Csv
# in order to make Excel see the values as Text and not convert them into scientific numbers
$csv | ForEach-Object {
# add a TAB character in front of the values in the column
$_.ValueThatMatters = "`t{0}" -f $_.ValueThatMatters
}
$csv | Export-Csv -Path 'X:\ExcelFriendly.csv' -UseCulture -NoTypeInformation
"$psIP
$psName
$psPower $psVar1
$psWattage $psVar2
$psRED $psVar3
$psHE $psVar4
$psL1 $psVar5
$psP1 $psVar6
$psStat1 $psVar7
$psL2 $psVar8
$psP2 $psVar9
$psStat2 $psVar10"| Out-File -FilePath P:\root\ResultFolder\Fingers1.csv
Sorry I am very new to powershell but I do have some background in coding.
I have set two sets of variables all in the same script. The goal is to have them in two columns so that way when the script executes it can be easily filtered. I need to know how to specify which VAR goes to which column. Please ask any question.
If you're looking to create a csv file with specific values in each row/column - see if this is what you're looking for:
#"
$psIP,$psName
$psPower,$psVar1
$psWattage,$psVar2
$psRED,$psVar3
$psHE,$psVar4
$psL1,$psVar5
$psP1,$psVar6
$psStat1,$psVar7
$psL2,$psVar8
$psP2,$psVar9
$psStat2,$psVar10
"# -split [System.Environment]::NewLine | Out-File -FilePath P:\root\ResultFolder\Fingers1.csv -Encoding ascii
The solution uses a Here String piped to split operator to break up the rows.
Also notice i'm using -Encoding ascii - this is so that Excel will see the file correctly when you open it.
I am actually trying to make a .txt file looking like this:
text1
text 2
text3
text4
in something like this:
text1,text2
text3,text4
I have tried a lot of things, like:
foreach($line in $file){
$line=$line -replace'`r`n', ','
}
Or directly by replacing in the file and not the string:
$file=Get-Content('testo.txt')
$file=$file -replace '`r`n',''
But nothing worked. I have never been able to replace the CRLF.
If someone has an idea or anything!
Something even simpler I think. This would depend on the size of the text file since it has to be all read into memory. However this would will work with variable lines of "test" data and white space. I am going to assume that your data does not already contain commas.
$file = (Get-Content "c:\temp\data.txt") -join "," -replace ",{2,}","`r`n"
Set-Content -Path "C:\temp\Newfile.txt" -Value $file
Take the file and convert it into a comma delimited string. What will happen is that you will see a series of groups commas. Using your test data
text1,text 2,,,text3,text4
The we use regex to replace and consecutive comma groups of 2 or more with newlines. Getting the desired output.
text1,text 2
text3,text4
In fact i have done something like this:
$Writer = New-Object IO.StreamWriter "$($PWD.Path)\$OutputFile"
$Writer.Write( [String]::Join(".", $data) )
$Writer.Close()
$data=get-content temp.txt
$data=$data -replace '[#]{2,}',"`r`n"
$data=$data -replace '[#]',","
$data=$data -replace '( ){1,},',"no data"
$data>>temp1.txt
It works and do what I want. It is not exactly efficient, but I wanted it to work quickly.
I will try all the answers when I have the time and say if it works or not.
And maybe I will change this part of my script if I find something more efficient.
Thanks to all!
I have a pipe-delimited text file with a header row. (I said CSV in the question to make it a a bit more immediately understandable ... I imagine most solutions would be applicable to either format.)
The file looks like this:
COLUMN1|COLUMN2|COLUMN3|COLUMN4|...|
Field1|Field2|Field3|Field4|...|
...
I need to obscure the data in (for example) columns 3 and 9, without affecting any of the other entries in the file.
I want to do this using a hashing algorithm like SHA1 or MD5, so that the same strings will resove to the same hash values anywhere they are encountered.
EDIT - Why I want to do this
I need to send some data to a third party, and certain columns contain sensitive information (e.g. customer names). I need the file to be complete, and where a string is replaced, I need it to be done in the same way every time it is encountered (so that any mapping or grouping remains). It does not need military encryption, just to be difficult to reverse. As I need to to this intermittently, a scripted solution would be ideal.
/EDIT
What is the easiest way to achieve this using a command line tool or script?
By preference, I would like a batch script or PowerShell script, since that does not require any additional software to achieve...
Try
(Import-Csv .\my.csv -delimiter '|' ) | ForEach-Object{
$_.column3 = $_.column3.gethashcode()
$_.column4 = $_.column4.gethashcode()
$_
} | Export-Csv .\myobfuscated.csv -NoTypeInformation -delimiter '|'
$md5 = new-object -TypeName Security.Cryptography.MD5CryptoServiceProvider
$utf8 = new-object -TypeName Text.UTF8Encoding
import-csv original.csv -delimiter '|' |
foreach {
$_.Column3 = [BitConverter]::ToString($md5.ComputeHash($utf8.GetBytes($_.Column3)))
$_.Column9 = [BitConverter]::ToString($md5.ComputeHash($utf8.GetBytes($_.Column9)))
$_
} |
export-csv encrypted.csv -delimiter '|' -noTypeInformation