I have the following code which removes all the commas in my csv file but there is an issue:
I only want to remove commas on Column P and and the remaining data should be untouched. Currently it appends data underneath the csv data.
$inform = Get-Content C:\Users\bmd\Desktop\report.csv
$inform.ToString()
$y=$inform -replace ',' -replace ''
$y | Out-file C:\Users\bmd\Desktop\report.csv -Append
Using Import-Csv and Export-Csv is usually going to be easier than trying to manipulate strings in a CSV. With no sample of the CSV file, we can only make assumptions. Assuming it contains true and qualified as needed CSV data, you can try the following:
$csv = Import-Csv C:\Users\bmd\Desktop\report.csv
foreach ($row in $csv) {
$row.UNSTRUCTURED_VARS = $row.UNSTRUCTURED_VARS -replace ','
}
$csv | Export-Csv C:\Users\bmd\Desktop\report.csv -NoType
Note that if you are on PowerShell Core or PowerShell 7, you can simply use Export-Csv with -UseQuotes AsNeeded and only the required fields will be qualified.
Related
I have a scenario where I need to write replace the column values if it contains INC** number instead of entire value
Duration starttime task
3 hours 24-09-2020 #{display_value=INC*****6; link=https://8888.service-8888888888888}
8 hours 23-09-2020 #{display_value=INC*****7; link=https://8888.service-888888888888888}
Now in task column I have to fill this by
task
INC6
INC7
Actually I got this csv file by accessing servicenow apis and converted the payload to csv file using powershell script. Can someone guide me
It will be easiest to edit the INC###### value when you receive the payload from the API as a PowerShell object.
$payload.task.display_value = $payload.task.display_value -replace '\d(?=\d)','*'
If you must use this CSV file, which should be unnecessary, you will have to do some parsing:
# I assumed the delimiter/separator is a tab. Change it to the appropriate character if needed.
$csv = import-csv file.csv -delimiter "`t"
foreach ($row in $csv) {
$row.task = $row.task -replace '(?<=display_value=INC\d*)\d(?=\d)','*'
}
$csv | Export-Csv output.csv -delimiter "`t"
See Regex for regex explanation.
I have a CSV file where I need to change the delimiter from comma to semicolon.
I fixed this issue with following command
(Import-CSV $File) | Export-Csv $File -delimiter ';' -notypeinformation
But when the imported CSV file contains only one line which I assume is being handled as the headers. It deletes this line in the output file and it becomes an empty file.
However when I have more than one line and the csv contains the header and some data the exported file is being executed correctly with the headers and the data.
Code that I used to set the header file in the beginning of my script
#set csv headers
Set-Content -Path "$FolderNameCiTypeDelta" -Value "name,type,location,size"
I've tried adding the code
(Import-CSV $File -Header "name,type,location,size") | Export-Csv $File -delimiter ';' -notypeinformation
But this made just an extra line with the headers I specified.
[EDIT]
The 2 csv files that I test with contain the following lines
The file that becomes empty with the import --> export command contains
name,type,location,size
The file that performs what I need with the import --> export command contains
name,type,location,size
test,extra,line
You can do the following:
$file = 't.txt'
$contents = Get-Content $file -totalcount 2
if ($contents.Count -gt 1) {
(Import-Csv $file) | Export-Csv $file -Delimiter ';' -NoType
}
if ($contents.Count -eq 1) {
$contents -replace ',',';' | Set-Content $file
}
The idea here is to read two lines of your file. If it only contains one line, it will use -replace to fix your delimiters. If it has more than two lines, it will use the *-Csv commands.
The -replace section only works if you don't have commas in your field values. If you do have commas in your fields, then you should have text qualifiers (double quotes most likely) around your fields. In that case, you can simply use -replace '","','";"'.
I have a Powershell script that I am trying to work out part of it, so the text input to this is listing the user group they are part of. This PS script is supposed to replace the group with the groups that I am assigning them in active directory(I am limited to only changing groups in active directory). My issue is that when it reaches HR and replaces it, it will then proceed to contine and replace all the new but it all so replaces the HR in CHRL, so my groups look nuts right now. But I am looking it over and it doesn't do it with every line. But for gilchrist it will put something in there for the HR in the name. Is there anything can I do to keep it for changing or am I going to have to change my HR to Human Resources? Thanks for the help.
$lookupTable = #{
'Admin' = 'W_CHRL_ADMIN_GS,M_CHRL_ADMIN_UD,M_CHRL_SITE_GS'
'Security' = 'W_CHRL_SECURITY_GS,M_CHRL_SITE_GS'
'HR' = 'M_CHRL_HR_UD,W_CHRL_HR_GS,M_CHRL_SITE_GS'
$original_file = 'c:\tmp\test.txt'
$destination_file = 'c:\tmp\test2.txt'
Get-Content -Path $original_file | ForEach-Object {
$line = $_
$lookupTable.GetEnumerator() | ForEach-Object {
if ($line -match $_.Key)
{
$line = $line -replace $_.Key, $_.Value
}
}
$line
} | Set-Content -Path $destination_file
Get-Content $destination_file
test.txt:
user,group
john.smith,Admin
joanha.smith,HR
john.gilchrist,security
aaron.r.smith,admin
abby.doe,secuity
abigail.doe,admin
Your input appears to be in CSV format (though note that your sample rows have trailing spaces, which you'd have to deal with, if they're part of your actual data).
Therefore, use Import-Csv and Export-Csv to read / rewrite your data, which allows a more concise and convenient solution:
Import-Csv test.txt |
Select-Object user, #{ Name='group'; Expression = { $lookupTable[$_.group] } } |
Export-Csv -NoTypeInformation -Encoding Utf8 test2.txt
Import-Csv reads the CSV file as a collection of custom objects whose properties correspond to the CSV column values; that is, each object has a .user and .name property in your case.
$_.group therefore robustly reports the abstract group name only, which you can directly pass to your lookup hashtable; Select-Object is used to pass the original .user value through, and to replace the original .group value with the lookup result, using a calculated property.
Export-Csv re-converts the custom objects to a CSV file:
-NoTypeInformation suppresses the (usually useless) data-type-information line at the top of the output file
-Encoding Utf8 was added to prevent potential data loss, because it is ASCII encoding that is used by default.
Note that Export-Csv blindly double-quotes all field values, whether they need it or not; that said, CSV readers should be able to deal with that (and Import-Csv certainly does).
As for what you tried:
The -replace operator replaces all occurrences of a given regex (regular expression) in the input.
Your regexes amounts to looking for (case-insensitive) substrings, which explains why HR matches both the HR group name and substring hr in username gilchrist.
A simple workaround would be to add assertions to your regex so that the substrings only match where you want them; e.g.: ,HR$ would only match after a , at the end of a line ($).
However, your approach of enumerating the hashtable keys for each input CSV row is inefficient, and you're better off splitting off the group name and doing a straight lookup based on it:
# Split the row into fields.
$fields = $line -split ','
# Update the group value (last field)
$fields[-1] = $lookupTable[$fields[-1]]
# Rebuild the line
$line = $fields -join ','
Note that you'd have to make an exception for the header row (e.g., test if the lookup result is empty and refrain from updating, if so).
Why don't you load your text file as a CSV file, using Import-CSV and use "," as a delimiter?
This will allow you to have a Powershell Object you can work on. and then export it as text o CSV. if I use your file & lookup table this code may help you :
$file = Import-Csv -Delimiter "," -Path "c:\ps\test.txt"
$lookupTable = #{
'Admin' = 'W_CHRL_ADMIN_GS,M_CHRL_ADMIN_UD,M_CHRL_SITE_GS'
'Security' = 'W_CHRL_SECURITY_GS,M_CHRL_SITE_GS'
'HR' = 'M_CHRL_HR_UD,W_CHRL_HR_GS,M_CHRL_SITE_GS'}
foreach ($i in $file) {
#Compare and replace
...
}
Export-CSV $file -Delimiter ","
You can then iterate over $file and compare and replace. you can also Export-CSV after you're done.
I have a text file. Each line record an event and the fields are delimited by '|'.
Is it possible using some cmdlet like "get-content event.log | export-csv event.csv" to convert text file to csv file?
xxx(1365)|2016-09-29 06:00:00.0|2016-09-29 06:30:00.0|
bbb(110)|2016-09-29 06:30:00.0|2016-09-29 07:00:00.0|
ccc(5243)|2016-09-29 07:00:00.0|2016-09-29 07:30:00.0|
ddd(1950)|2016-09-29 07:30:00.0|2016-09-29 08:00:00.0|
eee(10)|2016-09-29 08:00:00.0|2016-09-29 09:00:00.0|
fff(464)|2016-09-29 09:00:00.0|2016-09-29 10:00:00.0|
dddd(874)|2016-09-29 10:00:00.0|2016-09-29 11:00:00.0|
ggggg(6)|2016-09-29 11:00:00.0|2016-09-29 12:00:00.0|
kkkk(272)|2016-09-29 12:00:00.0|2016-09-29 12:30:00.0|
The Import-Csv cmdlet allows you to specify a delimiter
$file = Import-Csv .\event.log -Delimiter '|'
so in your case, it can be as simple as
Import-Csv .\event.log -Delimiter "|" | Export-Csv .\event.csv -NoTypeInformation
If the file isn't too big (under a few megabytes) you could do it with a straight-forward String.Replace operation:
$content = ( $fileName | Get-Content )
$content = $content -replace '|' ','
$content | Out-File $fileName
Or more succinctly:
( $fileName | Get-Content ) -replace '|' ',' | Out-File $fileName
This won't work very well for large files (more than a few megabytes) because the entire file is loaded into memory as a System.String instance, then the Replace operation will then create a new instance (thus doubling the memory requirement).
A faster version might read from the input file line-by-line and perform the -replace operation for each line - or even character-by-character.
In Powershell Script, how do I convert a | (pipe) delimited CSV file to a , (comma) delimited CSV file?
When we use the following command in Windows Powershell Encoding 'UTF8' -NoType to convert from | (pipe delimiter) to , (comma delimiter), the file is converted with , delimited but the string was surrounded by " " (double quotes). Like given below:
Source file data:
ABC|1234|CDE|567|
Converted file data:
"ABC","1234","CDE","567",
I want to generate the following:
ABC,1234,CDE,567,
What command can I use to convert the delimiter from | to ,?
I would use:
(Get-Content -Path $file).Replace('|',',') | Set-Content -Path $file
You must escape the pipe, so:
(get-content "d:\makej\test.txt" ) -replace "\|","," | set-content "d:\makej\test.csv"
Seems easy enough:
(get-content $file) -replace '|',',' | set-content $file
In general, you should use the commands Import-Csv and Export-Csv which properly handle delimiters embedded in the field values, such as Field,1|Field2. The Get-Content based solutions would turn this into 3(!) fields Field,1,Field2, while the output actually should be quoted like "Field,1",Field2 or "Field,1","Field2".
Import-Csv input.csv -Delimiter '|' | Export-Csv output.csv -Delimiter ','
This always quotes fields in "output.csv". Since PowerShell (Core) 7+, the new Export-Csv parameters -UseQuotes and -QuoteFields allow us to control the quoting of the output file.
E. g. to quote only if necessary (when a field value contains the delimiter or quotation marks):
Import-Csv input.csv -Delimiter '|' | Export-Csv output.csv -Delimiter ',' -UseQuotes AsNeeded
Be careful with -UseQuotes Never, because it can render the output file unreadable, if a field value contains embedded delimiter or quotation marks.
Here is a function to convert to unquoted CSV for PowerShell 5.x (possibly supports older versions as well). This is like -UseQuotes Never, so make sure your data doesn't contain the delimiter. Additionally you may omit the header by passing the -NoHeader switch.
Function ConvertTo-CsvUnquoted {
[CmdletBinding()]
param (
[Parameter(Mandatory, ValueFromPipeline)] $InputObject,
[string] $Delimiter = ',',
[switch] $NoHeader
)
process {
if( -not $NoHeader ) {
$_.PSObject.Properties.Name -join $Delimiter
$NoHeader = $true
}
$_.PSObject.Properties.Value -join $Delimiter
}
}
Usage example:
Import-Csv input.csv | ConvertTo-CsvUnquoted -Delimiter '|' | Set-Content output.csv
Sorry this may need some tweaking on your part, but it does the job. Note that this also changes the file type from .txt to .csv which I dont think you wanted.
$path = "<Path>"
$outPath = $path -replace ".txt",".csv"
Get-Content -path $path |
ForEach-Object {$_ -replace "|","," } |
Out-File -filepath $outPath
I view the suggested answers as a little risky, because you are getting the entire contents of the existing file into memory, and therefore won't scale well, and risks using a lot of memory. My suggestion would be to use the string replace as the previous posts suggested, but to use streams instead for both reading and writing. That way you only need memory for each line in the file rather than the entire thing.
Have a look here at one of my other answers here:
https://stackoverflow.com/a/32337282/380016
And in my sample code you'd just change the string replace to:
$s = $line -replace '|', ','
And also adjust your input and output filenames accordingly.