Powershell replace text once per line - powershell

I have a Powershell script that I am trying to work out part of it, so the text input to this is listing the user group they are part of. This PS script is supposed to replace the group with the groups that I am assigning them in active directory(I am limited to only changing groups in active directory). My issue is that when it reaches HR and replaces it, it will then proceed to contine and replace all the new but it all so replaces the HR in CHRL, so my groups look nuts right now. But I am looking it over and it doesn't do it with every line. But for gilchrist it will put something in there for the HR in the name. Is there anything can I do to keep it for changing or am I going to have to change my HR to Human Resources? Thanks for the help.
$lookupTable = #{
'Admin' = 'W_CHRL_ADMIN_GS,M_CHRL_ADMIN_UD,M_CHRL_SITE_GS'
'Security' = 'W_CHRL_SECURITY_GS,M_CHRL_SITE_GS'
'HR' = 'M_CHRL_HR_UD,W_CHRL_HR_GS,M_CHRL_SITE_GS'
$original_file = 'c:\tmp\test.txt'
$destination_file = 'c:\tmp\test2.txt'
Get-Content -Path $original_file | ForEach-Object {
$line = $_
$lookupTable.GetEnumerator() | ForEach-Object {
if ($line -match $_.Key)
{
$line = $line -replace $_.Key, $_.Value
}
}
$line
} | Set-Content -Path $destination_file
Get-Content $destination_file
test.txt:
user,group
john.smith,Admin
joanha.smith,HR
john.gilchrist,security
aaron.r.smith,admin
abby.doe,secuity
abigail.doe,admin

Your input appears to be in CSV format (though note that your sample rows have trailing spaces, which you'd have to deal with, if they're part of your actual data).
Therefore, use Import-Csv and Export-Csv to read / rewrite your data, which allows a more concise and convenient solution:
Import-Csv test.txt |
Select-Object user, #{ Name='group'; Expression = { $lookupTable[$_.group] } } |
Export-Csv -NoTypeInformation -Encoding Utf8 test2.txt
Import-Csv reads the CSV file as a collection of custom objects whose properties correspond to the CSV column values; that is, each object has a .user and .name property in your case.
$_.group therefore robustly reports the abstract group name only, which you can directly pass to your lookup hashtable; Select-Object is used to pass the original .user value through, and to replace the original .group value with the lookup result, using a calculated property.
Export-Csv re-converts the custom objects to a CSV file:
-NoTypeInformation suppresses the (usually useless) data-type-information line at the top of the output file
-Encoding Utf8 was added to prevent potential data loss, because it is ASCII encoding that is used by default.
Note that Export-Csv blindly double-quotes all field values, whether they need it or not; that said, CSV readers should be able to deal with that (and Import-Csv certainly does).
As for what you tried:
The -replace operator replaces all occurrences of a given regex (regular expression) in the input.
Your regexes amounts to looking for (case-insensitive) substrings, which explains why HR matches both the HR group name and substring hr in username gilchrist.
A simple workaround would be to add assertions to your regex so that the substrings only match where you want them; e.g.: ,HR$ would only match after a , at the end of a line ($).
However, your approach of enumerating the hashtable keys for each input CSV row is inefficient, and you're better off splitting off the group name and doing a straight lookup based on it:
# Split the row into fields.
$fields = $line -split ','
# Update the group value (last field)
$fields[-1] = $lookupTable[$fields[-1]]
# Rebuild the line
$line = $fields -join ','
Note that you'd have to make an exception for the header row (e.g., test if the lookup result is empty and refrain from updating, if so).

Why don't you load your text file as a CSV file, using Import-CSV and use "," as a delimiter?
This will allow you to have a Powershell Object you can work on. and then export it as text o CSV. if I use your file & lookup table this code may help you :
$file = Import-Csv -Delimiter "," -Path "c:\ps\test.txt"
$lookupTable = #{
'Admin' = 'W_CHRL_ADMIN_GS,M_CHRL_ADMIN_UD,M_CHRL_SITE_GS'
'Security' = 'W_CHRL_SECURITY_GS,M_CHRL_SITE_GS'
'HR' = 'M_CHRL_HR_UD,W_CHRL_HR_GS,M_CHRL_SITE_GS'}
foreach ($i in $file) {
#Compare and replace
...
}
Export-CSV $file -Delimiter ","
You can then iterate over $file and compare and replace. you can also Export-CSV after you're done.

Related

Remove additional commas in CSV file using Powershell

I have a csv file that I'll like to import to sql but isn't properly formatted. I am not able to format the generated file (excel file) so I'm looking to do this with the CSV file using. I want to remove the extra commas and also replace the department name (,,,,,,) with the correct department as seen in the example below. Thank you in advance.
Example:
Current Format:
Department,,,,,,First Name,,,,Last Name,,,,,,,School Year,Enrolment Status
Psychology ,,,,,,,,,,,,,,,,,,,,,,, (Remove this line)
,,,,,,Jane,,,,Doe,,,,,,,2022,Enrolled
,,,,,,Jeff,,,,Dane,,,,,,,2019,Enrolled
,,,,,,Tate,,,,Anderson,,,,,,,2019,Not Enrolled
,,,,,,Daphne,,,,Miller,,,,,,,2021,Enrolled
,,,,,,Cora,,,,Dame,,,,,,,2022,Enrolled
Computer Science ,,,,,,,,,,,,,,,,,,,,,,, (Remove this line)
,,,,,,Dora,,,,Explorer,,,,,,,2022,Not Enrolled
,,,,,,Peppa,,,,Diggs,,,,,,,2020,Enrolled
,,,,,,Conrad,,,,Strat,,,,,,,2020,Enrolled
,,,,,,Kat,Noir,,,,2019,,,,,,,Enrolled
,,,,,,Lance,,,,Bug,2018,,,,,,,Enrolled
Ideal format:
Department,First Name,Last Name,School Year,Enrolment Status
Psychology ,,,,,,,,,,,,,,,,,,,,,,, (Remove this line)
Psychology,Jane,Doe,2022,Enrolled
Psychology,Jeff,Dane,2019,Enrolled
Psychology,Tate,Anderson,2019,Not Enrolled
Psychology,Daphne,Miller,2021,Enrolled
Psychology,Cora,Dame,2022,Enrolled
Computer Science ,,,,,,,,,,,,,,,,,,,,,,, (Remove this line)
Computer Science,Dora,Explorer,2022,Not Enrolled
Computer Science,Peppa,Diggs,2020,Enrolled
Computer Science,Conrad,Strat,2020,Enrolled
Computer Science,Kat,Noir,2019,Enrolled
Computer Science,Lance,Bug,2018,Enrolled
here you go:
$csvArray = new-object System.Collections.Generic.List[string]
#Import the file
$text = (gc "C:\tmp\testdata.txt") -replace ",{2,}",","
$arrayEnd = $text.count -1
$text[1..$arrayEnd] | %{
If ($_ -notmatch "^(,)"){
$department = $_ -replace ","
}
Else {
$csvArray.add($department + $_)
}
}
$csvArray.Insert(0,$text[0])
$csvArray | set-content 'C:\tmp\my.csv'
Using the Csv cmdlets:
$Csv = #'
Department,,,,,,First Name,,,,Last Name,,,,,,,School Year,Enrolment Status
Psychology ,,,,,,,,,,,,,,,,,,,,,,, (Remove this line)
,,,,,,Jane,,,,Doe,,,,,,,2022,Enrolled
,,,,,,Jeff,,,,Dane,,,,,,,2019,Enrolled
,,,,,,Tate,,,,Anderson,,,,,,,2019,Not Enrolled
,,,,,,Daphne,,,,Miller,,,,,,,2021,Enrolled
,,,,,,Cora,,,,Dame,,,,,,,2022,Enrolled
Computer Science ,,,,,,,,,,,,,,,,,,,,,,, (Remove this line)
,,,,,,Dora,,,,Explorer,,,,,,,2022,Not Enrolled
,,,,,,Peppa,,,,Diggs,,,,,,,2020,Enrolled
,,,,,,Conrad,,,,Strat,,,,,,,2020,Enrolled
,,,,,,Kat,Noir,,,,2019,,,,,,,Enrolled
,,,,,,Lance,,,,Bug,2018,,,,,,,Enrolled
'#
$List = ConvertFrom-Csv $Csv -Header #(1..20) # |Import-Csv .\Your.Csv -Header #(1..20)
$Columns = $List[0].PSObject.Properties.Where{ $_.Value -and $_.Value -ne 'Department' }.Name
$List |Select-Object -Property $Columns |Where-Object { $_.$($Columns[0]) } |
ConvertTo-Csv -UseQuote Never |Select-Object -Skip 1 # |Set-Content -Encoding utf8 out.csv
First Name,Last Name,School Year,Enrolment Status
Jane,Doe,2022,Enrolled
Jeff,Dane,2019,Enrolled
Tate,Anderson,2019,Not Enrolled
Daphne,Miller,2021,Enrolled
Cora,Dame,2022,Enrolled
Dora,Explorer,2022,Not Enrolled
Peppa,Diggs,2020,Enrolled
Conrad,Strat,2020,Enrolled
Kat,,,Enrolled
Lance,Bug,,Enrolled
Use a switch statement:
& {
$first = $true
switch -Wildcard -File in.csv { # Loop over all lines in file in.csv
',*' { # intra-department line
# Prepend the department name, eliminate empty fields and output.
$dept + (($_ -split ',' -ne '') -join ',')
}
default {
if ($first) { # header line
# Eliminate empty fields and output.
($_ -split ',' -ne '') -join ','
$first = $false
}
else { # department-only line
$dept = ($_ -split ',')[0] # save department name
}
}
}
} | Set-Content -Encoding utf8 out.csv
Note:
$_ -split ',' splits each line into fields by ,, and -ne '' filters out empty fields from the resulting array; applying -join ',' rejoins the nonempty fields with ,, which in effect removes multiple adjacent , and thereby eliminates empty fields.
If you don't mind the complexity of a regex, you can perform the above more simply with a single -replace operation, as shown in Toni's helpful answer.
Using switch -File is an efficient way to read files line by line and perform conditional processing based on sophisticated matching (as an alternative to -Wildcard you can use -Regex for regex matching, and you can even use script blocks ({ ... } as conditionals).
As a language statement, switch cannot be used directly in a pipeline.
This limitation can be overcome by enclosing it in a script block ({ ... }) invoked with &, which enables the usual, memory-friendly streaming behavior in the pipeline; that is, the lines are processed one by one, as are the modified output lines relayed to Set-Content, so that the input file needn't be read into memory as a whole.
In your case, plain-text processing of your CSV file enabled a simple solution, but in general it is better to parse CSV files into objects whose properties you can work with, using the Import-Csv cmdlet, and, for later re-exporting to a CSV file, Export-Csv,

Save property with comma to CSV

I'm creating a CSV file in powershell.
Right now my code is:
Add-content -Path $filePath -Value "$($variable.Property)"
This works fine for the most part EXCEPT if the property contained a comma ie. "test, organization".
When I open up the CSV, the comma is taken with it (which is what i want) causing a extra separation. How do save "test, organization" to one column?
Referring to the documentation for Export-CSV, you will need to use a different delimiter, like a semi-colon.
When you read the CSV you should specify the delimiter as well: Import-CSV.
Try to quote your properties:
Add-content -Path $filePath -Value "'$($variable.Property)'"
Or use one of the built-in CSV commands, which automatically quote all values:
$foo.Bar | Export-Csv -Path $filePath
$foo.Bar| ConvertTo-Csv | Out-File -Path $filePath
If you just want to avoid issues with commas, you can change the delimiter between fields:
$foo | Export-Csv -Path $filePath -Delimiter '|'
Here is an article on how to use out-file or add-member with some cells of the row having commas in the variable values and some not.
https://imjustanengineer.blogspot.com/2022/01/so-youre-trying-to-use-powershell-out.html
Here is a code snippet, a more detailed explanation with a full working function is in the link. $outputArr is an array of all the lines of csv data you want to write to the csv. The loop checks each line to see if it contains commas inside of the individual cell entries and puts quotes around that entry if it does. If it does not, no adjustment is necessary and then a new array is appended to afterwards.
$index = 0;
foreach ($outputTemp in $outputArr)
{
if ($outputTemp.ToString().Contains(","))
{
$output += "`"$outputTemp`",";
}
else
{
$output += $outputTemp + ",";
}
$index++;
if ($index -eq $outputArr.Count)
{
if ($output.EndsWith(","))
{
$output = $output.Remove($output.Length - 1);
}
}
}
I found a diabolically simply answer after I opened a csv file in Excel and added text and commas to one column. When I saved, closed and reopened, the column still had all the words and commas properly formatted. So, then I opened the file in notepad++ and this is what I found:
column1text, column2text,"column3,text,with,commas"
In case it's not clear, and it took me a fair bit to recognize the little detail that makes all the difference, the opening double quote cannot have a space after the preceding comma.
column1text, column2text, "column3,text,with,commas"
splits all the words into separate columns because there is a space between
column2text, "column3,etc"
Take that space away
column2text,"column3,etc"
and everything within the double quotes stays in one column.
Example using active directory distinguishedName such as CN=somename,OU=Domain Controllers,DC=foo,DC=bar
$computers = get-adcomputer -filter *
foreach ($computer in $computers) {
$deviceName = $computer.Name
$dn = '"' + $computer.DistinguishedName + '"'
$guid = $computer.objectGUID
$lastLogon = $computer.LastLogonDate
$serialNumber = $computer.serialNumber
$whenCreated = $computer.whenCreated
"$guid, $lastLogon, $deviceName, $serialNumber, $whenCreated,$dn" | add-content "c:\temp\filename.csv"
}
It does not work if a space is added between $whenCreated, and $dn like so:
"$guid, $lastLogon, $deviceName, $serialNumber, $whenCreated, $dn" | add-content "c:\temp\filename.csv"
This took up an afternoon, so I hope this saves somebody some time and frustration.

Is there a way to merge similar lines using Powershell?

Suppose I have two csv files. One is
id_number,location_code,category,animal,quantity
12212,3,4,cat,2
29889,7,6,dog,2
98900,
33221,1,8,squirrel,1
the second one is:
98900,2,1,gerbil,1
The second file may have a newline or something at the end (maybe or maybe not, I haven't checked), but only the one line of content. There may be three or four or more different varieties of the "second" file, but each one will have a first element (98900 in this example) that corresponds to an incomplete line in the first file similar to what is in this example.
Is there a way using powershell to automatically merge the line in the second (plus any additional similar) csv file into the matching line(s) of the first file, so that the resulting file is:
12212,3,4,cat,2
29889,7,6,dog,2
98900,2,1,gerbil,1
33221,1,8,squirrel,1
main.csv
id_number,location_code,category,animal,quantity
12212,3,4,cat,2
29889,7,6,dog,2
98900,
33221,1,8,squirrel,1
correction_001.csv
98900,2,1,gerbil,1
merge code used at the commandline, or in the .ps1 file of your choice
$myHeader = #('id_number','location_code','category','animal','quantity')
#Stage all the correction files: last correction in the most recent file wins
$ToFix = #{}
filter Plumbing_Import-Csv($Header){import-csv -LiteralPath $_ -Header $Header}
ls correction*.csv | sort -Property LastWriteTime | Plumbing_Import-Csv $myHeader | %{$ToFix[$_.id_number]=$_}
function myObjPipe($Header){
begin{
function TextTo-CsvField([String]$text){
#text fields which contain comma, double quotes, or new-line are a special case for CSV fields and need to be accounted for
if($text -match '"|,|\n'){return '"'+($text -replace '"','""')+'"'}
return $text
}
function myObjTo-CsvRecord($obj){
return ''+
$obj.id_number +','+
$obj.location_code +','+
$obj.category +','+
(TextTo-CsvField $obj.animal)+','+
$obj.quantity
}
$Header -join ','
}
process{
if($ToFix.Contains($_.id_number)){
$out = $ToFix[$_.id_number]
$ToFix.Remove($_.id_number)
}else{$out = $_}
myObjTo-CsvRecord $out
}
end{
#I assume you'd append any leftover fixes that weren't used
foreach($out in $ToFix.Values){
myObjTo-CsvRecord $out
}
}
}
import-csv main.csv | myObjPipe $myHeader | sc combined.csv -encoding ascii
You could also use ConvertTo-Csv, but my preference is to not have all the extra " cruft.
Edit 1: reduced code redundancy, accounted for \n, fixed appends, and used #OwlsSleeping suggestion about the -Header commandlet parameter
also works with these files:
correction_002.csv
98900,2,1,I Win,1
correction_new.csv
98901,2,1,godzilla,1
correction_too.csv
98902,2,1,gamera,1
98903,2,1,mothra,1
Edit 2: convert gc | ConvertTo-Csv over to Import-Csv to fix the front-end \n issues. Now also works with:
correction_003.csv
29889,7,6,"""bad""
monkey",2
This is a simple solution assuming there's always exactly one match, and you don't care about output order. Change the output path to csv1 to overwrite.
I added headers manually in both input files, but you can specify them in Import-Csv instead if you'd rather avoid changing your files.
[array]$MissingLine = Import-Csv -Path "C:\Users\me\Documents\csv2.csv"
[string]$MissingId = $MissingLine[0].id_number
[array]$BigCsv = Import-Csv -Path "C:\Users\me\Documents\csv1.csv" |
Where-Object {$_.id_number -ne $MissingId}
($BigCsv + $MissingLine) |
Export-Csv -Path "C:\Users\me\Documents\Combined.csv"

Append text to certain values in text file with PowerShell

I have a CSV text file separated with ; and it's in the format as:
USER_EMPLOYEE_ID;SYSTEM1;USERNAME1
The first column is an identity and the following pairs of columns are user's account on different active directories. I have placed garbage data but the idea is there.
ay7suve0001;ADDPWN;ay7suve0001
AAXMR3E0001;ADDPWN;AAXMR3E0001
ABABIL;ADDPWN;ABABIL
ABDF17;ADDPWN;ABDF17;
ABKMPPE0001;ADDPWN;ABKMPPE0001
ABL1FL;ADDPWN;ABL1FL
AB6JG8E0004;ADDPWN;AB6JG8E0004;
ACB4YB;ADDPWN;ACB4YB
ACK7J9;ADDPWN;ACK7J9
ACLZFS;ADDPWN;ACLZFS;
ACQXZ3;ADDPWN;ACQXZ3
Now there is a requirement that I have to append a fixed string like #ADDPWN.com to all the USERNAME1 values. Some records are having a ; and some don't.
Is there a quick way to append the #ADDPWN.com to each line taking care of:
any ;
any already #ADDPWN.com
From PowerShell?
Import-Csv is your friend. The following should get you on the right track.
Import-Csv "import.csv" -Delimiter ';' |
foreach {
if ($_.username1 -notlike '*#ADDPWN.com') { $_.username1 += '#ADDPWN.com' }
$_
} |
Export-Csv "export.csv" -Delimiter ';'
This assumes the first line of your csv file is your header line. If it's not, you can pass -Header 'USER_EMPLOYEE_ID','SYSTEM1','USERNAME1' as another parameter to Import-Csv.
Export-Csv adds some extra stuff like quotes around parameters, so you may need to play with the output format if you don't want that.
For another explanation how this works, check out Changes last name, first name to first name, last name in last column CSV powershell
This was a solution that worked for me.........
#opens list of file names
$file2 ="F:\OneDrive_Biz\PowerApps\SecurityCameraVideoApp\file_list_names.csv"
$x = Get-Content $file2
#appends URl to beginning of file name list
for($i=0; $i -lt $x.Count; $i++){
$x[$i] = "https://analytics-my.sharepoint.com/personal/gpowell_analytics_onmicrosoft_com/Documents/PowerApps/SecurityCameraVideoApp/Video_Files/" + $x[$i]
}
$x
#remove all files in target directory prior to saving new list
get-childitem -path C:\_TEMP\file_list_names.csv | remove-item
Add-Content -Path C:\_TEMP\file_list_names_url.csv -Value $x

Export-CSV cmdlet rewriting entire CSV during each iteration of a FOREACH statement

I'm working with some code that is going to take a series of performance counters, and then put the counters in a .csv file that rolls over every time it hits 1MB.
$Folder="C:\Perflogs\BBCRMLogs" # Change the bit in the quotation marks to whatever directory you want the log file stored in
$Computer = $env:COMPUTERNAME
$1GBInBytes = 1GB
$p = LOTS OF COUNTERS;
# If you want to change the performance counters, change the above list. However, these are the recommended counters for a client machine.
$num = 0
$file = "$Folder\SQL_log_${num}.csv"
if( !(test-path $folder)) {New-Item $Folder -type directory}
Get-Counter -counter $p -SampleInterval 2 -Continuous | Foreach {
if ((Get-Item $file -ErrorAction SilentlyContinue ).Length -gt 1mb)
{
$num +=1
$file = "$Folder\SQL_log_${num}.csv"
}
$_
} | Foreach-Object { $_ | Export-Csv $file -Force -Append}
Right now, it's working quite well. The iteration works fine, and it does create a new file each time the .csv reaches 1MB. However, each .CSV after the first is being created after 2 minutes already at 1MB, causing a new file to be created. I'm not quite sure why this is occurring, although I believe it's because Powershell is just rewriting the entirety of the .csv each time it creates it.
[I'm posting this as a new answer rather than editing the original because it's completely different. Replacing or appending to the original answer would make the ensuing discussion confusing.]
What you need to do is use a regex to extract the values from the Readings property of the output of Get-Counter, and manually construct CSV output from the timestamp and those values. Change the last line to this (format according to your preferred style):
| %{'"' + (Get-Date $_.Timestamp -f 's') + '","' + (([regex]::matches($_.Readings, '(?<=\\\\.+?:\n)(.+?)(?=\n)') | select -ExpandProperty Value) -join '","') + '"'} | Out-File $file -Append -Encoding ASCII
To break that down:
(Get-Date $_.Timestamp -f 's') This part is not strictly necessary, though I think it will make your results easier to follow. The 's' format puts the date in an ISO 8601 sortable pattern. You could substitute 'u' for another sortable format, or use your favorite custom format string. Or just replace it with $_.Timestamp to retain the original format.
[regex]::matches($_.Readings, '(?<=\\\\.+?:\n)(.+?)(?=\n)') The regex matches the contents of any line that is preceded by a line that begins with \\ and ends with : (those pesky counter names you wanted to get rid of). Note that I'm using [regex]::matches, which performs a global match, as opposed to [regex]::match or -match, which will just give you the first match for each string (the Readings property is a single string, so only the first counter reading would be returned).
| select -ExpandProperty Value Produces an array of all the matches, which you can then join with "," and surround with "'s to produce CSV output.
Since you're not using a conversion function, you also need to construct a header row. Add this line right above the pipeline:
`'"Timestamp","' + ($p -join '","') + '"' | Out-File $file -Append -Encoding ASCII`
That's assuming that $p is an array (which it should be). If it's a string, then depending on the format you can either use it as-is, or -split it and rejoin it in CSV format.
Change the last line to this, to convert each line to CSV format and then append it to the output file:
} | Foreach-Object {($_ | ConvertTo-Csv -NoTypeInformation)[1] | Out-File $file -Append -Encoding ASCII}.
A few notes:
The -Encoding ASCII is not strictly necessary, but you might have trouble with a Unicode CSV file in some applications (Excel, for example, won't open it as a CSV file by default, and everything will be in Column A)
The reason for the index in ($_ | ConvertTo-Csv -NoTypeInformation)[1] is that ConvertTo-Csv -NoTypeInformation still outputs the header row each time, so you want to grab the second line of the two-line output (($_ | ConvertTo-Csv -NoTypeInformation)[0] is the header row)
Since you're not outputting a header row, you'll need to output one to $file before the loop