I am trying to write a script which runs on multiple client machines and writes to a single text file on a network share.
I want to ensure that only one machine can maniputale the file at any one time, whilst the other machines run a loop to check if the file is available.
the script runs this first:
Set-Content -Path $PathToHost -Value (get-content -Path $PathToHost | Select-String -Pattern "$HostName " -NotMatch) -ErrorAction Stop
Which removes some lines if they are matching the criteria. Then I want to append a new line with this:
Add-Content $PathToHost "$heartbeat$_" -ErrorAction Stop
The problem is that between the execution of those two commands another client has access to the file and begins to write to the file as well.
I have explored the solution here: Locking the file while writing in PowerShell
$PathToHost = "C:\file.txt"
$mode = "Open"
$access = "ReadWrite"
$share = "None"
$file = [System.IO.File]::Open($path, $mode, $access, $share)
$file.close()
Which can definitely lock the file, but I am not sure how to proceed to then read and write to the file.
Any help is much appreciated.
EDIT: Solution as below thanks to twinlakes' answer
$path = "C:\Users\daniel_mladenov\hostsTEST.txt"
$mode = "Open"
$access = "ReadWrite"
$share = "none"
$file = [System.IO.File]::Open($path, $mode, $access, $share)
$fileread = [System.IO.StreamReader]::new($file, [Text.Encoding]::UTF8)
# Counts number of lines in file
$imax=0
while ($fileread.ReadLine() -ne $null){
$imax++
}
echo $imax
#resets read position to beginning
$fileread.basestream.position = 0
#reads content of whole file and discards mathching lines
$content=#()
for ($i=0; $i -lt $imax; $i++){
$ContentLine = $fileread.ReadLine()
If($ContentLine -notmatch "$HostIP\s" -and $ContentLine -notmatch "$HostName\s"){
$content += $ContentLine
}
}
echo $content
#Writes remaining lines back to file
$filewrite = [System.IO.StreamWriter]::new($file)
$filewrite.basestream.position = 0
for ($i=0; $i -lt $content.length; $i++){
$filewrite.WriteLine($content[$i])
}
$filewrite.WriteLine($heartbeat)
$filewrite.Flush()
$file.SetLength($file.Position) #trims file to the content which has been written, discarding any content past that point
$file.close()
$file is a System.IO.FileStream object. You will need to call the write method on that object, which requires a byte array.
$string = # the string to write to the file
$bytes = [Text.Encoding]::UTF8.GetBytes($string)
$file.Write($bytes, 0, $bytes.Length)
I want to do this
read the file
go through each line
if the line matches the pattern, do some changes with that line
save the content to another file
For now I use this script:
$file = [System.IO.File]::ReadLines("C:\path\to\some\file1.txt")
$output = "C:\path\to\some\file2.txt"
ForEach ($line in $file) {
if($line -match 'some_regex_expression') {
$line = $line.replace("some","great")
}
Out-File -append -filepath $output -inputobject $line
}
As you can see, here I write line by line. Is it possible to write the whole file at once ?
Good example is provided here :
(Get-Content c:\temp\test.txt) -replace '\[MYID\]', 'MyValue' | Set-Content c:\temp\test.txt
But my problem is that I have additional IF statement...
So, what could I do to improve my script ?
You could do it like that:
Get-Content -Path "C:\path\to\some\file1.txt" | foreach {
if($_ -match 'some_regex_expression') {
$_.replace("some","great")
}
else {
$_
}
} | Out-File -filepath "C:\path\to\some\file2.txt"
Get-Content reads a file line by line (array of strings) by default so you can just pipe it into a foreach loop, process each line within the loop and pipe the whole output into your file2.txt.
In this case Arrays or Array List(lists are better for large arrays) would be the most elegant solution. Simply add strings in array until ForEach loop ends. After that just flush array to a file.
This is Array List example
$file = [System.IO.File]::ReadLines("C:\path\to\some\file1.txt")
$output = "C:\path\to\some\file2.txt"
$outputData = New-Object System.Collections.ArrayList
ForEach ($line in $file) {
if($line -match 'some_regex_expression') {
$line = $line.replace("some","great")
}
$outputData.Add($line)
}
$outputData |Out-File $output
I think the if statement can be avoided in a lot of cases by using regular expression groups (e.g. (.*) and placeholders (e.g. $1, $2 etc.).
As in your example:
(Get-Content .\File1.txt) -Replace 'some(_regex_expression)', 'great$1' | Set-Content .\File2.txt
And for the good example" where [MYID\] might be somewhere inline:
(Get-Content c:\temp\test.txt) -Replace '^(.*)\[MYID\](.*)$', '$1MyValue$2' | Set-Content c:\temp\test.txt
(see also How to replace first and last part of each line with powershell)
I have a very large file (hence .ReadLines) which I need to efficiently and quickly parse and split into other files. For each line which contains a keyword I need to copy that line and append to a specific file. This is what I have so far, the script runs but the files aren't getting populated.
$filename = "C:\dev\powershell\test1.csv"
foreach ($line in [System.IO.File]::ReadLines($filename)) {
if ($line | %{$_ -match "Apple"}){Out-File -Append Apples.txt}
elseif($line | %{$_ -match "Banana"}){Out-File -Append Bananas.txt}
elseif($line | %{$_ -match "Pear"}){Out-File -Append Pears.txt}
}
Example content of the csv file:
Apple,Test1,Cross1
Apple,Test2,Cross2
Apple,Test3,Cross3
Banana,Test4,Cross4
Pear,Test5,Cross5
I want Apples.txt to contain:
Apple,Test1,Cross1
Apple,Test2,Cross2
Apple,Test3,Cross3
Couple of things:
Your if conditions don't need %/foreach-object - -match will do on its own:
foreach ($line in [System.IO.File]::ReadLines($filename)) {
if($line -match "Apple"){
# output to apple.txt
}
else($line -match "Banana"){
# output to banana.txt
}
# etc...
}
The files aren't getting populated because you're not actually sending any output to Out-File:
foreach ($line in [System.IO.File]::ReadLines($filename)) {
if($line -match "Apple"){
# send $line to the file
$line |Out-File apple.txt -Append
}
# etc...
}
If your files are really massive and you expect a lot of matching lines, I'd recommend using a StreamWriter for the output files - otherwise Out-File will be opening and closing the file all the time:
$OutFiles = #{
'apple' = New-Object System.IO.StreamWriter $PWD\apples.txt
'banana' = New-Object System.IO.StreamWriter $PWD\bananas.txt
'pear' = New-Object System.IO.StreamWriter $PWD\pears.txt
}
foreach ($line in [System.IO.File]::ReadLines($filename)) {
foreach($keyword in $OutFiles.Keys){
if($line -match $keyword){
$OutFiles[$keyword].WriteLine($line)
continue
}
}
}
foreach($Writer in $OutFiles.Values){
try{
$Writer.Close()
}
finally{
$Writer.Dispose()
}
}
This way you also only have to maintain the $OutFiles hashtable if you need to update the keywords for example.
$check = $args[1]
$numArgs = $($args.count)
$totMatch = 0
#reset variables for counting
for ( $i = 2; $i -lt $numArgs; $i++ )
{
$file = $args[$i]
if ( Test-Path $file ) {
#echo "The input file was named $file"
$match = #(Select-String $check $file -AllMatches | Select -Expand Matches | Select -Expand Value).count
echo "There were $match Matches in $file"
echo "There were $match Matches in $file" >> Output.txt
$totMatch = $totMatch + $match
}
else {
echo "File $file does not exist"
echo "File $file does not exist" >> Output.txt
}
}
echo "Total Matches Found: $totMatch"
Esentially i created a quick app to find the word searched and check the instances in the file, would anyone know how to edit this to send the whole Line that the word was found in to the Ouput.txt file, So rather on top of instances add the whole line itself? Thanks in advance
I can't see that your code works properly; even though you don't say how it's supposed to work (why is $check taken from args[1] instead of args[0]?).
Your Select-String line is getting the matching lines, then doing some selecting which throws away the line data you want, but doesn't seem to be necessary.
I've reworked it as:
$check = $args[0]
$totalMatches = 0
foreach ( $file in $args[1..$args.Length] )
{
if ( Test-Path $file ) {
$matches = Select-String $check $file -AllMatches -SimpleMatch
Write-Output "There were $($matches.Count) Matches in $file" | Tee-Object -FilePath "output.txt" -Append
foreach ($match in $matches) {
Write-Output $match.Line | Tee-Object -FilePath "output.txt" -Append
}
Write-Host
$totalMatches = $totalMatches + $matches.Count
}
else {
Write-Output "File $file does not exist" | Tee-Object -FilePath "output.txt" -Append
}
}
echo "Total Matches Found: $totalMatches"
Changes:
Take the $check as the first argument
Iterate over the arguments directly instead of counting through them
Added -SimpleMatch so it doesn't work with regexes, since you didn't mention them
Removed the select-object -expand bits, just grab the select-string results
Loop through the results and get the line from $match.line
Added Tee-Object which both writes to screen and to file in one line
Using PowerShell, I want to replace all exact occurrences of [MYID] in a given file with MyValue. What is the easiest way to do so?
Use (V3 version):
(Get-Content c:\temp\test.txt).replace('[MYID]', 'MyValue') | Set-Content c:\temp\test.txt
Or for V2:
(Get-Content c:\temp\test.txt) -replace '\[MYID\]', 'MyValue' | Set-Content c:\temp\test.txt
I prefer using the File-class of .NET and its static methods as seen in the following example.
$content = [System.IO.File]::ReadAllText("c:\bla.txt").Replace("[MYID]","MyValue")
[System.IO.File]::WriteAllText("c:\bla.txt", $content)
This has the advantage of working with a single String instead of a String-array as with Get-Content. The methods also take care of the encoding of the file (UTF-8 BOM, etc.) without you having to take care most of the time.
Also the methods don't mess up the line endings (Unix line endings that might be used) in contrast to an algorithm using Get-Content and piping through to Set-Content.
So for me: Fewer things that could break over the years.
A little-known thing when using .NET classes is that when you have typed in "[System.IO.File]::" in the PowerShell window you can press the Tab key to step through the methods there.
(Get-Content file.txt) |
Foreach-Object {$_ -replace '\[MYID\]','MyValue'} |
Out-File file.txt
Note the parentheses around (Get-Content file.txt) is required:
Without the parenthesis the content is read, one line at a time, and flows down the pipeline until it reaches out-file or set-content, which tries to write to the same file, but it's already open by get-content and you get an error. The parenthesis causes the operation of content reading to be performed once (open, read and close). Only then when all lines have been read, they are piped one at a time and when they reach the last command in the pipeline they can be written to the file. It's the same as $content=content; $content | where ...
The one above only runs for "One File" only, but you can also run this for multiple files within your folder:
Get-ChildItem 'C:yourfile*.xml' -Recurse | ForEach {
(Get-Content $_ | ForEach { $_ -replace '[MYID]', 'MyValue' }) |
Set-Content $_
}
I found a little known but amazingly cool way to do it from Payette's Windows Powershell in Action. You can reference files like variables, similar to $env:path, but you need to add the curly braces.
${c:file.txt} = ${c:file.txt} -replace 'oldvalue','newvalue'
You could try something like this:
$path = "C:\testFile.txt"
$word = "searchword"
$replacement = "ReplacementText"
$text = get-content $path
$newText = $text -replace $word,$replacement
$newText > $path
This is what I use, but it is slow on large text files.
get-content $pathToFile | % { $_ -replace $stringToReplace, $replaceWith } | set-content $pathToFile
If you are going to be replacing strings in large text files and speed is a concern, look into using System.IO.StreamReader and System.IO.StreamWriter.
try
{
$reader = [System.IO.StreamReader] $pathToFile
$data = $reader.ReadToEnd()
$reader.close()
}
finally
{
if ($reader -ne $null)
{
$reader.dispose()
}
}
$data = $data -replace $stringToReplace, $replaceWith
try
{
$writer = [System.IO.StreamWriter] $pathToFile
$writer.write($data)
$writer.close()
}
finally
{
if ($writer -ne $null)
{
$writer.dispose()
}
}
(The code above has not been tested.)
There is probably a more elegant way to use StreamReader and StreamWriter for replacing text in a document, but that should give you a good starting point.
Credit to #rominator007
I wrapped it into a function (because you may want to use it again)
function Replace-AllStringsInFile($SearchString,$ReplaceString,$FullPathToFile)
{
$content = [System.IO.File]::ReadAllText("$FullPathToFile").Replace("$SearchString","$ReplaceString")
[System.IO.File]::WriteAllText("$FullPathToFile", $content)
}
NOTE: This is NOT case sensitive!!!!!
See this post: String.Replace ignoring case
If You Need to Replace Strings in Multiple Files:
It should be noted that the different methods posted here can be wildly different with regard to the time it takes to complete. For me, I regularly have large numbers of small files. To test what is most performant, I extracted 5.52 GB (5,933,604,999 bytes) of XML in 40,693 separate files and ran through three of the answers I found here:
## 5.52 GB (5,933,604,999 bytes) of XML files (40,693 files)
$xmls = (Get-ChildItem -Path "I:\TestseT\All_XML" -Recurse -Filter *.xml).FullName
#### Test 1 - Plain Replace
$start = Get-Date
foreach ($xml in $xmls) {
(Get-Content $xml).replace("'", " ") | Set-Content $xml
}
$end = Get-Date
New-TimeSpan –Start $Start –End $End
# TotalMinutes: 103.725113128333
#### Test 2 - Replace with -Raw
$start = Get-Date
foreach ($xml in $xmls) {
(Get-Content $xml -Raw).replace("'", " ") | Set-Content $xml
}
$end = Get-Date
New-TimeSpan –Start $Start –End $End
# TotalMinutes: 10.1600227983333
#### Test 3 - .NET, System.IO
$start = Get-Date
foreach ($xml in $xmls) {
$txt = [System.IO.File]::ReadAllText("$xml").Replace("'"," ")
[System.IO.File]::WriteAllText("$xml", $txt)
}
$end = Get-Date
New-TimeSpan –Start $Start –End $End
# TotalMinutes: 5.83619516833333
Since this comes up often, I defined a function for it. I defaulted to case-sensitive, regex-based matching, but I included switches for targeting literal text and ignoring case.
# Find and replace text in each pipeline string. Omit the -Replace parameter to delete
# text instead. Use the -SimpleMatch switch to work with literal text instead of regular
# expressions. Comparisons are case-sensitive unless the -IgnoreCase switch is used.
Filter Edit-String {
Param([string]$Find, [string]$Replace='', [switch]$SimpleMatch, [switch]$IgnoreCase)
if ($SimpleMatch) {
if ($IgnoreCase) {
return $_.Replace($Find, $Replace,
[System.StringComparison]::OrdinalIgnoreCase)
}
return $_.Replace($Find, $Replace)
}
if ($IgnoreCase) {
return $_ -replace $Find, $Replace
}
return $_ -creplace $Find, $Replace
}
Set-Alias replace Edit-String
Set-Alias sc Set-Content
Usage
# 1 file
$f = a.txt; gc $f | replace '[MYID]' 'MyValue' -SimpleMatch | sc $f
# 0 to many files
gci *.txt | % { gc $_ | replace '\[MYID\]' 'MyValue' | sc $_ }
# Several replacements chained together
... | replace '[1-9]' T | replace a b -IgnoreCase | replace 'delete me' | ...
# Alias cheat sheet
# gci Get-ChildItem
# gc Get-Content
# sc Set-Conent
# % ForEach-Object
This worked for me using the current working directory in PowerShell. You need to use the FullName property, or it won't work in PowerShell version 5. I needed to change the target .NET framework version in ALL my CSPROJ files.
gci -Recurse -Filter *.csproj |
% { (get-content "$($_.FullName)")
.Replace('<TargetFramework>net47</TargetFramework>', '<TargetFramework>net462</TargetFramework>') |
Set-Content "$($_.FullName)"}
A bit old and different, as I needed to change a certain line in all instances of a particular file name.
Also, Set-Content was not returning consistent results, so I had to resort to Out-File.
Code below:
$FileName =''
$OldLine = ''
$NewLine = ''
$Drives = Get-PSDrive -PSProvider FileSystem
foreach ($Drive in $Drives) {
Push-Location $Drive.Root
Get-ChildItem -Filter "$FileName" -Recurse | ForEach {
(Get-Content $_.FullName).Replace($OldLine, $NewLine) | Out-File $_.FullName
}
Pop-Location
}
This is what worked best for me on this PowerShell version:
Major.Minor.Build.Revision
5.1.16299.98
Here's a fairly simple one that supports multiline regular expressions, multiple files (using the pipeline), specifying output encoding, etc. Not recommended for very large files due to the ReadAllText method.
# Update-FileText.ps1
#requires -version 2
<#
.SYNOPSIS
Updates text in files using a regular expression.
.DESCRIPTION
Updates text in files using a regular expression.
.PARAMETER Pattern
Specifies the regular expression pattern.
.PARAMETER Replacement
Specifies the regular expression replacement pattern.
.PARAMETER Path
Specifies the path to one or more files. Wildcards are not supported. Each file is read entirely into memory to support multi-line searching and replacing, so performance may be slow for large files.
.PARAMETER CaseSensitive
Specifies case-sensitive matching. The default is to ignore case.
.PARAMETER SimpleMatch
Specifies a simple match rather than a regular expression match (i.e., the Pattern parameter specifies a simple string rather than a regular expression).
.PARAMETER Multiline
Changes the meaning of ^ and $ so they match at the beginning and end, respectively, of any line, and not just the beginning and end of the entire file. The default is that ^ and $, respectively, match the beginning and end of the entire file.
.PARAMETER UnixText
Causes $ to match only linefeed (\n) characters. By default, $ matches carriage return+linefeed (\r\n). (Windows-based text files usually use \r\n as line terminators, while Unix-based text files usually use only \n.)
.PARAMETER Overwrite
Overwrites a file by creating a temporary file containing all replacements and then replacing the original file with the temporary file. The default is to output but not overwrite.
.PARAMETER Force
Allows overwriting of read-only files. Note that this parameter cannot override security restrictions.
.PARAMETER Encoding
Specifies the encoding for the file when -Overwrite is used. Possible values for this parameter are ASCII, BigEndianUnicode, Unicode, UTF32, UTF7, and UTF8. The default value is ASCII.
.INPUTS
System.IO.FileInfo.
.OUTPUTS
System.String (single-line file) or System.String[] (file with more than one line) without the -Overwrite parameter, or nothing with the -Overwrite parameter.
.LINK
about_Regular_Expressions
.EXAMPLE
C:\> Update-FileText.ps1 '(Ferb) and (Phineas)' '$2 and $1' Story.txt
This command replaces the text 'Ferb and Phineas' with the text 'Phineas and Ferb' in the file Story.txt and outputs the content. Note that the pattern and replacement strings are enclosed in single quotes to prevent variable expansion.
.EXAMPLE
C:\> Update-FileText.ps1 'Perry' 'Agent P' Story2.txt -Overwrite
This command replaces the text 'Perry' with the text 'Agent P' in the file Story2.txt.
#>
[CmdletBinding(SupportsShouldProcess = $true,ConfirmImpact = "High")]
param(
[Parameter(Mandatory = $true,Position = 0,ValueFromPipeline = $true)]
[String[]] $Path,
[Parameter(Mandatory = $true,Position = 1)]
[String] $Pattern,
[Parameter(Mandatory = $true,Position = 2)]
[AllowEmptyString()]
[String] $Replacement,
[Switch] $CaseSensitive,
[Switch] $SimpleMatch,
[Switch] $Multiline,
[Switch] $UnixText,
[Switch] $Overwrite,
[Switch] $Force,
[ValidateSet("ASCII","BigEndianUnicode","Unicode","UTF32","UTF7","UTF8")]
[String] $Encoding = "ASCII"
)
begin {
function Get-TempName {
param(
$path
)
do {
$tempName = Join-Path $path ([IO.Path]::GetRandomFilename())
}
while ( Test-Path $tempName )
$tempName
}
if ( $SimpleMatch ) {
$Pattern = [Regex]::Escape($Pattern)
}
else {
if ( -not $UnixText ) {
$Pattern = $Pattern -replace '(?<!\\)\$','\r$'
}
}
function New-Regex {
$regexOpts = [Text.RegularExpressions.RegexOptions]::None
if ( -not $CaseSensitive ) {
$regexOpts = $regexOpts -bor [Text.RegularExpressions.RegexOptions]::IgnoreCase
}
if ( $Multiline ) {
$regexOpts = $regexOpts -bor [Text.RegularExpressions.RegexOptions]::Multiline
}
New-Object Text.RegularExpressions.Regex $Pattern,$regexOpts
}
$Regex = New-Regex
function Update-FileText {
param(
$path
)
$pathInfo = Resolve-Path -LiteralPath $path
if ( $pathInfo ) {
if ( (Get-Item $pathInfo).GetType().FullName -eq "System.IO.FileInfo" ) {
$fullName = $pathInfo.Path
Write-Verbose "Reading '$fullName'"
$text = [IO.File]::ReadAllText($fullName)
Write-Verbose "Finished reading '$fullName'"
if ( -not $Overwrite ) {
$regex.Replace($text,$Replacement)
}
else {
$tempName = Get-TempName (Split-Path $fullName -Parent)
Set-Content $tempName $null -Confirm:$false
if ( $? ) {
Write-Verbose "Created file '$tempName'"
try {
Write-Verbose "Started writing '$tempName'"
[IO.File]::WriteAllText("$tempName",$Regex.Replace($text,$Replacement),[Text.Encoding]::$Encoding)
Write-Verbose "Finished writing '$tempName'"
Write-Verbose "Started copying '$tempName' to '$fullName'"
Copy-Item $tempName $fullName -Force:$Force -ErrorAction Continue
if ( $? ) {
Write-Verbose "Finished copying '$tempName' to '$fullName'"
}
Remove-Item $tempName
if ( $? ) {
Write-Verbose "Removed file '$tempName'"
}
}
catch [Management.Automation.MethodInvocationException] {
Write-Error $Error[0]
}
}
}
}
else {
Write-Error "The item '$path' must be a file in the file system." -Category InvalidType
}
}
}
}
process {
foreach ( $PathItem in $Path ) {
if ( $Overwrite ) {
if ( $PSCmdlet.ShouldProcess("'$PathItem'","Overwrite file") ) {
Update-FileText $PathItem
}
}
else {
Update-FileText $PathItem
}
}
}
Also available as a gist on Github.
Sample to replace all strings inside a folder:
$path=$args[0]
$oldString=$args[1]
$newString=$args[2]
Get-ChildItem -Path $path -Recurse -File |
ForEach-Object {
(Get-Content $_.FullName).replace($oldString,$newString) | Set-Content $_.FullName
}
Small correction for the Set-Content command. If the searched string is not found the Set-Content command will blank (empty) the target file.
You can first verify if the string you are looking for exist or not. If not it will not replace anything.
If (select-string -path "c:\Windows\System32\drivers\etc\hosts" -pattern "String to look for") `
{(Get-Content c:\Windows\System32\drivers\etc\hosts).replace('String to look for', 'String to replace with') | Set-Content c:\Windows\System32\drivers\etc\hosts}
Else{"Nothing happened"}