There's a GNU program called sponge that soaks up input before writing to a file so you can do something like this: cat myFile | grep "myFilter" | sponge myFile
Is there a powershell equivalent, so I can work on a file in place, without having to pipe to a temporary file?
Thanks
In Powershell, judicious use of parentheses will force an operation to completely finish before passing data to the next command in the pipeline. The default for piping Get-Content is to pipe line by line to the next command, but with parentheses it must form a complete data set (e.g., load all lines) before continuing:
(Get-Content myFile) | Select-String 'MyFilter' | Set-Content myFile
An alternative that may use less memory (I have not benchmarked it) is to only force the results of Select-String to complete before continuing:
(Get-Content myFile | Select-String 'MyFilter') | Set-Content myFile
You could also assign things to a variable as an additional step. Any technique will load the contents into the Powershell session's memory, so be careful with big files.
Addendum: Select-String returns MatchInfo objects. Using Out-File adds pesky extra blank lines due to the way it tries to format the results as a string, but Set-Content correctly converts each object to its own string as it writes, producing better output. Being that you're coming from *nix and are used to everything returning strings (whereas Powershell returns objects), one way to force string output is to pipe them through a foreach that converts them:
(Get-Content myFile | Select-String 'MyFilter' | foreach { $_.tostring() }) | Set-Content myFile
You can try this :
(Get-content myfile) | where {$_ -match "regular-expression"} | Set-content myfile
or
${full-path-file-name-of-myfile} | where {$_ -match "regular-expression"} | add-content Anotherfile
more easier to keep in mind
two other ways come to mind - they are both the same really just one is a function the other is on the command line. (I don't know sponge on unix so I can't say for certain they mimic it).
here's the first on the command line
Get-Content .\temp.txt |
Select-String "grep" |
foreach-object -begin { [array] $out = #()} -process { $out = $out + ($_.tostring())} -end {write-output $out}
and the second is two create a function to do it
function sponge {
[cmdletbinding()]
Param(
[Parameter(
Mandatory = $True,
ValueFromPipeline = $True)]
[string]$Output
)
Begin {
[array] $out = #()
}
Process {
$out = $out + $Output
}
End {
Write-Output $Out
}
}
Get-Content .\temp2.txt | Select-String "grep" | sponge
HTH,
Matt
Related
I want to do this
read the file
go through each line
if the line matches the pattern, do some changes with that line
save the content to another file
For now I use this script:
$file = [System.IO.File]::ReadLines("C:\path\to\some\file1.txt")
$output = "C:\path\to\some\file2.txt"
ForEach ($line in $file) {
if($line -match 'some_regex_expression') {
$line = $line.replace("some","great")
}
Out-File -append -filepath $output -inputobject $line
}
As you can see, here I write line by line. Is it possible to write the whole file at once ?
Good example is provided here :
(Get-Content c:\temp\test.txt) -replace '\[MYID\]', 'MyValue' | Set-Content c:\temp\test.txt
But my problem is that I have additional IF statement...
So, what could I do to improve my script ?
You could do it like that:
Get-Content -Path "C:\path\to\some\file1.txt" | foreach {
if($_ -match 'some_regex_expression') {
$_.replace("some","great")
}
else {
$_
}
} | Out-File -filepath "C:\path\to\some\file2.txt"
Get-Content reads a file line by line (array of strings) by default so you can just pipe it into a foreach loop, process each line within the loop and pipe the whole output into your file2.txt.
In this case Arrays or Array List(lists are better for large arrays) would be the most elegant solution. Simply add strings in array until ForEach loop ends. After that just flush array to a file.
This is Array List example
$file = [System.IO.File]::ReadLines("C:\path\to\some\file1.txt")
$output = "C:\path\to\some\file2.txt"
$outputData = New-Object System.Collections.ArrayList
ForEach ($line in $file) {
if($line -match 'some_regex_expression') {
$line = $line.replace("some","great")
}
$outputData.Add($line)
}
$outputData |Out-File $output
I think the if statement can be avoided in a lot of cases by using regular expression groups (e.g. (.*) and placeholders (e.g. $1, $2 etc.).
As in your example:
(Get-Content .\File1.txt) -Replace 'some(_regex_expression)', 'great$1' | Set-Content .\File2.txt
And for the good example" where [MYID\] might be somewhere inline:
(Get-Content c:\temp\test.txt) -Replace '^(.*)\[MYID\](.*)$', '$1MyValue$2' | Set-Content c:\temp\test.txt
(see also How to replace first and last part of each line with powershell)
I am trying to get all the lines from an Input file starting with %% and paste it into Output file using powershell.
Used the following code, however I am only getting last line in Output file starting with %% instead of all the lines starting with %%.
I have only started to learn powershell, please help
$Clause = Get-Content "Input File location"
$Outvalue = $Clause | Foreach {
if ($_ -ilike "*%%*")
{
Set-Content "Output file location" $_
}
}
You are looping over the lines in the file, and setting each one as the whole content of the file, overwriting the previous file each time.
You need to either switch to using Add-Content instead of Set-Content, which will append to the file, or change the design to:
Get-Content "input.txt" | Foreach-Object {
if ($_ -like "%%*")
{
$_ # just putting this on its own, sends it on out of the pipeline
}
} | Set-Content Output.txt
Which you would more typically write as:
Get-Content "input.txt" | Where-Object { $_ -like "%%*" } | Set-Content Output.txt
and in the shell, you might write as
gc input.txt |? {$_ -like "%%*"} | sc output.txt
Where the whole file is filtered, and then all the matching lines are sent into Set-Content in one go, not calling Set-Content individually for each line.
NB. PowerShell is case insensitive by default, so -like and -ilike behave the same.
For a small file, Get-Content is nice. But if you start trying to do this on heavier files, Get-Content will eat your memory and leave you hanging.
Keeping it REALLY simple for other Powershell starters out there, you'll be better covered (and with better performance). So, something likes this would do the job:
$inputfile = "C:\Users\JohnnyC\Desktop\inputfile.txt"
$outputfile = "C:\Users\JohnnyC\Desktop\outputfile.txt"
$reader = [io.file]::OpenText($inputfile)
$writer = [io.file]::CreateText($outputfile)
while($reader.EndOfStream -ne $true) {
$line = $reader.Readline()
if ($line -like '%%*') {
$writer.WriteLine($line);
}
}
$writer.Dispose();
$reader.Dispose();
I tried the following to turn a text file into a document by leveraging import-csv where each item in the original document was a new line
Sample file.txt
James Cameron
Kirk Cobain
Linda Johnson
Code:
$array = import-csv file.txt | ConvertFrom-Csv -Delim `r
foreach ($Data in $array)
{
if (sls $Data Master.txt -quiet)
{Add-Content file.txt $Data}
}
It never created the document
Import-Csv takes a CSV and outputs PSCustomObjects. It's intended for when the file has a header row, and it reads that as the properties of the objects. e.g.
FirstName,LastName
James,Cameron
Kirk,Cobain
# ->
#{FirstName='James';LastName='Cameron'}
#{FirstName='Kirk';LastName='Cobain'}
etc.
If your file has no header row, it will take the first row and then ruin everything else afterwards. You need to provide the -Header 'h1','h2',... parameter to fix that. So you could use -Header Name, but your data only has one property, so there's not much benefit.
ConvertFrom-Csv is intended to do the same thing, but from CSV data in a variable instead of a file. They don't chain together usefully. It will try, but what you end up with is...
A single object, with a property called '#{James=Kirk}' and a value of '#{James=Linda}', where 'James' was taken from line 1 as a column header, and the weird syntax is from forcing those objects through a second conversion.
It's not at all clear why you are reading in from file.txt and adding to file.txt. But since you don't have a CSV, there's no benefit from using the CSV cmdlets.
$lines = Get-Content file.txt
$master = Get-Content master.txt
foreach ($line in $lines)
{
if ($master -contains $line)
{
Add-Content file2.txt $line
}
}
or just
gc file.txt |? { sls $_ master.txt -quiet } | set-content file2.txt
Auto-generated PS help links from my codeblock (if available):
gc is an alias for Get-Content (in module Microsoft.PowerShell.Management)
? is an alias for Where-Object
sls is an alias for Select-String (in module Microsoft.PowerShell.Utility)
Set-Content (in module Microsoft.PowerShell.Management)
I'm looking for a way to export all lines from within a text file where part of the line matches a certain string. The string is actually the first 4 bytes of the file and I'd like to keep the command to only checking those bytes; not the entire row. I want to write the entire row. How would I go about this?
I am using Windows only and don't have the option to use many other tools that might do this.
Thanks in advance for any help.
Do you want to perform a simple "grep"? Then try this
select-string .\test.txt -pattern "\Athat" | foreach {$_.Line}
or this (very similar regex), also writes to an outfile
select-string .\test.txt -pattern "^that" | foreach {$_.Line} | out-file -filepath out.txt
This assumes that you want to search for a 4-byte string "that" at the beginning of the string , or beginning of the line, respectively.
Something like the following Powershell function should work for you:
function Get-Lines {
[cmdletbinding()]
param(
[string]$filename,
[string]$prefix
)
if( Test-Path -Path $filename -PathType Leaf -ErrorAction SilentlyContinue ) {
# filename exists, and is a file
$lines = Get-Content $filename
foreach ( $line in $lines ) {
if ( $line -like "$prefix*" ) {
$line
}
}
}
}
To use it, assuming you save it as get-lines.ps1, you would load the function into memory with:
. .\get-lines.ps1
and then to use it, you could search for all lines starting with "DATA" with something like:
get-lines -filename C:\Files\Datafile\testfile.dat -prefix "DATA"
If you need to save it to another file for viewing later, you could do something like:
get-lines -filename C:\Files\Datafile\testfile.dat -prefix "DATA" | out-file -FilePath results.txt
Or, if I were more awake, you could ignore the script above, use a simpler solution such as the following one-liner:
get-content -path C:\Files\Datafile\testfile.dat | select-string -Pattern "^DATA"
Which just uses the ^ regex character to make sure it's only looking for "DATA" at the beginning of each line.
To get all the lines from c:\somedir\somefile.txt that begin with 'abcd' :
(get-content c:\somedir\somefile.txt) -like 'abcd*'
provided c:\somedir\somefile.txt is not an unusually large (hundreds of MB) file. For that situation:
get-content c:\somedir\somefile.txt -readcount 1000 |
foreach {$_ -like 'abcd*'}
I am trying to change dates programmatically in a file. The line I need to fix looks like this:
set ##dateto = '03/15/12'
I need to write a powershell V2 script that replaces what's inside the single quotes, and I have no idea how to do this.
The closest I've come looks like this:
gc $file | ? {$_ -match "set ##dateto ="} | % {$temp=$_.split("'");$temp[17]
=$CorrectedDate;$temp -join ","} | -outfile newfile.txt
Problems with this: It gives an error about the index 17 being out of range. Also, the outfile only contains one line (The unmodified line). I'd appreciate any help with this. Thanks!
You can do something like this ( though you may want to handle the corner cases) :
$CorrectedDate = '10/09/09'
gc $file | %{
if($_ -match "^set ##dateto = '(\d\d/\d\d/\d\d)'") {
$_ -replace $matches[1], $CorrectedDate;
}
else {
$_
}
} | out-file test2.txt
mv test2.txt $file -force