I'm working with an application that creates a log file. Due to an error in the software itself, it keeps producing three errors I'm not interested in. Each line has a unique identifier so I can't just replace the line since each one is different.
I have two main issues with this: I need to save it with the same name, and while it works the file should be available (in case the logger needs to write something).
I can't hard-code the original app to prevent it from writing that part of the log.
I have tried so far:
Get-Content log.log | Where-Object {$_-notmatch 'ERROR1' -And $_-notmatch 'ERROR2' -And $_-notmatch 'ERROR3' } `|Set-Content log_stripped.log
^ It only works if the output file has a different name.
Get-Content error.log | foreach-object { Where-Object {$_-notmatch 'ERROR1' -And $_-notmatch 'ERROR2' -And $_-notmatch 'ERROR3' } } | Set-Content error.log
^ This one froze my PS session.
I also tried reading the file to a variable:
$logcontent = ${h:error.log}
but I got System.OutOfMemoryException.
Ideally, what I need is something that reads the log file, takes away all the lines I don't want, and then save it with its original name.
Ideas? (Keep in mind that the log file is +/- 900 MB with the unnecesary data and 45mb once I strip the data with the first method - but I need it to save the file with its original name)
You can't save the file back to the same name while you're still reading from it, which means you'd have to read the whole 900MB into memory before you start writing. Not a good idea.
Try this:
Remove-Item log_stripped.log
Get-Content log.log -ReadCount 1000 |
foreach {$_ -notmatch 'ERROR1|ERROR2|ERROR3' | Add-Content log_stripped.log }
Remove-item log.log
Rename-Item log_stripped.log log.log
I know you said you want to save to the same filename, but if the reason you want that is that you want the log to be continuously updated, then you could do the following:
Get-Content -Wait log.log |
? {$_ -notmatch 'ERROR1|ERROR2|ERROR3' } |
Out-File log_stripped.log
Note the -Wait on the Get-Content.
log_stripped.log will be continuously updated as log.log is updated.
Related
I would like to use PowerShell to add a line break for every tilde it finds in a file.
The source could contain main .in files which contain tildes.
I have this script so far, and could benefit by some assistance in how to tweak it.
This will work for one file, but not for many:
(Get-Content -Path '.\amalgamatedack.in') |
ForEach-Object {$_.Replace('~', "~`r`n")} |
Set-Content -Path '.\amalgamatedack.in'
You can use Get-ChildItem to find all your .in files, then follow the same logic, just replace the input and output hardcoded file name for the absolute path of each file (.FullName property).
Your code could also benefit by using Get-Content -Raw, assuming these files are not very big and they fit in memory, reading the content as single multi-line string is always faster.
# If you need to search recursively for the files use `-Recurse`
Get-ChildItem path\to\sourcefolder -Filter *.in | ForEach-Object {
($_ | Get-Content -Raw).Replace('~', "~`r`n") |
Set-Content -Path $_.FullName
}
I'm finishing a script in PowerShell and this is what I must do:
Find and retrieve all .txt files inside a folder
Read their contents (there is a number inside that must be less than 50)
If any of these files has a number greater than 50, change a flag which will allow me to send a crit message to a monitoring server.
The piece of code below is what I already have, but it's probably wrong because I haven't given any argument to Get-Content, it's probably something very simple, but I'm still getting used to PowerShell. Any suggestions? Thanks a lot.
Get-ChildItem -Path C:\temp_erase\PID -Directory -Filter *.txt |
ForEach-Object{
$warning_counter = Get-Content
if ($warning_counter -gt '50')
{
$crit_counter = 1
Write-Host "CRITICAL: Failed to kill service more than 50 times!"
}
}
but it's probably wrong because I haven't given any argument to Get-Content
Yes. That is the first issue. Have a look at Get-Help <command> and or docs like TechNet when you are lost. For the core cmdlets you will always see examples.
Second, Get-Content, returns string arrays (by default), so if you are doing a numerical comparison you need to treat the value as such.
Thirdly you have a line break between foreach-object cmdlet and its opening brace. That will land you a parsing problem and PS will prompt for the missing process block. So changing just those mentioned ....
Get-ChildItem -Path C:\temp_erase\PID -Directory -Filter *.txt | ForEach-Object{
[int]$warning_counter = Get-Content $_.FullName
if ($warning_counter -gt '50')
{
$crit_counter = 1
Write-Host "CRITICAL: Failed to kill service more than 50 times!"
}
}
One obvious thing missing from this is you do not show which file triggered the message. You should update your notification/output process. You also have no logic validating file contents. The could easily fail, either procedural or programically, on files with non numerical contents.
I have a script that searches for the lastest modified log file. It then is suppose to read that text file and pick up a key phrase then display the line after it.
So far i have this
$logfile = get-childitem 'C:\logs' | sort {$_.lastwritetime} | where {$_ -notmatch "X|Zr" }| select -last 1
$error = get-content $logfile | select-string -pattern "Failed to Modify"
an example line it reads is this
20150721 12:46:26 398fbb92 To CV Failed to Modify
CN=ROLE-x-USERS,OU=Role Groups,OU=Groups,DC=gyp,DC=gypuy,DC=net
MDS_E_BAD_MEMBERSHIP One or more members do not exist in the directory
They key bit of information im trying to get here is
Can anyone help?
Thanks
Try this:
$error = get-content $logfile |
Where-Object { $_ -like "*Failed to Modify*" } |
Select-Object -First 1
This is provided you are looking for the first match in the file. The Select-String cmdlet returns a MatchInfo object. Depending on your requirements there might be no reason to add that level of complexity if you're just looking to pull the first occurrence of this error in the file.
Failing this, my recommendation would be to debug this and step through it. Break on the Get-Content call and see what $logfile is. Run Get-Content $logfile and see what that content looks like. Then do your Select-String on that output. See what MatchInfo.ToString() looks like. Maybe you'll see some disconnect.
Again, my recommendation would be to just parse manually through the file and work with the Where-Object cmdlet at this point.
This shoul work:
get-childitem 'c:\logs' | where {$_.Name -notmatch "X|Zr" } | sort {$_.lastwritetime} | select -last 1 | select-string "Failed to Modify"
But I don't like "X|Zr" part. If your log files have .txt extension, it'll not list them because you're saying you don't want any file containing "x" or "zr" in entire name. Use $_.BaseName (name without extension), or modify regular expression.
My Powershell script seems slow, when I run the below code in ISE, it keeps running, doesn't stop.
I am trying to write the list of subfolders in a folder(the folder path is in $scratchpart) to a text file. There are >30k subfolders
$limit = (Get-Date).AddDays(-15)
$path = "E:\Data\PathToScratch.txt"
$scratchpath = Get-Content $path -TotalCount 1
Get-ChildItem -Path $scratchpath -Recurse -Force | Where-Object { $_.PSIsContainer -and $_.CreationTime -lt $limit } | Add-Content C:\Data\eProposal\POC\ScratchContents.txt
Let me know if my approach is not optimal. Ultimately, I will read the text file, zip the subfolders for archival and delete them.
Thanks for your help in advance. I am new to PS, watched few videos on MVA
Add-Content, Set-Content, and even Out-File are notoriously slow in PowerShell. This is because each call opens the file, writes to it, and closes the handle. It never does anything more intelligently than that.
That doesn't sound bad until you consider how pipelines work with Get-ChildItem (and Where-Object and Select-Object). It doesn't wait until it's completed before it begins passing objects into the pipeline. It starts passes objects as soon as the provider returns them. For a large result set, this means that the objects are still feeding in the pipeline long after several have finished processing. Generally speaking, this is great! It means the system will function more efficiently, and it's why stuff like this:
$x = Get-ChildItem;
$x | ForEach-Object { [...] };
Is significantly slower than stuff like this:
Get-ChildItem | ForEach-Object { [...] };
And it's why stuff like this appears to stall:
Get-ChildItem | Sort-Object Name | ForEach-Object { [...] };
The Sort-Object cmdlet needs to waits until it's received all pipeline objects before it sorts. It kind of has to to be able to sort. The sort itself is nearly instantaneous; it's just the cmdlet waiting until it has the full results.
The issue with Add-Content is that, well, it experiences the pipeline not as, "Here's a giant string to write once," but instead as, "Here's a string to write. Here's a string to write. Here's a string to write. Here's a string to write." You'll be sending content to Add-Content here line by line. Each line will instantiate a new call to Add-Content, requiring the file to open, write, and close. You'll likely see better performance if you assign the result of Get-ChildItem [...] | Where-Object [...] to a variable, and then write the entire variable to the file at once:
$limit = (Get-Date).AddDays(-15);
$path = "E:\Data\PathToScratch.txt";
$scratchpath = Get-Content $path -TotalCount 1;
$Results = Get-ChildItem -Path $scratchpath -Recurse -Force -Directory | `
Where-Object{$_.CreationTime -lt $limit } | `
Select-Object -ExpandPropery FullName;
Add-Content C:\Data\eProposal\POC\ScratchContents.txt -Value $Results;
However, you might be concerned about memory usage if your results are actually going to be extremely large. You can actually use System.IO.StreamWriter for this purpose, too. My process improved in speed by nearly two orders of magnitude (from 12 hours to 20 minutes) by switching to StreamWriter and also only calling StreamWriter when I had about 250 lines to write (that seemed to be the break-even point for StreamWriter's overhead). But I was parsing all ACLs for user home and group shares for about 10,000 users and nearly 10 TB of data. Your task might not be as large.
Here's a good blog explaining the issue.
Do you have at least PowerShell 3.0? If you do you should be able to reduce the time by filtering out the files since you are returning those as well.
Get-ChildItem -Path $scratchpath -Recurse -Force -Directory | ...
Currently you are returning all files and folders then filtering out the files with $_.PSIsContainer which would be slower. So should end up with something like this
Get-ChildItem -Path $scratchpath -Recurse -Force -Directory |
Where-Object{$_.CreationTime -lt $limit } |
Select-Object -ExpandPropery FullName |
Add-Content C:\Data\eProposal\POC\ScratchContents.txt
Can someone tell me why this script won't work?
Get-ChildItem "\\fhnsrv01\home\aborgetti\Documentation\Stage" -Filter *.EDIPROD | `
Foreach-Object{
$content = Get-Content $_.FullName
#filter and save content to a new file
$content | Where-Object {$_ -match 'T042456'} | Rename-Item `
($_.BaseName+'_834.txt')
I found this syntax from another question on here and changed the environment variables.
For some reason it won't change the name of the file. The filename is
'AIDOCCAI.D051414.T042456.MO.EDIPROD'
Help much appreciated.
UPDATE
Thanks to TheMadTechnician I was able to get some working stuff. Great stuff actually. Figure I should share with the world!
#Call Bluezone to do file transfer
#start-process "\\fhnsrv01\home\aborgetti\Documentation\Projects\Automation\OpenBZ.bat"
#Variable Declarations
$a = Get-Date
$b = $a.ToString('MMddyy')
$source = "\\fhnsrv01\home\aborgetti\Documentation\Stage\"
$dest = "\\fhnsrv01\home\aborgetti\Documentation\Stage\orig"
#Find all the files that have EDIPROD extension and proceed to process them
#First copy the original file to the orig folder before any manipulation takes place
Copy-item $source\*.EDIPROD $dest
# Now we must rename the items that are in the table
Switch(GCI \\fhnsrv01\home\aborgetti\Documentation\Stage\*.EDIPROD){
{(GC $_|Select -first 1) -match "834*"}{$_ | Rename-Item -NewName {$_.BaseName+'_834.txt'}}
{(GC $_|Select -first 1) -match "820*"}{$_ | Rename-Item -NewName {$_.BaseName+'_820.txt'}}
}
Get-ChildItem's -Filter has issues, I really hesitate to use it in general. If it were up to me I'd do something like this:
Get-ChildItem "\\fhnsrv01\home\aborgetti\Documentation\Stage" |
?{$_.Extension -match ".EDIPROD" -and $_.name -match "T042456"}|
%{$_.MoveTo($_.FullName+"_834.txt")}
Well, I would put it all on one line, but you can line break after the pipe and it does make it a little easier to read, so there you have it. I'm rambling, sorry.
Edit: Wow, I didn't even address what was wrong with your script. Sorry, kind of distracted at the end of my work day here. So, why doesn't your script work? Here's why:
You pull a file and folder listing for the chosen path. That's great, it should work, more or less, I have almost no faith in the -Filter capabilities of the file system provider, but anyway, moving on!
You take that list and run it through a ForEach loop processing each file that matches your filter as such:
You read the contents of the file, and store them in the variable $content
You run the contents of the file, line by line, there a Where filter looking for the text "T042456"
For each line that matches that text you attempt to rename something to that line's basename plus _834.txt (the line of text is a string, it doesn't have a basename property, and it's not an object that can be renamed, so this is going to fail)
So, that's where the issue is. You're pulling the contents of the file, and parsing that line by line trying to match the text instead of matching against the file name. If you removed Everything after the first pipe up to the Where statement, and then for your rename-item put -newname before your desired name, and change the ( ) to { } that goes around the new name, and you would be set. Your code would work. So, your code, modified as I said, would look like:
Get-ChildItem "\\fhnsrv01\home\aborgetti\Documentation\Stage" -Filter *.EDIPROD |
Where-Object {$_ -match 'T042456'} | Rename-Item -NewName {$_.BaseName+'_834.txt'}
Though I have a feeling you want $.Name and not $.BaseName. Using $_.BaseName will leave you with (to use your example file name):
'AIDOCCAI.D051414.T042456.MO_834.txt`
Edit2: Really that's a whole different question, how to match multiple criteria, but the question is here, I'm here, why not just get it done?
So, you have multiple criteria for matching the file names. That really doesn't affect your loop to be honest, what it does affect is the Where statement. If you have multiple options what you probably want is a RegEx match. Totally doable! I'm only going to address the Where statement (?{ }) here, this won't change anything else in the script.
We leave the extension part, but we're going to need to modify the file name part. With RegEx you can match against alternative text by putting it in parenthesis and splitting up the various options with a pipe character. So it would look something like this:
"(T042456|T195917|T048585)"
Now we can incorporate that into the rest of the Where statement and it looks like this:
?{$_.Extension -match ".EDIPROD" -and $_.name -match "(T042456|T195917|T048585)"}
or in your script:
Where-Object {$_ -match "(T042456|T195917|T048585)"}
Edit3: Hm, need the first line for the qualifier. That complicates things a bit. Ok, so what I'm thinking is to get our directory listing, get the first line of each file with the desired extension, make an object that has two properties, the first property is the fileinfo object for the file, and the other property will be the first line of the file. Wait, I think we can do better. Switch (GCI *.EDIPROD){(get-content|select -first 1) -match 820}{Rename 820};{blah blah -match 834}{rename 834}}. Yeah, that should work. Ok, actual script, not theoretical gibberish script time. This way if you have other things to look for you can just add lines for them.
Switch(GCI \\fhnsrv01\home\aborgetti\Documentation\Stage\*.EDIPROD){
{(GC $_|Select -first 1).substring(177) -match "^834"}{$_ | Rename-Item -NewName {"834Dailyin$b"};Continue}
{(GC $_|Select -first 1).substring(177) -match "^820"}{$_ | Rename-Item -NewName {$_.BaseName+'_820.txt'};Continue}
}
Again, if you want the EDIPROD part to remain in the file name change $_.BaseName to $_.Name. Switch is pretty awesome if you're trying to match against different things and perform different actions depending on what the results are. If you aren't familiar with it you may want to go flex your google muscles and check it out.
Hm, alternatively we could have gotten the first line inside the Where filter, run a regex match against that, and renamed the file based on the regex match.
GCI \\fhnsrv01\home\aborgetti\Documentation\Stage\*.EDIPROD | ?{(GC $_ | Select -First 1) -match "(820|834)"}|Rename-Item -NewName {$_.Name+"_"+$Matches[1]+".txt"}
Then you just have to update the Where statement to include anything you're trying to match against. That's kind of sexy, though not as versatile as the switch. But for just simple search and rename it works fine.
Try it like this way
Get-ChildItem -Filter "*T042456*" -Recurse | % {Rename-Item $_ "$_ _834.txt"}