I have a list of EDI text files with specific text in them. Currently in order for our custom scripting to convert them into an SQL table, we need to be able to see the X12 file type in the filename. Because we are using SQL script to get the files into tables this solution needs to be a one line solution. We have a definition table of client files which specify which field terminator and file types to look for so we will be later substitute those values into the one line solution to be executed individually. I am currently looking at Powershell (v.3) to do this for maximum present and future compatibility. Also, I am totally new to Powershell, and have based my script generation on posts in this forum.
Files example
t.text.oxf.20170815123456.out
t.text.oxf.20170815234567.out
t.text.oxf.20170815345678.out
t.text.oxf.20170815456789.out
Search strings to find within files: (To find EDI X12 file type uniquely, which may be duplicated within the same file n times)
ST*867
ST*846
ST~867
ST~846
ST|867
ST|846
Here is what I have so far which does not show itself doing anything with the whatif parameter:
(Get-ChildItem .\ -recurse | Select-String -pattern 'ST~867' -SimpleMatch).Path | Foreach -Begin {$i=1} -Process {Rename-Item -LiteralPath $_ -NewName ($_ -replace 'out$','867.out' -f $i++) -whatif}
The fist part:
(Get-ChildItem .\ -recurse | Select-String -pattern 'ST~867' -SimpleMatch).Path
Simply gets a list of paths that we need to input to be renamed
The second part after the | pipe:
Foreach -Begin {$i=1} -Process {Rename-Item -LiteralPath $_ -NewName ($_ -replace '\.out','.867.out' -f $i++) -whatif}
will supposedly loop through that list and rename the files adding the EDI type to the end of the file. I have tried 'out$','867.out' with no change.
Current Errors:
The first part shows duplicated path elements probably because there are multiple Transaction Set Headers in the files, is there any way to force it to be unique?
The command does not show any Errors (red text) but with the whatif parameter shows that it does not rename any files (tried running it without as well).
1) remove duplicates using -List switch in Select-String
2) you need to really pipe the objects into the for loop
Try this?
Select-String -Path .\*.out -pattern 'ST~867' -SimpleMatch -List | Select-Object Path | ForEach-Object { Rename-Item $_.path ($_.path -replace 'out$','867.out') }
Related
I am searching for all lines with '.png' and '.jpg' strings in them across multiple folders of TXT files.
Tried:
(Get-ChildItem K:\FILES -Recurse -Include '*.txt') | ForEach-Object {
(Get-Content $_) -match '\.png','\.jpg' | out-file K:\Output.txt
}
but it does not output anything. No error either. I did something similar recently and it was working. I am scratching my head wondering what am I doing wrong here...
By placing your Out-File call inside the ForEach-Object script block, you're rewriting your output file in full for every input file, so that the last input file's results - which may be none - end up as the sole content of the file.
The immediate fix is to move the Out-File call to its own pipeline segment, so that it receives all output, across all files:
Get-ChildItem K:\FILES -Recurse -Include '*.txt' |
ForEach-Object {
#(Get-Content $_) -match '\.png', '\.jpg'
} |
Out-File K:\Output.txt
Note: Technically, adding -Append to your Out-File call inside the ForEach-Object could have worked too, but this approach should be avoided:
Every Out-File call must open and close the output file, which makes the operation much slower.
You need to ensure that there is no preexisting output file beforehand - otherwise you'll end up appending to that file's existing content.
However, consider speeding up your command with the help of Select-String:
Get-ChildItem K:\FILES -Recurse -Include '*.txt' |
Select-String -Pattern '\.png', '\.jpg' |
ForEach-Object Line |
Out-File K:\Output.txt
Note:
In PowerShell (Core) 7+, you can use the -Raw switch with Select-String, which directly outputs only the text of all matching lines, in which case ForEach-Object Line isn't needed.
If you want to prefix each matching line with the source file path:
Get-ChildItem K:\FILES -Recurse -Include '*.txt' |
Select-String -Pattern '\.png', '\.jpg' |
ForEach-Object { '{0}: {1}' -f $_.Path, $_.Line } |
Out-File K:\Output.txt
Note: If you pipe Select-String output directly (without -Raw or ForEach-Object Line) to Out-File (or if you use >), you'll get similar output (even including a character position), but with limitations:
You'll get a blank line at the top and the bottom of the file.
Long line texts may be truncated.
The reason is that Out-File and its virtual alias > send the for-display representations of the input objects to the output file, which aren't meant for programmatic processing and can incur truncation of the data based on the line length (number of columns) of the current console window.
I'm working on a script to output some data from multiple files based on a string search. It outputs the string found, followed by the following six characters. I can get this to work for an exact location. However, I want to search across files inside multiple subfolders in the path. Using the below script, I get PermissionDenied errors...
[regex] $pattern = '(?<=(a piece of text))(?<chunk>.*)'
Get-Content -Path 'C:\Temp\*' |
ForEach-Object {
if ($_ -match $pattern) {
$smallchunk = $matches.chunk.substring(0, 6)
}
}
"$smallchunk" | Out-File 'C:\Temp\results.txt'
If I change -Path to one of the subfolders, it works fine, but I need it to go inside each subfolder and execute the get-content.
e.g., look inside...
C:\Temp\folder1\*
C:\Temp\folder2\*
C:\Temp\folder3\*
And so on...
Following up on boxdog's suggestion of Select-String, the only limitation would be folder recursion. Unfortunately, Select-String only allows the searching of multiple files in one directory.
So, the way around this is piping the output of Get-ChildItem with a -Recurse switch into Select-String:
$pattern = "(?<=(a piece of text))(?<chunk>.*)"
Get-ChildItem -Path "C:\Temp\" -Exclude "results.txt" -File -Recurse |
Select-String -Pattern $pattern |
ForEach-Object -Process {
$_.Matches[0].Groups['chunk'].Value.Substring(0,6)
} | Out-File -FilePath "C:\Temp\results.txt"
If there's a need for the result to be saved to $smallchunk you can still do so inside the loop if need be.
Abraham Zinala's helpful answer is the best solution to your problem, because letting Select-String search your files' content is faster and more memory-efficient than reading and processing each line with Get-Content.
As for what you tried:
Using the below script I get PermissionDenied errors...
These stem from directories being among the file-system items output by Get-ChildItem, which Get-Content cannot read.
If your files have distinct filename extensions that your directories don't, one option is to pass them to the (rarely used with Get-Content) -Include parameter; e.g.:
Get-Content -Path C:\Temp\* -Include *.txt, *.c
However, as with Select-String, this limits you to a single directory's content, and it doesn't allow you to limit processing to files fundamentally, if extension-based filtering isn't possible.
For recursive listing, you can use Get-ChildItem with -Recurse, as in Abraham's answer, and pipe the file-info objects to Get-Content:
Get-ChildItem -Recurse C:\Temp -Include *.txt, *.c | Get-Content
If you want to simply limit output to files, whatever their name is, use the -File switch (similarly, -Directory limits output to directories):
Get-ChildItem -File -Recurse C:\Temp | Get-Content
i am quite new to powershell and i am trying to make a script that copy files to certain folders that are declared in a CSV file. But till now i am getting errors from everywhere and can't find nothing to resolve this issue.
I have this folders and .txt files created in the same folder as the script.
Till now i could only do this:
$files = Import-Csv .\files.csv
$files
foreach ($file in $files) {
$name = $file.name
$final = $file.destination
Copy-Item $name -Destination $final
}
This is my CSV
name;destination
file1.txt;folderX
file2.txt;folderY
file3.txt;folderZ
As the comments indicate, if you are not using default system delimiters, you should make sure to specify them.
I also recommend typically to use quotes for your csv to ensure no problems with accidentally including an entry that includes the delimiter in the name.
#"
"taco1.txt";"C:\temp\taco2;.txt"
"# | ConvertFrom-CSV -Delimiter ';' -Header #('file','destination')
will output
file destination
---- -----------
taco1.txt C:\temp\taco2;.txt
The quotes make sure the values are correctly interpreted. And yes... you can name a file foobar;test..txt. Never underestimate what users might do. 😁
If you take the command Get-ChildItem | Select-Object BaseName,Directory | ConvertTo-CSV -NoTypeInformation and review the output, you should see it quoted like this.
Sourcing Your File List
One last tip. Most of the time I've come across a CSV for file input lists a CSV hasn't been needed. Consider looking at grabbing the files you in your script itself.
For example, if you have a folder and need to filter the list down, you can do this on the fly very easily in PowerShell by using Get-ChildItem.
For example:
$Directory = 'C:\temp'
$Destination = $ENV:TEMP
Get-ChildItem -Path $Directory -Filter *.txt -Recurse | Copy-Item -Destination $Destination
If you need to have more granular matching control, consider using the Where-Object cmdlet and doing something like this:
Get-ChildItem -Path $Directory -Filter *.txt -Recurse | Where-Object Name -match '(taco)|(burrito)' | Copy-Item -Destination $Destination
Often you'll find that you can easily use this type of filtering to keep CSV and input files out of the solution.
example
Using techniques like this, you might be able to get files from 2 directories, filter the match, and copy all in a short statement like this:
Get-ChildItem -Path 'C:\temp' -Filter '*.xlsx' -Recurse | Where-Object Name -match 'taco' | Copy-Item -Destination $ENV:TEMP -Verbose
Hope that gives you some other ideas! Welcome to Stack Overflow. 👋
I've researched this issue and have perused the knowledge base here. I did find a few topics that dabbled in removing certain characters in a file/folder name via PowerShell. Unfortunately, trying to do the below cmdlet resulted in the following error.
cmd: Get-Item * | ForEach-Object {Rename-Item $_ ($_.Name -replace "%", "")}
error: rename-item: Source and destination path must be different
There are many thousands of files. I'm trying to remove % and # symbols from any and all filenames within many subfolders.
Example: Before
FileName%1%#1.doc
after
FileName11.doc
Don't use cmd but powershell. Here is the correct syntax from your example:)
Get-Item * | ForEach-Object {Rename-Item $_ -NewName ($_.name -replace '#','' -replace '%','')}
How can I delete all files in a directory that contain a string using powershell?
I've tried something like
$list = get-childitem *.milk | select-string -pattern "fRating=2" | Format-Table Path
$list | foreach { rm $_.Path }
And that worked for some files but did not remove everything. I've tried other various things but nothing is working.
I can easily get the list of file names and can create an array with the path's only using
$lista = #(); foreach ($f in $list) { $lista += $f.Path; }
but can't seem to get any command (del, rm, or Remove-Item) to do anything. Just returns immediately without deleting the files or giving errors.
Thanks
First we can simplify your code as:
Get-ChildItem "*.milk" | Select-String -Pattern "fRating=2" | Select-Object -ExcludeProperty path | Remove-Item -Force -Confirm
The lack of action and errors might be addressable by one of two things. The Force parameter which:
Allows the cmdlet to remove items that cannot otherwise be changed,
such as hidden or read-only files or read-only aliases or variables.
I would aslo suggest that you run this script as administrator. Depending where these files are located you might not have permissions. If this is not the case or does not work please include the error you are getting.
Im going to guess the error is:
remove-item : Cannot remove item C:\temp\somefile.txt: The process cannot access the file 'C:\temp\somefile.txt'
because it is being used by another process.
Update
In testing, I was also getting a similar error. Upon research it looks like the Select-String cmd-let was holding onto the file preventing its deletion. Assumption based on i have never seen Get-ChildItem do this before. The solution in that case would be encase the first part of this in parentheses as a sub expression so it would process all the files before going through the pipe.
(Get-ChildItem | Select-String -Pattern "tes" | Select-Object -ExpandProperty path) | Remove-Item -Force -Confirm
Remove -Confirm if deemed required. It exists as a precaution so that you don't open up a new powershell in c:\windows\system32 and copy paste a remove-item cmdlet in there.
Another Update
[ and ] are wildcard searches in powershell in order to escape those in some cmdlets you use -Literalpath. Also Select-String can return multiple hits in files so we should use -Unique
(Get-ChildItem *.milk | Select-String -Pattern "fRating=2" | Select-Object -ExpandProperty path -Unique) | ForEach-Object{Remove-Item -Force -LiteralPath $_}
Why do you use select-string -pattern "fRating=2"? You would like to select all files with this name?
I think the Format-Table Path don't work. The command Get-ChildItem don't have a property called "Path".
Work this snipped for you?
$list = get-childitem *.milk | Where-Object -FilterScript {$_.Name -match "fRating=2"}
$list | foreach { rm $_.FullName }
The following code gets all files of type *.milk and puts them in $listA, then uses that list to get all the files that contain the string fRating=[01] and stores them in $listB. The files in $listB are deleted and then the number of files deleted versus the number of files that contained the match is displayed(they should be equal).
sv -name listA -value (Get-ChildItem *.milk); sv -name listB -value ($listA | Select-String -Pattern "fRating=[01]"); (($listB | Select-Object -ExpandProperty path) | ForEach-Object {Remove-Item -Force -LiteralPath $_}); (sv -name FCount -value ((Get-ChildItem *.milk).Count)); Write-Host -NoNewline Files Deleted ($listA.Count - $FCount)/($listB.Count)`n;
No need to complicate things:
1. $sourcePath = "\\path\to\the\file\"
2. Remove-Item "$sourcePath*whatever*"
I tried the answer, unfortunately, errors seems to always come up, however, I managed to create a solution to get this done:
Without using Get-ChilItem; You can use select-string directly to search for files matching a certain string, yes, this will return the filename:count:content ... etc, but, internally these have names that you can chose or omit, the one you need is the "filename" to do this pipe this into "select-object" choosing the "FileName" from the output.
So, to select all *.MSG files that has the pattern of "Subject: Webservices restarted", you can do the following:
Select-String -Path .*.MSG -Pattern 'Subject: WebServices Restarted'
-List | select-object Filename
Also, to remove these files on the fly, you could pip into a ForEach statement with the RM command as follows:
Select-String -Path .*.MSG -Pattern 'Subject: WebServices Restarted'
-List | select-object Filename | foreach { rm $_.FileName }
I tried this myself, works 100%.
I hope this helps