I'm working on a script that takes a directory name as input, and writes a log file to that directory.
This works in most cases:
param([string]$output_folder)
$logfile="$output_folder" + "\myscript.log"
Write-Output "logfile: " $logfile
(Get-Date -Format "[yyyy-MM-dd HH:mm:ss]") | Out-file $logfile -Append
except when the file path contains a wildcard:
E:\Publishing\User manual [en]\
This results in an error 'Cannot perform operation because the wildcard path E:\Publishing\User manual [en]\myscript.log did not resolve to a file.
I spent some time trying to sanitize the input, but I haven't found a way that works yet.
$logfile='$output_folder' + "\myscript.log"
This treats the string as a literal, but that prevents $output_folder from being expanded.
param([io.directoryinfo]$output_folder)
doesn't help either.
Edit:
A comment suggested this:
Out-file -LiteralPath $logfile -Append
That works for the instances where I use Out-file.
I also call an external program, and I can't use -LiteralPath here:
"C:\Program Files\Saxonica\SaxonHE9.8N\bin\Transform.exe" -s:"$inputFile" -xsl:"$output_folder\$transformation_filename" -o:"$outputFile" -t 1>>$logfile 2>&1
in fact I have more arguments here that need to be treated as literals. Is there a way so specify this when I create the string?
$logfile="$output_folder" + "\myscript.log"
Related
I've got a Powershell script that takes a filename as input.
param([string]$folder, [string]$input_filename)
$logfile=$folder + "\duplicate.log"
Write-Output "`nScript: `n$transformation_filename`nLogfile: " | Out-file $logfile -Append
This works fine for most filenames, but it breaks down when the filename or folder name contains [].
dupfile.ps1 "C:\foldername [w]" "filename.xml"
Line 3 throws this error:
Cannot perform operation because the wildcard path C:foldername [w]\duplicate.log did not resolve to a file.
So the variables have to be treated more literally.
$logfile='$folder' + "\duplicate.log"
Nope, that's too literal.
$logfile="$folder" + "\duplicate.log"
Nope, same error.
How can I tell Powershell to insert the variable value but don't try to interpret anything in that value as wildcards?
Edit: I'm using Powershell 5.
My dupfile.ps1 contains only the 3 lines above.
Using this from the PS shell to run the script:
C:_SCRIPTS_\dupfile.ps1 "C:\foldername [w]" "test.txt"
One more addition, if I may. I have a few lines that redirect their error messages to the logfile.
Copy-Item -LiteralPath $inputPath -Destination $outputPath 2>&1|%{ "$_" } >>$logfile
The redirection trips over the [] again. I don't think I can use -LiteralPath here.
tl;dr
To ensure that file-path argument $var in redirection operations > $var / >> $var is treated literally if it contains [ and ], use
| Out-File -LiteralPath $var / | Out-File -LiteralPath $var -Append instead.
Copy-Item -LiteralPath $inputPath -Destination $outputPath 2>&1 |
ForEach-Object { "$_" } |
Out-File -LiteralPath $logfile # literal alternative to >> $logfile
As in your case, you may have to combine it with a redirection in order to ensure that additional streams beside the success output stream are targeted:
# Literal equivalent of *>> $logfile
... *>&1 | Out-File -LiteralPath $logfile -Append
Unfortunately, it gets tricky with capturing a given other stream only, as you then need to use the appropriate -*Variable common parameter, such as the common -ErrorVariable parameter:
# Literal equivalent of 2>> $logfile
... -ErrorVariable errs
$errs | Out-File -LiteralPath $logfile -Append
Caveat:
It is tempting to try to bypass the above solutions by escaping the literal path so that when it is interpreted as a wildcard, it is treated literally, using [WildcardPattern]::Escape().
However, this does not work as expected as of PowerShell 7.2.6, because the escaped form of the literal path is then used as the literal file path - see GitHub issue #9475 and also the discussion about whether > / >>, ... should treat their arguments as wildcards to begin with.
# !! Arguably SHOULD work, but doesn't as of 7.2.6:
# The file that is created is LITERALLY named for the ESCAPED form:
# `[1`].txt
'hi' > ([WildcardPattern]::Escape('[1].txt'))
Background information:
The -Path (-FilePath) parameter of PowerShell's provider cmdlets expects wildcard expressions for targeting file-system items by a name or path pattern. This parameter is implicitly targeted by the first positional argument, i.e. if you don't explicitly name the target parameter - e.g., Get-ChildItem foo is implicitly the same as Get-ChildItem -Path foo.
Surprisingly, this doesn't just apply to file-reading cmdlets (e.g., Get-ChildItem, Get-Content), but also to file-writing cmdlets (e.g., Set-Content, Out-File).
Note that the parameter named -Path is called -FilePath in Out-File for historical reasons. In PowerShell (Core) 7+, -Path was introduced as an alias name, for consistency.
Arguably, in file-writing/creating cmdlets this behavior is inappropriate - given that you usually know exactly what literal path you want to target - and there are several GitHub issues discussing this; an overview of all relevant discussions can be found in this comment on GitHub issue #17106.
In order for such arguments to be treated literally (verbatim), you must
use the -LiteralPath parameter instead.
In effect, > $var and >> $var are aliases for | Out-File -FilePath $var and | Out-File -FilePath $var -Append, which implies that $var is interpreted as a wildcard.
Using explicit Out-File calls with -LiteralPath, as shown at the top, avoids this problem.
I have a fairly simple PS script that was working perfectly, and now has suddenly started giving errors. I have narrowed the problem portion to a couple of Get-Content statements. Here's what the affected part of the script looks like:
$pathSource = "D:\FileDirectory"
Set-Location -Path $pathSource
Get-Content -Encoding UTF8 -Path FilesA*.txt | Out-File -Encoding ASCII FilesA_Digest.txt
Get-Content -Encoding UTF8 -Path FilesB*.txt | Out-File -Encoding ASCII FilesB_Digest.txt
This part of the script gathers up a collection of like-named files and concatenates them into a single text file for uploading to an FTP site. The Get-Content/Out-File was needed as the original files are encoded incorrectly for the FTP site. The script was working perfectly, running once each night for several weeks. Now, it gets the following error when the Get-Content statements are reached:
Get-Content : A parameter cannot be found that matches parameter name 'Encoding'.
At D:\FileDirectory\Script.ps1
Environment is Windows Server 2016. I've tried different variations on the Get-Content parameters, but nothing has worked. I know there is a bug that affects network-mapped drives, but that's not the case here -- all files are local.
Any ideas/suggestions?
The only plausible explanation I can think of is that a custom Get-Content command that lacks an -Encoding parameter is shadowing (overriding) the standard Get-Content cmdlet in the PowerShell session that's executing your script.
To demonstrate:
# Define a custom Get-Content command (function) that accepts only
# a (positional) -Path parameter, not also -Encoding.
function Get-Content { [CmdletBinding()] param([string] $Path) }
# Now try to use Get-Content -Encoding
Get-Content -Encoding Utf8 FilesA*.txt
You'll see the same error message as in your question.
Use Get-Command Get-Content -All to see all commands named Get-Content, with the effective command listed first.
Then examine where any custom commands may come from; e.g., your $PROFILE script may contain one.
To rule out $PROFILE as the culprit, start PowerShell without loading the profile script and examine Get-Content then:
powershell -noprofile # Windows PowerShell
pwsh -noprofile # PowerShell Core
A simple way to rule out custom overrides ad hoc is to call a command by its module-qualified name:
Microsoft.Powershell.Management\Get-Content ...
You can determine a built-in cmdlet's module name of origin as follows:
PS> (Get-Command Get-Content -All)[-1].ModuleName
Microsoft.PowerShell.Management
In a pinch you can also infer the originating module name from the URL of the help topic:
Googling Get-Content will take you to https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/get-content - note how the cmdlet's module name, microsoft.powershell.management (case doesn't matter), is the penultimate (next to last) URI component.
It seems an issue with the out command. Can you please try below code :
$pathSource = "D:\FileDirectory"
Set-Location -Path $pathSource
Get-Content -Encoding UTF8 -Path FilesA*.txt | Set-Content -Encoding ASCII -path FilesA_Digest.txt
Get-Content -Encoding UTF8 -Path FilesB*.txt | Set-Content -Encoding ASCII -path FilesB_Digest.txt
Well, I don't know why it failed, but I can say that I have completely re-written the script and now it works. I have to note that, given the errors that were occurring, I also don't know why it is now working.
I am using the exact same calls to the Get-Content commandlet, with the -Encoding parameter and the pipe to Out-File with its own -Encoding parameter. I am doing the exact same actions as the previous version of the script. The only part that is significantly different is the portion that performs the FTP transfer of the processed files. I'm now using only PowerShell to perform the transfer rather than CuteFTP and it all seems to be working correctly.
Thanks to everyone who contributed.
Cheers
Norm
Not sure if it helps, but I was running into the same with:
$n = ni '[hi]' -value 'some text'
gc $n -Encoding Byte
$f = ls *hi*
$f.where{$_.name -eq '[hi]'}.Delete()
also looks like there's already a chain of SOs about this known bug see this answer
I am trying to pass a .txt file with arguments to an .exe file via powershell. Currently, this is what I have.
Write-Host "starting upgrade at $(Get-Date -format 'U')"
C:\dev\temp.exe.exe /DIR="C:\TEST" /BTPServerHost="Test" /DBInstance="testDB" /Log=C:\path\to\test\testlog.txt
This is calling a function within an InnoScript file that accepts command line input.
How would I format the .txt file, and how would I be able to pass it into the .exe? Any help would be appreciated! Thanks!
If you are saying, in this text file, there are just these argument line on individual rows and you are saying you've already tried something like the below and were not successful?
You also don't need the Write-Host for the message line, since the default is output to screen. You normal only need Write-Host for colorizing screen text, and a few other formatting cases, depending on what you are doing. All-in-All, Write-Host should be avoided.
"starting upgrade at $(Get-Date -format 'U')"
($ConsoleCommand = Get-Content -Path 'd:\temp\input.txt' -Raw)
# Results - showing the commands in the file before process them
whoami
get-date
'hello world'
Without using the -Wait switch, this will spawn 3 separate PowerShell consoles with the results
ForEach($CmdLine in $ConsoleCommand)
{ Start-Process -FilePath powershell -ArgumentList "-NoExit","-Command &{ $CmdLine }" }
you can of course point to your .exe vs what I am doing here.
Start-Process
By adding the -Raw after specifying the .txt file path it ignores newline characters and returns the entire contents of a file in one string with the newlines preserved. By default, newline characters in a file are used as delimiters to separate the input into an array of strings.
This script takes parameters from a txt file and passes them into an executable and auto-populates the fields in the installation wizard. This is what I'm looking to do, but I don't want to start a new process for each argument in the input txt file.
Write-Host "starting upgrade at $(Get-Date -format 'U')" Get-Content -Path C:\TestInput.txt -Raw | foreach {Start-Process C:\dev\test.exe -ArgumentList $_}
The TestInput.txt file passed in looks like this:
/DIR="C:\TEST"
/ServerHost="Test"
/DBInstance="testDB"
/Log=C:\testlog.txt
I have a simple PowerShell script that replaces "false" or "true" with "0" or "1":
$InputFolder = $args[0];
if($InputFolder.Length -lt 3)
{
Write-Host "Enter a path name as your first argument" -foregroundcolor Red
return
}
if(-not (Test-Path $InputFolder)) {
Write-Host "File path does not appear to be valid" -foregroundcolor Red
return
}
Get-ChildItem $InputFolder
$content = [System.IO.File]::ReadAllText($InputFolder).Replace("`"false`"", "`"0`"").Replace("`"true`"", "`"1`"").Replace("`"FALSE`"", "`"0`"").Replace("`"TRUE`"", "`"1`"")
[System.IO.File]::WriteAllText($InputFolder, $content)
[GC]::Collect()
This works fine for almost all files I have to amend, with the exception of one 808MB CSV.
I have no idea how many lines are in this CSV, as nothing I have will open it properly.
Interestingly, the PowerShell script will complete successfully when invoked manually via either PowerShell directly or via command prompt.
When this is launched as part of the SSIS package it's required for, that's when the error happens.
Sample data for the file:
"RowIdentifier","DateProfileCreated","IdProfileCreatedBy","IDStaffMemberProfileRole","StaffRole","DateEmploymentStart","DateEmploymentEnd","PPAID","GPLocalCode","IDStaffMember","IDOrganisation","GmpID","RemovedData"
"134","09/07/1999 00:00","-1","98","GP Partner","09/07/1999 00:00","14/08/2009 15:29","341159","BRA 871","141","B83067","G3411591","0"
Error message thrown:
I'm not tied to PowerShell - I'm open to other options. I had a cribbed together C# script previously, but that died on small files than this - I'm no C# developer, so was unable to debug it at all.
Any suggestions or help gratefully received.
Generally, avoiding read large files all at once, as you can run out of memory, as you've experienced.
Instead, process text-based files line by line - both reading and writing.
While PowerShell generally excels at line-by-line (object-by-object) processing, it it is slow with files with many lines.
Using the .NET Framework directly - while more complex - offers much better performance.
If you process the input file line by line, you cannot directly write back to it and must instead write to a temporary output file, which you can replace the input file with on success.
Here's a solution that uses .NET types directly for performance reasons:
# Be sure to use a *full* path, because .NET typically doesn't have the same working dir. as PS.
$inFile = Convert-Path $Args[0]
$tmpOutFile = [io.path]::GetTempFileName()
$tmpOutFileWriter = [IO.File]::CreateText($tmpOutFile)
foreach ($line in [IO.File]::ReadLines($inFile)) {
$tmpOutFileWriter.WriteLine(
$line.Replace('"false"', '"0"').Replace('"true"', '"1"').Replace('"FALSE"', '"0"').Replace('"TRUE"', '"1"')
)
}
$tmpOutFileWriter.Dispose()
# Replace the input file with the temporary file.
# !! BE SURE TO MAKE A BACKUP COPY FIRST.
# -WhatIf *previews* the move operation; remove it to perform the actual move.
Move-Item -Force -LiteralPath $tmpOutFile $inFile -WhatIf
Note:
UTF-8 encoding is assumed, and the rewritten file will not have a BOM. You can change this by specifying the desired encoding to the .NET methods.
As an aside: Your chain of .Replace() calls on each input line can be simplified as follows, using PowerShell's -replace operator, which is case-insensitive, so only 2 replacements are needed:
$line -replace '"false"', '"0"' -replace '"true"', '"1"'
However, while that is shorter to write, it is actually slower than the .Replace() call chain, presumably because -replace is regex-based, which incurs extra processing.
You could read the file Per line with get-content -readcount, Out-file a temp file, then delete old file and rename-item the temp file the old files name.
Small things that would need fixing. This will add a new empty line at end of file. This will change the encoding. You could try and get the current file encoding and set the encoding on the Out-file -encoding
function Replace-LargeFilesInFolder(){
Param(
[string]$DirectoryPath,
[string]$OldString,
[string]$NewString,
[string]$TempExtention = "temp",
[int]$LinesPerRead = 500
)
Get-ChildItem $DirectoryPath -File | %{
$File = $_
Get-Content $_.FullName -ReadCount $LinesPerRead |
%{
$_ -replace $OldString, $NewString |
out-file "$($File.FullName).$($TempExtention)" -Append
}
Remove-Item $File.FullName
Rename-Item "$($File.FullName).$($TempExtention)" -NewName $($File.FullName)
}
}
Replace-LargeFilesInFolder -DirectoryPath C:\TEST -LinesPerRead 1 -OldString "a" -NewString "5"
I am trying to replicate the functionality of the cat command in Unix.
I would like to avoid solutions where I explicitly read both files into variables, concatenate the variables together, and then write out the concatenated variable.
Simply use the Get-Content and Set-Content cmdlets:
Get-Content inputFile1.txt, inputFile2.txt | Set-Content joinedFile.txt
You can concatenate more than two files with this style, too.
If the source files are named similarly, you can use wildcards:
Get-Content inputFile*.txt | Set-Content joinedFile.txt
Note 1: PowerShell 5 and older versions allowed this to be done more concisely using the aliases cat and sc for Get-Content and Set-Content respectively. However, these aliases are problematic because cat is a system command in *nix systems, and sc is a system command in Windows systems - therefore using them is not recommended, and in fact sc is no longer even defined as of PowerShell Core (v7). The PowerShell team recommends against using aliases in general.
Note 2: Be careful with wildcards - if you try to output to inputFiles.txt (or similar that matches the pattern), PowerShell will get into an infinite loop! (I just tested this.)
Note 3: Outputting to a file with > does not preserve character encoding! This is why using Set-Content is recommended.
Do not use >; it messes up the character encoding. Use:
Get-Content files.* | Set-Content newfile.file
In cmd, you can do this:
copy one.txt+two.txt+three.txt four.txt
In PowerShell this would be:
cmd /c copy one.txt+two.txt+three.txt four.txt
While the PowerShell way would be to use gc, the above will be pretty fast, especially for large files. And it can be used on on non-ASCII files too using the /B switch.
You could use the Add-Content cmdlet. Maybe it is a little faster than the other solutions, because I don't retrieve the content of the first file.
gc .\file2.txt| Add-Content -Path .\file1.txt
To concat files in command prompt it would be
type file1.txt file2.txt file3.txt > files.txt
PowerShell converts the type command to Get-Content, which means you will get an error when using the type command in PowerShell because the Get-Content command requires a comma separating the files. The same command in PowerShell would be
Get-Content file1.txt,file2.txt,file3.txt | Set-Content files.txt
I used:
Get-Content c:\FileToAppend_*.log | Out-File -FilePath C:\DestinationFile.log
-Encoding ASCII -Append
This appended fine. I added the ASCII encoding to remove the nul characters Notepad++ was showing without the explicit encoding.
If you need to order the files by specific parameter (e.g. date time):
gci *.log | sort LastWriteTime | % {$(Get-Content $_)} | Set-Content result.log
You can do something like:
get-content input_file1 > output_file
get-content input_file2 >> output_file
Where > is an alias for "out-file", and >> is an alias for "out-file -append".
Since most of the other replies often get the formatting wrong (due to the piping), the safest thing to do is as follows:
add-content $YourMasterFile -value (get-content $SomeAdditionalFile)
I know you wanted to avoid reading the content of $SomeAdditionalFile into a variable, but in order to save for example your newline formatting i do not think there is proper way to do it without.
A workaround would be to loop through your $SomeAdditionalFile line by line and piping that into your $YourMasterFile. However this is overly resource intensive.
To keep encoding and line endings:
Get-Content files.* -Raw | Set-Content newfile.file -NoNewline
Note: AFAIR, whose parameters aren't supported by old Powershells (<3? <4?)
I think the "powershell way" could be :
set-content destination.log -value (get-content c:\FileToAppend_*.log )