How do I get WSL Vim to accept pipeline input from Powershell? - powershell

I'm trying to create a Powershell function that acts as a wrapper for Vim in WSL. I would like this wrapper to be able to accept input from the pipeline.
In WSL, it's possible to do something like:
$ cat manylines.txt | vim -
The - indicates that Vim should read the input from stdin, which is then redirected through the pipeline. This will open vim with the command output already in the buffer, without any file associated.
(You could also simply open the file with vim manylines.txt, but that wouldn't work in general for the outputs of other commands or executables).
The same is possible in Powershell, with Windows native Vim:
Get-Content manylines.txt | vim -
It's possible to go one step further and execute Vim within another instance of Powershell or cmd:
Get-Content manylines.txt | powershell -Command vim -
Get-Content manylines.txt | cmd /c vim -
However, the same does not work with WSL Vim.
# doesn't work, Vim freezes
Get-Content manylines.txt | wsl vim -
Get-Content manylines.txt | wsl bash -c "vim -"
# doesn't work, Vim quits reporting "Error reading Input"
Get-Content manylines.txt | wsl bash -c vim -
In the first two cases, Vim opens correctly with the contents of manylines.txt in the buffer... and then refuses to accept any keyboard input. It is not possible to navigate, edit, or even quit with :q!.
Using a temporary powershell variable seems to fix things, but only when the input consists of a single line
# "works as long as the file only has a single line of text"
# "$tmpvar has type System.String (not an array)"
$tmpvar = Get-Content singleline.txt
wsl bash -c "echo $tmp | vim -"
But this breaks when $tmpvar becomes a String[] array or a String with any newlines
# sort of works, but newlines are converted to a single space
$tmpvar = [string[]] Get-Content manylines.txt
wsl bash -c "echo $tmp | vim -"
# combines the strings in the array, inserts a newline between each pair of elements
$tmpvar = ([string[]] Get-Content manylines.txt) -join "`n"
# doesn't work, bash interprets the newline as the start of the next command
wsl bash -c "echo $tmp | vim -"
How do I get WSL Vim to accept pipeline input from Powershell?
Other Notes:
I could use a temporary file and pass the file to WSL Vim, but that's not an elegant solution and leaves a file that has to be cleaned up. Unless the temporary file is in the current directory, it also involves extra shenanigans with wslpath

Indeed it seems that piping input to vim via wsl seems to break vim's keyboard interface with respect to cursor movements (I was still able to type :q! to quit, for instance).
The workaround is to refine the approach you've discovered, namely to provide the input as part of a shell command passed to wsl instead of using the pipeline:
Doing so requires careful escaping and transformations, as shown in the following wrapper function:
# PowerShell wrapper function for invoking vim via WSL
function vim {
if ($MyInvocation.ExpectingInput) { # pipeline input present
$allInput = ($input | Out-String) -replace '\r?\n', "`n" -replace "'", "'\''" -replace '"', '\"'
wsl -e sh -c "printf %s '$allInput' | vim $args -"
}
elseif ($args) { # no pipeline input, arguments that may include file paths given
wsl -e vim ($args.ToLower() -replace '^([a-z]):', '/mnt/$1' -replace '\\', '/')
} else { # no pipeline input, no pass-through arguments
wsl -e vim
}
}
Caveats:
When passing command output, you may run into the max. command-line length.
When passing file paths, relative file paths work fine, but absolute ones work only for local drives (mapped drives / UNC paths require mounting a volume on the WSL side first).
With this function defined, you can call, say (+, as a sample option, tells vim to place the cursor at the end of the input):
vim + somefile.txt
or
Get-ChildItem | vim +

Related

Script in powershell not working with array variables

I am creating a PowerShell script just to backup my WSL distros, but when I try to run the command with variables it's not working, it displays the usage text as though I provided the incorrect arguments.
$DistrosArray1 = (wsl -l --quiet) | where {$_ -ne ""}
$DistrosArray2 = 'Arch', 'Ubuntu-22.04', 'docker-desktop-data', 'docker-desktop'
$CheckArrayDifference = Compare-Object -ReferenceObject $DistrosArray1 -DifferenceObject $DistrosArray2 -PassThru
echo $CheckArrayDifference
# Does not return anything (there is no difference)
foreach ($Distro in $DistrosArray1) {
wsl --export $Distro "$Distro.tar"
# This method is not working
}
foreach ($Distro in $DistrosArray2) {
wsl --export $Distro "$Distro.tar"
# This method is working
}
It sounds like you are running into complications from issue #4607 -- The wsl.exe command outputs some oddly mangled UTF16 encoding that creates issues when attempting to process it from PowerShell (or even from inside WSL).
This is now fixed in the latest WSL Preview release 0.64.0, but you do have to "opt-in" to the fix so that older workarounds (like the ones #Bender and I provided) don't inadvertently break.
Simply set:
$env:WSL_UTF8=1
... before your code, and WSL will no longer spit out the "mangled UTF16."
Other examples in my answers to:
Why cannot I match for strings from wsl.exe output?
Powershell - Strange WSL output string encoding
How to ask WSL to check if the distribution exists, using Bash and wsl.exe?
Older solution:
Let's simplify the problem and make a "safe" example that doesn't attempt to export:
$DistrosArray1 = (wsl -l --quiet) | where {$_ -ne ""}
wsl -d $DistrosArray1[0]
Results in:
There is no distribution with the supplied name.
I've successfully used the method in this comment to handle it. For this particular example:
$console = ([console]::OutputEncoding)
[console]::OutputEncoding = New-Object System.Text.UnicodeEncoding
$DistrosArray1 = (wsl -l --quiet) | where {$_ -ne ""}
wsl -d $DistrosArray1[0]
This will run the first distribution in the list correctly.
Reset the encoding after with:
[console]::OutputEncoding = $console
That shouldn't be a problem for most non-interactive scripts since it will just be the final line of the "wrapper", but as #BendertheGreatest pointed out in the comments, it's a critical step.
This is part of a known issue with wsl.exe output. Here is what I put together from workarounds provided on that issue:
$DistrosArray1 = wsl -l --quiet | wsl iconv -c -f utf16 -t ascii
foreach ($Distro in $DistrosArray1) {
wsl --export $Distro "$Distro.tar"
}
Unfortunately I could not get this working with a conversion to UTF8 (changing ascii to utf8 produces additional garbage characters although they are consistent and detectable in my limited testing), so only any characters outside of the ASCII range will likely cause problems for you.

Perl: List files and directories recursively but exclude some directories and files that passed

Please give any suggestion or snippet or anything that may work.
I have already tried wanted function but how do I exclude some directory while recursing?
In Linux, you can make use of the Linux "find" and "grep" commands and run those Linux commands in Perl using qx to store Linux command result in Perl.
e.g.
$cmd = "find . | grep -v 'dir1\|dir2\|...\|dirn'";
$result=qx($cmd);
The above command combinations do the following:
The find command will list the all the directory and
files recursively.
The pipe "|" will pass the find result to grep command
The grep -v command will print on screen only the string not exist
in the "dir1", "dir2"..."dirn" to be ignored
At last, the qx command will execute the find and grep Linux
commands and stored the output to $result variable.
You can do the similar thing in Windows. The only difference is to use the Windows command line.
e.g.
$result=qx('dir /b/s | find /v "workspace" | find /v "TVM"')
The above command will list all the directory recursively except the directory has name "workspace" or "TVM".

Powershell fails to run multi-line commands from stdin?

I'm wanting to pass arbitrary scripts to Powershell via stdin.
(In practice, I'd like to avoid having to put the script into a temporary file, but for the purposes of this question I will pipe the contents of a file to powershell.)
So I'm doing something like so (in this example, from a Windows cmd shell):
type myfile.txt | powershell -
It works if myfile.txt contains something like this:
1..3 | % { $_ *2 }
echo done
(It outputs 2\n4\n6\ndone.)
However, if I split this first statement across multiple lines like so, then Powershell simply exists without generating any output at all:
1..3 |
% { $_ *2 }
echo done
This seems to fail for any multiline statement. For example, this also fails to produce output:
1..3 | % {
$_ *2 }
echo done
I'm surprised by this since each are legal Powershell scripts that would work normally if placed into a .ps1 file and run as normal.
I've tried various things including escaping the EOL using line continuation chars, to no avail. The same effect occurs if the parent shell is Powershell, or even Python (using subprocess.Popen with stdin=PIPE). In each case, Powershell exits without any error, and the exit code is 0.
Interestingly, if I run the following, only "before.txt" gets created.
"before" | out-file before.txt
1..3 |
% { $_ *2 }
"after" | out-file after.txt
echo done
Any ideas why Powershell would have trouble reading a multi-line command, if read from stdin?
I'm going to consider this answered by this:
How to end a multi-line command in PowerShell since it shows that an extra newline is required.
However, I'm going to raise this to MS as a bug since this should not be required when reading from a non-tty, or when -NonInteractive switch is specified.
Please vote on my bug report to the Powershell team.
This is not a complete answer, but from what I can tell, the problem has to do with the input being sent in line by line.
To demonstrate the line-by-line issue, I invoke powershell this way:
powershell.exe -command "gc myfile.txt" | powershell.exe -
vs
powershell.exe -command "gc myfile.txt -raw" | powershell.exe -
The first example replicates what you see with type, the second reads the entire contents of the file, and it works as expected.
It also works from within PowerShell if you put the script contents in a string and pipe it into powershell.exe -.
I had a theory that it had to do with line-by-line input lacking line breaks, but it's not so clear cut. If that were the case, why would the first option work but not the second (removing the line break splitting the single pipeline should have no effect, while removing the line break between the pipeline and the echo should make it fail). Maybe there's something unclear about the way powershell is handling the input with or without line breaks.

Compressing to tar.xz using 7-zip through a pipe on windows

My command line is this (powershell):
$7z ="`"c:\Program Files\7-Zip\7z.exe`""
&$7z a -r -ttar -bd -so . | &$7z a -r -txz -bd $archive -si
The produced archive file indeed contains a tar file, but that tar file is corrupt.
Note, that breaking the pipe into two commands works correctly:
&$7z a -r -ttar -bd ${archive}.tmp .
&$7z a -r -txz -bd $archive ${archive}.tmp
The produced archive is perfectly valid.
So, what is wrong with my pipeline?
(I am using Powershell)
Nothing is wrong with your pipeline it is the way that the pipeline works that's causing the error.
PowerShell pipe works in an asynchronous way. Meaning that output of the first command is available to the second command immediately one object at the time even if the first one has not finished executing, See here.
Both Unix and PowerShell pipes operate in the same way. The reason why you might be seeing a difference from Unix to PowerShell is the way in which they go about it is different.
Unix passes Strings between the commands. Where as a Powershell pipe will pass full-fledged .net object between commands. This difference in the data type being past between command will be why it works on unix and not in PowerShell. If 7z.exe can not huddle these .net objects correctly the files will be come corrupt, See here.
Try adding | %{ "$_" } in between the pipes like
&$7z a -r -ttar -bd -so . | %{ "$_" } | &$7z a -r -txz -bd $archive -si
The point is that the second call to 7z expects unmodified data on STDIN, but PowerShell is converting the output from the first call to 7z to (multiple) (string) objects. % is an alias for foreach-object, so what the additional command does is to loop over each object and convert it to a plain string before passing it on to the second call to 7z.
Edit: Reading through PowerShell’s Object Pipeline Corrupts Piped Binary Data it looks to me now as if my suggestion would not work, and there's also no way to fix it. Well, other than wrapping the whole pipeline into a cmd /c "..." call to make cmd and not PowerShell handle the pipeline.
Edit2: I also was trying this solution from the PowerShell Cookbook, but it was very slow.
In the end, I created a .cmd script with the 7z pipes that I'm calling from my PowerShell script.

How do I reference variables when executing a shell command in PowerShell?

I'm a newbie to PowerShell. What's wrong with my script below? It's not wanting to emit the value of $config. However, when I wrap that command in double quotes, everything looks okay.
param($config, $logfolder)
# Must run log analysis in chronological order.
ls $logfolder | Sort-Object LastWriteTime | % {
perl D:\Websites\_awstats\wwwroot\cgi-bin\awstats.pl -LogFile="$($_.FullName)" -config=$config update
}
# Execute with - .\regen-logs.ps1 webgenesis "C:\inetpub\logs\LogFiles\W3SVC5"
# Returns for each file - Error: Couldn't open config file "awstats.config.conf" nor "awstats.conf" after searching in path "D:\Websites\_awstats\wwwroot\cgi-bin,/etc/awstats,/usr/local/etc/awstats,/etc,/etc/opt/awstats": No such file or directory
As-is, what gets emitted and executed seems to have "-config=$config" passed as an argument. At least, that's my best guess. I don't know if $_ is working correctly either.
If I put quotes around the perl command like so, I get the command I do want to execute.
ls $logfolder | Sort-Object LastWriteTime | % {
"perl D:\Websites\_awstats\wwwroot\cgi-bin\awstats.pl -LogFile=`"$($_.FullName)`" -config=$config update"
}
# Outputs for each log file something like - perl D:\Websites\_awstats\wwwroot\cgi-bin\awstats.pl -LogFile="C:\inetpub\logs\LogFiles\W3SVC5\u_ex110602.log" -config=webgenesis update
If putting quotes around it produces the correct commandline, one way to execute the contents of a string is with Invoke-Expression (alias iex):
$v = "myexe -myarg1 -myarg2=$someVar"
iex $v
Put double quotes around "-config=$config". Without this, PowerShell will interpret -config=$config as one string argument that just happens to contain a $ sign in it.
I think you need to start your perl command out with & so that PowerShell interprets things as a command and not a string.
& perl D:\Websites\_awstats\wwwroot\cgi-bin\awstats.pl -LogFile=`"$($_.FullName)`" -config=$config update
Also, see: Run a program in a foreach