I am creating a PowerShell script just to backup my WSL distros, but when I try to run the command with variables it's not working, it displays the usage text as though I provided the incorrect arguments.
$DistrosArray1 = (wsl -l --quiet) | where {$_ -ne ""}
$DistrosArray2 = 'Arch', 'Ubuntu-22.04', 'docker-desktop-data', 'docker-desktop'
$CheckArrayDifference = Compare-Object -ReferenceObject $DistrosArray1 -DifferenceObject $DistrosArray2 -PassThru
echo $CheckArrayDifference
# Does not return anything (there is no difference)
foreach ($Distro in $DistrosArray1) {
wsl --export $Distro "$Distro.tar"
# This method is not working
}
foreach ($Distro in $DistrosArray2) {
wsl --export $Distro "$Distro.tar"
# This method is working
}
It sounds like you are running into complications from issue #4607 -- The wsl.exe command outputs some oddly mangled UTF16 encoding that creates issues when attempting to process it from PowerShell (or even from inside WSL).
This is now fixed in the latest WSL Preview release 0.64.0, but you do have to "opt-in" to the fix so that older workarounds (like the ones #Bender and I provided) don't inadvertently break.
Simply set:
$env:WSL_UTF8=1
... before your code, and WSL will no longer spit out the "mangled UTF16."
Other examples in my answers to:
Why cannot I match for strings from wsl.exe output?
Powershell - Strange WSL output string encoding
How to ask WSL to check if the distribution exists, using Bash and wsl.exe?
Older solution:
Let's simplify the problem and make a "safe" example that doesn't attempt to export:
$DistrosArray1 = (wsl -l --quiet) | where {$_ -ne ""}
wsl -d $DistrosArray1[0]
Results in:
There is no distribution with the supplied name.
I've successfully used the method in this comment to handle it. For this particular example:
$console = ([console]::OutputEncoding)
[console]::OutputEncoding = New-Object System.Text.UnicodeEncoding
$DistrosArray1 = (wsl -l --quiet) | where {$_ -ne ""}
wsl -d $DistrosArray1[0]
This will run the first distribution in the list correctly.
Reset the encoding after with:
[console]::OutputEncoding = $console
That shouldn't be a problem for most non-interactive scripts since it will just be the final line of the "wrapper", but as #BendertheGreatest pointed out in the comments, it's a critical step.
This is part of a known issue with wsl.exe output. Here is what I put together from workarounds provided on that issue:
$DistrosArray1 = wsl -l --quiet | wsl iconv -c -f utf16 -t ascii
foreach ($Distro in $DistrosArray1) {
wsl --export $Distro "$Distro.tar"
}
Unfortunately I could not get this working with a conversion to UTF8 (changing ascii to utf8 produces additional garbage characters although they are consistent and detectable in my limited testing), so only any characters outside of the ASCII range will likely cause problems for you.
Related
I'm trying to create a Powershell function that acts as a wrapper for Vim in WSL. I would like this wrapper to be able to accept input from the pipeline.
In WSL, it's possible to do something like:
$ cat manylines.txt | vim -
The - indicates that Vim should read the input from stdin, which is then redirected through the pipeline. This will open vim with the command output already in the buffer, without any file associated.
(You could also simply open the file with vim manylines.txt, but that wouldn't work in general for the outputs of other commands or executables).
The same is possible in Powershell, with Windows native Vim:
Get-Content manylines.txt | vim -
It's possible to go one step further and execute Vim within another instance of Powershell or cmd:
Get-Content manylines.txt | powershell -Command vim -
Get-Content manylines.txt | cmd /c vim -
However, the same does not work with WSL Vim.
# doesn't work, Vim freezes
Get-Content manylines.txt | wsl vim -
Get-Content manylines.txt | wsl bash -c "vim -"
# doesn't work, Vim quits reporting "Error reading Input"
Get-Content manylines.txt | wsl bash -c vim -
In the first two cases, Vim opens correctly with the contents of manylines.txt in the buffer... and then refuses to accept any keyboard input. It is not possible to navigate, edit, or even quit with :q!.
Using a temporary powershell variable seems to fix things, but only when the input consists of a single line
# "works as long as the file only has a single line of text"
# "$tmpvar has type System.String (not an array)"
$tmpvar = Get-Content singleline.txt
wsl bash -c "echo $tmp | vim -"
But this breaks when $tmpvar becomes a String[] array or a String with any newlines
# sort of works, but newlines are converted to a single space
$tmpvar = [string[]] Get-Content manylines.txt
wsl bash -c "echo $tmp | vim -"
# combines the strings in the array, inserts a newline between each pair of elements
$tmpvar = ([string[]] Get-Content manylines.txt) -join "`n"
# doesn't work, bash interprets the newline as the start of the next command
wsl bash -c "echo $tmp | vim -"
How do I get WSL Vim to accept pipeline input from Powershell?
Other Notes:
I could use a temporary file and pass the file to WSL Vim, but that's not an elegant solution and leaves a file that has to be cleaned up. Unless the temporary file is in the current directory, it also involves extra shenanigans with wslpath
Indeed it seems that piping input to vim via wsl seems to break vim's keyboard interface with respect to cursor movements (I was still able to type :q! to quit, for instance).
The workaround is to refine the approach you've discovered, namely to provide the input as part of a shell command passed to wsl instead of using the pipeline:
Doing so requires careful escaping and transformations, as shown in the following wrapper function:
# PowerShell wrapper function for invoking vim via WSL
function vim {
if ($MyInvocation.ExpectingInput) { # pipeline input present
$allInput = ($input | Out-String) -replace '\r?\n', "`n" -replace "'", "'\''" -replace '"', '\"'
wsl -e sh -c "printf %s '$allInput' | vim $args -"
}
elseif ($args) { # no pipeline input, arguments that may include file paths given
wsl -e vim ($args.ToLower() -replace '^([a-z]):', '/mnt/$1' -replace '\\', '/')
} else { # no pipeline input, no pass-through arguments
wsl -e vim
}
}
Caveats:
When passing command output, you may run into the max. command-line length.
When passing file paths, relative file paths work fine, but absolute ones work only for local drives (mapped drives / UNC paths require mounting a volume on the WSL side first).
With this function defined, you can call, say (+, as a sample option, tells vim to place the cursor at the end of the input):
vim + somefile.txt
or
Get-ChildItem | vim +
I hope can you help me, essentially, I'm looking for the Powershell equivalent of the awk command:
awk '/"Box11"/ { print $0 }' test.txt|awk '{ SUM += $4} END { print SUM} '
What his does is print lines that contain the string Box11, then piping it to another awk that prints the total of the 4th column (delimited by spaces).
Multiple ways of doing it but this would do the trick:
Get-Content c:\temp\test.txt | Where-Object{$_ -match '"Box11"'} |
ForEach-Object{($_ -split "\s+")[3]} | Measure-Object -Sum |
Select-Object -ExpandProperty Sum
Get a string array of the file. For each line that contains the string "Box11" we split the line on each group of spaces. Then pass the 4 element of each match to Measure-Object.
A short hand, if you value that, would look like this:
gc c:\temp\test.txt | ?{$_ -match '"Box11"'} | %{($_ -split "\s+")[3]} |
Measure -Sum | Select -Exp Sum
If this file/string input had header this would be a good start as well. Assuming of course that your file is delimited with one space exactly.
Get-Content c:\temp\test.txt | ConvertFrom-Csv -Delimiter " "
I know this post is old but i thought I'd add to this. Currently if yo have WSL (windows sub system for Linux) enabled, (windows 10 all version on systems that support virtualization, in the turn windows features on) with a distribution installed in the subsystem. You can can call Linux commands directly from windows
wsl -e awk '/"Box11"/{sum += $4} END{print sum}' test.txt
(borrowed from #Ed Moritn)
( or any awk command of your choice. )
Basically cmd or PowerShell takes the command and pipes it into the subsystem and the results are returned (bit of an over simplification but in effect accurate). But the -e flag allows you to execute the command without opening an instance.
edit
Since writing this initial response I have found two answers which are better solutions. The first is GNUwin32 This is a collection of Gnutils which have been ported to windows standalone .exe files including sed, awk, grep and many more, allowing you to call get-childitem | awk.exe '{print $1}' directly. These tools are fully portable with no installation required. The second option is Msys32, a platform that grew out of chocolatey (though it is almost fully code in dependant now) designed for cross compiling binaries. Once installed in the /bin folder, are many Linux utilities as exe files. most of these executable can be pulled from the bin and are portable with no required installation of dependencies. The reason msys32 is preferred (in my books) over the gnuwin32 is the fact that that gnuwin32 has gawk version 3.1 and msys32 has nawk and gawk vs 5.1.
You can get get awk for Windows now. I have been using it as a direct replacement and haven't had any problems yet.
It can be easily installed via Chocolatey
I have curl command stored in variable ($command). The command looks like this:
curl -F "login=xxx" -F "password=xxx" "title=Some title" -F "img=#/path/to/image" https://example.com/api/import
Then i execute the command:
Invoke-Expression ($command)
Everything is fine unless title contains special characters like "č,š,ý..." because server expects UTF8 encoded parameters. In such case special characters are replaced with questionmarks on the website.
I tried setting [Console]::OutputEncoding and $OutputEncoding to UTF8 but it didn't solve the problem.When i run the command on linux (ubuntu) everything is fine because it uses UTF8 as default encoding, so i rewrote the script to bash to get the job done. But i'm still wondering if it's possible in powershell somehow. Any suggestions appreciated.
Settting [Console]::OutputEncoding works for me. Are you sure you're setting it correctly?
C:\PS> [console]::OutputEncoding = [text.encoding]::UTF8
C:\PS> echoargs "title=č,š,ý"
Arg 0 is <title=č,š,ý>
Command line:
"C:\Program Files (x86)\PowerShell Community Extensions\Pscx3\Pscx\Apps\EchoArgs.exe" title=č,š,ý
Echoargs is a tool from PSCX.
I've searched everywhere, but I can't seem to find a solution for my issue. Probably, it is code related.
I'm trying to catch the exit code from a novell program called DXCMD, to check whether certain "drivers" are running. This is no problem in bash, but I need to write a more complex perl script (easier working with arrays for example).
This is the code:
#Fill the driverarray with the results from ldapsearch (in ldap syntax)
#driverarray =`ldapsearch -x -Z -D "$username" -w "$password" -b "$IDM" -s sub "ObjectClass=DirXML-Driver" dn | grep ^dn:* | sed 's/^....//' | sed 's/cn=//g;s/dc=//g;s/ou=//;s/,/./g'`;
#iterate through drivers and get the exit code:
foreach $driverdn (#driverarray)
{
my $cmd = `/opt/novell/eDirectory/bin/dxcmd -user $username -password $password -getstate "$driverdn"`;
my $driverstatus = $?>>8;
}
I've come this far; the rest of the code is written (getting the states).
But the $?>>8 code always returns 60. When I copy the command directly into the shell and echo the $?, the return code is always 2 (which means the driver is running fine). In bash, the code also works (but without the >>8, obviously).
I've looked into the error code 60, but I cannot find anything, so I think it is due to my code.
How can I rectify this error? Or how can I track the error? Anyone? :)
Wrong value passed to -getstate. You didn't remove the newline. You're missing
chomp(#driverarray);
I'm a newbie to PowerShell. What's wrong with my script below? It's not wanting to emit the value of $config. However, when I wrap that command in double quotes, everything looks okay.
param($config, $logfolder)
# Must run log analysis in chronological order.
ls $logfolder | Sort-Object LastWriteTime | % {
perl D:\Websites\_awstats\wwwroot\cgi-bin\awstats.pl -LogFile="$($_.FullName)" -config=$config update
}
# Execute with - .\regen-logs.ps1 webgenesis "C:\inetpub\logs\LogFiles\W3SVC5"
# Returns for each file - Error: Couldn't open config file "awstats.config.conf" nor "awstats.conf" after searching in path "D:\Websites\_awstats\wwwroot\cgi-bin,/etc/awstats,/usr/local/etc/awstats,/etc,/etc/opt/awstats": No such file or directory
As-is, what gets emitted and executed seems to have "-config=$config" passed as an argument. At least, that's my best guess. I don't know if $_ is working correctly either.
If I put quotes around the perl command like so, I get the command I do want to execute.
ls $logfolder | Sort-Object LastWriteTime | % {
"perl D:\Websites\_awstats\wwwroot\cgi-bin\awstats.pl -LogFile=`"$($_.FullName)`" -config=$config update"
}
# Outputs for each log file something like - perl D:\Websites\_awstats\wwwroot\cgi-bin\awstats.pl -LogFile="C:\inetpub\logs\LogFiles\W3SVC5\u_ex110602.log" -config=webgenesis update
If putting quotes around it produces the correct commandline, one way to execute the contents of a string is with Invoke-Expression (alias iex):
$v = "myexe -myarg1 -myarg2=$someVar"
iex $v
Put double quotes around "-config=$config". Without this, PowerShell will interpret -config=$config as one string argument that just happens to contain a $ sign in it.
I think you need to start your perl command out with & so that PowerShell interprets things as a command and not a string.
& perl D:\Websites\_awstats\wwwroot\cgi-bin\awstats.pl -LogFile=`"$($_.FullName)`" -config=$config update
Also, see: Run a program in a foreach