Groovy Jenkins powershell pipe not working - powershell

I want to replace some text in a file after Jenkins does its build step.
I have the code below in my post build step.
powershell.exe -Command(Get-Content "./PublishedWebsites/app.config") |
Foreach-Object {$_.replace({{MyDummValue}}, "${env.BUILD_NUMBER}")} |
Set-Content "./PublishedWebsites/app.config"
The first part works (Get-Content "./PublishedWebsites/app.config") up until the pipe ("|").
After the pipe it throws an error that Foreach-Object is not recognized by an internal or external command.
I am thinking it has something to do with the escape sequence and the syntax.
So I have tried this as well
"""powershell.exe -Command(Get-Content "./PublishedWebsites/app.config") |
Foreach-Object {$_.replace({{MyDummValue}}, "${env.BUILD_NUMBER}")} |
Set-Content "./PublishedWebsites/app.config"""
Then for some reason jenkins is trying to evaluate the $_.replace({{MyDummValue}} before the post build step is even reached and gives me the error of 'Cannot get property replace on null object'
So I am at a loss. All I want to do is on a post build step, rename a value in a file with the Build Number.

When calling PowerShell (or other) scripts from Groovy, always use single-quoted (or triple-single-quoted) string literals for the script. With double-quoted (or triple-double-quoted) strings, Groovy tries to interpolate the string, so you'd have to escape $ as \$, to use it literally. While this is just a nuisance, double-quoted strings are also prone to script injection, where a pipeline user might run arbitrary PowerShell scripts, when pipeline parameters are interpolated by Groovy within the script.
Another mistake: the value {{MyDummValue}} isn't quoted.
Also, I don't know why you are calling powershell.exe explicitly. Use the powershell step instead:
post {
always { // or one of the other available sections
powershell '''
Get-Content "./PublishedWebsites/app.config" |
Foreach-Object {$_.replace("{{MyDummValue}}", $env:BUILD_NUMBER)} |
Set-Content "./PublishedWebsites/app.config"
'''
}
}

Related

Can you pipe a string or variable into a downloaded .ps1 file?

I have file with the following contents hosted on dropbox:
(I know it is not currently calling anything)
function test_echo {
[CmdletBinding()]
param (
[Parameter (Position=0,Mandatory = $True, ValueFromPipeline = $True)]
[string]$e
)
echo $e
}
Is there a way to pipe information into this file before it is downloaded and executed?
ex:
"test" | iwr DropBoxLink | iex
and to make this echo out test
this honestly probably has no practical application, but I found myself wondering if it is possible so thought I'd ask.
I know I could define the string as a variable first and execute it but I just want to know if you can pipe it for principles sake
$testEcho = "string"; iwr DropBoxLink | iex
> string
"test" | % -begin { iex (irm $DropBoxUrl) } -process { $_ | test_echo }
The above uses a ForEach-Object call's -Begin block to download and evaluate the script (the usual caveats re iex (Invoke-Expression) apply - you should trust the source), which defines the test_echo function contained in the downloaded script.
The -Begin script block executes before pipeline input is processed, which means that by the time the -Process script block processes (each) pipeline input object, the test_echo function is already defined.
Also note that irm (Invoke-RestMethod) rather than iwr (Invoke-WebRequest) is used, given that you're only interested in the content of the script.
Of course, this doesn't gain you much, as you could simply use two statements, which has the added advantage that all pipeline input (should there be multiple input objects) are handled by a single test_echo invocation:
iex (irm $DropBoxUrl) # Download and effectively dot-source the script.
"test" | test_echo # Pipe to the script's `test_echo` function.
A general caveat is that if the downloaded script contains an exit statement that is hit during evaluation, the calling shell exits as a whole.
Since in effect you need to dot-source the downloaded script in order to make its functions available, the only way to solve the exit problem is to download to a (temporary) file first, and dout-source that.
If dot-sourcing isn't needed, calling via the PowerShell CLI (child process) may be an option - see below.
GitHub issue #5909 proposes a future enhancement that would allow piping Invoke-WebRequest (iwr) calls to Invoke-Command (icm) for direct-from-the-web downloading and execution, without the exit problem (iwr $DropboxLink | icm).
Note that if your downloaded script were to accept pipeline input directly, you could use [scriptblock]::Create() as follows (Invoke-Expression is not an option, because it doesn't accept pipeline input):
# Assumes that the script located at $DropBoxUrl
# *itself* accepts pipeline input.
# Use . (...) if you want to *dot-source* the script, as
# Invoke-Expression would implicitly do.
"test" | & ([scriptblock]::Create((irm $DropBoxUrl))
To work around the exit problem, you can call via the CLI and a script block, using pwsh, the PowerShell (Core) 7+ CLI, in this example; use powershell.exe for Windows PowerShell:
# Assumes that the script located at $DropBoxUrl
# *itself* accepts pipeline input.
'test' |
pwsh -NoProfile {
$input | & ([scriptblock]::Create($args[0]))
} -args (irm $DropBoxUrl)

In Windows power shell, how do you extract a properties file value and save it to an env var?

I have a properties file with entries like the below ...
...
USERNAME=myuser
...
In my Makefile, I have the below which uses Unix like commands to get the value of the variables ...
export USERNAME=$(shell grep USERNAME my_properties.txt | cut -d'=' -f 2-)
However, in a Windows power shell (maybe command prompt is the right phrase?), the above doesn't work because "grep" is not a standard command (among others). What's the equivalent way to extract a property from a properties file in a Windows power shell environment?
We could achieve this in PowerShell by following the below steps
Read the contents of the file
Convert the contents into key-value pairs
Create environment variable with the required value
(you can combine the steps if you like, I've kept them separate for better understanding)
Here's the script
$content = Get-Content .\user.properties -raw
$hashTable = ConvertFrom-StringData -StringData $content
$Env:USERNAME = $hashTable.USERNAME
Assuming that cmd.exe is the default shell:
export USERNAME=$(shell powershell -noprofile -c "(Select-String 'USERNAME=(.+)' my_properties.txt).Matches.Group[1]")
Note: -NoProfile suppresses loading of PowerShell's profiles, which unfortunately happens by default. Should you need the -File parameter to execute a script file, you may additionally need -ExecutionPolicy Bypass, unless your effective execution policy allows script execution.
The above uses the PowerShell CLI's -c (-Command) parameter to pass a command that uses the Select-String cmdlet, PowerShell's grep analog.
A closer analog to your command would be the following, which additionally uses -split, the string-splitting operator (showing the raw PowerShell command only; place it inside the "..." above):
((Select-String USERNAME my_properties.txt) -split '=', 2)[-1]

Remove Orphaned Carriage Returns from File

I have a daily process that downloads and parses pipe-delimited files into a database. Sometimes these files contain a single carriage return (no newline) at the end of one of the fields within a row, which breaks my process. My scheduler (not a windows scheduler, internal to our company) allows me to run batch or PowerShell processes on these files before loading them into the database, so I want to trim out any instance of orphaned carriage returns at the end of a field (\r|).
This PowerShell command works when run manually:
(Get-Content CarriageReturnTest.txt -Raw).Replace("`r|","|") | Set-Content CarriageReturnTest.txt -Force
However when I put this in the scheduler, I get an error that '","' is not recognized as an internal or external command. I realized the scheduler must be running a windows command of PowerShell "{command}" so the double quotes are making the command line think that I'm running several different commands. I tried replacing the double quotes in the PowerShell command with single quotes like so:
(Get-Content CarriageReturnTest.txt -Raw).Replace('`r|','|') | Set-Content CarriageReturnTest.txt -Force
This runs through the scheduler, but doesn't actually trim anything out of the file because PowerShell interprets the ` as an escape character when it's enclosed in single quotes.
I've tried escaping the double quotes with "", `", \", and ^" but nothing seems to work.
I realize there's some longer batch scripts I could write to handle this but ideally this would be a one liner to fit in the scheduler. I've also looked into saving the script as a ps1 file and running that with the file paths as arguments but I haven't gotten that to work either.
Based off what #JosefZ said I tried this and it worked:
(Get-Content CarriageReturnTest.txt -Raw) -replace('\r\|','|') | Set-Content CarriageReturnTest.txt -Force
The -replace let me use regex in single quotes '\r' instead of "`r"in double, which kept me from escaping

problem with powershell and cmd with pipes

I have this command that works ok on powershell
Compare-Object (Get-Content "tex1.txt") (Get-Content "tex2.txt") | Where-Object{$_.SideIndicator -eq "<="} | select inputobject | ft -hidetableheaders
I'm trying to running in cmd by doing this:
powershell -Command " & {Compare-Object (Get-Content "tex1.txt") (Get-Content "tex2.txt") | Where-Object{$_.SideIndicator -eq "<="} | select inputobject | ft -hidetableheaders}"
but it says something like: the name, the directory or the volume syntax is incorrect (is in spanish so i dont know the exact translation)
I think the problem is the pipes, since running everything before the pipe: Compare-Object (Get-Content "tex1.txt") (Get-Content "tex2.txt") works
PD: I also tried to write ^ before the pipes but I haven't succeeded.
tl;dr
When calling the PowerShell CLI (powershell.exe for Windows PowerShell, pwsh for the cross-platform PowerShell [Core] 6+ edition):
Using embedded " in an overall "..." string comes with escaping challenges.
If feasible for a given command, formulating it without embedded " is the easiest solution:
powershell -Command "Compare-Object (Get-Content tex1.txt) (Get-Content tex2.txt) | Where-Object {$_.SideIndicator -eq '<='} | select inputobject | ft -hidetableheaders"
Read on, if you do need to use embedded ".
eryksun points out that your problem is your lack of escaping of embedded " chars. inside the overall "..." string, which causes cmd.exe to see multiple strings, including parts it considers unquoted, which causes problems with special characters such as | and < - they never reach PowerShell.
Nesting double-quote strings from cmd.exe is tricky business:
To make cmd.exe happy, you need to double the embedded " chars. ("")
Separately, to make powershell.exe happy, you need to \-escape " chars.
Note: PowerShell [Core] 6+, the cross-platform edition, on Windows now also accepts "" by itself, which is the most robust choice.
Generally, dealing with quoting and escaping arguments when calling from cmd.exe is a frustrating experience with no universal solutions, unlike in the Unix world. Sadly, PowerShell has its own challenges, even in the Unix world.[1]
In short: Escape embedded " chars. when calling the Windows PowerShell CLI, powershell.exe, from cmd.exe as follows (for the PowerShell (Core) 7+ CLI, pwsh.exe, "" is the robust choice):
Use "^"" (sic) when using powershell.exe -Command, which works robustly.
Caveat: "^"" does not work for calling other programs.
This saves you from additional escaping, as would be necessary if you used \"
Example: powershell -command " Write-Output "^""a & b"^"" " yields a & b, as expected, and the & didn't need escaping.
If you use the simpler - and customary - \", you may need to perform additional escaping: Specifically, you must individually ^-escape the following cmd.exe metacharacters with ^ inside \"...\" runs: & | < > ^Thanks, LotPings.
Example: powershell -command " Write-Output \"a ^& b\" " yields a & b; that is, the & needed escaping with ^.
Additionally, to treat % (and, with enabledelayedexpansion , !) verbatim, the escaping syntax unfortunately depends on whether you're calling from the command line or a batch file: use %^USERNAME% (!^USERNAME) from the former, and %%USERNAME%% (^!USERNAME^! / ^^!USERNAME^^! inside \"...\" runs) from the latter - see this answer for the gory details.
It is unfortunate that cmd.exe makes use of \" treacherous, given that it is supported by virtually all programs (except batch files), and if it weren't for these extra escaping requirements, command lines that use it have the potential to work across different platforms and shells - with the notable exception of calling from PowerShell, where, sadly, an additional layer of escaping is needed and " inside "..." must be escaped as \`" (sic); see this answer.
See the bottom section for ways to ease the escaping pain by avoiding use of nested ".
Other programs, including PowerShell Core:
Use just "" for programs compiled with Microsoft compilers and, on Windows, also Python and Node.js as well as PowerShell Core (pwsh.exe).
Regrettably, this robust option does not work with powershell.exe, i.e. Windows PowerShell.
Use \" for programs with Unix heritage, such as Perl and Ruby - which comes with the escaping headaches discussed above.
Avoiding embedded ":
When you call PowerShell's CLI, you can often get away without needing to embed double quotes:
There may be arguments in your string that don't require quoting at all, such as text1.txt and text2.txt
You can alternatively use single-quoting ('...') inside the overall command string, which require no escaping; note that such strings, from PowerShell's perspective, are string literals.
To put it all together:
powershell -Command "Compare-Object (Get-Content tex1.txt) (Get-Content tex2.txt) | Where-Object {$_.SideIndicator -eq '<='} | select inputobject | ft -hidetableheaders"
Note that I've also removed the & { ... } around your command, as it isn't necessary.
[1] eryksun puts it as follows: "This is the inescapable frustration of the Windows command line. Every program parses its own command line, using whatever rules it wants. So the syntax of a command line has to work with not only the shell (CMD) but also all programs invoked in the pipeline. In the Unix world the shell parses the command line into argv arrays, so typically you only have to get the syntax right to make the shell happy."
The problems with PowerShell Core, even on Unix, stem from how it re-quotes arguments behind the scenes before passing them on - see this GitHub docs issue.

Powershell: Pipe external command output to another external command

How can I get PowerShell to understand this type of thing:
Robocopy.exe | Find.exe "Started"
The old command processor gave a result, but I'm confused about how to do this in PowerShell:
&robocopy | find.exe "Started" #error
&robocopy | find.exe #("Started") #error
&robocopy #("|", "find.exe","Started") #error
&robocopy | &find #("Started") #error
&(robocopy | find "Started") #error
Essentially I want to pipe the output of one external command into another external command. In reality I'll be calling flac.exe and piping it into lame.exe to convert FLAC to MP3.
Cheers
tl;dr
# Note the nested quoting. CAVEAT: May break in the future.
robocopy.exe | find.exe '"Started"'
# Alternative. CAVEAT: doesn't support *variable references* after --%
robocopy.exe | find.exe --% "Started"
# *If available*, use PowerShell's equivalent of an external program.
# In lieu of `findstr.exe`, you can use Select-String (whose built-in alias is scs):
# Note: Outputs are *objects* describing the matching lines.
# To get just the lines, pipe to | % ToString
# or - in PowerShell 7+ _ use -Raw
robocopy.exe | sls Started
For an explanation, read on.
PowerShell does support piping to and from external programs.
The problem here is one of parameter parsing and passing: find.exe has the curious requirement that its search term must be enclosed in literal double quotes.
In cmd.exe, simple double-quoting is sufficient: find.exe "Started"
By contrast, PowerShell by default pre-parses parameters before passing them on and strips enclosing quotes if the verbatim argument value doesn't contain spaces, so that find.exe sees only Started, without the double quotes, resulting in an error.
There are three ways to solve this:
PS v3+ (only an option if your parameters are only literals and/or environment variables): --%, the stop-parsing symbol, tells PowerShell to pass the rest of the command line as-is to the target program (reference environment variables, if any, cmd-style (%<var>%)):
robocopy.exe | find.exe --% "Started"
This answer details the limitations of --%.
PS v2 too, or if you need to use PowerShell variables in the parameters: apply an outer layer of PowerShell quoting (PowerShell will strip the single quotes and pass the contents of the string as-is to find.exe, with enclosing double quotes intact):
robocopy.exe | find.exe '"Started"'
Caveat: It is only due to broken behavior that this technique works. If this behavior gets fixed (the fix may require opt-in), the above won't work anymore, because PowerShell would then pass ""Started"" behind the scenes, which breaks the call - see this answer for more information.
If an analogous PowerShell command is available, use it, which avoids all quoting problems. In this case, the Select-String cmdlet, PowerShell's more powershell analog to findstr.exe can be used, as shown above.
#Jobbo: cmd and PowerShell are two different shells. Mixing them is sometimes possible but as you realized from Shay's answer, it won't get you too far. However, you may be asking the wrong question here.
Most of the time, the problem you are trying to solve like piping to find.exe are not even necessary.
You do have equivalent of find.exe, in fact more powerful version, in Powershell: select-string
You can always run a command and assign results to a variable.
$results = Robocopy c:\temp\a1 c:\temp\a2 /MIR
Results are going to be STRING type, and you have many tools to slice and dice it.
PS > $results |select-string "started"
Started : Monday, October 07, 2013 8:15:50 PM
Invoke it via cmd:
PS> cmd /c 'Robocopy.exe | Find.exe "Started"'