I am using jenkins and mongodb on centos 7 server.
I want to insert some jenkins build data on success to my mongo database.
this is the code i'm running in my post build task shell:
echo 'password' | su -
mongo jenkinsdb <<\EOF
db.history.insert({wokspace:$WORKSPACE,remote_url:$GIT_URL,branch:$GIT_BRANCH,type:"back",date:$(date
'+%Y-%m-%d %H:%M:%S'),description:$short_description})
db.history.find()
EOF
The problem is mongo is interpreting the Jenkins environnement variables as in simple strings.
How can i pass those variables to the mongo script ?
You are preventing the variable interpolation by using \ before EOF. See https://tldp.org/LDP/abs/html/here-docs.html:
Quoting or escaping the "limit string" at the head of a here document disables parameter substitution within its body. The reason for this is that quoting/escaping the limit string effectively escapes the $, `, and \ special characters, and causes them to be interpreted literally.
serene% a=1
serene% cat <<T
heredoc> $a
heredoc> T
1
serene% cat <<\T
heredoc> $a
heredoc> T
$a
Remove the backslash if you want interpolation to happen.
I have YAML file for GitLab CI/CD Pipeline
.myanchor: &myanchor
- 'echo "Some Test"'
My Job:
script:
- 'echo "Some Text"'
- '*myanchor'
When I run pipeline, I get an error "myanchor : The term '*myanchor' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again."
I tried removing quotes in a string - '*myanchor' but then CLint says that YAML is incorrect.
What am I doing wrong?
'*myanchor' is a YAML scalar and will be loaded as string. If you want to refer to the node with the anchor &myanchor, you must not use quotes.
If you drop the quotes, the structure in My Job will be
My Job:
script:
- 'echo "Some Text"'
- - 'echo "Some Text"'
It is likely that nested sequences are not supported for the content of script: (this is not a limitation of YAML per se, but a limitation of the tool that processes this structure).
To fix it, you need to put the anchor on the sequence item instead of the list:
.myanchor:
- &myanchor 'echo "Some Test"'
My Job:
script:
- 'echo "Some Text"'
- *myanchor
Now the problem is probably that you want to reference multiple items with *myanchor and put each of them as item directly into the script: sequence. That is simply not possible with YAML so you need to find another solution – perhaps preprocessing your YAML with a templating engine (lots of YAML's heavy users like e.g. Ansible and SaltStack do this).
I have a bash script with the following functionality:
# usage: setOutput <name> <value>
function setOutput {
echo "##vso[task.setvariable variable=$1]$2"
}
setOutput environment "dev"
This normally sets the variable correctly as ENVIRONMENT=dev - however, sometimes this randomly appends a ' at the end, i.e. ENVIRONMENT=dev'
I tried re-running the same commit on the pipeline multiple times, and sometimes it works, sometimes it doesn't. Any ideas?
I tested it with your sample, it can be displayed normally .
You can try to remove the double quotes of dev to see if this will happen.
function setOutput {
echo "##vso[task.setvariable variable=$1]$2"
}
setOutput environment dev
Base on the fact that #Hugh Lin - MSFT could not reproduce the issue, I looked a littel closer and found out, what was actually going wrong:
In the beginning of my script, I used:
set -ex
This lead to there beeing two echos:
+echo '##vso[task.setvariable variable=key]value'
##vso[task.setvariable variable=key]value
Depending on the order of the two (since the first is written to stderr, the second one to stdout), sometimes the value was set to the correct value, and somtimes to the debug echo one. And since that one has the addition ' at the end, so does the exported output.
My suggestion: Only process stdout for these echo commands, not stderr, or add a special filter for set -x. I removed the debug output and now it works.
How would something like diff <(echo aoeu) <(echo snth) be done in Scala?
I've tried using the sys.process interface as follows:
"diff <(echo aoeu) <(echo snth)".!
...however, this doesn't interpret the <() as subprocess substitution.
import scala.sys.process._
def diff(one: String, two: String):
String = Seq(
"bash", "-c", """
diff <(printf '%s\n' "$1") \
<(printf '%s\n' "$2"); retval=$?
(( retval == 1 )) || exit "$retval"
""", "_", one, two).!!
This can be tested in practice:
scala> diff("hello", "world")
res1: String =
"1c1
< hello
---
> world
"
To break down the reasoning:
Invoking a sequence, rather than a string, allows data (in my examples hello and world; in yours, aoeu and snth) to be passed out-of-band from code. This is critical to avoiding injection attacks when such content is parameterized.
Invoking bash as your executable ensures that process substitution syntax is available.
Checking for the exit status of 1 (and coercing it to 0) avoids scala treating a case where diff returns an exit status indicating that the two inputs are not identical as an error, while ensuring that other errors still become exceptions in scala.
Using printf '%s\n' "$1" instead of echo "$1" avoids ambiguities in the POSIX definition of echo (see in particular the APPLICATION USAGE section).
Passing an explicit argument of _ fills in the argv[0] slot, (aka $0).
Note that invoking a sequence rather than a string also prevents you from needing a shell at all in many cases: Seq("hello", "world").! doesn't need to invoke any shell, but can be implemented so as to directly starts an executable named hello, whereas "hello world".! is equivalent to Seq("sh", "-c", "hello world").!, with an extra executable invocation with both performance cost and potential security vulnerabilities required for implementation. See Shellshock for an example of a (now-near-universally-patched) case where a shell invocation with no explicit user-controlled parameters could still be vulnerable in practice (when invoked from a web server following CGI conventions for exporting request parameters as environment variables); avoiding unnecessary shells is thus preferable behavior where feasible.
I'm trying to pass a JSON string from within a powershell script to the build.phonegap.com api, using curl.
According to phonegap's forum, when running on a Windows machine, the JSON data has to be formatted as:
curl.exe -ku user#email:mypass -X PUT -d "data={\"password\":\"keypass\"}" https://build.phonegap.com/api/v1/key
Indeed, this does run fine when invoked from the command line.
However, when I try to invoke this from within a powershell script, the double quotes seem to be stripped.
So far, I have tried:
Putting the JSON in single quoted string:
curl.exe -ku user#email:mypass -X PUT -d '"data={\"password\":\"keypass\"}"' https://build.phonegap.com/api/v1/key
Putting the JSON in single quoted string, without the DOS escape backslashes:
curl.exe -ku user#email:mypass -X PUT -d '"data={"password":"keypass"}"' https://build.phonegap.com/api/v1/key
Putting the JSON in single quoted string, escaping the double quotes and backslashes (DOS style with a backslash):
curl.exe -ku user#email:mypass -X PUT -d '\"data={\\\"password\\\":\\\"keypass\\\"}\"' https://build.phonegap.com/api/v1/key
Putting the JSON in a double quoted string, escaping the double quotes with the powershell backtick character (`):
curl.exe -ku user#email:mypass -X PUT -d "`"data={\`"password\`":\`"build*2014`\`"}`"" https://build.phonegap.com/api/v1/key
Any idea how to achieve this?
Thanks for your time,
Koen
Try using the --% operator to put PowerShell into simple (dumb) argument parsing mode:
curl.exe --% -ku user#email:mypass -X PUT -d "data={\"password\":\"keypass\"}" https://build.phonegap.com/api/v1/key
This is quite often useful for invoking exes with argument syntax that runs afoul of PowerShell's argument syntax. This does require PowerShell V3 or higher.
Update:
PowerShell 7.3.0 mostly fixed the problem, with selective exceptions on Windows, though a future version may requires opt-in - see this answer for details.
For cross-version, cross-edition code, the Native module discussed below may still be of interest.
tl;dr:
Up to at least PowerShell 7.2.x, manual \-escaping of " characters inside PowerShell strings is - unfortunately - required.
# From inside PowerShell:
# Note the outer '...' quoting and the unexpected need to escape
# the embedded " chars. as \" (unexpected, because PowerShell itself doesn't
# require " inside '...' to be escaped; also, PowerShell's escape char. is `).
# If outer "..." quoting must be used, use \`" (sic) to escape the embeded "
curl.exe -ku user#email:mypass -X PUT -d 'data={\"password\":\"keypass\"}' https://build.phonegap.com/api/v1/key
Read on for why that is necessary, and about a potential future fix.
PowerShell's escape character is ` (the so-called backtick), so in order to embed " characters in a "..." (double-quoted, interpolating) string, use `" (or "") rather than \"; by contrast, inside a '...' (single-quoted, verbatim) string, " need not be escaped.
In your attempt, PowerShell didn't see \" as an escaped " and therefore saw multiple "..." strings, which ultimately - when PowerShell of necessity applied its on demand re-quoting behind the scenes, passed two separate string arguments that individually didn't need double-quoting, due to not containing spaces, namely: verbatim data={\ and password\:\keypass\}
Using PowerShell's quoting rules, you should have used:
either:
"data={`"password`":`"keypass`"}"
or, more simply, given that no string interpolation is needed, via a verbatim, single-quoted string, inside of which " chars. don't require escaping:
'data={"password":"keypass"}'
Unfortunately, however, as of PowerShell 7.2.x this is NOT enough, though the experimental PSNativeCommandArgumentPassing feature available since PowerShell Core 7.2.0-preview.5 may fix this at least for some external programs, including curl.exe; read on for details.
As of PowerShell 7.2.x, an unexpected extra layer of escaping of embedded " characters is needed, using \-escaping when calling (most) external programs:
In Windows PowerShell there are edge cases where this approach doesn't work, in which case use of --% is required (see below). Notably, escaping '"foo bar"' as '\"foo bar\"' doesn't work, due to the enclosing \" being at the very start and end of the string - see this answer for details.
Also, some external programs on Windows understand ""-escaping only (e.g. msiexec); for them, use -replace '"', '""' in order to programmatically perform the extra escaping, assuming the value contains at least one space. Do the same for programs that do not support embedded " chars. at all (WSH), so that the embedded " at least do not break argument boundaries (but they will be stripped).
For programs that expect \"-escaping, use the following -replace operation to robustly perform the extra escaping programmatically:
'...' -replace '([\\]*)"', '$1$1\"'
If the input string contains no preexisting verbatim \" sequences, you can simplify to '...' -replace '"', '\"'
# Note: Escaping the embedded " chars. as `" is enough for PowerShell itself,
# but, unfortunately, not when calling *external programs*.
# The `-replace` operation performs the necessary additional \-escaping.
$passwd = 'foo'
curl.exe -ku user#email:mypass -X PUT -d (
"data={`"password`": `"$passwd`"}" -replace '([\\]*)"', '$1$1\"'
) https://build.phonegap.com/api/v1/key
This shouldn't be required, but is due to a bug since v1 that hasn't been fixed for fear of breaking backward compatibility - see this answer.
A - presumably - opt-in fix is now being considered for some future version, post v7.2 - see GitHub issue #14747 - and using it, once available, would obviate the need for the manual escaping.
Since PowerShell Core 7.2.0-preview.5, experimental feature PSNativeCommandArgumentPassing with an attempted fix is available, but, unfortunately, it looks like it will lack important accommodations for high-profile CLIs on Windows (though curl.exe wouldn't be affected) - see this summary from GitHub issue #15143.
A backward- and forward-compatible helper function is the ie function from the Native module (Install-Module Native), which obviates the need for the extra escaping, contains important accommodations for high-profile CLIs on Windows, and will continue to work as expected even with the opt-in fix in place:
ie curl.exe ... -d "data={`"password`": `"$passwd`"}" ... )
Using --%, the stop-parsing symbol, as in Keith Hill's answer is a suboptimal workaround that also doesn't require the extra \-escaping, however:
--% has inherent limitations - see GitHub docs issue #6149 - and is virtually useless on Unix-like platforms - see GitHub docs issue #4963.
The only - awkward and side effect-producing - way to embed PowerShell variable values in the arguments following --% is to (a) define them as environment variables (e.g., $env:passwd = 'foo') and (b) to reference these variables cmd.exe-style, even on Unix (e.g., %passwd%).
An alternative workaround - especially if you need to include the values of PowerShell variables or expressions in your calls - is to call via cmd /c with a single argument containing the entire command line; for quoting convenience, the following example uses a here-string (see the bottom section of this answer for an overview of PowerShell's string literals):
# Use #"<newline>...<newline>"# if you need to embed PowerShell variables / expressions.
cmd /c #'
curl.exe -ku user#email:mypass -X PUT -d "data={\"password\":"\keypass\"}" https://build.phonegap.com/api/v1/key
'#
Here's how I did manage to send a json request in PowerShell 7.2
Finally found the solution. Istead of using one " use 3 of them ("""), and thats it. So it would be:
data={"""password""":"""keypass"""}
Set the content type:
curl -H "Content-Type: application/json" -d '{"password":"keypass"}' https://build.phonegap.com/api/v1/key