I want to replace a variable in my .yml file with a variable from the gitlab ci pipeline.
gitlab-ci.yml
deploy_test:
stage: deploy
script:
- sed -i 's/$TAG/$CI_COMMIT_REF_NAME/g' deploy/test.yml
- kubectl apply -f deploy/test.yml
when: manual
only:
- master
- tags
This says within the deploy/test.yml file it should replace $TAG with the value of $CI_COMMIT_REF_NAME?
deploy/test.yml
image: git.<removed>.com:5005/eng/my-group/my-image:$TAG
Use double quotes(") instead of single quotes (') in sed and also take note that forward slashes (/) need to be escaped too (like \/).
so given the following (assuming you are using docker hub)
$CI_COMMIT_REF_NAME=docker_user/repo:v1
You will firstly need to escape the '/' character first, like so
$CI_COMMIT_REF_NAME=docker_user\/repo:v1
Then finally use double quotes in the sed command
sed -i "s/$TAG/$CI_COMMIT_REF_NAME/g" deploy/test.yml
Note: In addition to this you can also prefer using | instead of / if there's too many escapes.
Like:
sed -i "s|$TAG|$CI_COMMIT_REF_NAME|g" deploy/test.yml
If your string consists of / , as a workaroud you can use different delimiting character instead of /
for example,
sed -i -E "s%BASE_URL: '.*'%BASE_URL: '\$BACKEND_URL'%g" $FRONTEND_PATH/app/config/index.js
Here I am using % as delimiting character as none of my strings has that character, you can use any other depending on the symbols used in your strings.
- sed -i "s|TAG|${CI_COMMIT_REF_NAME}|g" deploy/test.yml
and for simpler situation remove the $ from $TAG, so the deploy/test.yml:
image: git.<removed>.com:5005/eng/my-group/my-image:TAG
Related
I have file (db-chk.txt) which has multiple rows which start with single quote and I would like to search for this pattern using bash "sed" and remove all those lines which start with single quote (').
Tried few option, but they don't work
$ sed '/^'-/d' db-check.txt -- This one don't work
$ sed '/^\'-/d' db-check.txt
e.g. Sample data from db-check.txt which i want to get these lines removed.
---[BLOCKED/-SESSION]--
----[SESSION/-COUNT]---
In case you don't have any parameter expansion in your sed command, you can use double quote (as mentioned in comment):
sed -i "/'^-/d" db-check.txt
If you don't want parameter expansion, use single quote:
sed -i '/^\x27-/d' db-check.txt
\x27 is the hex code for the '
Is there a way in (Gnu) sed to replace all characters in a matching part of a string? For example I might have a list of file paths with several (arbitrary number of) paths in each line, e.g.:
/a/b/c/d/e /f/g/XXX/h/i /j/k/l/m
/n/o/p /q/r/s/t/u /v/x/x/y
/z/XXX/a/b /c/d/e/f
I would like to replace all the slashes in paths containing XXX keping all the others untouched, e.g.:
/a/b/c/d/e #f#g#XXX#h#i /j/k/l/m
/n/o/p /q/r/s/t/u /v/x/x/y
#z#XXX#a#b /c/d/e/f
Unfortunately I cannot come up with a solution. Maybe it's even impossible with sed. But I'm curious if somebody find a way to solve the problem.
We can replace any / preceding XXX with no intervening spaces like this:
# Using extended regex syntax
s!/([^ ]*XXX)!#\1!
It's a very similar substitution for those that follow XXX.
Putting them together in a loop makes this program:
#!/bin/sed -rf
:loop
s!/([^ ]*XXX)!#\1!
s!(XXX[^ ]*)/!\1#!
tloop
Output:
/a/b/c/d/e #f#g#XXX#h#i /j/k/l/m
/n/o/p /q/r/s/t/u /v/x/x/y
#z#XXX#a#b /c/d/e/f
That said, it might be simpler to use a pipeline, to break the file paths into individual lines and then reassemble them after the substitution:
sed -e 's/ *$//;s/ */&\n/g' \
| sed -e '/XXX/y,/,#,' \
| sed -e ':a;/ $/{N;s/\n//;ba}'
I'm using sed to replace some strings in some of my files. I'm trying to match a string like "namespace Foo\BarBundle\Tests\blah\blah" with this pattern:
^\\(namespace\|use\\)\s\*Foo\BarBundle\Tests\\(.\*\\)$
But it's not working. The Complete command goes as follows:
sed -i -e "s/$pattern/\1 Tests\\Foo\BarBundle\2/g" <file_name>
Where $pattern is the pattern stated above. (which is the output from echo $pattern).
I've ran it on multiple files to no avail. Is there something wrong with the first pattern?
Use 3 backslashes \\\ to specify a regular backsplash \:
pattern="^\(namespace\|use\)\s*Foo\\\BarBundle\\\Tests\(.*\)$"
sed -e "s/$pattern/\1 Tests\\\Foo\\\BarBundle\2/g" <file_name>
The first two \\ become one in the shell which then escape the third one \ in sed
You have to use 4 backslashes if you want to match a single backslash - both in standard input and in the script.
This question already has answers here:
Using different delimiters in sed commands and range addresses
(3 answers)
Closed 1 year ago.
I have a Visual Studio project, which is developed locally. Code files have to be deployed to a remote server. The only problem is the URLs they contain, which are hard-coded.
The project contains URLs such as ?page=one. For the link to be valid on the server, it must be /page/one .
I've decided to replace all URLs in my code files with sed before deployment, but I'm stuck on slashes.
I know this is not a pretty solution, but it's simple and would save me a lot of time. The total number of strings I have to replace is fewer than 10. A total number of files which have to be checked is ~30.
An example describing my situation is below:
The command I'm using:
sed -f replace.txt < a.txt > b.txt
replace.txt which contains all the strings:
s/?page=one&/pageone/g
s/?page=two&/pagetwo/g
s/?page=three&/pagethree/g
a.txt:
?page=one&
?page=two&
?page=three&
Content of b.txt after I run my sed command:
pageone
pagetwo
pagethree
What I want b.txt to contain:
/page/one
/page/two
/page/three
The easiest way would be to use a different delimiter in your search/replace lines, e.g.:
s:?page=one&:pageone:g
You can use any character as a delimiter that's not part of either string. Or, you could escape it with a backslash:
s/\//foo/
Which would replace / with foo. You'd want to use the escaped backslash in cases where you don't know what characters might occur in the replacement strings (if they are shell variables, for example).
The s command can use any character as a delimiter; whatever character comes after the s is used. I was brought up to use a #. Like so:
s#?page=one&#/page/one#g
A very useful but lesser-known fact about sed is that the familiar s/foo/bar/ command can use any punctuation, not only slashes. A common alternative is s#foo#bar#, from which it becomes obvious how to solve your problem.
add \ before special characters:
s/\?page=one&/page\/one\//g
etc.
In a system I am developing, the string to be replaced by sed is input text from a user which is stored in a variable and passed to sed.
As noted earlier on this post, if the string contained within the sed command block contains the actual delimiter used by sed - then sed terminates on syntax error. Consider the following example:
This works:
$ VALUE=12345
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
MyVar=12345
This breaks:
$ VALUE=12345/6
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
sed: -e expression #1, char 21: unknown option to `s'
Replacing the default delimiter is not a robust solution in my case as I did not want to limit the user from entering specific characters used by sed as the delimiter (e.g. "/").
However, escaping any occurrences of the delimiter in the input string would solve the problem.
Consider the below solution of systematically escaping the delimiter character in the input string before having it parsed by sed.
Such escaping can be implemented as a replacement using sed itself, this replacement is safe even if the input string contains the delimiter - this is since the input string is not part of the sed command block:
$ VALUE=$(echo ${VALUE} | sed -e "s#/#\\\/#g")
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
MyVar=12345/6
I have converted this to a function to be used by various scripts:
escapeForwardSlashes() {
# Validate parameters
if [ -z "$1" ]
then
echo -e "Error - no parameter specified!"
return 1
fi
# Perform replacement
echo ${1} | sed -e "s#/#\\\/#g"
return 0
}
this line should work for your 3 examples:
sed -r 's#\?(page)=([^&]*)&#/\1/\2#g' a.txt
I used -r to save some escaping .
the line should be generic for your one, two three case. you don't have to do the sub 3 times
test with your example (a.txt):
kent$ echo "?page=one&
?page=two&
?page=three&"|sed -r 's#\?(page)=([^&]*)&#/\1/\2#g'
/page/one
/page/two
/page/three
replace.txt should be
s/?page=/\/page\//g
s/&//g
please see this article
http://netjunky.net/sed-replace-path-with-slash-separators/
Just using | instead of /
Great answer from Anonymous. \ solved my problem when I tried to escape quotes in HTML strings.
So if you use sed to return some HTML templates (on a server), use double backslash instead of single:
var htmlTemplate = "<div style=\\"color:green;\\"></div>";
A simplier alternative is using AWK as on this answer:
awk '$0="prefix"$0' file > new_file
You may use an alternative regex delimiter as a search pattern by backs lashing it:
sed '\,{some_path},d'
For the s command:
sed 's,{some_path},{other_path},'
I have a file named check.txt which has the below contents:
$ cat check.txt
~/bin/tibemsadmin -server $URL-user $USER -password $PASWRD
$
I have a main script where the values of $URL, $USER, $PASWRD are obtained from the main script. I want to use the SED utility to replace the $URL, $USER, $PASWRD to the actual values in the check.txt.
I am trying like this but it fails.
emsurl=tcp://myserver:3243
emsuser=test
emspasswd=new
sed s/$URL/${emsurl}/g check.txt >> check_new.txt
sed s/$USER/${emsuser}/g check.txt_new.txt >> check_new_1.txt
sed s/PASWRD/${emspasswd}/g check_new_1.txt >> final.txt
My final.txt output is desired as below:
~/bin/tibemsadmin -server tcp://myserver:3243 -user test -password new
Could you please help me?
You have to be rather careful with your use of quotes. You also need to learn how to do multiple operations in a single pass, and/or how to use pipes.
emsurl=tcp://myserver:3243
emsuser=test
emspasswd=new
sed -e "s%\$URL%${emsurl}%g" \
-e "s%\$USER%${emsuser}%g" \
-e "s%\$PASWRD%${emspasswd}%g" check.txt >final.txt
Your problem is that the shell expanded the '$URL' in your command line (probably to nothing), meaning that sed got to see something other than what you intended. By escaping the $ with the \, sed gets to see what you intended.
Note that I initially used / as the separator in the substitute operations; however, as DarkDust rightly points out, that won't work since there are slashes in the URLs. My normal fallback character is % - as now shown - but that can appear in some URLs and might not be appropriate. I'd probably use a control character, such as control-A, if I needed to worry about that - or I'd use Perl which would be able to play without getting confused.
You can also combine the three separate -e expressions into one with semi-colons replacing them. However, I prefer the clarity of the three operations clearly separated.
You could take a slightly different approach by modifying your main script as follows :-
export URL="tcp://myserver:3243"
export USER=test
export PASWRD=new
. ./check.txt
This sets up the variables and then runs check.txt within the context of your main script
Although you don't say what's failing I guess I see the problems.
I suggest you do this:
sed "s|\$URL|${emsurl}|g"
That is, the first $ needs to be escaped because you want it literally. Then, instead of / I suggest you use | (pipe) as delimiter since it's not used in your strings. Finally, use " to ensure the content is interpreted as string by the shell.
You can then pipe everything together to not need any temporary files:
sed "s|\$URL|${emsurl}|g" | sed "s|\$USER|${emsuser}|g" | sed "s|\$PASSWRD|${emspasswd}|g"
Variable substitution should be outside sed expression and '$' should be escaped; in your case something like this:
sed -e 's/\$URL/'$emsurl'/g' -e 's/\$USER/'$emsuser'/g' -e 's/\$PASSWORD/'$emaspasswd'/g'
Anyway in your place I would avoid using $ to match placeholders in a template file, because it's causing confusion with BASH variables, use a different pattern instead (for instance #URL#).