how to insert tab in sed command? - sed

I have a text file in which there's a YAML formatted code. Here's the code:
trigger:
branches:
include:
- develop
- release/*
- rapid/*
- Sprint/*
- Unity/*
Now, I have a bash script through which I want to add a string after this string; " - develop". Here's the point I want to maintain the spacing format.
Here;s my bash file code.
$target_string= " - develop"
$replacment_string=" - TerraformTest/*"
sed -i "/$target_string/a$replacment_string" test.txt
But the result I got it this:
trigger:
branches:
include:
- develop
- TerraformTest/*
- release/*
- rapid/*
- Sprint/*
- Unity/*
Is there any way, I can put tab in sed command to obtain the output like this:
trigger:
branches:
include:
- develop
- TerraformTest/*
- release/*
- rapid/*
- Sprint/*
- Unity/*

You could use a YAML-aware tool such as yq. This command inserts TerraformTest/* as the second element in the array at .trigger.branches.include:
yq '
.trigger.branches.include |=
([.[0], "TerraformTest/*", del(.[0])] | flatten)
' infile.yml

You can match match and grab space of search string in a capture group and use it in replacement:
sed -E 's~^([[:blank:]]*)- develop~&\n\1- TerraformTest/*~' file
trigger:
branches:
include:
- develop
- TerraformTest/*
- release/*
- rapid/*
- Sprint/*
- Unity/*
Or using variables:
target_string='- develop'
replacment_string='- TerraformTest/*'
sed -E "s~^([[:blank:]]*)$target_string~&\n\1$replacment_string~" file
trigger:
branches:
include:
- develop
- TerraformTest/*
- release/*
- rapid/*
- Sprint/*
- Unity/*
Breakup:
^: Start
([[:blank:]]*): Match 0 or more whitespaces in capture group #1
$target_string: Match string contained in $target_string
Replacement:
&: Place whole match back
\n: Place a line break
\1: Place spaces string on next line
$replacment_string: Place new replacement value
TO save changes inline use:
sed -i.bak -E "s~^([[:blank:]]*)$target_string~&\n\1$replacment_string~" file

With your shown samples only, please try following awk code. Written and tested in GNU awk.
awk -v RS= '
match($0,/^trigger:\n[[:space:]]+branches:\n[[:space:]]+include:\n[[:space:]]+- develop/){
print substr($0,RSTART,RLENGTH)"\n - TerraformTest/*" substr($0,RSTART+RLENGTH)
}
' Input_file

It is best practice to use a YAML aware tool such as yq, Perl, Python, etc.
Here is a Ruby:
ruby -r yaml -e 'y=YAML.load($<.read)
begin
a=y["trigger"]["branches"]["include"]
a.insert(a.find_index("develop")+1,"TerraformTest/*")
puts YAML.dump(y)
rescue
puts "Not Found"
end
' file
Prints:
---
trigger:
branches:
include:
- develop
- TerraformTest/*
- release/*
- rapid/*
- Sprint/*
- Unity/*

Using sed
$ sed -Ei.bak '/ +- develop/{p;s/[a-z]+/TerraformTest\/*/}' file
trigger:
branches:
include:
- develop
- TerraformTest/*
- release/*
- rapid/*
- Sprint/*
- Unity/*

Related

GitHub Actions Workflow Not Triggering

I have a project in which I have two yml files under .github/workflows/ as below:
build.yml
release.yml
I use annotated tags to do releases and here is how the trigger looks like in build.yml:
on:
push:
paths-ignore:
- 'images/**'
- README.md
branches:
- master
tags:
- 'v*.*.*'
pull_request:
branches:
- master
And here is how it looks like in release.yml:
on:
push:
# Sequence of patterns matched against refs/tags
tags:
- '[0-9]+.[0-9]+.[0-9]+'
I did the following to push a new annotated tag:
git tag -a v0.0.3-SNAPSHOT -m "My very third tag with release"
git push origin --tags
I was actually expecting my release.yml to get triggered, but it does not. Is there anything that I'm missing?
The regex will not match your tag 'v0.0.3-SNAPSHOT'. Missing the 'v' and the trailing text section. You could match it with the following:
- 'v[0-9]+.[0-9]+.[0-9]+-[a-zA-Z]*'
Example can be found here. Not sure why you cant use the '.*' as any character any number of times.
See working example here -> https://github.com/jnus/trigger-semver-tags/blob/main/.github/workflows/workflow.yml

Jobs in azure depending on jobs from other stages (inside templates)

Migrating from .gitlab-ci.yml to azure-pipelines.yml. In the .gitlab-ci.yml, I havee a scenario where one job(in the deploy stage) needs two other jobs(from test stages) for its execution
.deploy
stage: deploy
needs:
- testmethod1
- testmethod2
deployPROD:
extends: .deploy
Now here, does the deployProd job executes the testmethods again or just checks if they have been executed?
Moving to azure context, I created a templatefolder in my repository, with test file just to replicate this scenario.
My azure-pipelines.yml file is as shown below:
trigger:
- azure-pipelines
pool:
vmImage: ubuntu-latest
jobs:
- job: InitialA
steps:
- script: echo hello from initial A
- job: InitialB
steps:
- script: echo hello from initial B
- job: Subsequent
dependsOn:
- templates/test1.yml/testme
steps:
- script: echo hello from subsequent
I used the dependsOn key to show the depending jobs. Now the structure of the repo, along with the template file, looks like this:
But I end up getting the following error :
So is my approach correct? Am I using the correct keywords in azure? if yes, what is the path that I need to consider in the dependsOnkey?
Suggestions welcome.
You can add a template job and the Subsequent job can dependsOn the template job.
Please refer Template types & usage for more template usages in Azure Pipeline.
Code sample:
trigger:
- none
pool:
vmImage: ubuntu-latest
jobs:
- job: InitialA
steps:
- script: echo hello from initial A
- job: InitialB
steps:
- script: echo hello from initial B
- job: templates
steps:
- template: test.yml # Template reference
- job: Subsequent
dependsOn: templates
steps:
- script: echo hello from subsequent

YAML: Mapping values are not allowed in this context at line 4 column 5

This throws an error, any yaml experts can see why?
All my spaces and indentations seem to be correct:
trigger:​
- develop
pool:
vmImage: ubuntu-latest​
​
steps:​
- checkout: self
submodules: true
- task: AzureStaticWebApp#0​
inputs:​
app_location: "/" ​
api_location: "api​"
output_location: ""
env:​
azure_static_web_apps_api_token: $(deployment_token)
Most of your lines have the unicode character U+200B (zero width space) at the end of the line. Remove these characters, YAML doesn't recognize them as separation spaces, which is why parsing fails.

Can I substring a variable in Azure Pipelines?

I'm looking for a way to define a variable in my azure-pipelines.yml file where I can substring the 'Build.SourceVersion' -> Use the first 7 characters only.
Seems like there is no build-in function that can do such string operations in the documentation. Is there something I'm missing?
My other approach would be to use a bash task and overwrite the variable there but finding something build-in that can do this would be way better solution.
My other approach would be to use a bash task and overwrite the variable there but finding something build-in that can do this would be way better solution.
I agree with Lucas. There is no such built-in task to get the first 7 characters of $(Build.SourceVersion) in Azure DevOps.
We could use the command line/powershell task to split long sha into short sha:
echo $(Build.SourceVersion)
set TestVar=$(Build.SourceVersion)
set MyCustomVar=%TestVar:~0,7%
echo %MyCustomVar%
echo ##vso[task.setvariable variable=ShortSourceVersion]%MyCustomVar%
In this case, we could get the short versions of Build.SourceVersion and set it as environment variable.
Then we could set this command line task as a task group:
So, we could use this task to set the ShortSourceVersion directly.
Hope this helps.
Here is the shortest version that I use for this job;
- bash: |
longcommithash=$(Build.SourceVersion)
echo "##vso[task.setvariable variable=shorthash;]$(echo ${longcommithash::9})"
It gives you the output as shown below;
On Azure devops it's possible to be done using batch technique indeed - like other answers propose in here - however I wanted to remove string from the end of string - and with batch this gets more complex ( See following link ).
Then I've concluded to use Azure's built-in expressions, for example like this:
- name: sourceBranchName
value: ${{ replace(variables['Build.SourceBranch'], 'refs/heads/') }}
replace will have similar effect to substring - so it will remove unnecessary part from original string.
If however you want to concatenate some string and then remove it from original name - it's possible to be done using for instance format function:
- name: originBranchName
value: ${{ replace(variables.sourceBranchName, format('-to-{0}', variables.dynamicvar ) ) }}
If dynamicvar is a - then function will remove -to-a from branch name.
You are right, there is no native way to do it. You will have to write a script to transform the variable.
Here is an example:
trigger:
- master
resources:
- repo: self
stages:
- stage: Build
displayName: Build image
jobs:
- job: Build
displayName: Build
pool:
vmImage: 'ubuntu-latest'
steps:
- task: CmdLine#2
inputs:
script: ' x=`echo "$(Build.SourceVersion)" | head -c 7`; echo "##vso[task.setvariable variable=MyVar]$x"'
- task: CmdLine#2
inputs:
script: 'echo "$(MyVar)"'
Sure you can! If you absolutely must. Here is a runtime computation taking the first 7 characters of the Build.SourceVersion variable.
variables:
example: ${{ format('{0}{1}{2}{3}{4}{5}{6}', variables['Build.SourceVersion'][0], variables['Build.SourceVersion'][1], variables['Build.SourceVersion'][2], variables['Build.SourceVersion'][3], variables['Build.SourceVersion'][4], variables['Build.SourceVersion'][5], variables['Build.SourceVersion'][6]) }}
NB: I can't get it to work with $[...] syntax as the variable is apparently empty at the initial.

Make DEPLOY job automatic for release branches and manual for others with GitLab CI

The next solution should work.
deploy_release:
stage: deploy
tags:
- linux
only:
- master
- stable
retry: 2
script:
- do_action 1
- do_action 2
- git push artifacts
deploy_manual:
stage: deploy
tags:
- linux
except:
- master
- stable
when: manual
retry: 2
script:
- do_action 1
- do_action 2
- git push artifacts
But it has one ☝️ significant lack – script: is repeated 2 times.
I think it would be a good idea to write something like:
.deploy_base:
stage: deploy
tags:
- linux
retry: 2
script:
- do_action 1
- do_action 2
- git push artifacts
deploy_release:
include: .deploy_base
only:
- master
- stable
deploy_manual:
include: .deploy_base
except:
- master
- stable
when: manual
But I doubt this will work. Is it possible to do something similar in YAML?
The other straight-forward idea is to
move script: to separate file deploy_script.sh
and finish the problem in the bud.
Here it is
https://docs.gitlab.com/ce/ci/yaml/README.html#extends
extends
Introduced in GitLab 11.3
extends defines an entry name that a job, that uses extends is going to inherit from.
extends in an alternative to using YAML anchors that is a little more flexible and readable.
.tests:
only:
refs:
- branches
rspec:
extends: .tests
script: rake rspec
stage: test
only:
variables:
- $RSPEC
Thanks to this Q&A yaml repeated node that is a key
The solution is:
.deploy_base: &deploy_base
stage: deploy
tags:
- linux
retry: 2
script: &deploy_script
- do_action 1
- do_action 2
- git push artifacts
deploy_release:
only: &deploy_release_only
- master
- stable
script: *deploy_script
deploy_manual:
except: *deploy_release_only
when: manual
script: *deploy_script
And even better:
inherit .deploy_base:
.deploy_base: &deploy_base
stage: deploy
tags:
- DlpcsCore
- linux
retry: 2
variables:
URL: 'git#gitlab.com:Yahoo/HeavenShine-bin.git'
script: &deploy_script
- do_act_1
- do_action_2
deploy_release:
<< : *deploy_base
only: &deploy_release_only
- master
- stable
- CI
#- /^master[-_].+$/
#- /^(.+)[+]bin$/
deploy_manual:
<< : *deploy_base
except: *deploy_release_only
when: manual
To get know more search for YAML merge
Gitlab now support this via rules directive: https://docs.gitlab.com/ee/ci/yaml/#rules-clauses