Concourse CI git clone multiple repos to same directory - concourse

Can someone help me to achieve my request. I would like to git clone multiple repos into the same directory where the first git repo is downloaded. Below is my pipeline.yml file. Any help is much appreciated.
resources:
- name: workspace-repo1
type: git
source:
uri: <git-repo1-url>
branch: master
private_key: ((publishing-outputs-private-key))
- name: workspace-repo2
type: git
source:
uri: <git-repo2-url>
branch: master
private_key: ((publishing-outputs-private-key))
- name: workspace-repo3
type: git
source:
uri: <git-repo3-url>
branch: master
private_key: ((publishing-outputs-private-key))
jobs:
- name: job-test
public: true
plan:
- get: workspace-repo1
- get: workspace-repo2
- get: workspace-repo3
- task: app-tests
config:
platform: linux
image_resource:
type: docker-image
source: {repository: golang, tag: 1.14.15-stretch}
inputs:
- name: workspace-repo1 (git repo1, repo2,repo3 should be downloaded here)
outputs:
- name: workspace-repo-deb
run:
path: workspace-repo1/local_build.sh

You could potentially do something like
- task: prep
file: scripts/merge.yml
input_mapping: {repo1: workspace-1, repo1: workspace-2, ...}
---
platform: linux
image_resource:
type: docker-image
source:
repository: alpine
params:
#this could whatever path you want
SRC_PATH: src
inputs:
- name: ci-scripts
- name: repo1
- name: repo2
run:
path: scripts/merge.sh
#this output could be use in the next step, it contains the directories you copied in the script below
outputs:
- name: build
#!/bin/sh
set -e
mkdir -p build
#copy around anything you need
cp -rf repo1/../../. build/
cp -rf repo2/../../. build/
cp -rf repo3/../../. build/
#listing content of build folder just checking
ls -la build/

Related

Github action pointing to a different directory than the current directory

When running an action during a pull request the path emitted by mkdir while trying to create a folder was different than the current directory set by a previous step. This also continues to the next steps as follows..
name: Publish demo
on:
push:
branches:
- 'develop'
pull_request:
branches:
- 'develop'
jobs:
web-deploy:
name: Deploy
runs-on: windows-latest
steps:
- name: Get latest code
uses: actions/checkout#v3
- name: Setup MSBuild
uses: microsoft/setup-msbuild#v1
- name: Setup NuGet
uses: NuGet/setup-nuget#v1.1.1
- name: Navigate to Workspace
run: cd ${{ github.workspace }}/demo_project
- name: Create Build Directory
run: mkdir _build
- name: Restore Packages
run: nuget restore demo_project.csproj
- name: Build Solution
run: |
msbuild.exe demo_project.csproj /nologo /nr:false /p:DeployOnBuild=true /p:DeployDefaultTarget=WebPublish /p:WebPublishMethod=FileSystem /p:DeleteExistingFiles=True /p:platform="Any CPU" /p:configuration="Release" /p:PublishUrl="_build"
- name: Sync files
uses: SamKirkland/FTP-Deploy-Action#4.3.3
with:
local-dir: "_build"
server: <server>
username: <username>
password: ${{ secrets.password }}
So, in this line..
run: mkdir _build
the _build folder should be created in demo_project but instead it gets created in ${{ github.workspace }} which i think means that setting the current directory here..
run: cd ${{ github.workspace }}/demo_project
which in turn prints out this in job view..
Run cd D:\a\demo_solution\demo_solution/demo_project
and..
Input file does not exist: D:\demo_project.csproj.
did not take effect. So, what am i missing?
you could specify a different working directory for a job:
You can provide default shell and working-directory options for all run steps in a job
jobs:
web-deploy:
name: Deploy
runs-on: windows-latest
defaults:
run:
shell: bash
working-directory: ${{ github.workspace }}/demo_project
steps:
- name: Get latest code
uses: actions/checkout#v3

concourse pipeline - throwing error for sub folders "not a valid repository name"

I have repository which has two frontend application and one server folder. i need to create pipeline for two frontend(angular) and one server(nodejs) folder. if I create a pipeline for main folder(concourse-pipeline) its working fine. but when I try to create to pipeline for subfolders(frontend) its throwing an error as "not a valid repository name". I'm not sure whats going wrong here.
- name: repo
type: git
source:
uri: git#github.com:test-repo/concourse-pipeline.git
branch: master
private_key: ((repo.private-key))
- name: frontend
type: git
source:
uri: git#github.com:test-repo/concourse-pipeline/frontend.git
branch: master
private_key: ((repo.private-key))
- name: version
type: semver
source:
driver: git
initial_version: 0.0.1
uri: git#github.com:test-repo/concourse-pipeline.git
private_key: ((repo.private-key))
branch: master
file: version
- name: run-server
type: git
source:
uri: git#github.com:test-repo/concourse-pipeline.git
branch: master
private_key: ((repo.private-key))
jobs:
- name: run-server
build_logs_to_retain: 20
max_in_flight: 1
plan:
- get: run-server
trigger: true
- task: run-tests
config:
platform: linux
image_resource:
type: registry-image
source:
repository: node
inputs:
- name: run-server
run:
path: /bin/sh
args:
- -c
- |
echo "Node Version: $(node --version)"
echo "NPM Version: $(npm --version)"
cd run-server
npm install
npm test
- name: run-frontend
build_logs_to_retain: 20
max_in_flight: 1
plan:
- get: frontend
trigger: true
- task: run-tests
config:
platform: linux
image_resource:
type: registry-image
source:
repository: node
inputs:
- name: frontend
run:
path: /bin/sh
args:
- -c
- |
echo "Node Version: $(node --version)"
echo "NPM Version: $(npm --version)"
cd frontend
npm install
ng test
- name: bump-version
plan:
- get: repo
trigger: true
- put: version
params:
bump: patch
- name: build-repo
plan:
- get: repo
trigger: true
- get: version
params:
build: repo
tag_file: version/version
tag_as_latest: true
Any help would be appreciated
The uri of the frontend resource seems invalid.
uri: git#github.com:test-repo/concourse-pipeline/frontend.git
The Github address should be only git#github.com:(user):(repository).git

Not able to trigger jobs one after the other using gcs-resource in concourse

I have two jobs viz. build and publish. I want publish to trigger after build is done. So, I am using an external resource gcs-resourcehttps://github.com/frodenas/gcs-resource
Following is my pipeline.yml:
---
resource_types:
- name: gcs-resource
type: docker-image
source:
repository: frodenas/gcs-resource
resources:
- name: proj-repo
type: git
source:
uri: <my uri>
branch: develop
username: <username>
password: <password>
- name: proj-gcr
type: docker-image
source:
repository: asia.gcr.io/myproject/proj
tag: develop
username: _json_key
password: <my password>
- name: proj-build-output
type: gcs-resource
source:
bucket: proj-build-deploy
json_key: <my key>
regexp: Dockerfile
jobs:
- name: build
serial_groups: [proj-build-deploy]
plan:
- get: proj
resource: proj-repo
- task: build
config:
platform: linux
image_resource:
type: docker-image
source: {repository: node, tag: 10.13.0}
inputs:
- name: proj
run:
path: sh
args:
- -exc
- |
<do something>
- put: proj-build-output
params:
file: proj/Dockerfile
content_type: application/octet-stream
- name: publish
serial_groups: [proj-build-deploy]
plan:
- get: proj-build-output
trigger: true
passed: [build]
- put: proj-gcr
params:
build: proj-build-output
I am using the external resource proj-build-output to trigger the next job. I can run the individual jobs without any problem, however the the publish job doesn't automatically get triggered after completion of build job.
Am I missing something?
The regexp of the gcs-resource is misconfigured:
...
regexp: Dockerfile
...
while regexp, as the original S3 resource from which it comes from, wants:
regexp: the pattern to match filenames against within GCS. The first grouped match is used to extract the version, or if a group is explicitly named version, that group is used.
The https://github.com/frodenas/gcs-resource#example-configuration shows its correct usage:
regexp: directory_on_gcs/release-(.*).tgz
This is not specific to the GCS or S3 resource; Concourse needs a "version" to move artifacts from jobs to storage and back. It is one of the fundamental concepts of Concourse. See https://web.archive.org/web/20171205105324/http://concourse.ci:80/versioned-s3-artifacts.html for an example.
As Marco mentioned, the problem was with versioning.
I solved my issue using these two steps:
Enabled versioning on my GCS Bucket https://cloud.google.com/storage/docs/object-versioning#_Enabling
Replaces regexp with versioned_file as mentioned in the docs https://github.com/frodenas/gcs-resource#file-names

Concourse CI: static_buildpack issue

I am deploying a simple angular4 application to cloud foundry using static_buildpack. While accessing the application I am always getting nginx 403 issue.
jobs:
- name: app
serial: true
plan:
- get: develop-repo
- task: npm-build
config:
platform: linux
image_resource:
type: docker-image
source:
repository: node
run:
path: sh
args:
- -exec
- |
cd develop-repo
npm install
npm run dist
inputs:
- name: develop-repo
outputs:
- name:
- put: develop
params:
manifest: develop-repo/manifest.yml
current_app_name: app
path: develop-repo
resources:
- name: develop-repo
type: git
- name: develop
type: cf
manifest.yml:
---
applications:
- name: app
instances: 1
memory: 512M
disk_quota: 512M
buildpack: staticfile_buildpack
stack: cflinuxfs2
All I am doing is git clone -> npm build -> cf deploy
Note: All resource variables are rightly set. Just ignored for better readability
After trying out couple of options, I found that by publishing the artifacts to the output folder we can push the app from the output folder
---
jobs:
- name: app
serial: true
plan:
- get: develop
- task: npm-build
config:
platform: linux
image_resource:
type: docker-image
source:
repository: node
inputs:
- name: develop
outputs:
- name: artifacts
run:
path: sh
args:
- -exec
- |
cd develop
npm install
npm run dist
ls
cp -R dist ../artifacts/
- put: deploy-cf
params:
manifest: develop/ci/manifests/manifest-int.yml
path: artifacts/dist
resources:
- name: develop
type: git
source:
uri: <<GITHUB-URI>>
branch:<<GITHUB-BRANCH>>
username:<<GITHUB-USERNAME>>
password: <<GITHUB-PASSWORD>>
- name: deploy-cf
type: cf
source:
api: <<CF-API>>
username: <<CF-USERNAME>>
password: <<CF-PASSWORD>>
organization: <<CF-ORG>>
space: <<CF-SPACE>>

Concourse CI, get and put a git-resource

I'm trying to use a git-resource to get, modify and push a file but it isn't work, can someone help me?
The two resources point to the same git repository and the goal is to add a file in the repository.
I can't understand where I'm wrong, concourse output is green but the repository doesn't have the new file
This is the job:
jobs:
- name: myjob
plan:
- get: input-repo
- get: output-repo
- task: simpletask
config:
platform: linux
image_resource:
type: docker-image
source:
repository: ubuntu
run:
path: sh
args:
- -exc
- |
cat a_file > output-repo/another_file
inputs:
- name: input-repo
- name: output-repo
- put: input-repo
params: { repository: output-repo }
you should not need two different resources for this. What you want to do is to get (git clone) the repo, modify it, git commit it and then put (git push) it.
So you want something like this.
jobs:
- name: myjob
plan:
- get: repo
- task: simpletask
config:
platform: linux
image_resource:
type: docker-image
source:
repository: ubuntu
run:
path: sh
args:
- -exc
- |
cp -r repo changed-repo
cat a_file > changed-repo/another_file
git add .
git commit -m "message"
inputs:
- name: repo
- put: repo
params: { repository: changed-repo }