Bosh Concourse error when trying to add external workers | failed to configure SSH server: ssh: no key found - worker

We are having a hard time setting up the cluster for an external worker. I think we don't understand all of the SSH keys that need to be created manually and exchanged. The TSA job errors with
failed to configure SSH server: ssh: no key found.
here is my manifest for bosh:
---
name: concourse
releases:
- name: concourse
version: latest
- name: garden-runc
version: latest
stemcells:
- alias: trusty
os: ubuntu-trusty
version: latest
instance_groups:
- name: web
instances: 1
vm_type: default
stemcell: trusty
azs: [z1]
networks: [{name: default}]
jobs:
- name: atc
release: concourse
properties:
no_really_i_dont_want_any_auth: true
postgresql_database: &atc_db atc
- name: tsa
release: concourse
properties:
host_key: ((tsa_host_private_key))
host_public_key: ((tsa_host_public_key))
authorized_keys:
- ((osx_worker_public_key))
- name: db
instances: 1
vm_type: default
stemcell: trusty
persistent_disk_type: default
azs: [z1]
networks: [{name: default}]
jobs:
- name: postgresql
release: concourse
properties:
databases:
- name: *atc_db
role: postgresql
password: password
- name: worker
instances: 1
vm_type: default
stemcell: trusty
azs: [z1]
networks: [{name: default}]
jobs:
- name: groundcrew
release: concourse
properties:
tsa:
host_public_key: ((tsa_host_public_key))
- name: baggageclaim
release: concourse
properties: {}
- name: garden
release: garden-runc
properties:
garden:
listen_network: tcp
listen_address: 0.0.0.0:7777
update:
canaries: 1
max_in_flight: 1
serial: false
canary_watch_time: 1000-60000
update_watch_time: 1000-60000

Related

terraform-cli : Duplicate value: "step-init". Maybe missing or invalid Task default/terraform-cli error

I've been attempting to use Tekton to deploy some AWS infrastructure via Terraform but not having much success.
The pipeline clones a Github repo containing TF code , it then attempts to use the terraform-cli task to provision the AWS infrastructure. For initial testing I just want to perform the initial TF init and provision the AWS VPC.
Expected behaviour
Clone Github Repo
Perform Terraform Init
Create the VPC using targeted TF apply
Actual Result
task terraform-init has failed: failed to create task run pod "my-infra-pipelinerun-terraform-init": Pod "my-infra-pipelinerun-terraform-init-pod" is invalid: spec.initContainers[1].name: Duplicate value: "step-init". Maybe missing or invalid Task default/terraform-cli
pod for taskrun my-infra-pipelinerun-terraform-init not available yet
Tasks Completed: 2 (Failed: 1, Cancelled 0), Skipped: 1
Steps to Reproduce the Problem
Prerequisites: Install Tekton command line tool, git-clone and terraform-cli
create this pipeline in Minikube
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: my-infra-pipeline
spec:
description: Pipeline for TF deployment
params:
- name: repo-url
type: string
description: Git repository URL
- name: branch-name
type: string
description: The git branch
workspaces:
- name: tf-config
description: The workspace where the tf config code will be stored
tasks:
- name: clone-repo
taskRef:
name: git-clone
workspaces:
- name: output
workspace: tf-config
params:
- name: url
value: $(params.repo-url)
- name: revision
value: $(params.branch-name)
- name: terraform-init
runAfter: ["clone-repo"]
taskRef:
name: terraform-cli
workspaces:
- name: source
workspace: tf-config
params:
- name: terraform-secret
value: "tf-auth"
- name: ARGS
value:
- init
- name: build-vpc
runAfter: ["terraform-init"]
taskRef:
name: terraform-cli
workspaces:
- name: source
workspace: tf-config
params:
- name: terraform-secret
value: "tf-auth"
- name: ARGS
value:
- apply
- "-target=aws_vpc.vpc -auto-approve"
Run the pipeline by creating a pipelinerun resource in k8s
Review the logs > tkn pipelinerun logs my-tf-pipeline -a
Additional Information
Pipeline version: v0.35.1
There is a known issue regarding "step-init" in some earlier versions - I suggest you upgrade to latest version (0.36.0) and try again.

concourse pipeline - throwing error for sub folders "not a valid repository name"

I have repository which has two frontend application and one server folder. i need to create pipeline for two frontend(angular) and one server(nodejs) folder. if I create a pipeline for main folder(concourse-pipeline) its working fine. but when I try to create to pipeline for subfolders(frontend) its throwing an error as "not a valid repository name". I'm not sure whats going wrong here.
- name: repo
type: git
source:
uri: git#github.com:test-repo/concourse-pipeline.git
branch: master
private_key: ((repo.private-key))
- name: frontend
type: git
source:
uri: git#github.com:test-repo/concourse-pipeline/frontend.git
branch: master
private_key: ((repo.private-key))
- name: version
type: semver
source:
driver: git
initial_version: 0.0.1
uri: git#github.com:test-repo/concourse-pipeline.git
private_key: ((repo.private-key))
branch: master
file: version
- name: run-server
type: git
source:
uri: git#github.com:test-repo/concourse-pipeline.git
branch: master
private_key: ((repo.private-key))
jobs:
- name: run-server
build_logs_to_retain: 20
max_in_flight: 1
plan:
- get: run-server
trigger: true
- task: run-tests
config:
platform: linux
image_resource:
type: registry-image
source:
repository: node
inputs:
- name: run-server
run:
path: /bin/sh
args:
- -c
- |
echo "Node Version: $(node --version)"
echo "NPM Version: $(npm --version)"
cd run-server
npm install
npm test
- name: run-frontend
build_logs_to_retain: 20
max_in_flight: 1
plan:
- get: frontend
trigger: true
- task: run-tests
config:
platform: linux
image_resource:
type: registry-image
source:
repository: node
inputs:
- name: frontend
run:
path: /bin/sh
args:
- -c
- |
echo "Node Version: $(node --version)"
echo "NPM Version: $(npm --version)"
cd frontend
npm install
ng test
- name: bump-version
plan:
- get: repo
trigger: true
- put: version
params:
bump: patch
- name: build-repo
plan:
- get: repo
trigger: true
- get: version
params:
build: repo
tag_file: version/version
tag_as_latest: true
Any help would be appreciated
The uri of the frontend resource seems invalid.
uri: git#github.com:test-repo/concourse-pipeline/frontend.git
The Github address should be only git#github.com:(user):(repository).git

How to wait until env for appid is created in jelastic manifest installation?

I have the following manifest:
jpsVersion: 1.3
jpsType: install
application:
id: shopozor-k8s-cluster
name: Shopozor k8s cluster
version: 0.0
baseUrl: https://raw.githubusercontent.com/shopozor/services/dev
settings:
fields:
- name: envName
caption: Env Name
type: string
default: shopozor
- name: topo
type: radio-fieldset
values:
0-dev: '<b>Development:</b> one master (1) and one scalable worker (1+)'
1-prod: '<b>Production:</b> multi master (3) with API balancers (2+) and scalable workers (2+)'
default: 0-dev
- name: version
type: string
caption: Version
default: v1.16.3
onInstall:
- installKubernetes
- enableSubDomains
actions:
installKubernetes:
install:
jps: https://github.com/jelastic-jps/kubernetes/blob/${settings.version}/manifest.jps
envName: ${settings.envName}
displayName: ${settings.envName}
settings:
deploy: cmd
cmd: |-
curl -fsSL ${baseUrl}/scripts/install_k8s.sh | /bin/bash
topo: ${settings.topo}
dashboard: version2
ingress-controller: Nginx
storage: true
api: true
monitoring: true
version: ${settings.version}
jaeger: false
enableSubDomains:
- jelastic.env.binder.AddDomains[cp]:
domains: staging,api-staging,assets-staging,api,assets
Unfortunately, when I run that manifest, the k8s cluster gets installed, but the subdomains cannot be created (yet), because:
[15:26:28 Shopozor.cluster:3]: enableSubDomains: {"action":"enableSubDomains","params":{}}
[15:26:29 Shopozor.cluster:4]: api [cp]: {"method":"jelastic.env.binder.AddDomains","params":{"domains":"staging,api-staging,assets-staging,api,assets"},"nodeGroup":"cp"}
[15:26:29 Shopozor.cluster:4]: ERROR: api.response: {"result":2303,"source":"JEL","error":"env for appid [5ce25f5a6988fbbaf34999b08dd1d47c] not created."}
What jelastic API methods can I use to perform the necessary waiting until subdomain creation is possible?
My current workaround is to split that manifest into two manifests: one cluster installation manifest and one update manifest creating the subdomains. However, I'd like to have everything in the same manifest.
Please change this:
enableSubDomains:
- jelastic.env.binder.AddDomains[cp]:
domains: staging,api-staging,assets-staging,api,assets
to:
enableSubDomains:
- jelastic.env.binder.AddDomains[cp]:
envName: ${settings.envName}
domains: staging,api-staging,assets-staging,api,assets

Concourse CI: static_buildpack issue

I am deploying a simple angular4 application to cloud foundry using static_buildpack. While accessing the application I am always getting nginx 403 issue.
jobs:
- name: app
serial: true
plan:
- get: develop-repo
- task: npm-build
config:
platform: linux
image_resource:
type: docker-image
source:
repository: node
run:
path: sh
args:
- -exec
- |
cd develop-repo
npm install
npm run dist
inputs:
- name: develop-repo
outputs:
- name:
- put: develop
params:
manifest: develop-repo/manifest.yml
current_app_name: app
path: develop-repo
resources:
- name: develop-repo
type: git
- name: develop
type: cf
manifest.yml:
---
applications:
- name: app
instances: 1
memory: 512M
disk_quota: 512M
buildpack: staticfile_buildpack
stack: cflinuxfs2
All I am doing is git clone -> npm build -> cf deploy
Note: All resource variables are rightly set. Just ignored for better readability
After trying out couple of options, I found that by publishing the artifacts to the output folder we can push the app from the output folder
---
jobs:
- name: app
serial: true
plan:
- get: develop
- task: npm-build
config:
platform: linux
image_resource:
type: docker-image
source:
repository: node
inputs:
- name: develop
outputs:
- name: artifacts
run:
path: sh
args:
- -exec
- |
cd develop
npm install
npm run dist
ls
cp -R dist ../artifacts/
- put: deploy-cf
params:
manifest: develop/ci/manifests/manifest-int.yml
path: artifacts/dist
resources:
- name: develop
type: git
source:
uri: <<GITHUB-URI>>
branch:<<GITHUB-BRANCH>>
username:<<GITHUB-USERNAME>>
password: <<GITHUB-PASSWORD>>
- name: deploy-cf
type: cf
source:
api: <<CF-API>>
username: <<CF-USERNAME>>
password: <<CF-PASSWORD>>
organization: <<CF-ORG>>
space: <<CF-SPACE>>

Cloud Foundry bosh Error 100: Can't find network

I'm attempting to setup a service broker to add postgres to our Cloud Foundry installation. We're running our system on vmWare. I'm using this release in order to do that:
cf-services-contrib-release
I need to setup the networks: section in the manifest, and what I'm setting there isn't working.
This is what my networks look like in the vmWare vCenter UI:
And this is what my clusters and resource pools look like in the vCenter UI:
I tried both with and without quotes, around the 'name' of the network. But I'm now getting an error saying that bosh can't find the network:
Failed compiling packages > rootfs_lucid64/9b3f611b46e076b94b37645c98f9100e7bcef5dd: Can't find network: VLAN1130_LB_100.114.130.0 (00:00:01)
Failed compiling packages > postgresql93/06163819b694f8d9836586d024f64c11efe30180: Can't find network: VLAN1130_LB_100.114.130.0 (00:00:01)
Failed compiling packages > postgresql92/2867893e714aae6e6b76bd06e7aa30d47023c46e: Can't find network: VLAN1130_LB_100.114.130.0 (00:00:01)
Error 100: Can't find network: VLAN1130_LB_100.114.130.0
Task 2430 error
This was my latest configuration attempt:
networks:
- name: default
type: manual
subnets:
- range: 100.114.130.0/24
gateway: 100.114.130.1
cloud_properties:
name: VLAN1130_LB_100.114.130.0
I also tried using single quotes as below. But I got the same error as above!
networks:
- name: default
type: manual
subnets:
- range: 100.114.130.0/24
gateway: 100.114.130.1
cloud_properties:
name: 'VLAN1130_LB_100.114.130.0'
Our network that we're on is this one: 100.114.130.0/24
So it makes sense to select VLAN1130_LB_100.114.130.0 in the config.
I've tried setting all of these options in the yaml file with no quotes. And none of them seem to work!
<ul>
<li>USH_UCS_CLOUD_FOUNDRY: <a href="https://gist.github.com/bluethundr/18ac490e96a5e02fad65">postgres_2432_debug.txt</li>
<li>USH_UCS_CLOUD_FOUNDRY_DVS: postgres_2433_debug.txt</li>
<li>USH_UCS_CLOUD_FO-DVUplinks-435272: postgres_2434_debug.txt </li>
<li>VLAN1129_LB_100.114.129.0: postgres_2435_debug.txt</li>
<li>VLAN1130_LB_100.114.130.0: postgres_2436_debug.txt</li>
<li>VLAN14-ESXI_MGMT-3.156.14.0: <a href="https://gist.github.com/bluethundr/dbde624e63842721a133">postgres_2437_debug.txt</li>
</ul>
I wouldn't expect VLAN1129_LB_100.114.129.0 to work, but I tried it anyway, just to be complete.
I've supplied debug dumps of each failed attempt next to each setting you see above. Surely one of them must work! But as you can see none of them did.
Here's my complete yaml file that I deployed with the 'bosh deploy' command:
name: cf-22b9f4d62bb6f0563b71
director_uuid: fd713790-b1bc-401a-8ea1-b8209f1cc90c
releases:
- name: cf-services-contrib
version: 6
compilation:
workers: 3
network: default
reuse_compilation_vms: true
cloud_properties:
ram: 5120
disk: 10240
cpu: 2
update:
canaries: 1
canary_watch_time: 30000-60000
update_watch_time: 30000-60000
max_in_flight: 4
networks:
- name: default
type: manual
subnets:
- range: 100.114.130.0/24
gateway: 100.114.130.1
cloud_properties:
name: VLAN1130_LB_100.114.130.0
resource_pools:
- name: 'USH_UCS_CLOUD_FOUNDRY_NONPROD_01_RP'
network: default
stemcell:
name: bosh-vsphere-esxi-ubuntu-trusty-go_agent
version: '2865.1'
cloud_properties:
cpu: 2
ram: 4096
disk: 10240
datacenters:
- name: 'Universal City'
clusters:
- USH_UCS_CLOUD_FOUNDRY_NONPROD_01: {resource_pool: 'USH_UCS_CLOUD_FOUNDRY_NONPROD_01_RP'}
jobs:
- name: gateways
release: cf-services-contrib
templates:
- name: postgresql_gateway_ng
instances: 1
resource_pool: 'USH_UCS_CLOUD_FOUNDRY_NONPROD_01_RP'
networks:
- name: default
default: [dns, gateway]
properties:
# Service credentials
uaa_client_id: "cf"
uaa_endpoint: http://uaa.devcloudwest.example.com
uaa_client_auth_credentials:
username: admin
password: secret
- name: postgresql_service_node
release: cf-services-contrib
template: postgresql_node_ng
instances: 1
resource_pool: 'USH_UCS_CLOUD_FOUNDRY_NONPROD_01_RP'
persistent_disk: 10000
properties:
postgresql_node:
plan: default
networks:
- name: default
default: [dns, gateway]
properties:
networks:
apps: default
management: default
cc:
srv_api_uri: http://api.devcloudwest.example.com
nats:
address: 100.114.130.11
port: 25555
user: nats #CHANGE
password: secret
authorization_timeout: 5
service_plans:
postgresql:
default:
description: "Developer, 250MB storage, 10 connections"
free: true
job_management:
high_water: 230
low_water: 20
configuration:
capacity: 125
max_clients: 10
quota_files: 4
quota_data_size: 240
enable_journaling: true
backup:
enable: false
lifecycle:
enable: false
serialization: enable
snapshot:
quota: 1
postgresql_gateway:
token: f75df200-4daf-45b5-b92a-cb7fa1a25660
default_plan: default
supported_versions: ["9.3"]
version_aliases:
current: "9.3"
cc_api_version: v2
postgresql_node:
supported_versions: ["9.3"]
default_version: "9.3"
max_tmp: 900
password: secret
How can we get past this issue?
From Amit's comment:
The name used in Cloud Properties must include any nested sub-folders. In the provided configuration the network is nested under USH_UCS_CLOUD_FOUNDRY, so the value for name should reflect that, i.e. USH_UCS_CLOUD_FOUNDRY/VLAN1130_LB_100.114.130.0 no quotes are required.
networks:
- name: default
type: manual
subnets:
- range: 100.114.130.0/24
gateway: 100.114.130.1
cloud_properties:
name: USH_UCS_CLOUD_FOUNDRY/VLAN1130_LB_100.114.130.0