Why isn't Cloud Code honoring my cloudbuild.yaml file but gcloud beta builds submit is? - kubernetes

I am using Google's Cloud Code extension with Visual Studio Code to use GCP's Cloud Build and deploy to a local kubernetes cluster (Docker Desktop). I have directed Cloud Build to run unit tests after installing modules.
When I build using the command line gcloud beta builds submit, the Cloud Build does the module install and successfully fails to build because I intentionally wrote a failing unit test. So that's great.
However, when I try to build and deploy using the Cloud Code extension, it is not using my cloudbuild.yaml at all. I know this because the
1.) The build succeeds even with the failing unit test
2.) No logging from the unit test appears in GCP logging
3.) I completely deleted cloudbuild.yaml and the build / deploy still succeeded, which seems to imply Cloud Code is using Dockerfile
What do I need to do to ensure Cloud Code uses cloudbuild.yaml for its build/deploy to a local instance of kubernetes?
Thanks!
cloudbuild.yaml
steps:
- name: node
entrypoint: npm
args: ['install']
- id: "test"
name: node
entrypoint: npm
args: ['test']
options:
logging: CLOUD_LOGGING_ONLY
scaffold.yaml
apiVersion: skaffold/v2beta19
kind: Config
build:
tagPolicy:
sha256: {}
artifacts:
- context: .
image: genesys-gencloud-dev
deploy:
kubectl:
manifests:
- kubernetes-manifests/**
profiles:
- name: cloudbuild
build:
googleCloudBuild: {}
launch.json
{
"configurations": [
{
"name": "Kubernetes: Run/Debug - cloudbuild",
"type": "cloudcode.kubernetes",
"request": "launch",
"skaffoldConfig": "${workspaceFolder}\\skaffold.yaml",
"profile": "cloudbuild",
"watch": true,
"cleanUp": false,
"portForward": true,
"internalConsoleOptions": "neverOpen",
"imageRegistry": "gcr.io/my-gcp-project",
"debug": [
{
"image": "my-image-dev",
"containerName": "my-container-dev",
"sourceFileMap": {
"${workspaceFolder}": "/WORK_DIR"
}
}
]
}
]
}

You will need to edit your skaffold.yaml file to use Cloud Build:
build:
googleCloudBuild: {}
See https://skaffold.dev/docs/pipeline-stages/builders/#remotely-on-google-cloud-build for more details.
EDIT: It looks like your skaffold.yaml enables cloud build for the cloudbuild profile, but that the profile isn't active.
Some options:
Add "profile": "cloudbuild" to your launch.json for 'Run on Kubernetes'.
Screenshot
Move the googleCloudBuild: {} to the top-level build: section. (In other words, skip using the profile)
Activate the profile using one of the other methods from https://skaffold.dev/docs/environment/profiles/#activation
UDPATE (from asker)
I needed to do the following:
Update skaffold.yaml as follows. In particular note the image, field under build > artifacts, and projectId field under profiles > build.
apiVersion: skaffold/v2beta19
kind: Config
build:
tagPolicy:
sha256: {}
artifacts:
- context: .
image: gcr.io/my-project-id/my-image
deploy:
kubectl:
manifests:
- kubernetes-manifests/**
profiles:
- name: cloudbuild
build:
googleCloudBuild:
projectId: my-project-id
Run this command to activate the profile: skaffold dev -p cloudbuild

Related

Azure Function run in docker compose but debug not working

I create sample project with Azure functions & Dapr and running in Linux Docker. I use docker compose and yml file to start project. But there are few issues. I'm using VS 2022 and VS code.
First in VS 2022 it will not start for Azure Function project. It works for asp.net Web and API. For Azure Function project it will be getting error as project not support running in Linux. Then I open in VS code command "docker compose up". Projects runs fine, but it was not able to attach debugger on it.
I first tried in VS 2022. followed instruction. I can see my container in list as follows.
After Attached container, it keep runs but not stop at any breakpoint.
Then I tried in VS code, in container I have mount vsdbg to /remote_debugger
Here is my docker-compose.yml file
daprfunctester:
image: ${DOCKER_REGISTRY-}daprfunctester
container_name: "daprfunctester"
build:
context: .
dockerfile: DaprFuncTester/Dockerfile
environment:
- DAPR_HTTP_PORT=3500
- AzureWebJobsStorage
- DOTNET_USE_POLLING_FILE_WATCHER
- NUGET_FALLBACK_PACKAGES
- NUGET_PACKAGES
- ASPNETCORE_CONTENTROOT=/azure-functions-host
- FUNCTIONS_WORKER_RUNTIME=dotnet
- DOTNET_RUNNING_IN_CONTAINER=true
- PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
volumes:
- F:\Work\Moneycorp\CodeForFun\DurableForFun:/src
- C:\Users\winso\vsdbg\vs2017u5:/remote_debugger:rw
- C:\Users\winso\AppData\Local\AzureFunctionsTools\Containers_2167102\Releases\4.20.0\linuxCLI:/functions_debugging_cli:ro
- ~/.nuget/packages:/root/.nuget/packages:ro
- C:\Users\winso\.nuget\packages:/root/.nuget/fallbackpackages:ro
networks:
- hello-dapr
And launch.json file
{
"name": "Docker .NET Core Attach (Preview)",
"type": "docker",
"request": "attach",
"platform": "netCore",
"netCore": {
"debuggerPath": "/remote_debugger/vsdbg",
//"appProject": "${workspaceFolder}/DaprFuncTester/DaprFuncTester.csproj",
//"enableDebugging": true
},
"sourceFileMap": {
"/home/site/wwwroot": "${workspaceFolder}/DaprFuncTester"
},
"processName": "Microsoft.Azure.WebJobs.Script.WebHost"
//"processId":"${command:pickProcess}"
},
When I do F5 run debugger, it was running and if I try to breakpoint, it also stop the Thread due to breakpoint, but it doesn't interact with code, it doesn't indicate or have pointer to the line of code. If press F10 "step over" it will pop error "Failed to step". I don't able to check any variable values. If I check on .Net ThreadPool Worker, it has message "Error processing 'stackTrace' request".
My question are, it looks from vs code it debugger does stop at breakpoint, why my code doesn't interact? was this something wrong with setting in my launch.json or docker compose files?

Cloudbuild can't access Artifacts Registery when building cloud run docker container

I'm using a package from Artifacts Registery in my cloud run nodejs container.
When I try to gcloud builds submit I get the following error:
Step #1: npm ERR! 403 403 Forbidden - GET https://us-east4-npm.pkg.dev/....
Step #1: npm ERR! 403 In most cases, you or one of your dependencies are requesting
Step #1: npm ERR! 403 a package version that is forbidden by your security policy.
Here is my cloudbuild.yaml:
steps:
- name: gcr.io/cloud-builders/npm
args: ['run', 'artifactregistry-login']
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/...', '.']
- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/...']
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: gcloud
args:
- 'run'
- 'deploy'
- 'admin-api'
- '--image'
- 'gcr.io/...'
- '--region'
- 'us-east4'
- '--allow-unauthenticated'
images:
- 'gcr.io/....'
and Dockerfile
FROM node:14-slim
WORKDIR /usr/src/app
COPY --chown=node:node .npmrc ./
COPY package*.json ./
RUN npm install
COPY . ./
EXPOSE 8080
CMD [ "npm","run" ,"server" ]
.npmrc file:
#scope_xxx:registry=https://us-east4-npm.pkg.dev/project_xxx/repo_xxx/
//us-east4-npm.pkg.dev/project_xxx/repo_xxx/:always-auth=true
the google build service account already has the permission "Artifact Registry Reader"
You have to connect the CloudBuild network in your docker build command. Like that
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/...', '--network=cloudbuild', '.']
I had the same root cause, my setup is close to #AmmAr, after hours of trial and error, found a solution.
Dislaimer, this might not be the reason for your issue, the gcp 403 error message is vague, you need to chip away and eliminate all possibilities, that is how I arrived on this page.
Comparing to #AmmArr above, the changes I made:-
In node.js package.json, add to "scripts" :{...} property
"artifactregistry-login": "npx google-artifactregistry-auth",
"artifactregistry-auth-npmrc": "npx google-artifactregistry-auth .npmrc"
In cloudbuild.yaml, I added two steps prior to the build step, these steps should result in .npmrc getting appended with an access token, thus allowing it to communicate with the gcp artifact repository, that resolved the 403 issue for my scenario.
steps:
- name: gcr.io/cloud-builders/npm
args: ['run', 'artifactregistry-login']
- name: gcr.io/cloud-builders/npm
args: ['run', 'artifactregistry-auth-npmrc']
- name: gcr.io/cloud-builders/docker
args: ['build', '-t', 'gcr.io/...', '.']
#- next steps in your process...
In Dockerfile, copy over .nmprc before package.json
COPY .npmrc ./
COPY package*.json ./
Screenshot of my cloud build config
Now run, and see if it gets past the build step where it pulls npm module from artifact registry.
The solution that worked with me can be found in this blog post:
https://dev.to/brianburton/cloud-build-docker-and-artifact-registry-cicd-pipelines-with-private-packages-5ci2

GCP: Cloud Run preview build fails because of a missing tag "latest"

If have an issue where cloud build is failing on creating a preview build for use in github pull requests.
I have
a github organization with the cloud build app installed.
a cloud build set-up with triggers to deploy to cloud run
functional build on master deploy (doesn't really matter here).
The following is my cloudbuild-preview.yaml file. The failing step is the last one: "link revision on pull request"
steps:
- id: "build image"
name: "gcr.io/cloud-builders/docker"
args:
[
"build",
"-t",
"$_GCR_HOSTNAME/${PROJECT_ID}/${_SERVICE_NAME}:${_PR_NUMBER}-${SHORT_SHA}",
".",
]
- id: "push image"
name: "gcr.io/cloud-builders/docker"
args:
[
"push",
"$_GCR_HOSTNAME/${PROJECT_ID}/${_SERVICE_NAME}:${_PR_NUMBER}-${SHORT_SHA}",
]
- id: "deploy revision with tag"
name: "gcr.io/google.com/cloudsdktool/cloud-sdk"
entrypoint: "gcloud"
args:
[
"beta",
"run",
"deploy",
"${_SERVICE_NAME}",
"--platform",
"managed",
"--region",
"${_REGION}",
"--image",
"$_GCR_HOSTNAME/${PROJECT_ID}/${_SERVICE_NAME}:${_PR_NUMBER}-${SHORT_SHA}",
"--tag",
"pr-${_PR_NUMBER}",
"--no-traffic",
]
- id: "link revision on pull request"
name: "$_GCR_HOSTNAME/${PROJECT_ID}/deployment-previews" # our custom builder
args:
[
"set",
"--project-id",
"${PROJECT_ID}",
"--region",
"${_REGION}",
"--service",
"${_SERVICE_NAME}",
"--pull-request",
"${_PR_NUMBER}",
"--repo-name",
"${_GITHUB_REPO}",
"--commit-sha",
"${SHORT_SHA}",
]
timeout: 1400s
options:
machineType: N1_HIGHCPU_8
substitutions:
_GCR_HOSTNAME: eu.gcr.io
_SERVICE_NAME: redacted-service
_REGION: europe-west4
_GITHUB_REPO: $(pull_request.pull_request.head.repo.full_name)
The execution fails with
Step #3 - "link revision on pull request": Error response from daemon: manifest for eu.gcr.io/redacted-org/deployment-previews:latest not found: manifest unknown: Failed to fetch "latest" from request "/v2/redacted-org/deployment-previews/manifests/latest".
Step #3 - "link revision on pull request": Using default tag: latest
Step #3 - "link revision on pull request": Pulling image: eu.gcr.io/redacted-org/deployment-previews
Starting Step #3 - "link revision on pull request"
What I don't undestand is why the sep is even looking for a :latest tag. There is none. The above steps don't create one. The container registry does not contain one.
How to tell that build step to use the proper image tagged with ${_PR_NUMBER}-${SHORT_SHA}?
Where can I dive into the magic here? Where is the definition of this magic build step?!
Thank you very much for any ideas.
When you don't specify an image tag, tools will always try to pull the :latest image. In Cloud Build, you can specify a specific version of a builder image by simply including the tag in the name for your build step:
- id: "link revision on pull request"
name: "$_GCR_HOSTNAME/${PROJECT_ID}/deployment-previews:${_PR_NUMBER}-${SHORT_SHA}" # our custom builder
args:
[
"set",
"--project-id",
"${PROJECT_ID}",
"--region",
"${_REGION}",
"--service",
"${_SERVICE_NAME}",
"--pull-request",
"${_PR_NUMBER}",
"--repo-name",
"${_GITHUB_REPO}",
"--commit-sha",
"${SHORT_SHA}",
]

AWS ECS Blue/Green CodePipeline: Exception while trying to read the image artifact

I wanted to create a CodePipeline which builds a container image from CodeCommit source and afterwards deploys the new image in Blue/Green fashion to my ECS service (EC2 launchtype).
The source stage is CodeCommit, which already includes appspec.json
as well as taskdef.json
The build stage is building the new
container & pushing it to ECR successfully, the file
imagedefinition.json is the BuildArtifact created at this step,
containing the container and the recently created image with its
tag corresponding to the CodeCommit commit-id.
The deploy stage
is made of action "Amazon ECS (Blue/Green)" using the
SourceArtifact and BuildArtifact as InputArtifacts, to take the
appspec and taskdef from the SourceArtifact and the image
description from the BuildArtifact, to finally deploy the new
container in Blue/Green manner.
The problem is with the image definition from the BuildArtifact. The pipeline fails in the Deploy phase with error:
""
Invalid action configuration
Exception while trying to read the image artifact file from the artifact: BuildArtifact.
""
How to properly configure the "Amazon ECS (Blue/Green)" deploy phase, so that it can use the recently created image and deploy it....by replacing placeholder IMAGE_NAME inside taskdef.json ?
Any hint highly appreciated :D
answering my own question here, hopefully it helps others who facing the same situation.
the file imagedefinitions.json is inappropriate for deploy action "Amazon ECS Blue/Green". For that you have to create file imageDetail.json within the build step and provide it as artifact to the deploy step. How ? This is how the bottom of my buildspec.yaml looks like:
- printf '{"ImageURI":"%s"}' $REPOSITORY_URI:$IMAGE_TAG > imageDetail.json
artifacts:
files:
- 'image*.json'
- 'appspec.yaml'
- 'taskdef.json'
secondary-artifacts:
DefinitionArtifact:
files:
- appspec.yaml
- taskdef.json
ImageArtifact:
files:
- imageDetail.json
In the Deploy phase of CodePipeline, use DefinitionArtifact and ImageArtifact as Input Artifacts and configure them in the corresponding section "Amazon ECS task definition" and "AWS CodeDeploy AppSpec file".
Ensure that your appspec.yaml contains placeholder for the task definition. Here is my appspec.yaml:
version: 0.0
Resources:
- TargetService:
Type: AWS::ECS::Service
Properties:
TaskDefinition: <TASK_DEFINITION>
LoadBalancerInfo:
ContainerName: "my-test-container"
ContainerPort: 8000
Also ensure that your taskdef.json contains placeholder for the final image, like
...
"image": <IMAGE1_NAME>,
...
use that placeholder in the codepipeline config of your blue/green deploy phase in the section "Dynamically update task definition image - optional" by choosing the input artifact as "ImageArtifact" and the placeholder <IMAGE1_NAME>
Amazon ECS Blue/Green (or CodeDeployToECS) CodePipeline action requires the TaskDefinitionTemplateArtifact parameter (see [1]).
In addition to the above file note an imageDetail.json is required for ECS Blue/Green deployments (not 'imagedefinition.json'). The file structure and details are available here [2]. Add this file to the root of your deployment artifact/version control. If you do not want to add this file manually you can use the ECR source action to the CodePipeline and configure this with the Image you are using in the ECS service/taskdef.json. This is all discussed at [2] for clarity.
To see how this is all brought together you can also follow the step by step instructions for ECS Blue/Green deployments here [3].
References:
[1] https://docs.aws.amazon.com/codepipeline/latest/userguide/reference-pipeline-structure.html#action-requirements : CodePipeline Pipeline Structure Reference - Action Structure Requirements in CodePipeline
[2] https://docs.aws.amazon.com/codepipeline/latest/userguide/file-reference.html#file-reference-ecs-bluegreen : Image Definitions File Reference - imageDetail.json File for Amazon ECS Blue/Green Deployment Actions
[3] https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-ecs-ecr-codedeploy.html : Tutorial: Create a Pipeline with an Amazon ECR Source and ECS-to-CodeDeploy Deployment
I ran into the same problem.
tl:dr
I was not passing the correct input artefact with the imageDetail.json to the pipeline CodeDeployToECS action.
Summary:
Instead of checking in a version of the task definition with the '<IMAGE1_NAME>' placeholder, I'm dynamically generating the task definition input to CodeDeploy inside the pipeline.
The task definition early in the project is quite volatile, with new variables etc being passed to the container. It's generated and registered within the pipeline (Cloudformation) and then read out via a Codebuild project, substituting the image placeholder with '<IMAGE1_NAME>' and passed to the next stage in the pipeline via a pipeline artefact.
Fixing it:
I have a CodeBuild project within the pipeline that produces the imageDetail.json:
{"ImageURI":"########.dkr.ecr.eu-west-1.amazonaws.com/##/#####:2739511dd87d4e4e1f65ed69c9e779b63fb72e36-master-fbe73fdc-6213-4bd6-a784-dcc3d2ae7845"}
It's pipeline output is named 'BuildDockerOutput'
I have another Codebuild project that produces:
taskdef.json
{
"containerDefinitions": [
{
"name": "ronantest1",
"image": "<IMAGE1_NAME>",
]
}
appspec.json
{
"version": 0.0,
"Resources": [
{
"TargetService": {
"Type": "AWS::ECS::Service",
"Properties": {
"TaskDefinition": "<TASK_DEFINITION>",
"LoadBalancerInfo": {
"ContainerName": "ronantest1",
"ContainerPort": "8080"
}
}
}
}
],
"Hooks": [
{
"AfterAllowTestTraffic": "arn:aws:lambda:eu-west-1:######:function:code-deploy-after-allow-test-traffic"
}
]
}
It's pipeline output is named 'PrepareCodeDeployOutputTesting'
My final CodeDeploy action is like the following:
- Name: BlueGreenDeploy
InputArtifacts:
- Name: BuildDockerOutput
- Name: PrepareCodeDeployOutputTesting
Region: !Ref DeployRegion1
ActionTypeId:
Category: Deploy
Owner: AWS
Version: '1'
Provider: CodeDeployToECS
RoleArn: !Sub arn:aws:iam::${TestingAccountId}:role/######/CrossAccountsDeploymentRole
Configuration:
AppSpecTemplateArtifact: PrepareCodeDeployOutputTesting
AppSpecTemplatePath: appspec.json
ApplicationName: !Ref ApplicationName
DeploymentGroupName: !Ref ApplicationName
TaskDefinitionTemplateArtifact: PrepareCodeDeployOutputTesting
TaskDefinitionTemplatePath: taskdef.json
Image1ArtifactName: BuildDockerOutput
Image1ContainerName: "IMAGE1_NAME"
RunOrder: 4
Note the different aspects of the CodeDeployToECS needed artefacts from different InputArtifacts, specifically 'Image1ArtifactName'
Thanks, to all, this gives me some light into solving the issue.
I would like to add that when you use aws cli, cloudformation, or Terraform to configure codepipeline, some parameters and options are not available with the console and setting some variables in these tools like the empty string "" will cause an exception error.
Always check for codepipeline settings in the console when you deploy using these tools.
so the error occur when you defined Image Artifact but not define the placeholder
imageDetail.json can be passed into codedeploy using the following methods:
git source ( codecommit or github ) the file that exist in your app codebase
ECR source - the file will be autogenerated by ECR, but will use SHA256 instead of the image tag
CodeBuild source - you update the file using codebuild buildspec.yml and pass it down to codedeploy stage.

Unable to run Sonarqube analysis from cloudbuild.yaml with Google Cloud build

I have integrated my github repo with Google cloud build to automatically build a docker images after every commit in github. This is working fine, but now I want to do sonarqube analysis on code before Docker image building process. So for that I have integrated the sonarqube part in cloudbuild.yaml file. But not able to run it.
I have followed the steps provided in link: https://github.com/GoogleCloudPlatform/cloud-builders-community/tree/master/sonarqube
and pushed the sonar-scanner image in google container registry.
My sonarqube server is running on a GCP instance. On every commit in github, cluod build automatically triggered and start doing task mentioned in cloudbuild.yaml file
Dockerfile:
FROM nginx
COPY ./ /usr/share/nginx/html
cloudbuild.yaml :
steps:
- name: 'gcr.io/PROJECT_ID/sonar-scanner:latest'
args:
- '-Dsonar.host.url=sonarqube_url'
- '-Dsonar.login=c2a7631a6e402c338739091ffbc30e5e3d66cf19'
- '-Dsonar.projectKey=sample-project'
- '-Dsonar.sources=.'
- name: 'gcr.io/cloud-builders/docker'
args: [ 'build', '-t', 'gcr.io/PROJECT_ID/html-css-website', '.' ]
images:
- 'gcr.io/PROJECT_ID/html-css-website'
Error:
Status: Build failed
Status detail: failed unmarshalling build config cloudbuild.yaml: yaml: line 3: did not find expected key
If the formatting you've pasted actually matches what you've got in your project then your issue is that the args property within the first steps block is indented too far: it should be aligned with the name property above it.
---
steps:
- name: "gcr.io/PROJECT_ID/sonar-scanner:latest"
args:
- "-Dsonar.host.url=sonarqube_url"
- "-Dsonar.login=c2a7631a6e402c338739091ffbc30e5e3d66cf19"
- "-Dsonar.projectKey=sample-project"
- "-Dsonar.sources=."
- name: "gcr.io/cloud-builders/docker"
args:
- "build"
- "-t"
- "gcr.io/PROJECT_ID/html-css-website"
- "."
images:
- "gcr.io/PROJECT_ID/html-css-website"