How to build azure pipeline tasks from https://github.com/microsoft/azure-pipelines-tasks locally and consume them in an on-prem Azure DevOps Server? - azure-devops

So I cloned their repository, but how do I actually build the tasks?
Here is my scenario. We use Azure DevOps Server 2020 (on prem). All of our build pipelines run the Index Sources & Publish Symbols task
However, it has a bug https://github.com/microsoft/azure-pipelines-tasks/issues/14852. Luckily a fix was merged to master. However, we are not going to see it until it is propagated to the Azure DevOps Server edition and only All Mighty knows when it would happen.
So, I would like to build that task locally and upload to our Azure DevOps server. But I cannot find instructions on how to do it.
So, how can I build and consume it in our Azure DevOps Server?
EDIT 1
Tried to follow the procedure in https://github.com/microsoft/azure-pipelines-tasks/blob/master/ci/build-all-steps.yml, but it did not work.
The first step that seems relevant to me is https://github.com/microsoft/azure-pipelines-tasks/blob/6ab084f52e582370880127132d1c449634c9bfbc/ci/build-all-steps.yml#L39:
- script: |
cd ci
cd verifyMinAgentDemands
npm install
node index.js
displayName: Verify all min agent demands are valid
And it is fine:
C:\work\azure-pipelines-tasks\ci\verifyMinAgentDemands [master ≡]> npm install
npm WARN verifyminagentdemands#1.0.0 No description
npm WARN verifyminagentdemands#1.0.0 No repository field.
added 52 packages from 78 contributors and audited 52 packages in 1.743s
found 2 low severity vulnerabilities
run `npm audit fix` to fix them, or `npm audit` for details
C:\work\azure-pipelines-tasks\ci\verifyMinAgentDemands [master ≡]> node .\index.js
##vso[task.debug]agent.TempDirectory=undefined
##vso[task.debug]agent.workFolder=undefined
##vso[task.debug]loading inputs and endpoints
##vso[task.debug]loaded 0
##vso[task.debug]Agent.ProxyUrl=undefined
##vso[task.debug]Agent.CAInfo=undefined
##vso[task.debug]Agent.ClientCert=undefined
##vso[task.debug]Agent.SkipCertValidation=undefined
Verifying min agent demands.
Latest version of the Agent that's fully rolled out is 2.195.0.
The next step (https://github.com/microsoft/azure-pipelines-tasks/blob/6ab084f52e582370880127132d1c449634c9bfbc/ci/build-all-steps.yml#L47) seems to be relevant too:
- script: node make.js build --task "$(task_pattern)"
displayName: Build
condition: ne(variables['numTasks'], 0)
But:
C:\work\azure-pipelines-tasks [master ≡]> node make.js build --task PublishSymbolsV2
> prepending PATH C:\work\azure-pipelines-tasks\node_modules\.bin
tsc tool:
Version 2.3.4
C:\work\azure-pipelines-tasks\node_modules\.bin\tsc
npm tool:
6.14.12
C:\Program Files\nodejs\npm
------------------------------------------------------------
Building: PublishSymbolsV2
------------------------------------------------------------
> getting task externals
Downloading file: https://vstsagenttools.blob.core.windows.net/tools/symstore/2/symbol.zip
Could not use "nc", falling back to slower node.js method for sync requests.
C:\work\azure-pipelines-tasks\node_modules\sync-request\index.js:77
throw new Error(res.stderr.toString());
^
Error
at doRequestWith (C:\work\azure-pipelines-tasks\node_modules\sync-request\index.js:77:11)
at doRequest (C:\work\azure-pipelines-tasks\node_modules\sync-request\index.js:20:10)
at downloadFile (C:\work\azure-pipelines-tasks\make-util.js:411:22)
at downloadArchive (C:\work\azure-pipelines-tasks\make-util.js:451:27)
at C:\work\azure-pipelines-tasks\make-util.js:644:33
at Array.forEach (<anonymous>)
at getExternals (C:\work\azure-pipelines-tasks\make-util.js:639:25)
at C:\work\azure-pipelines-tasks\make.js:193:13
at Array.forEach (<anonymous>)
at Function.target.build (C:\work\azure-pipelines-tasks\make.js:158:14)
C:\work\azure-pipelines-tasks [master ≡]>
What is now?

Related

Github Action failing to Build Images for the plugins being used in workflow

I am trying to use a plugin in my eks based k8s cluster,
I am using a Github Action controller that spawns on demand Container as Self Hosted runner
When the Github action start this plugin or any other that needs to build itself as a docker image fails with below error, any thoughts or ideas ?
This is my self hosted runner image Link
FYI : If i run a standalone alpine container in the cluster all typical cmd works, and this also works with default ubuntu based self hosted runner, so i dont think its the cluster
/usr/local/bin/docker build -t 60e226:1b6fc15462134e6fb8520b7df48cf7fd -f "/runner/_work/_actions/aquasecurity/trivy-action/master/Dockerfile" "/runner/_work/_actions/aquasecurity/trivy-action/master"
Sending build context to Docker daemon 644.6kB
Step 1/5 : FROM ghcr.io/aquasecurity/trivy:0.[3](https://github.com//docker-images/actions/runs/4134005760/jobs/7147011143#step:3:3)7.1
0.37.1: Pulling from aquasecurity/trivy
c158987b0551: Pulling fs layer
67a7d067ef7d: Pulling fs layer[6]Download complete
67a7d067ef7d: Pull complete
2ec1cdd48f38: Verifying Checksum
2ec1cdd48f38: Download complete
2ec1cdd48f38: Pull complete
fe56e6aa700e: Pull complete
Digest: sha256:7c[16](https://github.com//docker-images/actions/runs/4134005760/jobs/7147011143#step:3:16)7f7f3002948f1ec099555aa968bd8b8b097780603a38cc801fe965da0a69
Status: Downloaded newer image for ghcr.io/aquasecurity/trivy:0.37.1
---> c3e68408cd24
Step 2/5 : COPY entrypoint.sh /
---> 1f1da443ea86
Step 3/5 : RUN apk --no-cache add bash curl npm
---> Running in 647f7f479cac
fetch https://dl-cdn.alpinelinux.org/alpine/v3.[17](https://github.com//docker-images/actions/runs/4134005760/jobs/7147011143#step:3:17)/main/x86_64/APKINDEX.tar.gz
48ABC73BEB7F0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:[18](https://github.com//docker-images/actions/runs/4134005760/jobs/7147011143#step:3:18)89:
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.17/main: Permission denied
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz
48ABC73BEB7F0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1889:
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.17/community: Permission denied
ERROR: unable to select packages:
bash (no such package):
required by: world[bash]
curl (no such package):
required by: world[curl]
npm (no such package):
required by: world[npm]
The command '/bin/sh -c apk --no-cache add bash curl npm' returned a non-zero code: 3
Warning: Docker build failed with exit code 3, back off 6.807 seconds before retry.
It was expected to build the docker image and proceed with the github action workflow
Tried different flavors of image and nothing worked except for ubunut-latest
the plugin in question
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action#master
with:
image-ref: 'test:latest'
format: 'table'
exit-code: '1'
ignore-unfixed: true
vuln-type: 'os,library'
severity: 'CRITICAL,HIGH'

Cypress tests passing locally but not passing in Azure DevOps pipeline

My Cypress tests are passing locally when running this command:
npx cypress run --browser chrome --env api_endpoint="https://myApp.com",page="TestAppsForCdnIntegrations/myApp.html"
However, when I run this in my Azure DevOps pipeline, I'm getting the below error:
2021-09-29T15:51:07.2749462Z *****************************************
2021-09-29T15:51:07.2762053Z Run Cypress
2021-09-29T15:51:07.2771057Z *****************************************
2021-09-29T15:51:20.4370810Z (Run Starting)
2021-09-29T15:51:20.4448402Z │ Cypress: 6.9.1
2021-09-29T15:51:20.4449343Z │ Browser: Chrome 94
2021-09-29T15:51:20.4468328Z Running: examples\get_js_config.spec.js (1 of 7)
2021-09-29T15:52:20.4675196Z
2021-09-29T15:52:20.4676105Z Timed out waiting for the browser to connect. Retrying...
2021-09-29T15:53:17.2314823Z EPERM: operation not permitted, unlink 'C:\Users\admin\AppData\Roaming\Cypress\cy\production\browsers\chrome-stable\run-5596\CypressCache\Default\Cache\data_1'
2021-09-29T15:53:17.2316098Z Error: EPERM: operation not permitted, unlink 'C:\Users\admin\AppData\Roaming\Cypress\cy\production\browsers\chrome-stable\run-5596\CypressCache\Default\Cache\data_1'
I'm not sure why it's working locally, but not in the pipeline.
Can someone please advise what is causing this & how to resolve?

AzureDevOps - Cache Task in Release Pipeline

I'm trying to use the Cache task inside the AzureDevops release pipeline but I'm getting the following error: "Unable to find pipeline caching scopes."
The full log is:
2019-12-04T02:52:29.7320092Z ##[section]Starting: Cache Maven local repo
2019-12-04T02:52:29.7323359Z ==============================================================================
2019-12-04T02:52:29.7324096Z Task : Cache
2019-12-04T02:52:29.7324205Z Description : Cache files between runs
2019-12-04T02:52:29.7324288Z Version : 2.0.0
2019-12-04T02:52:29.7324386Z Author : Microsoft Corporation
2019-12-04T02:52:29.7324476Z Help : https://aka.ms/pipeline-caching-docs
2019-12-04T02:52:29.7324596Z ==============================================================================
2019-12-04T02:52:30.1883169Z Resolving key:
2019-12-04T02:52:30.2148089Z - maven [string]
2019-12-04T02:52:30.2157501Z - "Linux" [string]
2019-12-04T02:52:30.3775363Z - **/pom.xml [file pattern; matches: 1]
2019-12-04T02:52:30.3776471Z - /home/vsts/work/r1/a/_GHQ_ABI_Salesforce_OneApp/AutomationTestsPrj/... --> 14132C0725424D58200724C825592A7666659E231F87619340789DA110453697
2019-12-04T02:52:30.3840201Z Resolved to: maven|"Linux"|eoJcSbaPKzXrAAmPNfX9NFnxcoydrNu9yrhaLikHrF8=
2019-12-04T02:52:30.3891079Z Resolving restore key:
2019-12-04T02:52:30.3891618Z - maven [string]
2019-12-04T02:52:30.3892061Z - "Linux" [string]
2019-12-04T02:52:30.3892473Z Resolved to: maven|"Linux"|**
2019-12-04T02:52:30.3892596Z Resolving restore key:
2019-12-04T02:52:30.3892858Z - maven [string]
2019-12-04T02:52:30.3892979Z Resolved to: maven|**
2019-12-04T02:52:31.7549257Z Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 6a8d4cb9-7af8-4915-8dcc-46b9ded1e10b
2019-12-04T02:52:32.0675474Z Information, Getting a pipeline cache artifact with one of the following fingerprints:
2019-12-04T02:52:32.0676246Z Information, Fingerprint: `maven|"Linux"|eoJcSbaPKzXrAAmPNfX9NFnxcoydrNu9yrhaLikHrF8=`
2019-12-04T02:52:32.0676528Z Information, Fingerprint: `maven|"Linux"|**`
2019-12-04T02:52:32.0676755Z Information, Fingerprint: `maven|**`
2019-12-04T02:52:34.2400712Z Information, ApplicationInsightsTelemetrySender correlated 2 events with X-TFS-Session 6a8d4cb9-7af8-4915-8dcc-46b9ded1e10b
2019-12-04T02:52:34.2485748Z ##[error]Unable to find pipeline caching scopes.
2019-12-04T02:52:34.2528126Z ##[section]Finishing: Cache Maven local repo
And my configuration is:
Oh, the variables are created and this task is before the task that I execute the Maven:
Cache Task order
So, anybody got an idea of where I'm missing?
I know applying this Cache step before your Maven execution is very convenient for pipeline, such as reducing the whole execution time. Just, unfortunately, until now, we haven't provide this Cache task supported in Release pipeline:
Pipeline caching
Work around:
As I mentioned above, Cache task is very convenient because it can help reduce the time waste in file caching.
So, suggest you apply your release pipeline into Multi-stage YAML format. Though it is running a release phase, but its compile logic is following build.
Check this doc to convert your pipeline: YAML schema.

gatsby new [site] failing: Cannot find module 'gatsby/dist/commands/develop'

I'm pretty green here. I've run gatsby on this (macosx) before. It has stopped working - presumably due to an install or environment variable somewhere?
First noticed with Module build failed: Error: Couldn't find preset "flow" relative to directory "/Users/3Legs"
Then after installing babel presets:
npm install --global --save-dev babel-preset-flow
I get the above message:
gatsby develop
... Cannot find module 'gatsby/dist/commands/develop'
FULL TRAIL BELOW + babelrc
Michaels-MacBook-Air:mggatsby 3Legs$
gatsby new test
-bash: /usr/local/bin/gatsby: No such file or directory
Michaels-MacBook-Air:mggatsby 3Legs$
npm install --global gatsby-cli
WARN registry Unexpected warning for registry.npmjs.org: Miscellaneous Warning EINTEGRITY: sha1-xRn2KfhrOlvtuliojTETCe7Al/k= integrity checksum failed when using sha1: wanted sha1-xRn2KfhrOlvtuliojTETCe7Al/k= but got sha512-vE2hT1D0HLZCLLclfBSfkfTTedhVj0fubHpJBHKwwUWX0nSbhPAfk+SG9rTX95BYNmau8rGFfCeaT6T5OW1C2A==. (455516 bytes)
WARN registry Using stale package data from registry.npmjs.org/ due to a request error during revalidation.
WARN registry Unexpected warning for registry.npmjs.org/: Miscellaneous Warning EINTEGRITY: sha1-buxr+wdCHiFIx1xrunJCH4UwqCY= integrity checksum failed when using sha1: wanted sha1-buxr+wdCHiFIx1xrunJCH4UwqCY= but got sha512-+ktMAh1Jwas+TnGodfCfjUbJKoANqPaJFN0z0iqh41eqD8dvguNzcitVSBSVK1pidz0AqGbLKcoVuVLRVZ/aVg==. (42903 bytes)
WARN registry Using stale package data from registry.npmjs.org/ due to a request error during revalidation.
/usr/local/bin/gatsby -> /usr/local/lib/node_modules/gatsby-cli/lib/index.js
+ gatsby-cli#1.1.1
added 153 packages, removed 5 packages and updated 1 package in 10.573s
Michaels-MacBook-Air:mggatsby 3Legs$
gatsby new test
info Creating new site from git: git://github.com/gatsbyjs/gatsby-starter-default.git
Cloning into 'test'...
remote: Counting objects: 566, done.
remote: Total 566 (delta 0), reused 0 (delta 0), pack-reused 566
Receiving objects: 100% (566/566), 358.35 KiB | 187.00 KiB/s, done.
Resolving deltas: 100% (316/316), done.
success Created starter directory layout
info Installing packages...
yarn install v0.27.5
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning "eslint-config-fbjs#1.1.1" has unmet peer dependency "babel-eslint#^6.1.2".
warning "eslint-config-fbjs#1.1.1" has unmet peer dependency "eslint#^3.0.0".
warning "eslint-config-fbjs#1.1.1" has unmet peer dependency "eslint-plugin-babel#^3.3.0".
warning "eslint-config-fbjs#1.1.1" has unmet peer dependency "eslint-plugin-flowtype#^2.15.0".
warning "eslint-config-fbjs#1.1.1" has unmet peer dependency "eslint-plugin-react#^5.2.2".
[4/4] Building fresh packages...
success Saved lockfile.
Done in 47.94s.
Michaels-MacBook-Air:mggatsby 3Legs$
cd test
Michaels-MacBook-Air:test 3Legs$
gatsby develop
success delete html files from previous builds — 0.010 s
success open and validate gatsby-config.js — 0.006 s
success copy gatsby files — 0.028 s
success source and transform nodes — 0.045 s
success building schema — 0.134 s
success createLayouts — 0.039 s
success createPages — 0.016 s
success createPagesStatefully — 0.016 s
success extract queries from components — 0.118 s
success run graphql queries — 0.030 s
success write out page data — 0.006 s
success update schema — 0.094 s
info bootstrap finished - 3.856 s
error There was an error compiling the html.js component for the development server.
See our docs page on debugging HTML builds for help ...
Error: Module build failed: Error: Couldn't find preset "flow" relative to directory "/Users/3Legs"
Michaels-MacBook-Air:test 3Legs$
npm install --save-dev babel-preset-flow
npm WARN gentlyRm not removing /Users/3Legs/react/mggatsby/test/node_modules/.bin/gatsby as it wasn't installed by /Users/3Legs/react/mggatsby/test/node_modules/gatsby
npm WARN gentlyRm not removing /Users/3Legs/react/mggatsby/test/node_modules/.bin/semver as it wasn't installed by /Users/3Legs/react/mggatsby/test/node_modules/semver
npm WARN gentlyRm not removing /Users/3Legs/react/mggatsby/test/node_modules/jspm-registry/node_modules/.bin/semver as it wasn't installed by /Users/3Legs/react/mggatsby/test/node_modules/jspm-registry/node_modules/semver
npm WARN gentlyRm not removing /Users/3Legs/react/mggatsby/test/node_modules/.bin/browserslist as it wasn't installed by /Users/3Legs/react/mggatsby/test/node_modules/browserslist
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN gatsby-starter-default#1.0.0 No repository field.
+ babel-preset-flow#6.23.0
removed 1375 packages and updated 6 packages in 17.729s
Michaels-MacBook-Air:test 3Legs$
gatsby develop
/usr/local/bin/gatsby develop
Options:
-h, --help Show help [boolean]
-H, --host Set host. Defaults to localhost [string]
-p, --port Set port. Defaults to 8000 [string] [default: "8000"]
-o, --open Open the site in your browser for you. [boolean]
-v, --version Show version number [boolean]
error There was a problem loading the local develop command. Gatsby may not be installed.
Error: Cannot find module 'gatsby/dist/commands/develop'
- index.js:17 resolveFileName
[lib]/[gatsby-cli]/[resolve-from]/index.js:17:39
- index.js:31 resolveFrom
[lib]/[gatsby-cli]/[resolve-from]/index.js:31:9
- index.js:34 module.exports
[lib]/[gatsby-cli]/[resolve-from]/index.js:34:41
- index.js:4 module.exports.moduleId
[lib]/[gatsby-cli]/[resolve-cwd]/index.js:4:30
- create-cli.js:35 resolveLocalCommand
[lib]/[gatsby-cli]/lib/create-cli.js:35:22
- create-cli.js:66 Object.handler
[lib]/[gatsby-cli]/lib/create-cli.js:66:7
- command.js:233 Object.self.runCommand
[lib]/[gatsby-cli]/[yargs]/lib/command.js:233:22
- yargs.js:990 Object.Yargs.self._parseArgs
[lib]/[gatsby-cli]/[yargs]/yargs.js:990:30
- yargs.js:532 Object.Yargs.self.parse
[lib]/[gatsby-cli]/[yargs]/yargs.js:532:23
- create-cli.js:163 module.exports
[lib]/[gatsby-cli]/lib/create-cli.js:163:154
- index.js:122 Object.<anonymous>
[lib]/[gatsby-cli]/lib/index.js:122:1
Michaels-MacBook-Air:~ 3Legs$
cat .babelrc
{
"presets": ["flow"]
}
NPM seems to be removing modules because there isn't a package-lock.json. Not sure if this is new behavior but it's been hitting a lot of people in past few days.
All you need to do is delete node_modules and any lock file there and run npm install.

Does "Placing binaries" mean building k8s successfully?

I follow this Getting started with Kubernetes on Mesos to try to build Kubernetes-Mesos:
git clone https://github.com/kubernetes/kubernetes
cd kubernetes
export KUBERNETES_CONTRIB=mesos
make
But executing make, it outputs following log:
$ make
hack/build-go.sh
+++ [1204 04:38:23] Building go targets for linux/amd64:
cmd/kube-proxy
cmd/kube-apiserver
cmd/kube-controller-manager
cmd/kubelet
cmd/kubemark
cmd/hyperkube
cmd/linkcheck
plugin/cmd/kube-scheduler
contrib/mesos/cmd/k8sm-scheduler
contrib/mesos/cmd/k8sm-executor
contrib/mesos/cmd/k8sm-controller-manager
contrib/mesos/cmd/km
cmd/kubectl
cmd/integration
cmd/gendocs
cmd/genkubedocs
cmd/genman
cmd/mungedocs
cmd/genbashcomp
cmd/genconversion
cmd/gendeepcopy
cmd/genswaggertypedocs
examples/k8petstore/web-server/src
github.com/onsi/ginkgo/ginkgo
test/e2e/e2e.test
+++ [1204 04:38:34] Placing binaries
$
So is the build success or not? I can't find useful prompt info.
Yes, that indicates a successful build.