how to convert classic build job to yaml build in AzureDevops - azure-devops

We have a working classic build job in azure Devops with an self hosted agent pool. But when we tried to convert this build job to yaml method, while executing no agents are getting assigned and its hanging. Could you please correct me here if i am doing something task.
Error
"All eligible agents are disabled or offline"
below is the converted yaml file from classic build - agent job
pool:
name: MYpool
demands: maven
#Your build pipeline references an undefined variable named ‘Parameters.mavenPOMFile’. Create or edit the build pipeline for this YAML file, define the variable on the Variables tab. See https://go.microsoft.com/fwlink/?linkid=865972
steps:
- task: Maven#3
displayName: 'Maven pom.xml'
inputs:
mavenPomFile: '$(Parameters.mavenPOMFile)'
- task: CopyFiles#2
displayName: 'Copy Files to: $(build.artifactstagingdirectory)'
inputs:
SourceFolder: '$(system.defaultworkingdirectory)'
Contents: '**/*.war'
TargetFolder: '$(build.artifactstagingdirectory)'
condition: succeededOrFailed()
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: Root'
inputs:
PathtoPublish: '$(build.artifactstagingdirectory)'
ArtifactName: Root
condition: succeededOrFailed()
- task: CopyFiles#2
displayName: 'Copy wars to build directory'
inputs:
SourceFolder: '$(build.artifactstagingdirectory)/target'
TargetFolder: '/home/myadmin/builds/$(build.buildnumber)'
- task: CopyFiles#2
displayName: 'copying docker file to Build Directory'
inputs:
SourceFolder: Admin
TargetFolder: '/home/myadmin/builds/$(build.buildnumber)'
- bash: |
# Write your commands here
mv /home/myadmin/builds/$(build.buildnumber)/mypack0.0.1.war /home/myadmin/builds/$(build.buildnumber)/ROOT.war
displayName: 'Name war file Root.war'
- task: Docker#2
displayName: 'Build the docker image'
inputs:
repository: 'mycontainerregistry.azurecr.io/myservice'
command: build
Dockerfile: '/home/myadmin/builds/$(build.buildnumber)/Dockerfile'
tags: '$(Build.BuildNumber)-DEV'
- bash: |
# Write your commands here
docker login mycontainerregistry.azurecr.io
docker push mycontainerregistry.azurecr.io/myservice:$(Build.BuildNumber)-DEV
displayName: 'Push Docker Image'
- task: CopyFiles#2
displayName: 'Copy Deployment file'
inputs:
SourceFolder: /home/myadmin/kubernetes
TargetFolder: '/home/myadmin/builds/$(build.buildnumber)'
- task: qetza.replacetokens.replacetokens-task.replacetokens#3
displayName: 'Replace image in deployment file'
inputs:
rootDirectory: '/home/myadmin/builds/$(build.buildnumber)'
targetFiles: '**/*.yml'

In my previous answer, I said when I wait for nearly 20-30 mins, the interface of agent will prompt below message.
In fact, this is a process which upgrade the agent to latest version automatically.
Yes, when you using YAML with private agent, the agent version must be the latest one. No matter you add the demands or not.
For our system, the agent version is a implicit demand that your agent must satisfied with the latest one when you applying it in YAML.
If it is not satisfied, it will be blocked and the agent upgrade process will be forced to be performed automatically by system after some times.
So, to execute the private agent in YAML successfully, please upgrade the agent to latest one manually.
Since what my colleague and I talked are all private to microsoft in this ticket, sorry you could not get visible on this summary. So, here I take the screenshots about it, and you can refer to it: https://imgur.com/a/4OnzHp3
We are still working on why the system prompting so confusing message like: All eligible agents are disabled or offline. And, am trying to do some contribution to let this message more clear, for example: no agents meet demands: agent version xxx.

Related

Azure Devops - Can not retrieve artifact

I am publishing a artifact in a job (which completes successfully). In a separate job, I am trying to retrieve the artifact. However, nothings shows in the ls $(Build.ArtifactStagingDirectory). Why is this?
- task: PublishPipelineArtifact#1
displayName: 'Publish Artifact'
inputs:
targetpath: 'anatomy/dist'
artifact: 'anatomy-artifact'
publishLocation: 'pipeline'
- job: deployBucket
displayName: 'Deploy Bucket'
dependsOn: buildAnatomy
condition: eq(dependencies.buildAnatomy.result,'Succeeded')
pool:
vmImage: "ubuntu-20.04"
steps:
- task: Bash#3
displayName: "Upload contents to bucket"
inputs:
targetType: "inline"
script: |
ls -al '$(Build.ArtifactStagingDirectory)'
Each job runs in its own space and the agent cleans up after itself by default. If you want the contents of the artifact to be available in the 2nd job, you need to fetch it again.
Use the DownloadPipelineArtifact#1 task at the start of your 2nd job.
Or move the deploy command into the same job.

Git Checkout fails due to long file path names in the Azure DevOps yaml build pipeline

I have the Unity project code in Azure DevOps Repos and configured the below yaml pipeline to build the Unity project.
trigger:
- none
stages:
- stage: Build
displayName: Unity Build
jobs:
- job: 'UnityBuild'
displayName: 'Build the Unity application'
pool:
name: XXXXXXXXX
steps:
- checkout: none
- script: "git config system core.longpaths true"
- checkout: self
- task: UnityBuildTask#3
inputs:
buildTarget: 'standalone'
unityProjectPath: 'XXXXXXXXXX'
outputPath: '$(Build.BinariesDirectory)'
outputFileName: 'Standalone'
- task: UnityGetProjectVersionTask#1
inputs:
unityProjectPath: 'XXXXXXXXXX'
- task: CopyFiles#2
inputs:
SourceFolder: '$(Build.BinariesDirectory)'
Contents: '**'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'drop'
publishLocation: 'Container'
Whenever I ran the yaml build pipeline, the build failed before it even executed unity build tasks due to file path name length restrictions.
How to fix the issue of file path names being too long in the Azure DevOps YAML pipeline?
You can run a script before "checkout" that tells Git.exe how to handle long paths (i.e. git config --system core.longpaths true).
See here.
If the agent is running on your own Windows server, then you'll need to configure the server to Enable Long Paths support.

Azure Pipeline Copy Secure file into build folder

I've a vite/svelte project which uses .env files for environment settings. I also have an Azure Pipeline which contains a secure file .env.staging this is on the .gitignore list of the associated repo. I'd like to download this secure file, copy it to my build directory and then have it's contents read when I run vite build --mode staging (well, npm run build:staging which includes vite build...)
When run locally from my machine npm run build:staging works as expected and reads the .env.staging file, however it seems to get ignored when used in the pipeline, am I doing anything wrong?
Here's my yml.
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
steps:
- task: DownloadSecureFile#1
name: "dotenvStaging"
inputs:
secureFile: '.env.staging'
displayName: "Download .env.staging"
- task: NodeTool#0
inputs:
versionSpec: 14.15.4
displayName: "Install Node.JS"
- task: CopyFiles#2
inputs:
contents: "$(Agent.TempDirectory)/.env.staging"
targetFolder: "$(Agent.BuildDirectory)"
displayName: "Import .env.staging"
- script: npm install
displayName: "npm install"
- script: npm run build:staging
displayName: "npm run build:staging"
- task: ArchiveFiles#2
inputs:
rootFolderOrFile: 'dist'
archiveType: 'zip'
archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
#replaceExistingArchive: true
#verbose: # Optional
#quiet: # Optional
displayName: "Create archive"
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
ArtifactName: 'drop'
publishLocation: 'Container'
displayName: "Publish archive"
I'm not sure if CopyFiles#2 is doing what I expect or not as it just matches the content parameter to copy whatever files match, which could be 0 if I'm writing it wrong...
Another note, I also tried using $(dotenvStaging.secureFilePath) as the content parameter, but it doesn't seem to do anything either.
Naturally I figured it out as soon as I posted, I needed to update the CopyFiles part to specify sourceFolder, clearly it didn't like my absolute file path for content.
- task: CopyFiles#2
inputs:
sourceFolder: "$(Agent.TempDirectory)"
contents: ".env.staging"
targetFolder: "$(Agent.BuildDirectory)"
displayName: "Import .env.staging"

How can I execute and schedule Databricks notebook from Azure Devops Pipeline using YAML

I wanted to do CICD of my azure Databricks notebook using YAML file.
I have followed the below flow
Pushed my code from Databricks notebook to Azure Repos.
Created a Build using below YAML script.
stages:
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
steps:
- task: CopyFiles#2
displayName: 'Copy Files to: $(build.artifactstagingdirectory)'
inputs:
SourceFolder: '$(System.DefaultWorkingDirectory)'
TargetFolder: ' $(build.artifactstagingdirectory)'
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: notebooks'
inputs:
ArtifactName: dev_release
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'publish build'
publishLocation: 'Container'
By doing above I was able to create a Artifact.
Now I have added another task to deploy that artifact to my Databricks workspace. By using below YAML Script.
- stage: Deploy
displayName: Deploy stage
jobs:
- job: Deploy
displayName: Deploy
pool:
vmImage: 'vs2017-win2016'
steps:
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'current'
downloadType: 'single'
artifactName: 'dev_release'
downloadPath: '$(System.ArtifactsDirectory)'
- task: databricksDeployScripts#0
inputs:
authMethod: 'bearer'
bearerToken: 'dapj0ee865674cd9tfb583dbad61b78ce9b1-4'
region: 'Central US'
localPath: '$(System.DefaultWorkingDirectory)'
databricksPath: '/Shared'
Now i want to run the deployed notebook from here only. So I have "Configure Databricks CLI" task and "Execute Databricks" task to execute the note book.
Got below Error:
##[error]Error: Unable to locate executable file: 'databricks'. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also verify the file has a valid extension for an executable file.
##[error]The given notebook does not exist.
How can I execute notebook from Azure DevOps. My notebooks are in Scala Language.
Is there any other way to use in Production servers.
As you have deployed the Databricks Notebook using Azure DevOps and asking for any other way to run it, I would like to suggest you Azure Data Factory Service.
In Azure Data Factory, you can create pipeline that executes a Databricks notebook against the Databricks jobs cluster. You can also pass Azure Data Factory parameters to the Databricks notebook during execution.
Follow the official tutorial to Run Databricks Notebook with Databricks Notebook Activity in Azure Data Factory to deploy and run Databrick Notebook.
Additionally, you can schedule the pipeline trigger at any particular time or event to make the process completely automatic. Refer https://learn.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers
try this :
- job: job_name
displayName: test job
pool:
name: agent_name(selfhostedagent)
#pool:
workspace:
clean: all
steps:
- checkout: none
- task: DownloadBuildArtifacts#0
displayName: 'Download Build Artifacts'
inputs:
artifactName: app
downloadPath: $(System.DefaultWorkingDirectory)
- task: riserrad.azdo-databricks.azdo-databricks-configuredatabricks.configuredatabricks#0
displayName: 'Configure Databricks CLI'
inputs:
url: '$(Databricks_URL)'
token: '$(Databricks_PAT)'
- task: riserrad.azdo-databricks.azdo-databricks-deploynotebooks.deploynotebooks#0
displayName: 'Deploy Notebooks to Workspace'
inputs:
notebooksFolderPath: '$(System.DefaultWorkingDirectory)/app/path/to/notebbok'
workspaceFolder: /Shared
- task: riserrad.azdo-databricks.azdo-databricks-executenotebook.executenotebook#0
displayName: 'Execute /Shared/path/to/notebook'
inputs:
notebookPath: '/Shared/path/to/notebook'
existingClusterId: '$(cluster_id)'

Azure DevOps Pipeline with AWS CodeDeploy – Copy appspec.yml and hooks scripts to the same location as the build artifact

(Sorry for the long post! I wanted to put in as much detail as possible)
Using Azure DevOps I am trying to deploy using AWS CodeDeploy. I already have a successful AWS pipeline that utilizes AWS CodeBuild and AWS CodeDeploy So I know everything works in that environment. My organization now wants to convert some of our processes to use Azure DevOps.
In Azure I have successfully built the code into a war file that will be used in the deployment process. This proves that he azure-pipelines.yml file is set up correctly to build. Just to get familiar with Azure DevOps talking to AWS I was able to also upload the war artifact to S3 using the Azure releases process and a properly configured S3 agent. I now want deploy the war artifact directly from Azure to AWS CodeDeploy. Just to ensure that I can interface with AWS CodeDepoy from Azure I have successfully had the war artifact copied into the AWS CodeDeploy application/deployment group. It did copy it over but it fails to deploy because it does not know how to get to the AWS CodeDeploy appspec.yml and the hooks scripts outlined in the appspec.yml.
I have the hooks located in my code project under /aws
I have the following:
/aws/before_install.sh
/aws/after_install.sh
/aws/application_start.sh
/aws/application_stop.sh
/aws/before_allow_traffic.sh
/aws/after_allow_traffic.sh
The appspec.yml I placed in the root directory in my code project:
/appspec.yml.
I assume that I need to have the appspec.yml and all the shell scripts in “/aws” copied to the same location that the war file is being copied to (I assume that this would be the Revision Bundle location). I observed that the Azure Release process/CodeDeploy agent looks at the “Revision Bundle” location, zips all the files it finds there and passes that to AWS CodeDeploy.
What do I need to add to the azure-pipelines.yml to tell it to copy the appspec.yml and all the shell scripts in /aws to the same location that the war file is placed under and make this the Revision Bundle location?
Using the below azure-pipelines.yml it builds the war and as an experiment I do see that it does copy the one shell script I defined because I do see it stating that two artifacts were uploaded (pro-0.0.1-SNAPSHOT.war and after_install.sh) when I run it :
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Gradle#2
inputs:
workingDirectory: ''
gradleWrapperFile: 'gradlew'
gradleOptions: '-Xmx3072m'
javaHomeOption: 'JDKVersion'
jdkVersionOption: '1.11'
jdkArchitectureOption: 'x64'
publishJUnitResults: true
testResultsFiles: '**/TEST-*.xml'
tasks: 'build'
- publish: $(System.DefaultWorkingDirectory)/build/libs/
artifact: pro-0.0.1-SNAPSHOT.war
- task: CopyFiles#2
displayName: 'Copy the needed AWS files to the artifact directory'
inputs:
SourceFolder: '$(build.artifactstagingdirectory)'
Contents: 'aws/*.sh'
TargetFolder: '$(System.DefaultWorkingDirectory)/build/libs/'
- publish: $(System.DefaultWorkingDirectory)/aws/after_install.sh
artifact: after_install.sh
Now how do I get all the shell scripts uploaded? I tried wildcards but it didn’t work.
How do I have the Azure Codedeploy agent access/find all the files together to allow them to be zipped up and sent along to AWS CodeDeploy?
To summarize - I need all of the following files in one directory so that the Azure Codedeploy agent can access them to zip up and send to AWS CodeDeploy:
pro-0.0.1-SNAPSHOT.war, appspec.yml, after_install.sh, before_install.sh, application_start.sh, application_stop.sh¸ after_allow_traffic.sh, before_allow_traffic.sh,
FYI – I have the “Revision Bundle” location set to “$(System.DefaultWorkingDirectory)/_Trove /pro-0.0.1-SNAPSHOT.war/
Thanks for any help or advice!!
Try with below YAML script. Just do little changes to your CopyFiles and second Publish task:
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Gradle#2
inputs:
workingDirectory: ''
gradleWrapperFile: 'gradlew'
gradleOptions: '-Xmx3072m'
javaHomeOption: 'JDKVersion'
jdkVersionOption: '1.11'
jdkArchitectureOption: 'x64'
publishJUnitResults: true
testResultsFiles: '**/TEST-*.xml'
tasks: 'build'
- publish: $(System.DefaultWorkingDirectory)
artifact: pro-0.0.1-SNAPSHOT.war
- task: CopyFiles#2
displayName: 'Copy the needed AWS files to the artifact directory'
inputs:
SourceFolder: '$(System.DefaultWorkingDirectory)'
Contents: |
aws/*.sh
appspec.yml
TargetFolder: '$(build.artifactstagingdirectory)'
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'build.gradle'
publishLocation: 'Container'
For others that might have a similar situation, here is what finally worked for me :
trigger:
- '*'
name: $(SourceBranchName)_$(Year:yy).$(Month).$(DayOfMonth)$(Rev:.r)-$(BuildId)
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Gradle#2
inputs:
workingDirectory: ''
gradleWrapperFile: 'gradlew'
gradleOptions: '-Xmx3072m'
javaHomeOption: 'JDKVersion'
jdkVersionOption: '1.11'
jdkArchitectureOption: 'x64'
publishJUnitResults: true
testResultsFiles: '**/TEST-*.xml'
tasks: 'build'
# - publish: $(System.DefaultWorkingDirectory)
# artifact: pro-0.0.1-SNAPSHOT.war
- task: CopyFiles#2
displayName: 'Copy the needed AWS files to the artifact directory'
inputs:
SourceFolder: '$(System.DefaultWorkingDirectory)'
Contents: |
aws/*.sh
aws/appspec.yml
build/libs/*
TargetFolder: '$(build.artifactstagingdirectory)/output'
flattenFolders: true
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)/output'
ArtifactName: 'pro'
publishLocation: 'Container'