I am working on building a pipeline using AzureDevOps, and I face a strange problem.
This is my pipeline:
- stage: 'Test'
displayName: 'Deploy to the test environment'
dependsOn: Build
jobs:
- job: 'Deploy'
steps:
- download: current
artifact: lorehub-front
- bash: cd $(Pipeline.Workspace); echo $(ls)
- bash: cd $(Pipeline.Workspace)/lorehub-front; echo $(ls)
- task: AzureStaticWebApp#0
displayName: 'Publish to Azure Static WebApp'
inputs:
app_location: $(Pipeline.Workspace)/lorehub-front
azure_static_web_apps_api_token: xxxx
The first bash shows that the folder 'lorehub-front' is presented
The second bash shows that the inside folder is correct files (index.html and etc)
Script contents:
cd /home/vsts/work/1/lorehub-front; echo $(ls)
android-chrome-192x192.png android-chrome-512x512.png
apple-touch-icon.png css env-config.js favicon-16x16.png
favicon-32x32.png favicon.ico fonts index.html js site.webmanifest
But I am receiving this error:
App Directory Location: '/home/vsts/work/1/lorehub-front' is invalid. Could not
detect this directory. Please verify your deployment configuration
file reflects your repository structure.
App Directory Location: '/home/vsts/work/1/lorehub-front' is invalid.
The method to resolve this issue is that you need to change the path to /lorehub-front.
- task: AzureStaticWebApp#0
displayName: 'Publish to Azure Static WebApp'
inputs:
app_location: /lorehub-front
azure_static_web_apps_api_token: xxxx
For more detailed info, you could refer to this doc: Tutorial: Publish Azure Static Web Apps with Azure DevOps
Enter / if your application source code is at the root of the repository, or /app if your application code is in a directory called app.
In case anyone else comes across this post from Google or whatever, I had the exact same problem as Andrei above but for the life of me I couldn't get the accepted solution here to work.
No matter what I put in app_location:, the task just flat out refused to see any files.
Upon further investigation, I found this github issue which claims the following (Emphasis mine):
The AzureStaticWebApp task’s app location is relative to the current directory
Looking at the documentation, we can see that this task uses the default directory of $(System.DefaultWorkingDirectory), which is different to $(Pipeline.Workspace) for whatever reason.
So, if you find yourself stuck in the same position and cannot get this task to recognise your artifacts, the solution is to add cwd: $(Pipeline.Workspace) to your task, e.g.
strategy:
runOnce:
deploy:
steps:
- download: current
artifact: WebApp
- task: AzureStaticWebApp#0
inputs:
app_location: /
skip_app_build: true
azure_static_web_apps_api_token: $(deployment_token)
cwd: $(Pipeline.Workspace)/WebApp
Related
I have a sample project with a simple C file( hello world program). I am trying to get familiar with artifacts and hence I have started with pipeline artifacts. The pipeline has 2 stages, build and Test.
In the build stage, I compile the C file, then publish the artifact. In the test stage, I run the object file.
trigger:
branches:
include:
- '*'
pool:
vmImage: ubuntu-latest
stages:
- stage: build
jobs:
- job: buildjob
steps:
- script: |
echo "building the test.c file"
gcc test.c -o test
echo "build completed"
- task: PublishPipelineArtifact#1
inputs:
targetPath: $(System.DefaultWorkingDirectory)
artifactName: Test
- stage: test
jobs:
- job: testJob
steps:
- download: current
artifact: Test
- script: |
cd Test
echo "Running the object file"
./test
echo "job finished"
Error:
I can see that the artifacts have been published:
Concerns: What do I have to do to get the object file running`? Also can i just pass the object file alone to the artifact? How?
UPDATE
I have managed to find the correct path to the artifact folder. But I cannot seem to execute the file. It shows permission denied
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
trigger:
branches:
include:
- '*'
pool:
vmImage: ubuntu-latest
stages:
- stage: build
jobs:
- job: buildjob
steps:
- script: |
echo "building the test.c file"
gcc test.c -o test
echo "build completed"
- task: PublishPipelineArtifact#1
inputs:
targetPath: $(System.DefaultWorkingDirectory)
artifactName: Test
- stage: test
jobs:
- job: testJob
steps:
- download: current
artifact: Test
- script: |
echo "Running the object file"
cd $(Pipeline.Workspace)/Test
./test
echo "job finished"
New Error:
The execute permission bit on your compiled executable was lost between stages. When the file is downloaded in stage "test", it does not have the execute bit set anymore hence the "Permission denied" error when you try to runit.
Setting the x bit again by adding via chmod will solve this:
- stage: test
jobs:
- job: testJob
steps:
- download: current
artifact: Test
- script: |
echo "Running the object file"
cd $(Pipeline.Workspace)/Test
chmod +x ./test
./test
echo "job finished"
The issue is that under Linux the file permission get lost through the compression via zip.
A working solution would be to use tar as it preserves the file permissions.
PublishBuildArtifacts
StoreAsTar - Tar the artifact before uploading
boolean. Default value: false.
Adds all files from the publish path to a tar archive before uploading. This allows you to preserve the UNIX file permissions. Use extractTars option of theDownloadBuildArtifacts task to extract the downloaded items automatically. This setting is ignored on Windows agents.
DownloadBuildArtifacts
extractTars - Extract all files that are stored inside tar archives
boolean.
Set to true to extract all downloaded files that have the .tar extension. This is helpful because you need to pack your artifact files into tar if you want to preserve Unix file permissions. Enabling the StoreAsTar option in the Publish build artifacts task will store artifacts as .tar files automatically.
I have created a pipeline using only YAML.
I have defined the deployment part like this:
- stage: AzureDevOpsStaging
displayName: Deploy build artifacts to staging environment
dependsOn: BuildSolution
condition: succeeded('BuildSolution')
jobs:
- deployment: DeployArtifacts
displayName: Deploy artifacts
environment:
name: AzureDevOpsStaging
resourceType: VirtualMachine
strategy:
runOnce:
deploy:
steps:
- download: current
artifact: drop
- task: IISWebAppDeploymentOnMachineGroup#0
displayName: Deploy artifacts to IIS
inputs:
webSiteName: 'mysite-staging'
package: '$(Pipeline.Workspace)\drop\*.zip'
xmlTransformation: true
When I run this I get:
##[warning]Unable to apply transformation for the given package. Verify the following.
##[warning]1. Whether the Transformation is already applied for the MSBuild generated package during build. If yes, remove the <DependentUpon> tag for each config in the csproj file and rebuild.
##[warning]2. Ensure that the config file and transformation files are present in the same folder inside the package.
Things that I've checked:
Both Web.config and Web.AzureDevOpsStaging.config files are in the zip/artifact
Name of stage - The docs say stage must have the same name as your transform config file; that is: Web.AzureDevOpsStaging.config.
Name of .config transform file - the name of the .config transform file is Web.AzureDevOpsStaging.config
Name of environment (the docs doesn't say the name has to be the same as Web.ThisPart.config but I still named the environment
AzureDevOpsStaging just in case.)
But again doing all of the above results in the Web.config not being transformed.
I got it to work with using the file transform task instead which is referenced in the docs from the IIS Web App Deploy task:
- stage: AzureDevOpsStaging
displayName: Deploy build artifacts to staging environment
dependsOn: BuildSolution
condition: succeeded('BuildSolution')
jobs:
- deployment: DeployArtifacts
displayName: Deploy artifacts
environment:
name: AzureDevOpsStaging
resourceType: VirtualMachine
strategy:
runOnce:
deploy:
steps:
- download: current
artifact: drop
- task: FileTransform#1
inputs:
folderPath: '$(Pipeline.Workspace)\drop\*.zip'
enableXmlTransform: true
xmlTransformationRules: -transform **\*.AzureDevOpsStaging.config -xml **\*.config
- task: IISWebAppDeploymentOnMachineGroup#0
displayName: Deploy artifacts to IIS
inputs:
webSiteName: 'mysite-staging'
package: '$(Pipeline.Workspace)\drop\*.zip'
So can someone please explain to me how I am supposed to configure my YAML to get it to work using only the IISWebAppDeploymentOnMachineGroup#0 task?
And if this is not possible am I using the task FileTransform#1 properly?
Also, I saw there is a version FileTransform#2 as well. That task didn't have one of the properties that #1 has so I reverted to using v1 instead. But would be great if someone has a bit more info on this newer version and if it's going to deprecate #1 in the future?
Btw, I also got xmlTransformation: true to work with classic release pipeline under the Releases tab in Azure DevOps using the UI. But again I don't want to use the classic stuff, I want to do everything in YAML.
And if this is not possible am I using the task FileTransform#1 properly?
The answer is yes.
The task FileTransform is the one I am using and use frequently.
When I use it in the YAML pipeline as you set:
- task: FileTransform#1
displayName: 'File Transform'
inputs:
folderPath: '$(Pipeline.Workspace)\drop\*.zip'
enableXmlTransform: true
xmlTransformationRules: '-transform **\*.UAT.config -xml **\*.config'
fileType: xml
It works fine on my side:
In order to perform the conversion correctly, it is necessary to ensure that the syntax in config is correct, and the specified directory is correct
I have an Environment called 'Dev' that has a resource, which is a VM. As part of the 'Dev' pipeline I want to copy files from a specific folder on the develop branch of a specific repo to a specific folder on the VM that's on the Environment.
I've not worked with Environments before or yaml pipelines much but I gather I need to use the CopyFiles#2 task.
So I've got an azure pipeline yaml file something like this:
variables:
isDev: $[eq(variables['Build.SourceBranch'], 'refs/heads/develop')]
stages:
- stage: Build
jobs:
- job: Build
pool:
vmImage: 'windows-latest'
steps:
- task: CopyFiles#2
displayName: 'Copy Files'
inputs:
contents: 'myFolder\**'
Overwrite: true
targetFolder: $(Build.ArtifactStagingDirectory)
- task: PublishBuildArtifacts#1
inputs:
pathToPublish: $(Build.ArtifactStagingDirectory)
artifactName: myArtifact
- stage: Deployment
dependsOn: Build
condition: and(succeeded(), eq(variables.isDev, true))
jobs:
- deployment: Deploy
displayName: Deploy to Dev
pool:
vmImage: 'windows-latest'
environment: Dev
strategy:
runOnce:
deploy:
steps:
- script: echo Foo Bar
The first question is how to I get this to copy the files to a specific path on the Dev environment?
Is the PublishBuildArtifacts really needed? The reason I ask is that I want this to copy files every time the pipeline is run and not error if the artifact already exists.
It also feels a bit dirty to have to check the branch is the correct branch this way. Is there a better way to do it?
The deployment strategy you're using relies on specifying an agent pool, which means it doesn't run on the machines in the environment. If you use a strategy such as rolling, it will run the specified steps on those machines automatically, including any download steps to download artifacts.
Ref: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/deployment-jobs?view=azure-devops#deployment-strategies
You need to publish artifacts as part of the pipeline if you want them to be automatically available to down-stream jobs. Each run will get a different set of artifacts, even if the actual artifact contents are the same.
That said, based on the YAML you posted, you probably don't need to. In fact, you don't need the "build" stage at all. You could just add a checkout step during your rolling deployment, and the repo would be cloned on each of the target machines.
Ok, worked this out with help from this article: https://dev.to/kenakamu/azure-devops-yaml-release-pipeline-trigger-when-build-pipeline-completed-54d5.
I've taken the advice from Daniel Mann regarding the strategy being 'rolling'. I then split my pipeline into 2 pipelines; 1 for building the artifacts and 1 for releasing (copying them).
If you want just download the particular folders instead of all the source files from the repository, you can try using the REST API "Items - Get" to download each particular folder individually.
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/items?path={path}&download=true&$format=zip&versionDescriptor.version={versionDescriptor.version}&resolveLfs=true&api-version=6.0
For example:
Have the repository like as below.
Now, in the YAML pipeline, I just want to download the 'TestFolder01' folder from the main branch.
jobs:
- job: build
. . .
steps:
- checkout: none # Do not check out all the source files.
- task: Bash#3
displayName: 'Download particular folder'
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
inputs:
targetType: inline
script: |
curl -X GET \
-o TestFolder01.zip \
-u :$SYSTEM_ACCESSTOKEN 'https://dev.azure.com/MyOrg/MyProj/_apis/git/repositories/ShellScripts/items?path=/res/TestFolder01&download=true&$format=zip&versionDescriptor.version=main&resolveLfs=true&api-version=6.0'
This will download the 'TestFolder01' folder as a ZIP file (TestFolder01.zip) into the current working directory. You can use the unzip command to decompress it.
[UPDATE]
If you want to download the particular folders in the deploy job which target to your VM environment, yes, the folders will be download into the pipeline working directory on the VM.
Actually, you can consider a VM type environment resource is a self-hosted agent installed on the VM. So, when your deploy job is targeting to the VM environment resource, it is running on the self-hosted agent on the VM.
The pipeline working directory is under the directory where you install the VM environment resource (self-hosted agent). Normally, you can use the variable $(Pipeline.Workspace) to get value of this path (see here).
stages:
- stage: Deployment
jobs:
- deployment: Deploy
displayName: 'Deploy to Dev'
environment: 'Dev.VM-01'
strategy:
runOnce:
deploy:
steps:
- task: Bash#3
displayName: 'Download particular folder'
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
inputs:
targetType: inline
script: |
echo "Current working directory: $PWD"
curl -X GET \
-o TestFolder01.zip \
-u :$SYSTEM_ACCESSTOKEN 'https://dev.azure.com/MyOrg/MyProj/_apis/git/repositories/ShellScripts/items?path=/res/TestFolder01&download=true&$format=zip&versionDescriptor.version=main&resolveLfs=true&api-version=6.0'
I've a CD pipeline that builds a project (multiple times for different environments) and publishes / saves the ./dist directories as one stage. I can download each environment and run locally as expected.
Each environment build is a stage that needs a manual approval. This is where I am getting lost. Each stage shows the correct artifact being pulled into the stage BUT the AzureStaticWebApp#0 -> app_location input results in a "Could not detect this directory." error.
To recap:
After building the project and saving as an artifact (I can manually download and verify) I am unable to push that built code to Azure Static Web App as it cannot be found. I've tried any number of combinations to no effect. Any advice?
I'm using templates, here is the Push Built Project to Azure Static Web Apps template
When this template runs, I can see jobs running and successfully pulling down the right artifact with this output:
Successfully downloaded artifacts to /home/vsts/work/1/
Finishing: Download Artifact
But the AzureStaticWebApp#0 task gives this error:
App Directory Location: '/home/vsts/work/1/DEV' is invalid. Could not detect this directory. Please verify your deployment configuration file reflects your repository structure.
parameters:
- name: environment
default: development
type: string
- name: variableGroup
default: development-variables-group
type: string
jobs:
- deployment:
displayName: 'Deploy to'
environment: ${{parameters.environment}}
variables:
- group: ${{parameters.variableGroup}}
pool:
vmImage: ubuntu-latest
strategy:
runOnce:
deploy:
steps:
- task: AzureStaticWebApp#0
inputs:
app_location: '$(Pipeline.Workspace)/DEV'
api_location: 'api'
output_location: 'dist'
skip_app_build: true
env:
azure_static_web_apps_api_token: $(deployment-token)
EDIT
Does the task AzureStaticWebApp not have access to anything outside the project?
- deployment:
displayName: 'Deploy to'
environment: ${{parameters.environment}}
variables:
- group: ${{parameters.variableGroup}}
pool:
vmImage: ubuntu-latest
strategy:
runOnce:
deploy:
steps:
- checkout: self
submodules: true
# This step pulls down a complied site. E.g DEV/index.htm, ./images, staticwebapp.config.json
# That has an output like:
# Downloading DEV/index.html to /home/vsts/work/1/DEV/index.html
# Successfully downloaded artifacts to /home/vsts/work/1/
- download: current
artifact: DEV
- task: AzureStaticWebApp#0
inputs:
# I've tried many different values for app_location but all return back not found error
app_location: '/DEV'
api_location: 'api'
output_location: 'dist'
skip_app_build: true
env:
azure_static_web_apps_api_token: $(deploymenttoken)
Solved --- well found a way to make it work.
The build step created 'app/dist' directory and content
The 'app/dist' folder only is published as an
artifact
When downloading the artifact you need to 'put it back'
into the project. In this case DEV/ -> app/dist.
- task: DownloadPipelineArtifact#2
inputs:
artifact: DEV
path: ./app/dist # Put build artifact back into the project
displayName: "Download artifacts"
- task: AzureStaticWebApp#0
inputs:
app_location: 'app/dist'
api_location: 'api'
output_location: 'dist'
skip_app_build: true
env:
azure_static_web_apps_api_token: $(deploymenttoken)
app_location specifies the root of your application code. The property should point to a location in your repo.
Check the documentation here: https://learn.microsoft.com/en-us/azure/static-web-apps/publish-devops
Also, https://github.com/Azure/static-web-apps/issues/5#issuecomment-855309544
There are two other solutions described in #552 that do not require adding the DownloadPipelineArtifact#2 step.
The solution that worked for me was to set the workingDirectory to the pipeline's artifact location. Then set the app_location value relative to the workingDirectory
For example, if your artifact was downloaded to $(Pipeline.Workspace)/MyBuild/drop
Your properties would be:
app_location: /drop
workingDirectory: $(Pipeline.Workspace)/MyBuild
yml snippet:
- task: AzureStaticWebApp#0
displayName: Publish Static Web App
inputs:
app_location: /drop # The name of your artifact
output_location: "" # Leave this empty
skip_app_build: true
azure_static_web_apps_api_token: $(deployment-token)
# The path where your artifact was downloaded
workingDirectory: $(Pipeline.Workspace)/MyBuild
This is really confusing because every other pipeline task accepts absolute paths but AzureStaticWebApp#0 requires a relative path for app_location.
I am developing a Azure task template, and I have a large .py file that I want to be executed in one step
- task: PythonScript#0
displayName: 'Run a Python script'
inputs:
scriptSource: inline
script: |
... really long python code
It's possible to store the code in another file, at the same level of the yml template, and consume it from there? Or what would be the best approach to keep the template clean?
I know that it's possible to use scriptSource
- task: PythonScript#0
displayName: 'Run a Python script'
inputs:
scriptSource: 'filePath'
scriptPath: 'my_python.py'
arguments: '${{ parameters.my_param }}'
But as the template is in another repository than the repository ran in the pipeline, I don't think that I can reach that my_python.py without downloading it with a wget, or cloning, or doing additional steps. I am right?
Regards!
To use a template from another repo you need to define repository source like here:
# Repo: Contoso/LinuxProduct
# File: azure-pipelines.yml
resources:
repositories:
- repository: templates
type: github
name: Contoso/BuildTemplates
steps:
- template: common.yml#templates # Template reference
Once you have you need to just checkout this repo:
# Repo: Contoso/LinuxProduct
# File: azure-pipelines.yml
resources:
repositories:
- repository: templates
type: github
name: Contoso/BuildTemplates
steps:
- checkout: self
- checkout: templates #this download whole repo
- template: common.yml#templates # Template reference
Now you need to figure out where it is downloaded ;)
Multiple repositories: If you have multiple checkout steps in your job, your source code is checked out into directories named after the repositories as a subfolder of s in (Agent.BuildDirectory). If (Agent.BuildDirectory) is C:\agent\_work\1 and your repositories are named tools and code, your code is checked out to C:\agent\_work\1\s\tools and C:\agent\_work\1\s\code.
So if you have script in scripts folder in templates repo you will find it in $(Agent.BuildDirectory)\templates\scripts\script.py.
So then you can use it like this:
- task: PythonScript#0
displayName: 'Run a Python script'
inputs:
scriptSource: 'filePath'
scriptPath: '$(Agent.BuildDirectory)\templates\scripts\script.py'
arguments: '${{ parameters.my_param }}'