How to use the postgres db on the windows-latest agent used in the azure pipeline? - postgresql

I have a java maven project that I am building with an azure pipeline with as host "windows-latest" as it contains the correct java 13 version. However, for the integration tests, I need a postgres db and the "windows-latest" agent contains a postgres db, see: link. But how can I use this? I tried to use it by including it's serviceName in the Maven task as service:
services:
postgres: postgresql-x64-13
But then I get the error it can not find a service by that name.
I tried defining the db properties through env settings (see yml below), and then it shows the error:
Caused by: java.net.ConnectException: Connection refused
I also tried running it through a script task through the docker-compose.yml in the root of the project that I use during development, but docker-compose throws an error saying it can't find the compose file, I also doubt this the correct way.
So can I use the postgres db on the windows agent? and how?
My azure pipeline snippet:
variables:
MAVEN_CACHE_FOLDER: $(Pipeline.Workspace)/.m2/repository
MAVEN_OPTS: "-Dmaven.repo.local=$(MAVEN_CACHE_FOLDER)"
application_name: clearsky
service_name: backend
mygetUsername: myserUsername
mygetPassword: mytoken
SPRING_DATASOURCE_URL: jdbc:postgresql://localhost:5432/postgres
SPRING_DATASOURCE_USER: postgres
SPRING_DATASOURCE_PASSWORD: root
stages:
- stage: create_artifact
displayName: Create artifact
jobs:
- job: build
displayName: Build, test and publish artifact
steps:
- task: Maven#3
name: maven_package
displayName: Maven package
inputs:
goals: "package"
mavenPomFile: "backend/pom.xml"
options: '--settings backend/.mvn/settings.xml -DmygetUsername=$(mygetUsername) -DmygetPassword=$(mygetPassword)'
mavenOptions: "-Xmx3072m $(MAVEN_OPTS)"
javaHomeOption: "JDKVersion"
jdkVersionOption: "1.13"
mavenAuthenticateFeed: true

In Azure Devops Windows agen, the postgresql is disabled/stop by default.
Here is the configuration doc.
Property Value
ServiceName postgresql-x64-13
Version 13.2
ServiceStatus Stopped
ServiceStartType Disabled
You could try the following command to start the postgresql.
"C:\Program Files\PostgreSQL\13\bin\pg_ctl.exe" start -D "C:\Program Files\PostgreSQL\13\data" -w

Related

Azure yaml pipeline group variables not seen by task in a template file

I have a pipeline stage that is using a template as follows:
# Deploy to AKS
- stage: DeployTEST
displayName: Test env for my-app
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))
variables:
- group: 'my-app-var-group-test'
- group: 'package-variables'
- template: templates/shared-template-vars.yml#templates
jobs:
- deployment: TestDeployment
displayName: Deploy to AKS - Test
pool:
vmImage: $(vmImageName)
environment: env-test
strategy:
runOnce:
deploy:
steps:
- template: ./aks/deployment-steps.yml
...and the content of the template deployment-steps.yml is:
steps:
- script: |
echo AzureSubscription: '$(azureSubscription)'
echo KubernetesServiceConnection: '$(kubernetesServiceConnection)' # this is working
- task: KubernetesManifest#0
displayName: Create imagePullSecret
inputs:
action: createSecret
secretName: $(imagePullSecret)
dockerRegistryEndpoint: $(dockerRegistryServiceConnection)
kubernetesServiceConnection: $(kubernetesServiceConnection) # this is causing an error
I get an error like this:
There was a resource authorization issue: "The pipeline is not valid. Job TestDeployment: Step input kubernetesServiceConnection references service connection $(kubernetesServiceConnection) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
and like this when I try to select individual stages prior manual pipeline run:
Encountered error(s) while parsing pipeline YAML:
Job TestDeployment: Step input kubernetesServiceConnection references service connection $(kubernetesServiceConnection) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.
The errors above are misleading, because it is not an authorization issue:
the referenced K8s service connection is authorized
when I hardcode the value of the $(kubernetesServiceConnection) variable the pipeline runs just fine - no errors
variable group my-app-var-group-test is authorized - IMPORTANT: this is where the $(kubernetesServiceConnection) variable is defined
NOTE: The variable kubernetesServiceConnection is defined in the my-app-var-group-test variable group & when I comment out the KubernetesManifest task, the value of the $(kubernetesServiceConnection) variable is properly printed to the pipeline console output without any issues and the pipeline runs successfully!?
I know I could use parameters to pass values into the template, but this setup is already used by all other pipelines (variable group vars are used/references in templates) and this issue appeared on a newly created pipeline. I have used file comparison to compare the yaml of a working pipeline and this one and failed to spot anything...
I might be missing something obvious, but I spent hours on this failing to resolve the error...

Azure Devops Pipeline: There was a failure in sending the provision message: Unexpected response code from remote provider InternalServerError

I have created a new pipeline as part of investigating this issue as follows:
New Pipeline > Azure Repos Git > Start pipeline
I have not edited the base template pipeline code, so it looks like this:
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
- script: |
echo Add other tasks to build, test, and deploy your project.
echo See https://aka.ms/yaml
displayName: 'Run a multi-line script'
However randomly (at least 1 in every 15), I will see the following warning when running the pipeline:
There was a failure in sending the provision message: Unexpected response code from remote provider InternalServerError
Considering that im using their starter pipeline without any changes, I don't think this is an issue with the yaml, but an issue on Azure DevOps / Hosted Agents side.
I also enabled the diagnostic logs, and found the following in Agent_20220611-100355-utc.log:
[2022-06-11 10:03:56Z ERR JobNotification] Connection to monitor port 49100 failed!
[2022-06-11 10:03:56Z ERR JobNotification] System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (111): Connection refused [::ffff:127.0.0.1]:49100
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at System.Net.Sockets.Socket.Connect(IPAddress address, Int32 port)
at Microsoft.VisualStudio.Services.Agent.JobNotification.ConnectMonitor(String monitorSocketAddress)
[2022-06-11 10:03:56Z ERR JobNotification] Invalid socket address hello. Job Notification will be disabled.
Does anyone have any suggestions?
I switched from windows-latest to windows-2022 and that cured things for me. Which is weird because this says they are the same thing.
Maybe a similar trick will work for you? Try ubuntu-20.04.

Azure DevOps - Unable to load the service index for source

I am running a Azure DevOps pipeline, and I get the following error.
error: Unable to load the service index for source https://pkgs.dev.azure.com/companyName/_packaging/12345678-1234-1234-1234-1234567890ab/nuget/v3/index.json.
error: Response status code does not indicate success: 403 (Forbidden - User '12345678-1234-1234-1234-1234567890ab' lacks permission to complete this action. You need to have 'ReadPackages'. (DevOps Activity ID: 12345678-1234-1234-1234-1234567890ab)).
And I am getting the error from a task whose yaml looks like this.
steps:
- task: DotNetCoreCLI#2
displayName: 'Push the created package to Nuget feed. '
inputs:
command: push
publishVstsFeed: '12345678-1234-1234-1234-1234567890ab'
versioningScheme: byPrereleaseNumber
Any ideas what to be done?
Azure DevOps - Unable to load the service index for source
It seems you have not authenticated your private feed, please try to add the task NuGetAuthenticate
- task: NuGetAuthenticate#0
displayName: 'NuGet Authenticate'
inputs:
nuGetServiceConnections: TestNuGet
And create a service connection for the private feed:

Azure DevOps connecting to remote Artifactory

I am trying to figure out the appropriate way to connect an Azure DevOps pipeline that executes a Maven build to a remote JFrog Artifactory Maven repository.
I first looked at Feeds and Upstream Sources. I didn't see anything in the documentation that shows how to do this. Indeed it looks as though you can't actually do it.
Then I looked at Service Endpoints. Here, I was able to create a service endpoint that points at my Artifactory host. Great! I added a task of Maven Authenticate, which is the only one I saw that allows me to reference the mavenServiceConnection. But the pipeline still fails when trying to resolve artifacts, because it only looks at Maven Central.
# Docker
trigger:
- master
resources:
- repo: self
variables:
tag: '$(Build.BuildId)'
stages:
- stage: Build
displayName: Build image
jobs:
- job: Build
displayName: Build
pool:
vmImage: 'ubuntu-latest'
steps:
- task: MavenAuthenticate#0
inputs:
mavenServiceConnections: 'eti-libs-snapshots-local'
- task: Maven#3
inputs:
options: '-X'
mavenPomFile: 'pom.xml'
publishJUnitResults: true
testResultsFiles: '**/surefire-reports/TEST-*.xml'
javaHomeOption: 'JDKVersion'
mavenVersionOption: 'Default'
mavenAuthenticateFeed: false
effectivePomSkip: false
sonarQubeRunAnalysis: true
sqMavenPluginVersionChoice: 'latest'
- task: Docker#2
displayName: Build an image
inputs:
command: build
dockerfile: '$(Build.SourcesDirectory)/Dockerfile'
tags: |
$(tag)
Edited: Found some more relevant information
So as Kontekst pointed out, I needed to add the Jfrog extension for pipelines. I have done that and got it configured.

Seeing `permission: denied` errors in task when upgrading to version 3 of concourse

We just upgraded one of concourses to 3.3.0 and we're getting a weird error on one of our jobs.
runc create: exit status 1: container_linux.go:264: starting container process caused "process_linux.go:339: container init caused \"rootfs_linux.go:56: mounting \\\"/var/vcap/data/baggageclaim/volumes/live/17c7c6fb-a294-4274-4d3c-99d14980ab4f/volume\\\" to rootfs \\\"/var/vcap/data/garden/graph/aufs/mnt/9985cede6b6b24ac198ea4a6b252fcaa56eb1f0062cf102e9d45f293ec82ee9d\\\" at \\\"/var/vcap/data/garden/graph/aufs/mnt/9985cede6b6b24ac198ea4a6b252fcaa56eb1f0062cf102e9d45f293ec82ee9d/scratch\\\" caused \\\"mkdir /var/vcap/data/garden/graph/aufs/mnt/9985cede6b6b24ac198ea4a6b252fcaa56eb1f0062cf102e9d45f293ec82ee9d/scratch: permission denied\\\"\""
The configuration for the task is
- task: create-release
config:
platform: linux
run:
path: echo
As of version 3.0 of concourse and above, you must specify an image_resource in every task configuration.
Not having an image (now deprecated and renamed rootfs_uri) or image_resource defined in your task configuration used to be undocumented and unspecified behavior where it would let Garden choose the image based on Garden's default.
Try a task that looks like
- task: create-release
config:
platform: linux
image_resource:
type: docker-image
source:
repository: alpine
run:
path: echo