I have an Azure DevOps pipeline which is failing to run because it seems to be using an old connection string.
The pipeline is for a C# project, where a FileTransform task updates an appsettings.json file with variables set on the pipeline.
The variables were recently updated to use a new connection string, however, when running a Console.PrintLn before using it and viewing it on the pipeline, it shows an outdated value.
Many updates similar to this have been run in the past without issue.
I've also recently added a Powershell task to echo what the value is in the variables loaded while the pipeline is running, which does display the new value.
I've checked the order of precedence of variables and there shouldn't be any other variables being used.
There is no CacheTask being used in this pipeline.
Does anyone have any advice to remedy this? It seems that the pipeline itself is just ignoring the variables set on the pipeline.
There is a problem with the recent File transform task version v1.208.0.
It will shows the warning message and not update the variable value correctly.
Warning example:
Resource file haven't been set, can't find loc string for key: JSONvariableSubstitution
Refer to this ticket: File transform task failing to transform files, emitting "Resource file haven't been set" warnings
The issue is from Task itself instead of the Pipeline configuration. Many users have the same issue.
Workaround:
You can change to use the File Transform task Version 2 to update the appsettings.json file.
Here is an example: Please remove the content in XML Transformation rules field and set the JSON file path
I have 2 config files, ConnectionStrings.config and ConnectionStrings.Release.config. These files are used to store various connection strings we use in our app.
<?xml version="1.0" standalone="yes"?>
<ConnectionStrings xmlns="http://tempuri.org/ConnectionStrings.xsd">
<Service>
<Name>MyService</Name>
<Address>http://localhost/#{ConnectionString}#/MyServiceService.svc</Address>
<ClassName>MyClass, MyClass</ClassName>
</Service>
</ConnectionStrings>
When we deploy locally we use the normal ConnectionStrings.config without any replace tokens in the file. The .release.config is the one that has the tokens.
I'm using these 2 market place tasks
[Tasks in my release pipeline]
1
From what I understand, the transform task is taking the .release.config file and "renaming it" .config, then the replace tokens task replaces the necessary tokens with variables in my pipeline. Finally I copy these to my target. My source of the copy task is my Git working directory in Azure.
When I check the results, I don't see the ConnectionStrings.config changed at all and its just copied over like if it was a local deploy.
What am I doing wrong?
It seems that you are using this extension: XDT Transform. After installing it in my organization, there are 2 external tasks: XDT tranform task and Replace Tokens task in release pipeline.
below is my repository structure, the ConnectionStrings.config and ConnectionStrings.Release.config.
And my release pipeline looks like below.
I create a pipeline variable ConnectionStrings, and set its value to hello_world, use the PowerShell script gc _215\ConnectionStrings.config to print the replaced result. Creating a new release we can see below log.
Thus it works as expected.
I'm using Azure DevOps to deploy an ASP.NET application to an IIS servers on-prem. The IIS servers are using a shared configuration so they need a custom Machine Key setup.
I can use XML transform to add add the machineKey entry in the Web.Config
<system.web>
<machineKey decryptionKey="{hex-key value}" validationKey="{hex-key value}"/>
</system.web>
but I don't want to have the actual keys in source control so I'll need to replace those values at deploy time. Substitution is easy enough for appsettings and connection strings but how can I substitute values in the System.Web section of the Web.Config?
How can I substitute values in the System.Web section of the
Web.Config?
What about consider to use one extension Replace token? For why I recommend it is because it can achieve the demand that it can only be replaced during the pipeline running.
Also, its usage is very convenient. Just need to specify the prefix and suffix in the task, and then make apply them in to your web.config file.
Then specify the corresponding variables with same name in Variables tab.
Only this, during the pipeline running, the task could find the corresponding token correctly and replace the value into it.
For detailed steps, you could refer to my previous answer for details: Use replace token task.
Spring cloud config server supports reading property files with name ${spring.application.name}.properties. However I have 2 properties files in my application.
a.properties
b.properties
Can I get the config server to read both these properties files?
Rename your properties files in git or file system where your config server is looking at.
a.properties -> <your_application_name>.properties
a.properties -> <your_application_name>-<profile-name>.properties
For example, if your application name is test and you are running your application on dev profile, below two properties will be used together.
test.properties
test-dev.properties
Also you can specify additional profiles in bootstrap.properties of your config client to retrieve more properties files like below. For example,
spring:
profiles: dev
cloud:
config:
uri: http://yourconfigserver.com:8888
profile: dev,dev-db,dev-mq
If you specify like above, below all files will be used together.
test.properties
test-dev.properties
test-dev-db.prpoerties
test-dev-mq.properties
Note that the provided answer assumes your property files address different execution profiles. If they dont, i.e., your properties are split into different files for some other reason, e.g., maintenance purposes, divided by business/functional domain, or any other reason that suits your needs, then, by defining a profile for each such file, you are just "abusing" the profile feature, for achieving your goal (multiple property files per app).
You could then ask "OK, so what is the problem with that?". The problem is that you restrain yourself from various possibilities that you would otherwise have. If you actually want to customize your application configuration by profile you will have to create pseudo, sub, profiles for that since the file name is already a profile. Example:
Your application configuration could be customized by different profiles, which you use inside your springboot application (e.g. in #Profile() annotation), let them be dev, uat, prod. You can boot your application setting different profiles as active, e.g. 'dev' vs 'uat', and get the group of properties that you desire. For your a.properties b.properties and c.properties file, if different file names were supported, you would have a-dev.properties b-dev.properties and c-dev.properties files vs a-uat.properties b-uat.properties and c-uat.properties files for 'dev' and 'uat' profile.
Nevertheless, with the provided solution, you already have defined 3 profiles for each file: appname-a.properties appname-b.properties, and appname-c.properties: a, b, and c. Now imagine you have to create a different profile for each... profile(! it already shows something goes wrong here)! you would end up with a lot of profile permutations (which would get worse as files increase): The files would be appname-a-dev.properties, appname-b-dev.properties, app-c-dev.properties vs appname-a-uat.properties, appname-b-uat.properties, app-c-uat.properties, but the profiles would have been increased from ['dev', ' uat'] to ['a-dev', 'b-dev', 'c-dev', 'a-uat', 'b-uat', 'c-uat'] !!!
Even worse, how are you going to cope with all these profiles inside your code and more specifically your #Profile() annotations? Will you clutter the code space with "artificial" profiles just because you want to add one or two more different property files? It should have been sufficient to define your dev or uat profiles, where applicable, and define somewhere else the applicable property file names (which could then be further supported by profile, without any other configuration action), just as it happens in the externalized properties configuration for individual springboot apps
For argument completeness, I will just add here that if you want to switch to .yml property files one day, with the provided profile-based naming solution, you also loose the ability to define different "yaml document sections per profile" inside the same .yml file (Yes, in .yml you can have one property file yet define multiple logical yml documents inside, which its usually done for customizing the properties for different profiles, while having all related properties in one place). You loose the ability because you have already used the profile in the file name (appname-profile.yml)
I have issued a pull request with a minor fix for spring-cloud-config-server 1.4.x, which allows defining additionally supported file names (appart from "application[-profile]" and "{appname}[-profile]", that are currently supported) by providing a spring.cloud.congif.server.searchNames environment property - analogous to spring.config.name for springboot apps. I hope it gets reviewed and accepted.
I came across the same requirement lately with a little more constraint that I am not allowed to play around the environment profiles. So I wasn't allowed to do as the accepted answer. I'm sharing how I did it as an alternative to those who might have same case as me.
In my application, I have properties such as:
appxyz-data-soures.properties
appxyz-data-soures-staging.properties
appxyz-data-soures-production.properties
appxyz-interfaces.properties
appxyz-interfaces-staging.properties
appxyz-interfaces-production.properties
appxyz-feature.properties
appxyz-feature-staging.properties
appxyz-feature-production.properties
application.properties // for my use, contains local properties only
bootstrap.properties // for my use, contains management properties only
In my application, I have these particular properties set that allow me to achieve what I needed. But note I have the rest of needed config as well (enable cloud config, actuator refresh, eureka service discovery and so on) - just highlighting these for emphasis:
spring.application.name=appxyz
spring.cloud.config.name=appxyz-data-soures,appxyz-interfaces,appxyz-feature
You can observe that I didn't want to play around my application name but instead I used it as prefix for my config property files.
In my configuration server I configured in application.yml to capture pattern: 'appxyz-*':
spring:
cloud:
config:
server:
git:
uri: <git repo default>
repos:
appxyz:
pattern: 'appxyz-*'
uri: <another git repo if you have 1 repo per app>
private-key: ${git.appxyz.pk}
strict-host-key-checking: false
ignore-local-ssh-settings: true
private-key: ${git.default.pk}
In my Git repository I have the following. No application.properties and bootstrap because I didn't want those to be published and overridden/refreshed externally but you can do if you want.
appxyz-data-soures.properties
appxyz-data-soures-staging.properties
appxyz-data-soures-production.properties
appxyz-interfaces.properties
appxyz-interfaces-staging.properties
appxyz-interfaces-production.properties
appxyz-feature.properties
appxyz-feature-staging.properties
appxyz-feature-production.properties
It will be the pattern matching pattern: 'appxyz-*' that will capture and return the matching files from my git repository. The profile will also apply and fetch the correct property file accordingly. The prioritization of value is also preserved.
Furthermore, if you wish to add more file in your application (say appxyz-circuit-breaker.properties), we only need to do:
Add the name pattern in the spring.cloud.config.name=...,appxyz-circuit-breaker
The add the copies of the file locally and also externally (in the git repo.
No need to add/modify more or restart your configuration server later on. For new application, it's like a one time registration thing to add an entry under the repos of application.yml.
Hope it helps in one way or another!
In your application bootstrap.properties, you have to specify like below:
spring.application.name=a,b
I have a web project that has uses two service endpoints located in the Web.config file under the client --> endpoint --> address portion
I have found the following per Octopus Variables section but cannot seem to find any reference of how to address the changes using and actual variable like you normally would
I am using the webui for Octopus which would be
http://{server-name}/app#/projects/{project-name}/variables
variable-substitution-syntax
I attempted assigning the variables like so, but the values never updated
the original entries look like the following
<endpoint address="http://services-test.example.com/test.svc/soap" binding="basicHttpBinding" bindingConfiguration="soap" contract="test.service" name="soap" />
Name Address Instance
Endpoint[A].Address test-service-a.example.com 1
Endpoint[B].Address test-service-b.example.com 2
Is this something that is ever possible using Octopus Variables? (I know it can be done using regular Web.config Transforms, as we are doing that already).
If it is possible what is would the correct replacement value for the
endpoint address
be and how would I accomplish this for multiple different endpoint addresses?
Sounds like you are most of the way there. If it's already working with your Web.config transforms then all you need to do is replace the value IN the transform with the variable replacement token.
For example: Web.Release.Config
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<system.serviceModel>
<client>
<endpoint address="http://#{Server1}/test.svc/soap" name="x1"
xdt:Locator="Match(name)" xdt:Transform="SetAttributes(address)" />
<endpoint address="#{Endpoint2}" name="x2"
xdt:Locator="Match(name)" xdt:Transform="SetAttributes(address)" />
</client>
</system.serviceModel>
</configuration>
Of course there are lots of options here.
For filenames you could stick with default conventions 'Web.Release.config' or go with 'Web.[Environment].config' or go with something custom. We use 'Web.Octopus.Config' so that it wont get picked up by any other process.
More on naming transforms here: https://octopus.com/docs/deploying-applications/configuration-files#Configurationfiles-Namingconfigurationtransformfiles
More on custom transforms (Web.Octopus.com) here: https://octopus.com/docs/deploying-applications/configuration-files#Configurationfiles-AdditionalConfigurationTransforms
For variables, you could define a variable for just the server (name=x1) which is simpler or just put the whole address in a variable with gives Octopus a lot more control (name=x2).
The key part is getting the variable replacement tokens into the config. Octopus runs the variable substitution on config files first, and then runs the transforms. What that means is the first pass will replace the tokens in your Web.Release.Config, and then Octopus will run the transforms against Web.config
Hope that helps.