How can I allow Azure DevOps Hosted Agent access my Amazon RDS PostgreSql without setting the Security Group to Anywhere. I was looking for IP Range or something to whitelist Azure DevOps Agents but can't find it.
In Azure, I can check a box to grant all "Azure DevOps Services" access to my Azure SQL Database but of course its not present in AWS.
I don't think we can access the Amazon RDS PostgreSql directly from Azure DevOps Hosted Agent, I mean using the hosted service account.
However, Amazon RDS for PostgreSQL Supports User Authentication with Kerberos and Microsoft Active Directory, so we can try writing script to access it by using the specific credential. Then run the scripts in pipeline by adding corresponding tasks (e.g AWS CLI or AWS PowerShell).
Also check How do I allow users to connect to Amazon RDS with IAM credentials?
For the IP ranges, please refer to Allowed address lists and network connections and Microsoft-hosted Agents for details.
The IPs used for the hosted Agent IP ranges are linked through here. I have not had much success using it for hosted agents. The list is big and the documentation is not really clear about what types of services you need to whitelist.
I would go with whitelisting the hosted agent IP just-in-time during the pipeline run, then remove it as a final step. First you can grab the ip of the hosted agent:
$hostedIPAddress = Invoke-RestMethod http://ipinfo.io/json | Select -exp ip
Then you could use the AWS CLI or AWS PowerShell module to add the specific IP. Azure DevOps AWS tools task includes the CLI.
Do the needed work against the DB, then make sure you clean up the rule\temp security group at the end.
Related
I have a server in AWS, which is hosting a security tool. Azure DevOps supports this tool and I've installed the add on for it. I've added the step to my Pipeline and configured the service connection.
We are using Hosted Agents in a Cloud AZD instance.
When I run my pipeline, I get the following error:
##[error][TOOL] API GET '/api/server/version' failed, error was: {"errno":"ETIMEDOUT","code":"ETIMEDOUT","syscall":"connect","address":"1.1.1.1","port":443}
In my AWS security group, I have allowed the Inbound IP's for Azure DevOps listed here https://learn.microsoft.com/en-us/azure/devops/organizations/security/allow-list-ip-url?view=azure-devops&tabs=IP-V4#ip-addresses-and-range-restrictions
I have also allowed the Geographical IP's for listed in the json file here https://www.microsoft.com/en-us/download/details.aspx?id=56519
If I allow all traffic for 443 through the security group as a test, this works as expected. This is not a solution however as this is a security tool and should not be public.
In my pipeline, I added a task to run a curl command to inspect the IP's of the pipeline. Neither of these ranges appear in any list I can find published.
51.142.72.0/24
51.142.229.0/24
I was advised to post here by AzureDevOps on Twitter for some help, so hopefully someone can assist me here.
I have an Azure SQL server where I wish to deploy my database via dacpac using GitHub CI/CD. I am using the Azure SQL Deploy action with Service Principal for Azure login action
Due to policy restrictions on the client side, the "Deny Public Network Access" is always enabled and therefore while deploying even though the service principal login works, the GitHub action is unable to add the IP Address to the firewall rule.
We are using Self-Hosted GitHub runners. Is there any workaround to deploying the database via CI/CD under such circumstances where we cannot add the firewall rule to whitelist the agent/runners IP Address?
The solution was to do away with Azure Login action and add self-hosted runner virtual network in the Azure SQL Firewall settings:
The Azure Login action attempts to add IP Address of the runner to the Azure SQL Firewall. Hence, this action must not be used. I removed this action and relied on the second step for seamlessly accessing Azure SQL Database instead.
The Azure SQL Deploy action requires either above login step or "allow access to azure services" to be turned ON. Both of which were not an option for me. I decided to go with an action that runs sqlpackage.exe. https://learn.microsoft.com/en-us/sql/tools/sqlpackage/sqlpackage-publish?view=sql-server-ver15
I am using self-hosted runners which are hosted within the virtual network configured within Azure. However, the point that I had missed was to add the virtual network to the Firewall settings in Azure SQL. Once, I did that, I did not require an explicit login or whitelisting runner ip addresses.
After reading this documentation https://learn.microsoft.com/en-us/azure/devops/organizations/security/about-permissions?view=azure-devops-2020&tabs=preview-page#active-directory-and-azure-active-directory-security-groups it seems that Azure DevOps Server on premises can be used only with Active Directory but not with Azure Active Directory. Is it correct? Is there a way to use Azure DevOps Server with Azure Active Directory?
You are right, as on-premises Azure DevOps Server, is built on a SQL
Server back end which can work with on-prem AD .
Azure ad authentication for on prem directly is not feasible as it is
cloud based and involves azure based sql .
Note: On-premise SQL Server only supports Windows Authentication and
SQL Server Authentication. Windows authentication depends on Kerberos
(or NTLM), which needs an Active Directory domain to authenticate the
user in. Azure Active Directory does not handle Kerberos tokens.
But we can opt to migrate devops server to azure devops services to
work with AAD .Migration is when one wants to completely move to
cloud and it involves complex process as database/hardware is
involved.
If one wants to continue on premise server to use existing on-premises
identities with Azure DevOps, you can integrate directories with
Azure AD by using Azure AD Connect which connects an organization’s
on-prem Active Directory instance and cloud-based azure active
directory.It involves On prem users to be mapped to AAD tenant.
Please refer this Document.
Other References:
PTA
Migrate - Azure DevOps | Microsoft Docs
Will kubernetes service connections in azure devops work with an AKS cluster that is bound to AAD via openidconnect? Logging into such clusters goes through an openidconnect flow that involves a device login + browser. How is this possible w/ azure devops k8s service connections?
Will kubernetes service connections in azure devops work with an AKS
cluster that is bound to AAD via openidconnect?
Unfortunately to say, no, this does not support until now.
According to your description, what you want to connect with in Azure Devops Kubernetes service connection is Azure Kubernetes service. This means you would select Azure Subscription in Choose authentication. BUT, this connection method is using Service Principal Authentication (SPA) to authenticate, which does not yet supported for the AKS that is bound with AAD auth.
If you connect your AKS cluster as part of your CI/CD deployment in Azure Devops, and attempt to get the cluster credentials. You would get a warning response which inform you to log in since the service principal cannot handle it:
WARNING: To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code *** to authenticate.
You should familiar with this message, it needs you open a browser to login in to complete the device code authentication manually. But this could not be achieve in Azure Devops.
There has a such feature request raised on our forum which request us expand this feature to Support non-interactive login for AAD-integrated clusters. You can vote and comment there to advance the priority of this suggestion ticket. Then it could be considered into the develop plan by our Product Manager as soon as possible.
Though it could not be achieved directly. But there has 2 work around can for you refer now.
The first work around is change the Azure DevOps authenticate itself from AAD client to the server client.
Use az aks get-credentials command and specify the parameter --admin with it. This can help with bypassing the Azure AD auth since it can let you connect and retrieve the admin credentials which can work without Azure AD.
But, I do not recommend this method because subjectively, this method is ignoring the authentication rules set in AAD for security. If you want a quick method to achieve what you want and not too worry about the security, you can try with this.
The second one is using Kubernetes service accounts
You can follow this doc to create a service account. Then in Azure Devops, we could use this service account to communicate with AKS API. Here you also need to consider about the authorized IP address ranges in AKS.
After the service account created successfully, choose Service account in the service connection of Azure Devops:
Server URL: Get it from the AKS instance(API server address) in Azure portal, then do not forget append the https:// before it while you input it into this service connection.
Secret: Generate it by using command:
kubectl get secret -n <name of secret> -o yaml -n service-accounts
See this doc: Deploy Vault on Azure Kubernetes Service (AKS).
Then you can use this service connection in Azure Devops tasks.
Is there a way to deploy SQL Database project to Azure SQL Managed Database from Azure DevOps ? Are there ports to open on Azure Network Security Group ?
Thank you
Bertrand
No, not using Azure DevOps hosted agent. That is because a managed instance does not have a public ip address an DevOps can not find your server. If you want to use DevOps you must host your own agent on a separate VM and open up all the internal/external routing you need.
the 'Azure SQL Database Deployment' task in the release pipeline should work for you. With a service connection (Service Principal in Azure) with the correct permissions on the subscription or resource group you can write to the resources, no need to open specific ports in NSGs
This is now possible on MI and Azure DevOps.
Not sure what the NSG settings are if you have restricted access via a NSG.