Cannot use Azure SQL with Azure DevOps Server 2019 - (Azure Government) - azure-devops

I have been struggling to get my DevOpsServer 2019-RC1 installation to see my Azure SQL Server
My DevOpsServer install is on an Azure VM, as recommended
I have implemented everything from the below documentation regarding the set up of an MSI for the VM, and granting the MSI access to Azure SQL via AAD authentication:
https://learn.microsoft.com/en-us/tfs/server/install/install-azure-sql
I have also added the VNET where the VM NIC is located, as a VNET firewall rule on the Azure SQL Server to ensure there are no connection issues
My DevOpsServer refuses to see the Azure SQL Server or its databases
To confirm that Azure SQL is not blocked from the VM, I successfully created an ODBC system DSN connection on the VM, which allows me to see the Azure SQL Server, and its Databases
Per the reference documentation:
When setting up a new DevOps Server instance, i selected "This is a
new Azure DevOps Server deployment"
On the Database page of the configuration wizard, specify the Azure SQL
Database server instance, typically in the form of
"SQLInstanceName.database.windows.net"
Please let me know if there is anything else i can do to help the Devops Server Configuration Wizard see my Azure SQL Server and Databases
PS I am trying to get this working in Azure Government (MAG) if this changes the capability...
Error received when attempting to connect to the SQL Database programmatically via the following Powershell script:
# Retrieve the access token
$response = Invoke-WebRequest -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fdatabase.usgovcloudapi.net' -Method GET -Headers #{Metadata="true"}
$content = $response.Content | ConvertFrom-Json
$AccessToken = $content.access_token
# Create the connection
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Data Source = test-sqlsrv.database.usgovcloudapi.net; Initial Catalog = inldb"
$SqlConnection.AccessToken = $AccessToken
$SqlConnection.Open()
# Send a Query
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = "SELECT * from <TABLE>;"
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)

Try running ALTER USER [VMName] WITH DEFAULT_SCHEMA=dbo from both the collection and config DBs. I have a PR out to add this to the documentation and this might be causing your issue.

Try connecting to the SQL instance via Powershell instead of Azure DevOps Server.
You can run the following script to do that:
# Retrieve the access token
$response = Invoke-WebRequest -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fdatabase.windows.net%2F' -Method GET -Headers #{Metadata="true"}
$content = $response.Content | ConvertFrom-Json
$AccessToken = $content.access_token
# Create the connection
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Data Source = <AZURE-SQL-SERVERNAME>; Initial Catalog = <DATABASE>"
$SqlConnection.AccessToken = $AccessToken
$SqlConnection.Open()
# Send a Query
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = "SELECT * from <TABLE>;"
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
This should at least give you a more descriptive error to debug.

Gvazzana, I also recommend testing with the script danwkennedy provided, but be sure to change database.windows.net to database.usgovcloudapi.net for Azure Government.
I would pay close attention to any errors around obtaining an access token and errors associated with opening the SQL connection.
If nothing's standing out is it possible to capture a Fiddler trace?

I agree that you should perform the PowerShell test to test your MSI. It looks like you are having issues reaching the IP to get the access token. If you are attempting to connect from a Government network, it is likely that you are going through some type of web proxy service (Blue Coat, etc.) to get to the internet. You can maybe trick the proxy by emulating the user agent string from a browser by using the -UserAgent parameter on your Invoke-WebRequest command.
If that doesn't work, you may need to reach out to your proxy team to troubleshoot the issue.
Lastly, if you are able to test successfully in PowerShell, but you still get the connection test error in the DevOps install, make sure to follow the steps here:
https://learn.microsoft.com/en-us/tfs/server/install/install-azure-sql
I forgot to add the user to the master DB... Once I did that, everything worked. Good luck.

You can access to your Azure Devops database with your Windows Authentication level. So your Windows login user should have configured sysadmin privilege in your database (e.g Microsoft SQL server)

Related

Run SQL query using powershell and AAD authentication with an SSO account

I am trying to run a set of queries from Cloud Shell using powershell that require and Azure AD user. I generally use invoke-sqlcmd using the server admin and password but the specific query I want to run require an AD user. I would like to run the query as myself. I have found that I can use the ConnectionString parameter to do this, but my problem is I do not have a password as I login to Azure via SSO. I basically want to mimic the process of going to my DB in Azure, going to query editor, using AAD authentication and running a query, but with powershell. Is there a way I can do this? Example of what I am trying to do below:
$serverName = 'server'
$dbName = 'database'
$query = 'CREATE USER [AAD_User] FROM EXTERNAL PROVIDER;'
Invoke-Sqlcmd -ServerInstance $ServerName -Database $dbName -Query $query
When I attempt this I get:
Invoke-Sqlcmd: Cannot authenticate using Kerberos. Ensure Kerberos has been initialized on the client with 'kinit' and a Service Principal Name has been registered for the SQL Server to allow Kerberos authentication.
I tried to reproduce the same in my environment and got the results like below:
As SSO is enabled and you do not have a password, you can try generating access token to perform the action.
I used the below script to add the user in Azure Database:
Connect-AzAccount
$connectionString = "Server=testrukserver.database.windows.net;Initial Catalog=testdb;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30"
$accessToken = (Get-AzAccessToken -ResourceUrl https://database.windows.net).Token
$sqlConn = New-Object System.Data.SqlClient.SqlConnection
$sqlConn.ConnectionString = $connectionString
$sqlConn.AccessToken = $accessToken
$sqlConn.Open()
Invoke-Sqlcmd -ServerInstance testrukserver.database.windows.net -Database testdb -AccessToken $accessToken -query 'CREATE USER [test#xxx.onmicrosoft.com] FROM EXTERNAL PROVIDER'
$sqlConn.Close()
The user is added sccessfully in the Azure Database like below:
Reference:
Connecting to Azure SQL Database with AAD Authentication using Powershell by alex stuart

Programmatically Deploying Power BI Reports to Power BI Report Server and change Connection String

Is there any method to deploy Power BI reports to Power BI Report Server without having to manually copy these files, upload them to the server and finally change the data source connectivity information for each report on a report by report basis which is not practical in each customer sites.
Eg. PowerBI Report File - 'Report_1' need to Deploy on Customer server S1, S2, S3, & so on.
Now we doing manually copy these files, upload them to the server and finally change the data source connectivity information for each report on a report by report basis which is not practical in each customer sites.
How we can automate the deployment of PBIX reports to Power BI Report Server and changing Datasource connection string Pro-grammatically.?
Microsoft releasing feature in 2020 Jan to update connection string using API.
Microsoft releasing feature in 2020 Jan. But There is any way in 2019 ? any other way for update connection string ?
Microsoft Link
Finally invented one trick to update Connection String in PowerBI.
First Install PowerBI API in Powershell.
Microsoft API don’t give ability to update connection string but give permission to update username.
Both username and connection string are stored in encrypted format in database.
So logic is pass connection string to username and then copy encrypted string to connection string column in database.
Just check below example I have written and invented this trick. Thank you.
# Code By SB 2019
$ReportServerURI = 'http://localhost/PowerBIReports' # Input Local path of powerbi file
$filePath = "C:\12.pbix" # Input Local path of powerbi file
$PBIxfileName = "12" # INput your Powerbi File Name
$FolderName ='NewDataset' # Input PowerBI server Folder Name Where you wann to deploy
$Username ='admin'
$password ='password'
$ReportServerName ='localhost\SQl2017' #input SQL server where POWERBI database installed
$ReportServerDatabase = 'ReportServerPowerBI' #input PowerBi Database Name
$ConnectionString ='data source=Client01\SQL2019;initial catalog=Client_SB_1' # input New Connection String / Client ConnectionString
$FolderLocation = '/'
$FolderPath = $FolderLocation + $FolderName
write-host "Deployment Started ..." -ForeGroundColor Yellow
$session = New-RsRestSession -ReportPortalUri $ReportServerURI
Write-RsRestCatalogItem -WebSession $session -Path $filePath -RsFolder $folderPath -Description $Description -Overwrite
$datasources = Get-RsRestItemDataSource -WebSession $session -RsItem "$FolderPath/$PBIxfileName"
$dataSources[0].DataModelDataSource.AuthType = ‘Windows'
$dataSources[0].DataModelDataSource.Username = $ConnectionString
$dataSources[0].DataModelDataSource.Secret = $password
Set-RsRestItemDataSource -WebSession $session -RsItem "$folderPath/$PBIxfileName" -RsItemType PowerBIReport -DataSources $datasources
$ID = $dataSources[0].Id
$Query = " Update [DataModelDataSource] SET ConnectionString = Username From [dbo].[DataModelDataSource] Where DataSourceID ='" + $ID + "' "
Invoke-Sqlcmd -Query $Query -ServerInstance CPMSUNRSQL17\CPMSRINST17 -Database ReportServerPowerBI
$datasources = Get-RsRestItemDataSource -WebSession $session -RsItem "$FolderPath/$PBIxfileName"
$dataSources[0].DataModelDataSource.Username = $Username
$dataSources[0].DataModelDataSource.Secret = $password
Set-RsRestItemDataSource -WebSession $session -RsItem "$folderPath/$PBIxfileName" -RsItemType PowerBIReport -DataSources $datasources
write-host "Deployment Done . . ." -ForeGroundColor Green
This would only work if the change you need can be driven by a parameter, e.g. for a SQL Server source, can set database, schema or table name (but not server name).
First I would set up the query definitions to use query parameter(s) and test. The specifics of this would depend on your data sources and scenario - you have not provided any info on that.
Then I would call the appropriate REST API Update Parameters method - probably the Group version.
https://learn.microsoft.com/en-us/rest/api/power-bi/datasets/updateparametersingroup
You can deploy using to Power BI Report Server, and change connections and other setting using Powershell using the ReportingServiceTools library, As Power BI Report Service is SSRS you can use the same tools, to load reports, change data connections etc
Example of deploying a file and here
You can also change the connection settings directly in the PBIX file. If you change the extension from pbix to zip you can take a look inside.
If you open the 'Connections' file, it contains the setting via a JSON structured file
{"Version":1,"Connections":[{"Name":"EntityDataSource","ConnectionString":"Data Source=asazure://region.asazure.windows.net/somecubegoes her;Initial Catalog=SmartSpacesAnalysis;Cube=SmartSpacesModel","ConnectionType":"analysisServicesDatabaseLive"}]}
That can be read and changed if needed

How to run Powershell script on local computer but with credentials of a domain user

I have to implement a solution where I have to deploy a SSIS project (xy.ispac) from one machine to another. So far I've managed to copy-cut-paste the following stuff from all around the internet:
# Variables
$ServerName = "target"
$SSISCatalog = "SSISDB" # sort of constant
$CatalogPwd = "catalog_password"
$ProjectFilePath = "D:\Projects_to_depoly\Project_1.ispac"
$ProjectName = "Project_name"
$FolderName = "Data_collector"
# Load the IntegrationServices Assembly
[Reflection.Assembly]::LoadWithPartialName("Microsoft.SqlServer.Management.IntegrationServices")
# Store the IntegrationServices Assembly namespace to avoid typing it every time
$ISNamespace = "Microsoft.SqlServer.Management.IntegrationServices"
Write-Host "Connecting to server ..."
# Create a connection to the server
$sqlConnectionString = "Data Source=$ServerName;Initial Catalog=master;Integrated Security=SSPI;"
$sqlConnection = New-Object System.Data.SqlClient.SqlConnection $sqlConnectionString
$integrationServices = New-Object "$ISNamespace.IntegrationServices" $sqlConnection
$catalog = $integrationServices.Catalogs[$SSISCatalog]
# Create the Integration Services object if it does not exist
if (!$catalog) {
# Provision a new SSIS Catalog
Write-Host "Creating SSIS Catalog ..."
$catalog = New-Object "$ISNamespace.Catalog" ($integrationServices, $SSISCatalog, $CatalogPwd)
$catalog.Create()
}
$folder = $catalog.Folders[$FolderName]
if (!$folder)
{
#Create a folder in SSISDB
Write-Host "Creating Folder ..."
$folder = New-Object "$ISNamespace.CatalogFolder" ($catalog, $FolderName, $FolderName)
$folder.Create()
}
# Read the project file, and deploy it to the folder
Write-Host "Deploying Project ..."
[byte[]] $projectFile = [System.IO.File]::ReadAllBytes($ProjectFilePath)
$folder.DeployProject($ProjectName, $projectFile)
This seemed to be working surprisingly well on the development machine - test server pair. However, the live environment will be a bit different, the machine doing the deployment job (deployment server, or DS from now on) and the SQL Server (DB for short) the project is to be deployed are in different domains and since SSIS requires windows authentication, I'm going to need to run the above code locally on DS but using credentials of a user on the DB.
And that's the point where I fail. The only thing that worked is to start the Powershell command line interface using runas /netonly /user:thatdomain\anuserthere powershell, enter the password, and paste the script unaltered into it. Alas, this is not an option, since there's no way to pass the password to runas (at least once with /savecred) and user interactivity is not possible anyway (the whole thing has to be automated).
I've tried the following:
Simply unning the script on DS, the line $sqlConnection = New-Object System.Data.SqlClient.SqlConnection $sqlConnectionString would use the credentials from DS which is not recognized by DB, and New-Object does not have a -Credential arg that I could pass to
Putting everything into an Invoke-Command with -Credential requires using -Computername as well. I guess it would be possible to use the local as 'remote' (using . as Computername) but it still complains about access being denied. I'm scanning through about_Remote_Troubleshooting, so far without any success.
Any hints on how to overcome this issue?
A solution might be to use a sql user (with the right access rights) instead of an AD used.
Something like this should work.
(Check also the answer to correct the connection string)

How to run powershell script using default credential from sql job agent

I am downloading a file from sharepoint.
I have already scheduled the job in SQL job agent
Its working fine when i use the following code
$UserName = "xxxx"
$PswdPath = "D:\securestring.txt"
$SecurePassword = cat $PswdPath| convertto-securestring
$fileName = [System.IO.Path]::GetFileName($FileUrl)
$DownloadPath = "D:\Excel\"
$downloadFilePath = [System.IO.Path]::Combine($DownloadPath,$fileName)
$client = New-Object System.Net.WebClient
$client.Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($UserName, $SecurePassword)
$client.Headers.Add("X-FORMS_BASED_AUTH_ACCEPTED", "f")
$client.DownloadFile($FileUrl, $downloadFilePath)
$client.Dispose()
But the problem here i face is whenever i update my password i need to update the secure string as well
So i wanted to use the default credentials
so i used the following script
$webclient = New-Object System.Net.WebClient
$webclient.UseDefaultCredentials = $true
$webclient.Headers.Add("X-FORMS_BASED_AUTH_ACCEPTED", "f")
$webclient.DownloadFile($FileUrl, $DownloadPath)
but its getting failed with the following error
Exception calling "DownloadFile" with "2" argument(s): "The remote server returned an error: (401) Unauthorized."
went through different blogs all were suggesting the same approach which i have followed
Any help in this regard?
As far as I understand default credentials users the account and password the sql agent process is started and since it will not match the SharePoint online account it will fail. It would be easier if you create a powershell that updates all secure strings once password is changed.

how to connect oracle database and run query by powershell

How can I connect to an Oracle database (11g)? I have a list of oracle servers and I want to execute a query on them from Jump Server (the Jump server has Powershell v2).
Servers are accessible from Jump Server as I am able to access them by UNC path.
Update:
I know it can be done by using ODP.Net connector. But i don't want to use it due to approval.
You can try the method using an OleDbConnection described here
function Get-OLEDBData ($connectstring, $sql) {
$OLEDBConn = New-Object System.Data.OleDb.OleDbConnection($connectstring)
$OLEDBConn.open()
$readcmd = New-Object system.Data.OleDb.OleDbCommand($sql,$OLEDBConn)
$readcmd.CommandTimeout = '300'
$da = New-Object system.Data.OleDb.OleDbDataAdapter($readcmd)
$dt = New-Object system.Data.datatable
[void]$da.fill($dt)
$OLEDBConn.close()
return $dt
}