Reading and uploading zip files from Powershell - powershell

My below code successfully read a folder and uploades the subfolder one by one as a zip file in SharePoint. But if any of my file is of Zip extension, it gives me an error. I would like to know how can I fix my existing code below to handle the zip files as well so that it loads both the zip folder and non-zip files and folders in the SharePoint. I tried multiple tings like removing .zip extension in the Create-Archive and file but it does not work.
error received:
2020-12-09T15:49:05.7979920Z Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
2020-12-09T15:49:05.7981233Z At D:\a\r1\a\_DevOpsScripts\ReleaseNoteScripts\UploadInSharePoint.ps1:55 char:17
2020-12-09T15:49:05.7982104Z + ... $response = Invoke-RestMethod $uploadURLObject.uploadUrl -Method 'PUT ...
2020-12-09T15:49:05.7982819Z + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-12-09T15:49:05.7983312Z + CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebExc
2020-12-09T15:49:05.7983979Z eption
2020-12-09T15:49:05.7984369Z + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
param($ClientID,$ClientSecret,$WId,$RName,$RReqFor,$SPid,$LibId)
$tempDirectory = "temp"
# arguments defined from predefined variables in argument section of powershell stage of pipeline
$workingDirectory = "$WId"
$CLIENT_ID = "$ClientID"
$CLIENT_SECRET = "$ClientSecret"
$SharePointSiteId = "$SPid"
$LibraryId = "$LibId"
$azx = "$RName" +"_"+"$RReqFor"
$adx = ($azx.Split("[")[0])
$artifactname = $adx.Substring(0,$adx.Length-1)
# Get access token
Function GET-TOKEN
{
param($clientId, $clientSecret)
$tokenHeaders = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$tokenHeaders.Add("Content-Type", "application/x-www-form-urlencoded")
$tokenBody = "client_id="+ $clientId + "&scope=https%3A//graph.microsoft.com/.default&client_secret=" + $clientSecret + "&grant_type=client_credentials"
$tokenBody
$tokenResponse = Invoke-RestMethod 'https://login.microsoftonline.com/test.onmicrosoft.com/oauth2/v2.0/token' -Method 'POST' -Headers $tokenHeaders -Body $tokenBody
return $tokenResponse
}
# upload the file
Function UPLOAD-FILE
{
param($workingDir, $tempDir, $clientId, $clientSecret, $artifactname)
Get-ChildItem $workingDir | ForEach-Object {
$name = $_.FullName.Split("\")[-1]
CREATE-ARCHIVE -workingDir $_.FullName -tempDir $tempDir -name $name
$file = $_.FullName + "\" +$tempDir + "\" + "$($name).zip"
$fileSize = (Get-Item $file).length
$uploadURLObject = GET-UPLOADLINK -clientId $clientId -clientSecret $clientSecret -artifactname $name
$tokenObject = GET-TOKEN -clientId $clientId -clientSecret $clientSecret
$uploadHeaders = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$uploadHeaders.Add("Content-Type", "application/json")
$uploadHeaders.Add("Content-Range", "bytes " + 0 +"-" + ($fileSize-1) + "/" + $fileSize)
$uploadHeaders.Add("Content-Length", $fileSize)
$uploadHeaders.Add("Authorization", "Bearer "+ $tokenObject.access_token)
$uploadBody = [System.IO.File]::ReadAllBytes($file)
$response = Invoke-RestMethod $uploadURLObject.uploadUrl -Method 'PUT' -Headers $uploadHeaders -Body $uploadBody
$response | ConvertTo-Json
REMOVE-TEMPDIR -workingDir $_.FullName -tempDir $tempDirectory
}
}
Function GET-UPLOADLINK
{
param($clientId, $clientSecret, $artifactname)
$tokenObject = GET-TOKEN -clientId $clientId -clientSecret $clientSecret
$uploadLinkRequestHeaders = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$uploadLinkRequestHeaders.Add("Content-Type", "application/json")
$uploadLinkRequestHeaders.Add("Authorization", "Bearer "+ $tokenObject.access_token)
$uploadLinkRequestBody = ""
#update the actual link
$fullname = "https://graph.microsoft.com/v1.0/sites/$($SPid)/drives/$($LibId)/root/children/"
$comname = $fullname + $artifactname
$uploadLinkResponse = Invoke-RestMethod "$($comname).zip/createUploadSession" -Method 'POST' -Headers $uploadLinkRequestHeaders -Body $uploadLinkRequestBody
$uploadLinkResponse | ConvertTo-Json
return $uploadLinkResponse
}
Function CREATE-ARCHIVE
{
param($workingDir, $tempDir, $name )
write-host $workingDir
cd $workingDir
md $tempDir
#Compress-Archive -Path $workingDir -DestinationPath $workingDir\$tempDir\$artifactname
Compress-Archive -Path $workingDir -DestinationPath $workingDir\$tempDir\$name
}
Function REMOVE-TEMPDIR
{
param($workingDir, $tempDir)
#This will be the last step
rm $workingDir\$tempDir -Recurse
}
UPLOAD-FILE -workingDir $workingDirectory -tempDir $tempDirectory -clientId $CLIENT_ID -clientSecret $CLIENT_SECRET -artifactname $artifactname

If it's only failing on files that are .zip prior to the CREATE-ARCHIVE function being run.
What happens if CREATE-ARCHIVE ignores .zip files?
if (-not($_.name -like "*.zip")){
CREATE-ARCHIVE....
} else {
COPY-ITEM $workingDir\temp\$($_.name)
}
Seems like archiving an archive is unnecessary work, and potentially the source of an issue.

Related

Custom Azure Log from Azure Resource Graph query

I have Runbook under Azure automation account that should collect results from Resource graph query and pass it to Log analytics as custom log. I have managed to create a script that works fine.
$customerId = "xxxxxxxxx"
$SharedKey = "xxxxxxxxxxxxxxxx"
$LogType = "MyRecord"
$TimeStampField = ""
#function block
Function Connect-ToAzure {
$connectionName = "AzureRunAsConnection"
$automationAccountName = Get-AutomationVariable -Name 'automationAccountName'
Write-Output "Azure Automation Account Name - $automationAccountName"
$connectionName = "AzureRunAsConnection"
Write-Output "Connection Name - $connectionName"
$servicePrincipalConnection = Get-AutomationConnection -Name $connectionName
Write-Output "Logging in to Azure..."
Connect-AzAccount -ServicePrincipal -Tenant $servicePrincipalConnection.TenantId `
-ApplicationId $servicePrincipalConnection.ApplicationId `
-CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint `
-Subscription $servicePrincipalConnection.SubscriptionId
}
Function Build-Signature ($customerId, $sharedKey, $date, $contentLength, $method, $contentType, $resource)
{
$xHeaders = "x-ms-date:" + $date
$stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource
$bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash)
$keyBytes = [Convert]::FromBase64String($sharedKey)
$sha256 = New-Object System.Security.Cryptography.HMACSHA256
$sha256.Key = $keyBytes
$calculatedHash = $sha256.ComputeHash($bytesToHash)
$encodedHash = [Convert]::ToBase64String($calculatedHash)
$authorization = 'SharedKey {0}:{1}' -f $customerId,$encodedHash
return $authorization
}
# Create the function to create and post the request
Function Post-LogAnalyticsData($customerId, $sharedKey, $body, $LogType)
{
$method = "POST"
$contentType = "application/json"
$resource = "/api/logs"
$rfc1123date = [DateTime]::UtcNow.ToString("r")
$contentLength = $body.Length
$signature = Build-Signature `
-customerId $customerId `
-sharedKey $sharedKey `
-date $rfc1123date `
-contentLength $contentLength `
-method $method `
-contentType $contentType `
-resource $resource
$uri = "https://" + $customerId + ".ods.opinsights.azure.com" + $resource + "?api-version=2016-04-01"
$headers = #{
"Authorization" = $signature;
"Log-Type" = $LogType;
"x-ms-date" = $rfc1123date;
"time-generated-field" = $TimeStampField;
}
Write-Output "Sending a request"
$response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers $headers -Body $body -UseBasicParsing
return $response.StatusCode
Write-Output "Request has been sent"
}
try {
Write-Output "Starting runbook"
$customerId = "xxxxxxxxx"
$SharedKey = "xxxxxxxxxxxxxxxx"
$LogType = "MyRecord"
$Query = #'
resources
| where type == "microsoft.compute/disks"
| extend diskState = properties.diskState
| extend diskSizeGB = properties.diskSizeGB
| where properties['encryptionSettingsCollection'] != "enabled"
| where diskState == "Attached" or diskState == "Reserved"
| extend diskCreationTime = properties.timeCreated
| extend hostVM = split(managedBy,"/")[-1]
| project diskCreationTime, name, resourceGroup, hostVM, diskState
'#
Connect-ToAzure
$Result = (Search-AzGraph -Query $Query -First 1000)
$jsonResult = $Result | ConvertTo-Json
Post-LogAnalyticsData -customerId $customerId -sharedKey $sharedKey -body $jsonResult -logType $LogType
Write-Output "Runbook has been finished"
}
catch {
Write-Error -Message $_
break
}
However, I have an issue with collection of logs. I gather logs only from one subscription.
Can someone please help with code adjustment? How can I gather results from all subscriptions not just from one? I assume it should be foreach ($sub in $subs), but not sure how to do it combined with this graph query.
As the run as account will retire by Sep'23 so I would recommend using managed identities approach in your runbook i.e., configure role assignment for managed identity across multiple subscriptions as explained here and update your Connect-ToAzure function block something like shown below. For more context with regards to it, refer this Azure document.
$automationAccountName = Get-AutomationVariable -Name 'automationAccountName'
Write-Output "Azure Automation Account Name - $automationAccountName"
Disable-AzContextAutosave -Scope Process | Out-Null
Write-Output "Logging in to Azure..."
$AzureContext = (Connect-AzAccount -Identity).context
Also, you may use UseTenantScope parameter in your Search-AzGraph cmdlet to run the query across all available subscriptions in the current tenant.

Azure Function: Writing into ADLS Gen2 with PowerShell Script inside Azure Function App

I am newbie to Azure Functions. So I'm trying to write into ADLS Gen2 Storage Account Containers with the help of writing a PowerShell script inside Azure Function in Azure Function APP with Elastic Premium Plan.
Here, I am able to write the Data File to Storage Account but there is "No Data" in that file.
I'm attaching the Code of PowerShell Script written inside the Azure Function along with the Error File after execution.
Experts help me with this. Thanks in Advance.
Code
using namespace System.Net
# Input bindings are passed in via param block.
param($Request, $TriggerMetadata)
# Connect to Azure Account
# $username = "warehouse#rentpath.onmicrosoft.com"
# $password = Get-Content D:\PS\passwords\password.txt | ConvertTo-SecureString -Key (Get-Content D:\PS\passwords\aes.key)
# $credential = New-Object System.Management.Automation.PsCredential($username,$password)
# Connect-AzAccount -Credential $userCredential
$username = "warehouse#rentpath.onmicrosoft.com"
$pass = ConvertTo-SecureString "**********" -AsPlainText -Force
$cred = New-Object PSCredential($username,$pass)
Connect-AzAccount -Credential $cred
# Input Variables
$dataFactoryName="dna-production-gen2"
$resourceGroupName="DataLake-Gen2"
# get dataFactory triggers
$triggers=Get-AzDataFactoryV2Trigger -DataFactoryName $dataFactoryName -ResourceGroupName $resourceGroupName
$datas=#()
foreach ($trigger in $triggers) {
# get the trigger run history
$today = Get-Date
$yesterday = $today.AddDays(-1)
$splat = #{
ResourceGroupName = $trigger.ResourceGroupName
DataFactoryName = $trigger.DataFactoryName
TriggerName = $trigger.Name
TriggerRunStartedAfter = $yesterday
TriggerRunStartedBefore = $today
}
$historys =Get-AzDataFactoryV2TriggerRun #splat
if($historys -ne $null){
# create date
foreach($history in $historys){
$obj =[PsCustomObject]#{
'TriggerRunTimestamp ' = $history.TriggerRunTimestamp
'ResourceGroupName ' =$history.ResourceGroupName
'DataFactoryName' =$history.DataFactoryName
'TriggerName ' = $history.TriggerName
'TriggerRunId'= $history.TriggerRunId
'TriggerType'=$history.TriggerType
'Status' =$history.Status
}
# add data to an array
$datas += $obj
}
}
}
# convert data to csv string
$contents =(($datas | ConvertTo-Csv -NoTypeInformation) -join [Environment]::NewLine)
# upload to Azure Data Lake Store Gen2
#1. Create a sas token
$accountName="dna2020gen2"
# $path = New-Item -ItemType Directory -Path ".\$((Get-Date).ToString('yyyy-MM-dd'))"
$YY = (Get-Date).year
$MM = (Get-Date).month
$DD = get-date –f dd
$fileSystemName="dev"
$filePath="Input/Triggers/YYYY=$YY/MM=$MM/DD=$DD/data.csv"
$account = Get-AzStorageAccount -ResourceGroupName 'DataLake-Gen2' -Name $accountName
$sas= New-AzStorageAccountSASToken -Service Blob -ResourceType Service,Container,Object `
-Permission "racwdlup" -StartTime (Get-Date).AddMinutes(-10) `
-ExpiryTime (Get-Date).AddHours(2) -Context $account.Context
$baseUrl ="https://{0}.dfs.core.windows.net/{1}/{2}{3}" -f $accountName , $fileSystemName, $filePath, $sas
#2. Create file
$endpoint =$baseUrl +"&resource=file"
Invoke-RestMethod -Method Put -Uri $endpoint -Headers #{"Content-Length" = 0} -UseBasicParsing
$r = Invoke-WebRequest -Uri https://management.azure.com/subscriptions/{guid}/resourcegroups?api-version=2016-09-01 -Method GET -Headers $authHeaders
$r.Headers["x-ms-ratelimit-remaining-subscription-reads"]
#3 append data
$endpoint =$baseUrl +"&action=append&position=0"
Invoke-RestMethod -Method Patch -Uri $endpoint -Headers #{"Content-Length" = $contents.Length} -Body $contents -UseBasicParsing
#4 flush data
#$endpoint =$baseUrl + ("&action=flush&position={0}" -f $contents.Length)
#Invoke-RestMethod -Method Patch -Uri $endpoint -UseBasicParsing
#Check the result (get data)
Invoke-RestMethod -Method Get -Uri $baseUrl -UseBasicParsing
# Associate values to output bindings by calling 'Push-OutputBinding'.
Push-OutputBinding -Name Response -Value ([HttpResponseContext]#{
StatusCode = [HttpStatusCode]::OK
Body = $contents
})
Error File:
Connected!
2021-01-22T06:56:02.230 [Information] Executing 'Functions.HttpTrigger1' (Reason='This function was programmatically called via the host APIs.', Id=795e78c8-d6d7-4fcc-b4c0-115951b27807)
2021-01-22T06:56:16.039 [Warning] WARNING: TenantId '0c777d2e-f69e-41e4-8dc2-28fcf4c9604b' contains more than one active subscription. First one will be selected for further use. To select another subscription, use Set-AzContext.
2021-01-22T06:56:16.323 [Information] OUTPUT:
2021-01-22T06:56:31.207 [Information] OUTPUT: Account SubscriptionName TenantId Environment
2021-01-22T06:56:31.216 [Information] OUTPUT: ------- ---------------- -------- -----------
2021-01-22T06:56:31.216 [Information] OUTPUT: warehouse#rentpath.onmicrosoft.com Data Analytics 0c777d2e-f69e-41e4-8dc2-28fcf4c9604b AzureCloud
2021-01-22T06:56:31.216 [Information] OUTPUT:
2021-01-22T06:56:31.775 [Error] ERROR: The cmdlet cannot run because the -ContentType parameter is not a valid Content-Type header. Specify a valid Content-Type for -ContentType, then retry. To suppress header validation, supply the -SkipHeaderValidation parameter.Exception :Type : System.Management.Automation.ValidationMetadataExceptionErrorRecord :Exception :Type : System.Management.Automation.ParentContainsErrorRecordExceptionMessage : The cmdlet cannot run because the -ContentType parameter is not a valid Content-Type header. Specify a valid Content-Type for -ContentType, then retry. To suppress header validation, supply the -SkipHeaderValidation parameter.HResult : -2146233087CategoryInfo : MetadataError: (:) [], ParentContainsErrorRecordExceptionFullyQualifiedErrorId : RuntimeExceptionTargetSite :Name : ThrowTerminatingErrorDeclaringType : System.Management.Automation.MshCommandRuntime, System.Management.Automation, Version=7.0.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35MemberType : MethodModule : System.Management.Automation.dllStackTrace :at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)Message : The cmdlet cannot run because the -ContentType parameter is not a valid Content-Type header. Specify a valid Content-Type for -ContentType, then retry. To suppress header validation, supply the -SkipHeaderValidation parameter.InnerException :Type : System.FormatExceptionTargetSite :Name : ParseAndAddValueDeclaringType : System.Net.Http.Headers.HttpHeadersMemberType : MethodModule : System.Net.Http.dllStackTrace :at System.Net.Http.Headers.HttpHeaders.ParseAndAddValue(HeaderDescriptor descriptor, HeaderStoreItemInfo info, String value)at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.FillRequestStream(HttpRequestMessage request)Message : Cannot add value because header 'Content-Length' does not support multiple values.Source : System.Net.HttpHResult : -2146233033Source : System.Management.AutomationHResult : -2146233087CategoryInfo : InvalidArgument: (:) [Invoke-RestMethod], ValidationMetadataExceptionFullyQualifiedErrorId : WebCmdletContentTypeException,Microsoft.PowerShell.Commands.InvokeRestMethodCommandInvocationInfo :MyCommand : Invoke-RestMethodScriptLineNumber : 92OffsetInLine : 1HistoryId : 1ScriptName : C:\home\site\wwwroot\HttpTrigger1\run.ps1Line : Invoke-RestMethod -Method Patch -Uri $endpoint -Headers #{"Content-Length" = $contents.Length} -Body $contents -UseBasicParsingPositionMessage : At C:\home\site\wwwroot\HttpTrigger1\run.ps1:92 char:1+ Invoke-RestMethod -Method Patch -Uri $endpoint -Headers #{"Content-Le …+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~PSScriptRoot : C:\home\site\wwwroot\HttpTrigger1PSCommandPath : C:\home\site\wwwroot\HttpTrigger1\run.ps1InvocationName : Invoke-RestMethodCommandOrigin : InternalScriptStackTrace : at <ScriptBlock>, C:\home\site\wwwroot\HttpTrigger1\run.ps1: line 92Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException : Result: ERROR: The cmdlet cannot run because the -ContentType parameter is not a valid Content-Type header. Specify a valid Content-Type for -ContentType, then retry. To suppress header validation, supply the -SkipHeaderValidation parameter.Exception :Type : System.Management.Automation.ValidationMetadataExceptionErrorRecord :Exception :Type : System.Management.Automation.ParentContainsErrorRecordExceptionMessage : The cmdlet cannot run because the -ContentType parameter is not a valid Content-Type header. Specify a valid Content-Type for -ContentType, then retry. To suppress header validation, supply the -SkipHeaderValidation parameter.HResult : -2146233087CategoryInfo : MetadataError: (:) [], ParentContainsErrorRecordExceptionFullyQualifiedErrorId : RuntimeExceptionTargetSite :Name : ThrowTerminatingErrorDeclaringType : System.Management.Automation.MshCommandRuntime, System.Management.Automation, Version=7.0.3.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35MemberType : MethodModule : System.Management.Automation.dllStackTrace :at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)Message : The cmdlet cannot run because the -ContentType parameter is not a valid Content-Type header. Specify a valid Content-Type for -ContentType, then retry. To suppress header validation, supply the -SkipHeaderValidation parameter.InnerException :Type : System.FormatExceptionTargetSite :Name : ParseAndAddValueDeclaringType : System.Net.Http.Headers.HttpHeadersMemberType : MethodModule : System.Net.Http.dllStackTrace :at System.Net.Http.Headers.HttpHeaders.ParseAndAddValue(HeaderDescriptor descriptor, HeaderStoreItemInfo info, String value)at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.FillRequestStream(HttpRequestMessage request)Message : Cannot add value because header 'Content-Length' does not support multiple values.Source : System.Net.HttpHResult : -2146233033Source : System.Management.AutomationHResult : -2146233087CategoryInfo : InvalidArgument: (:) [Invoke-RestMethod], ValidationMetadataExceptionFullyQualifiedErrorId : WebCmdletContentTypeException,Microsoft.PowerShell.Commands.InvokeRestMethodCommandInvocationInfo :MyCommand : Invoke-RestMethodScriptLineNumber : 92OffsetInLine : 1HistoryId : 1ScriptName : C:\home\site\wwwroot\HttpTrigger1\run.ps1Line : Invoke-RestMethod -Method Patch -Uri $endpoint -Headers #{"Content-Length" = $contents.Length} -Body $contents -UseBasicParsingPositionMessage : At C:\home\site\wwwroot\HttpTrigger1\run.ps1:92 char:1+ Invoke-RestMethod -Method Patch -Uri $endpoint -Headers #{"Content-Le …+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~PSScriptRoot : C:\home\site\wwwroot\HttpTrigger1PSCommandPath : C:\home\site\wwwroot\HttpTrigger1\run.ps1InvocationName : Invoke-RestMethodCommandOrigin : InternalScriptStackTrace : at <ScriptBlock>, C:\home\site\wwwroot\HttpTrigger1\run.ps1: line 92Exception: The cmdlet cannot run because the -ContentType parameter is not a valid Content-Type header. Specify a valid Content-Type for -ContentType, then retry. To suppress header validation, supply the -SkipHeaderValidation parameter.Stack: at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)
2021-01-22T06:56:31.872 [Information] OUTPUT:
2021-01-22T06:56:31.872 [Information] OUTPUT:
2021-01-22T06:56:31.872 [Information] Executed 'Functions.HttpTrigger1' (Succeeded, Id=795e78c8-d6d7-4fcc-b4c0-115951b27807, Duration=29642ms)
Regarding the issue, please refer to the following script
using namespace System.Net
# Input bindings are passed in via param block.
param($Request, $TriggerMetadata)
# Write to the Azure Functions log stream.
Write-Host "PowerShell HTTP trigger function processed a request."
$username = "jimxxxx.onmicrosoft.com"
$pass = ConvertTo-SecureString "xxx" -AsPlainText -Force
$cred = New-Object PSCredential($username,$pass)
Connect-AzAccount -Credential $cred -Tenant "xxx.onmicrosoft.com"
$triggers=Get-AzDataFactoryV2Trigger -ResourceGroupName test001 -DataFactoryName testfactory05
$datas=#()
foreach ($trigger in $triggers) {
# get the trigger run history
$today = Get-Date
$yesterday = $today.AddDays(-1)
$splat = #{
ResourceGroupName = $trigger.ResourceGroupName
DataFactoryName = $trigger.DataFactoryName
TriggerName = $trigger.Name
TriggerRunStartedAfter = $yesterday
TriggerRunStartedBefore = $today
}
$historys =Get-AzDataFactoryV2TriggerRun #splat
if($historys -ne $null){
# create date
foreach($history in $historys){
$obj =[PsCustomObject]#{
'TriggerRunTimestamp ' = $history.TriggerRunTimestamp
'ResourceGroupName ' =$history.ResourceGroupName
'DataFactoryName' =$history.DataFactoryName
'TriggerName ' = $history.TriggerName
'TriggerRunId'= $history.TriggerRunId
'TriggerType'=$history.TriggerType
'Status' =$history.Status
}
# add data to an array
$datas += $obj
}
}
}
$contents =(($datas | ConvertTo-Csv -NoTypeInformation) -join [Environment]::NewLine)
$accountName="testadls05"
$YY = (Get-Date).year
$MM = (Get-Date).month
$DD = get-date –f dd
$fileSystemName="test"
$filePath="Input/Triggers/YYYY=$YY/MM=$MM/DD=$DD/data.csv"
$account = Get-AzStorageAccount -ResourceGroupName 'andywin7' -Name $accountName
$sas= New-AzStorageAccountSASToken -Service Blob -ResourceType Service,Container,Object `
-Permission "racwdlup" -StartTime (Get-Date).AddMinutes(-10) `
-ExpiryTime (Get-Date).AddHours(2) -Context $account.Context
$baseUrl ="https://{0}.dfs.core.windows.net/{1}/{2}{3}" -f $accountName , $fileSystemName, $filePath, $sas
#2. Create file
Write-Host "Create file"
$endpoint =$baseUrl +"&resource=file"
Invoke-RestMethod -Method Put -Uri $endpoint -Headers #{"Content-Length" = 0} -UseBasicParsing
$currentAzureContext = Get-AzContext
$azureRmProfile = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureRmProfileProvider]::Instance.Profile;
$profileClient = New-Object Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient($azureRmProfile);
$token=$profileClient.AcquireAccessToken($currentAzureContext.Subscription.TenantId).AccessToken;
$authHeaders=#{"Authorization" ="Bearer $token"}
$r = Invoke-WebRequest -Uri "https://management.azure.com/subscriptions/$($currentAzureContext.Subscription.Id)/resourcegroups?api-version=2016-09-01" -Method GET -Headers $authHeaders
$r.Headers["x-ms-ratelimit-remaining-subscription-reads"]
#3 append data
Write-Host "append data"
$endpoint =$baseUrl +"&action=append&position=0"
$body= [system.Text.Encoding]::UTF8.GetBytes($contents)
Invoke-RestMethod -Method Patch -Uri $endpoint -Body $body -UseBasicParsing
#4 flush data
$endpoint =$baseUrl + ("&action=flush&position={0}" -f $body.Length)
Invoke-RestMethod -Method Patch -Uri $endpoint -UseBasicParsing
#Check the result (get data)
Invoke-RestMethod -Method Get -Uri $baseUrl -UseBasicParsing
# Associate values to output bindings by calling 'Push-OutputBinding'.
Push-OutputBinding -Name Response -Value ([HttpResponseContext]#{
StatusCode = [HttpStatusCode]::OK
Body = $contents
})

Automation VSTS REST API to create new project

I'm trying to learn how to use REST API, but I'm at square one. The part of not knowing what tool to use to interact with VSTS REST API or how to configure it.
Any help on like "you should use this tool" and "this is how you connect to it".
Small goal is to be able to just get a list of projects in a VSTS account I own and I can build from there.
You can use PowerShell to use REST API of TFS/VSTS easily.
Here is a sample code to list all projects
param(
[Parameter(Mandatory=$true)]
[string] $tfsUri,
[Parameter(Mandatory=$true)]
[string] $token
<#[Parameter(Mandatory=$true)]
[string] $User,
[Parameter(Mandatory=$true)]
[string] $Password#>
)
<# Base64-encodes the Personal Access Token (PAT) appropriately #>
$User=''
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $User,$token)));
$header = #{Authorization=("Basic {0}" -f $base64AuthInfo)};
<#---------------------------------------------------------------------- #>
<#
$securePassword = $Password | ConvertTo-SecureString -AsPlainText -Force
$credential = New-Object System.Management.Automation.PSCredential($User, $securePassword)
#>
$reportName = 'TeamProjectList.html'
$teamProjectsReport = '<!DOCTYPE html><html><head>
<!--mce:0-->
</head><body>'
$teamProjectsReport = $teamProjectsReport + '<h2><u><center>' + 'Team Project List' + '</center></u></h2>'
$teamProjectsReport | Out-File -Force $reportName
$teamProjectsReport = '';
$Uri = $tfsUri + '/_apis/projectCollections?api-version=1.0'
$tfsCollections = Invoke-RestMethod -Method Get -ContentType application/json -Uri $Uri -Headers $header #-Credential $credential
foreach($tfsCollection in $tfsCollections.value)
{
$top=100;
$skip=0;
$collectionName = $tfsCollection.name;
if ($tfsUri.Contains('visualstudio.com'))
{
$collectionName = 'DefaultCollection'
}
write-host '==============================='
write-host $tfsCollection.Name
$teamProjectsReport = '<ul><h4>'+ $collectionName + '</h4>';
write-host '-------------------------------'
$collectionProjectList = #();
while($true)
{
$Uri = $tfsUri + '/' + $collectionName + '/_apis/projects?$top='+ $top + '&$skip='+ $skip + '&api-version=1.0'
$tfsProjects = Invoke-RestMethod -Method Get -ContentType application/json -Uri $Uri -Headers $header #-Credential $credential
$skip+=$top;
if($tfsProjects.count -le 0)
{
$orderedProjects = $collectionProjectList| Sort-Object -Property name
foreach($tfsProject in $orderedProjects)
{
write-host $tfsProject.Name
$teamProjectsReport = $teamProjectsReport + '<li> <a target="_blank" href="' + $tfsUri + '/' + $collectionName + '/'+ $tfsProject.Name + '" >' + $tfsProject.Name + '</a></li>';
}
$teamProjectsReport = $teamProjectsReport + '</ul>'
$teamProjectsReport | Out-File -Append -Force $reportName
$teamProjectsReport = '';
break;
}
$collectionProjectList += $tfsProjects.value
}
write-host '-------------------------------'
}
$teamProjectsReport = $teamProjectsReport + '</ul></body></html>'
$teamProjectsReport | Out-File -Append -Force $reportName
To execute this PowerShell in VSTS or TFS 2017 or 2018 you can create a Persona Access Token (PAT) with all scopes (all scopes required here to enable access to collection level information). You can call the script with you VSTS account as shown below.
.\GetAllProjects.ps1 -tfsUri 'https://youraccount.visualstudio.com' -token 'xxxxxxxxPATxxxxxxxxxxxxxx'
To call the script for TFS on premise 2017, 2018 you can use either of below depending on your TFS is setup with SSL.
.\GetAllProjects.ps1 -tfsUri 'https://yourtfs/tfs' -token 'xxxxxxxxPATxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
or
.\GetAllProjects.ps1 -tfsUri 'http://yourtfs:8080/tfs' -token 'xxxxxxxxPATxxxxxxxxxxxxxxxxxxxxxxxxxxx'
Project are listed as below
More information on script available here

Getting 404 on Powershell Query of Cosmos DB

I'm using this script from the download in this link.
https://gallery.technet.microsoft.com/scriptcenter/How-to-query-Azure-Cosmos-0a9aa517
However for some reason I am getting a 404 response.
I copy-pasted the url of the db directly. Putting in a fake URL gives me a "could not be resolved" error so I know the location exists.
Based on the Azure CosmosDB API documentation here:
https://learn.microsoft.com/en-us/rest/api/documentdb/databases
the $databaseID is user-set and just has to be unique, so I've set it to be the same as the db name and assigned that to the url.
Changing it to be different still gives me the same 404 response message (below).
Edit: Removed original commenting intro for readability
Powershell Script:
# add necessary assembly
#
Add-Type -AssemblyName System.Web
# generate authorization key
Function Generate-MasterKeyAuthorizationSignature
{
[CmdletBinding()]
Param
(
[Parameter(Mandatory=$true)][String]$verb,
[Parameter(Mandatory=$true)][String]$resourceLink,
[Parameter(Mandatory=$true)][String]$resourceType,
[Parameter(Mandatory=$true)][String]$dateTime,
[Parameter(Mandatory=$true)][String]$key,
[Parameter(Mandatory=$true)][String]$keyType,
[Parameter(Mandatory=$true)][String]$tokenVersion
)
$hmacSha256 = New-Object System.Security.Cryptography.HMACSHA256
$hmacSha256.Key = [System.Convert]::FromBase64String($key)
$payLoad = "$($verb.ToLowerInvariant())`n$($resourceType.ToLowerInvariant())`n$resourceLink`n$($dateTime.ToLowerInvariant())`n`n"
$hashPayLoad = $hmacSha256.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($payLoad))
$signature = [System.Convert]::ToBase64String($hashPayLoad);
[System.Web.HttpUtility]::UrlEncode("type=$keyType&ver=$tokenVersion&sig=$signature")
}
# query
Function Query-CosmosDb
{
[CmdletBinding()]
Param
(
[Parameter(Mandatory=$true)][String]$EndPoint,
[Parameter(Mandatory=$true)][String]$DataBaseId,
[Parameter(Mandatory=$true)][String]$CollectionId,
[Parameter(Mandatory=$true)][String]$MasterKey,
[Parameter(Mandatory=$true)][String]$Query
)
$Verb = "POST"
$ResourceType = "docs";
$ResourceLink = "dbs/$DatabaseId/colls/$CollectionId"
$dateTime = [DateTime]::UtcNow.ToString("r")
$authHeader = Generate-MasterKeyAuthorizationSignature -verb $Verb -resourceLink $ResourceLink -resourceType $ResourceType -key $MasterKey -keyType "master" -tokenVersion "1.0" -dateTime $dateTime
$queryJson = #{query=$Query} | ConvertTo-Json
$header = #{authorization=$authHeader;"x-ms-documentdb-isquery"="True";"x-ms-version"="2017-02-22";"x-ms-date"=$dateTime}
$contentType= "application/json "# The original said "application/query+json", I tried both
$queryUri = "$EndPoint$ResourceLink/docs"
$result = Invoke-RestMethod -Method $Verb -ContentType $contentType -Uri $queryUri -Headers $header -Body $queryJson
$result | ConvertTo-Json -Depth 10
}
# fill the target cosmos database endpoint uri, database id, collection id and masterkey
$DatabaseName = "" # name goes here
$MasterKey = "" #key goes here
$CollectionId = "transientUsers"
$DatabaseId = $DatabaseName
$CosmosDBEndPoint = "https://$DatabaseId.documents.azure.com:443/"
# query string
$Query = "SELECT * FROM transientUsers"
# execute
Query-CosmosDb -EndPoint $CosmosDBEndPoint -DataBaseId $DataBaseId -CollectionId $CollectionId -MasterKey $MasterKey -Query $Query
Error I'm getting:
Invoke-RestMethod : The remote server returned an error: (404) Not Found.
At D:\querycosmos\PowerShell\QueryCosmosDB.ps1:69 char:12
+ ... $result = Invoke-RestMethod -Method $Verb -ContentType $contentType ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
I noticed that you used $DatabaseId in two places:
$ResourceLink = "dbs/$DatabaseId/colls/$CollectionId"
and
$CosmosDBEndPoint = "https://$DatabaseId.documents.azure.com:443/"
If $DatabaseId refers to your account name, then you would need to change your $ResourceLink variable and use the name of the database inside your account containing the collection. If however $DatabaseId refers to the database name, then you would need to change $CosmosDBEndPoint and use account name there.

Azure DocumentDB Rest API PowerShell delete collection 401 Unathorized

Need to delete collection in my automation process.
Trying to execute script below. Get operations working fine, but Delete operation failed with "(401) Unathorized" error. It is strange cause Delete collection do not need additional headers. Can someone give a hint, what is wrong?
$accountName = 'someaccountname'
$connectionKey = 'masterkey'
$collectionName = 'mycollection'
$databaseName = 'mydatabase'
function GetKey([System.String]$Verb = '',[System.String]$ResourceId = '',
[System.String]$ResourceType = '',[System.String]$Date = '',[System.String]$masterKey = '') {
$keyBytes = [System.Convert]::FromBase64String($masterKey)
$text = #($Verb.ToLowerInvariant() + "`n" + $ResourceType.ToLowerInvariant() + "`n" + $ResourceId + "`n" + $Date.ToLowerInvariant() + "`n" + "`n")
$body =[Text.Encoding]::UTF8.GetBytes($text)
$hmacsha = new-object -TypeName System.Security.Cryptography.HMACSHA256 -ArgumentList (,$keyBytes)
$hash = $hmacsha.ComputeHash($body)
$signature = [System.Convert]::ToBase64String($hash)
[System.Web.HttpUtility]::UrlEncode($('type=master&ver=1.0&sig=' + $signature))
}
function BuildHeaders([string]$action = "get",[string]$resType, [string]$resourceId){
$authz = GetKey -Verb $action -ResourceType $resType -ResourceId $resourceId -Date $apiDate -masterKey $connectionKey
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Authorization", $authz)
$headers.Add("x-ms-version", '2015-12-16')
$headers.Add("x-ms-date", $apiDate)
$headers
}
function GetUTDate() {
$date = get-date
$date = $date.ToUniversalTime();
return $date.ToString("r", [System.Globalization.CultureInfo]::InvariantCulture);
}
function GetDatabases() {
$uri = $rootUri + "/dbs"
$hdr = BuildHeaders -resType dbs
$response = Invoke-RestMethod -Uri $uri -Method Get -Headers $hdr
$response.Databases
Write-Host ("Found " + $Response.Databases.Count + " Database(s)")
}
function GetCollections([string]$dbname){
$uri = $rootUri + "/" + $dbname + "/colls"
$hdr = BuildHeaders -resType colls -resourceId $dbname
$response = Invoke-RestMethod -Uri $uri -Method Get -Headers $hdr
$response.DocumentCollections
Write-Host ("Found " + $Response.DocumentCollections.Count + " DocumentCollection(s)")
}
function DeleteCollection([string]$dbname){
$uri = $rootUri + "/" + $dbname + "/colls" + "/" + $collectionName
$hdrs = BuildHeaders -action DELETE -resType colls -resourceId $collectionName
$response = Invoke-RestMethod -Uri $uri -Method Delete -Headers $hdrs
Write-Host "DELETE $uri"
}
$rootUri = "https://" + $accountName + ".documents.azure.com"
write-host ("Root URI is " + $rootUri)
#validate arguments
$apiDate = GetUTDate
$db = GetDatabases | where { $_.id -eq $databaseName }
if ($db -eq $null) {
write-error "Could not find database in account"
return
}
$dbname = "dbs/" + $databaseName
$collection = GetCollections -dbname $dbname | where { $_.id -eq $collectionName }
if($collection -eq $null){
write-error "Could not find collection in database"
return
}
Write-Host
$Delete = DeleteCollection -dbname $dbname | where { $_.id -eq $collectionName }
Normally, either the Authorization or x-ms-date header is not set (or the Authorization header with an invalid authorization token), 401 unauthorized error will be returned. I use fiddler to capture the request and check the response, and I find both Authorization and x-ms-date header are set, so it seems that the Authorization header is set to an invalid authorization token. Based on your code, I do some changes, and the function could work fine on my side.
function DeleteCollection([string]$dbname){
$uri = $rootUri + "/" + $dbname + "/colls" + "/" + $collectionName
$collectionName = $dbname + "/colls" + "/" + $collectionName
$hdrs = BuildHeaders -action DELETE -resType colls -resourceId $collectionName
#Write-Host "resourceId $collectionName"
$response = Invoke-RestMethod -Uri $uri -Method Delete -Headers $hdrs
Write-Host "resource $collectionName"
Write-Host "DELETE $uri"
}
The $collectionName should be dbs/{yourdbname}/colls/{yourcollectionname}
This can happen if there are pending operations. Either retry until it succeeds or add a delay before deleting. For us, a half second is enough to assure we never see this again.