Having trouble uploading large files with Invoke-RestMethod - powershell

I am trying to use the Hashicorp Atlas restful api to upload box files into but I am running into problems when uploading large files.
I am currently using the following command which works with small files like 100mb but most box files are well over 4GB where I start to see the problem:
$Filename = "c:\box\mybox.box"
$uploadpath = "https://archivist.hashicorp.com/v1/object/example"
$Timeout = 86400 #24 hours
Invoke-RestMethod -Uri $uploadPath -Method Put -InFile $Filename -TimeoutSec $Timeout -ContentType "multipart/form-data"
The error:
Invoke-RestMethod : The underlying connection was closed: An unexpected error occurred on a send.
At C:\Program Files\WindowsPowerShell\Modules\atlasbox\1.1.15\AtlasBox.psm1:642 char:5
+ Invoke-RestMethod -Uri $uploadPath -Method Put -InFile $Filename ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
If I use CURL command as per the documentation it works fine.
curl -X PUT --upload-file /path/to/my.box https://archivist.hashicorp.com/v1/object/example
Any idea how to make Invoke-RestMethod behave the same way curl does?
I can't take a dependency on curl because not all systems I run this on will have WSL installed and no curl.
Documentation for Atlas API is here: https://atlas.hashicorp.com/help/api/vagrant/box-providers

Looks like it might be getting weird on the SSL cert? I would try some of the recommendations here to start with, definitely an oddity specific to those cmdlets though:
Powershell v3 Invoke-WebRequest HTTPS error

Related

Google Text-to-Speech : Error could not auto authenticate

I'm using the google text to speech API in the powershell. I've created the project, enabled the api, created the key, when I tell it this
$cred = gcloud auth application-default print-access-token
$headers = #{ "Authorization" = "Bearer $cred" }
Invoke-WebRequest -Method POST
-Headers $headers -ContentType: "application/json; charset=utf-8"
-InFile request.json `
-Uri "https://texttospeech.googleapis.com/v1/text:synthesize" | Select-Object -Expand Content
And this is after I set the environment, I get this error message
ERROR: (gcloud.auth.application-default.print-access-token) Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
Invoke-WebRequest : The remote server returned an error: (403) Forbidden.
At line:4 char:1
Invoke-WebRequest `
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebExc
eption
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
I have attempted to recreate the key thinking that was maybe the issue but I keep getting this error here no matter what I do, I am using this code to point to the key
$env:GOOGLE_APPLICATION_CREDENTIALS=C:\IVR2\IVR2-ba991f7cce6a.json
For an application, you would set the env variable GOOGLE_APPLICATION_CREDENTIALS which is the location of your service account json file.
It looks like you're using gcloud to get creds and including an authorization header. That should work too. Does your gcloud command return an error?

Connect to WebHDFS using powershell: How to set the Different Credentials

I am rtying to connect te WebHDFS by powershell and have been retrieving some errors. I think the 401 error is because of the Credentials.
The code I've been using is:
Invoke-RestMethod -UseDefaultCredentials -Method Put -inFile c:\Users\petter\Documents\example.txt -Uri "https://drona-haproxy.corp.com:2000/gateway/drona/webhdfs/v1/user/petter/example.txt?op=CREATE"
How could I set different parameters in the Credentials under "Invoke-RestMethod"?
THe error that I've been retrieving is:
Invoke-RestMethod : The remote server returned an error: (401) Unauthorized.
At line:1 char:1
+ Invoke-RestMethod -Method Put -inFile c:\Users\petter\Documents\load ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
You can use the Credential parameter with Invoke-RestMethod.
I would have thought you'd need to remove -UseDefaultCredentials but the Invoke-RestMethod documentation says they're part of the same parameter set - not sure how that works.
related: WebHDFS Authentication

downloading a web page with powershell using a x.509 certificate

I am having a problem with PowerShell on Windows 10 trying to download a file from a site that uses an X.509 certificate to authenticate the client. I have this certificate installed in my certificate store in IE and I also have exported the PFX file. Here's what I have tried:
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::ssl3 and also tls12
trying to get the certificate using System.Security.Cryptography.X509Certificates.X509Certificates2
trying to set the credentials via Get-Credential and then passing it
I've used the Invoke-WebRequest command as well as System.Net.WebClient trying to get this to work. The error message I'm getting is as follows:
PS P:\PSScripts> P:\PSScripts\DownloadSslHtml.ps1
VERBOSE: GET https://somewebsite.com/GetReports.do?reportTypeId=1234
with 0-byte payload
Invoke-WebRequest : The request was aborted: Could not create SSL/TLS secure channel.
At P:\PSScripts\DownloadSslHtml.ps1:20 char:1
+ Invoke-WebRequest -Uri $url -UserAgent ([Microsoft.PowerShell.Command ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-
WebRequest], WebException + FullyQualifiedErrorId :
WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
Now I have seen a lot of people with this "SSL/TLS secure channel" error and they fixed it by specifying the encryption method as TLS12 or SSL3, neither of which is working for me. I'm confused how to proceed. Any thoughts? The website is using TLS1.2 AES with 256 bit encryption ECDH 256 bit exchange.
EDIT: Here's my powershell code:
$url = "https://somewebsite.com/GetReports.do?reportTypeId=1234"
$outputFile = "P:/DownloadedData/file.html"
$PFXPath = "P:/Properties/Config/mycert.pfx"
$PFXPassword = "cert_password"
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::tls12
$cert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2
$cert.Import($PFXPath,$PFXPassword,'DefaultKeySet')
Invoke-WebRequest -Uri $url -UserAgent ([Microsoft.PowerShell.Commands.PSUserAgent]::InternetExplorer) -Certificate $cert -OutFile $outputFile -Verbose
I changed this line:
$cert.Import($PFXPath,$PFXPassword,'DefaultKeySet')
to this:
$cert.Import($PFXPath,$PFXPassword,'UserKeySet')
And now it works!

Rest API call works with curl but doesn't work with PowerShell

I have following curl request that works
curl.exe -X GET "https://host.crm.dynamics.com/api/data/v9.1/accounts?$select=name" -H "Authorization: Bearer xxxxx"
And want to implement it in PowerShell
Invoke-RestMethod "https://host.crm.dynamics.com/api/data/v9.1/accounts?`$select=name" `
-Headers #{Authorization = "Bearer xxxxx"} `
-Method Get
I've also tried to do request with Invoke-WebRequest as well as curl version in Powershell and it didn't work
Invoke-RestMethod : The underlying connection was closed: An unexpected error occurred on a send.
At E:\Repos\myrepo\host\connection_test.ps1:51 char:1
+ Invoke-RestMethod $hostName `
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
When making connection using Invoke-RestMethod, you want to make sure that encryption your client is using is accepted by the URI.
PowerShell by default uses TLS 1.0 for web requests but the URI you are connecting to probably uses 1.1 or 1.2. You can use the following to force the use of TLS v1.1 or v1.2.
[Net.ServicePointManager]::SecurityProtocol = `
[Net.SecurityProtocolType]::Tls11, [Net.SecurityProtocolType]::Tls12;
Invoke-RestMethod "https://host.crm.dynamics.com/api/data/v9.1/accounts?`$select=name" `
-Headers #{Authorization = "Bearer xxxxx"} `
-Method Get

powershell: bitbucket api post request returns 400 error

I'm using powershell to work with Bitbucket API. Powershell sends command to check, if pull request could be merged and then merges it.
First command works without any problems:
Invoke-WebRequest -Headers #{Authorization = "Basic $base64AuthInfo"} -Method Get -ContentType "application/json" -Uri "$BaseUrl/rest/api/1.0/projects/$ProjectKey/repos/$RepoSlug/pull-requests/$PullRequestId/merge?version=$PRVersion"
I'm getting a valid JSON response. Example:
{"canMerge":true,"conflicted":false,"outcome":"CLEAN","vetoes":[]}
The second command must use POST request in order to merge the pull request. Same command with Post instead of Get :
Invoke-WebRequest -Headers #{Authorization = "Basic $base64AuthInfo"} -Method Post -ContentType "application/json" -Uri "$BaseUrl/rest/api/1.0/projects/$ProjectKey/repos/$RepoSlug/pull-requests/$PullRequestId/merge?version=$PRVersion" -Verbose
This one returns 400 error without any other information.
VERBOSE: POST https://bitbucket.example.com/rest/api/1.0/projects/TEAM/repos/tmp-test-repo/pull-requests/8/merge?version=2 with 0-byte payload
Invoke-WebRequest : The remote server returned an error: (400) Bad Request.
At line:1 char:1
+ Invoke-WebRequest -Headers #{Authorization = "Basic $base64AuthInfo"} ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
How can make POST request work?
Reference: Bitbucket documentation
The problem is that Powershell doesn't output expanded details in Invoke-WebRequest output, so this makes investigation really hard.
Turned out that the problem was in account I was using. After I used Fiddler to get the response headers, I got a detailed response from Bitbucket API, so I was able to fix my problem.