I have a need to make a few historical data pulls via REST API call using Powershell. I'm stuck with one of the endpoints where the data volume is over 4 GB. When I run the Powershell script in the Powershell ISE it works as expected. But, When I run the same via SQL Server Integration Services Execute Process Task it's not able to download the JSON payload.
Is there a better way to tackle this? Are there any limitations one should be mindful of?
$Header = #{
"authorization" = "Bearer $token"
}
#make REST API call
$Parameters = #{
Method = "GET"
Uri = "https://api.mysite.com/v1/data"
Headers = $Header
ContentType = "application/json"
Body = $BodyJson
}
Invoke-RestMethod #Parameters
Error: System.OutOfMemoryException
Related
I have a Powershell script that calls a Google App Script function.
When I run the Powershell script I can see the following error on the Error Reporting on my GCP project:
Exception: You do not have permission to call SpreadsheetApp.getActiveSpreadsheet. Required permissions: (https://www.googleapis.com/auth/spreadsheets.currentonly || https://www.googleapis.com/auth/spreadsheets)
at toSpreadsheet (Código:3)
at fromPS (Código:14)
I have understood that I have to authorize the scope, so I've been trying to do that by editing the manifest file.
The Authorization Scopes Documentation says,
"During the authorization flow, Apps Script presents human-readable descriptions of the required scopes to the user. For example, if your script needs read-only access to your spreadsheets, the manifest may have the scope https://www.googleapis.com/auth/spreadsheets.readonly. During the authorization flow, a script with this scope asks the user to allow this application to "View your Google Spreadsheets"."
In my case I edited the manifest file appscript.json to add the scope https://www.googleapis.com/auth/spreadsheets, then I saved it, published the Google App Script project as API Executable, and finally I run the Powershell code again, but I still get the same error as above. During all this flow, I was not asked to allow anything. I cannot understand what is missing authorizing the script have the required permission.
I also added the spreadsheets scope to OAuth consent screen, but it seems to do not make any difference. I am suspecting I should use a Service Account to accomplish that since I see no way to go through a OAuth Client Verification since my script on Google is called from Powershell script. I dont want to believe on that because getting to know how config OAuth2 took me a lot of time :(
A few considerations:
The function that the run method calls by Powershell just works fine when I run it directly from Google Script Editor.
The script project is deployd as an API executable
Google Apps Script API is enabled in the GCP project
It is associated to a Standard GCP project
The OAuth credential is Web Application type
The script for writing and reading values from Powershell to Google Sheets works fine
Google script:
function toSpreadsheet(text2write)
{
var sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("HIL_APP");
var LastRow = sheet.getLastRow();
for (var i = 1; i < LastRow; i++)
{
sheet.getRange(i+1, 8, 1).setValue(text2write)
}
return "myreturn"
}
function fromPS(params)
{
Logger.log(params)
var rtn = toSpreadsheet(params)
return rtn
}
manifest file:
{
"oauthScopes": [
"https://www.googleapis.com/auth/spreadsheets"
],
"timeZone": "America/Argentina/Buenos_Aires",
"dependencies": {
},
"webapp": {
"access": "ANYONE",
"executeAs": "USER_DEPLOYING"
},
"exceptionLogging": "STACKDRIVER",
"executionApi": {
"access": "MYSELF"
},
"runtimeVersion": "V8"
}
Powershell code:
function doit{
$json = ".\client_id.json"
$jdata = get-content $json | convertfrom-json
<#
$jdata | ForEach-Object {
$_.PSObject.Properties.Value
}
#>
$ClientID = $jdata.web.client_id.ToString()
$ClientSecret = $jdata.web.client_secret.ToString()
$refreshToken = "1//04VvG_FTyDGhiCgYIARAAGAQSNwF-L9IrZ-o1kaZQQccvzL5m4TUTNz6b9Q4KCb16t4cH11gGCshWZWvgaCoMlg73FgpLAGOYTEk"
$grantType = "refresh_token"
$requestUri = "https://accounts.google.com/o/oauth2/token"
$GAuthBody = "refresh_token=$refreshToken&client_id=$ClientID&client_secret=$ClientSecret&grant_type=$grantType"
$GAuthResponse = Invoke-RestMethod -Method Post -Uri $requestUri -ContentType "application/x-www-form-urlencoded" -Body $GAuthBody
$accessToken = $GAuthResponse.access_token
$headers = #{"Authorization" = "Bearer $accessToken"
"Content-type" = "application/json"}
$spreadsheetId = "1htbeGlqZ4hojQBWl9fxE4nW_KZI9uVwi0ApzNOIbwnY"
$currentDate = (Get-Date).ToString('MM/dd/yyyy')
$currentTime = (Get-Date).ToString('HH:mm:sstt')
$json = #”
{
"range": "HIL_APP!A1:G1",
"majorDimension": "ROWS",
"values":
[[
"HIL_NAME",
"$env:ComputerName",
"$currentDate",
"$currentTime",
"$env:UserName",
"input from user",
"attempt"
],]
}
“#
$write = Invoke-WebRequest -Uri "https://sheets.googleapis.com/v4/spreadsheets/${spreadsheetId}/values/HIL_APP!A1:G1:append?valueInputOption=USER_ENTERED" -Method Post -ContentType "application/json" -Body $json -Headers #{"Authorization"="Bearer $accessToken"}
$read = Invoke-WebRequest -Uri "https://sheets.googleapis.com/v4/spreadsheets/${spreadsheetId}/values/HIL_APP!A1:G1" -Headers #{"Authorization"="Bearer $accessToken"}
$read
Write-Output "read: " ($read.Content | ConvertFrom-Json)
$scriptId = "1eF7ZaHH-pw2-AjnRVhOgnDxBUpfr0wALk1dVFg7B220bg_KuwVudbALh"
$json = #"
{
"function": "fromPS",
"parameters": ["myparam"],
"devMode": true
}
"#
$resp = Invoke-WebRequest -Uri "https://script.googleapis.com/v1/scripts/${scriptId}:run" -Method Post -ContentType "application/json" -Body $json -Headers #{"Authorization"="Bearer $accessToken"}
$resp
Write-Output "script response: " ($resp.Content | ConvertFrom-Json)
}
$error.Clear()
clear
doit
In order to run the function of Google Apps Script (GAS) using Apps Script API, it is required to do a bit complicated settings. In this case, I would like to propose for testing to run the GAS function as follows. This flow might be too careful.
Flow:
Link the Cloud Platform Project to Google Apps Script Project. Ref
Install for running the GAS function with the scripts.run method in Apps Script API. Ref
Put the script you want to run to the script editor of Google Apps Script.
Here, please run the function by the script editor and confirm whether the script works. By this, the issue of the script can be avoided.
Put the following script for testing to run. This is used for 1st test of Apps Script API.
function test() {
return "ok";
}
Put a following sample script for retrieving the access token. This is used for testing it. Please run this at the script editor, and copy the returned access token.
function getToken() {
Logger.log(ScriptApp.getOAuthToken());
}
Test to run the GAS function of test() using the retrieved access token. In this case, the script of powershell is used by replacing $accessToken = $GAuthResponse.access_token.
When an error occurs, please confirm the settings of Apps Script API. In this case, it can be said that the GAS script is correct.
When no error occurs, please test to run the function you want to run. In this case, the required scopes have already been included in the access token. By this, the issue of scopes can be avoided.
When above test is done and your script for using Apps Script API works, please retrieve the refresh token using the scopes. The scopes can be seen at the script editor. By this, the valid access token can be retrieved by the refresh token. And your script can be used at the local PC.
References:
Linking Cloud Platform Project to Google Apps Script Project
Executing Functions using the Apps Script API
I am trying to POST information into a table in ServiceNow via a Powershell script. When I run it I get an error
Invoke-WebRequest : The remote server returned an error: (500) Internal Server Error.
Can someone help me figure out how to solve this? Thank you all in advance.
$userName = 'helpMe'
$password = 'iAmStuck' | ConvertTo-SecureString -asPlainText -Force
$cred = New-Object System.Management.Automation.PSCredential($userName, $password)
$uri = 'stuff'
$postParams = "test"
#[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
Invoke-WebRequest -Uri $uri -Method Post -Body $postParams -Credential $cred
ServiceNow has a REST API explorer with various code examples to start working with.
Below is an example that I threw together that posts to the incident table with an admin account. Two important factors here, the user must have roles (here for info https://docs.servicenow.com/bundle/istanbul-servicenow-platform/page/integrate/inbound-rest/reference/r_RESTAPIRoles.html) to use the API and must have access to the table you are trying to post to. Also, note that the body of the post needs to be RAW JSON and all the correct header data is supplied in the URL. If successful ServiceNow will return JSON data about the post.
# Eg. User name="admin", Password="admin" for this code sample.
$user = "admin"
$pass = "noPassword"
# Build auth header
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $user, $pass)))
# Set proper headers
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add('Authorization',('Basic {0}' -f $base64AuthInfo))
$headers.Add('Accept','application/json')
$headers.Add('Content-Type','application/json')
# Specify endpoint uri
$uri = "https://xxxxx.service-now.com/api/now/table/incident"
# Specify HTTP method
$method = "post"
# Specify request body
{request.body ? "$body = \"" :""}}{\"active\":\"true\",\"number\":\"123\",\"short_description\":\"test\"}"
# Send HTTP request
$response = Invoke-WebRequest -Headers $headers -Method $method -Uri $uri -Body $body
# Print response
$response.RawContent
Even though you posted code, you posted nothing relevant to the problem. Whatever ServiceNow is might have an API for reference on what it's expecting. Often when interacting with web-based APIs, there is a structure that is required for the API to be able to understand the data you're supplying in $postParams. Sometimes it can just be key->value pairs in the case of a flat HTTP POST but often times for RESTful APIs you'll need to structure a JSON header as defined by the API documentation.
If you do a search for "servicenow powershell interaction" it looks like there's a GitHub project for interacting with ServiceNow via PowerShell and also a PDF that specifcally covers this topic.
I am using a PowerShell module provided by Citrix to invoke the Nitro REST API. Calling the function I can successfully add and remove load balanced services from the load. However when I try to do a GET method to get the status of a service I get the error:
Invoke-RestMethod : The underlying connection was closed: An unexpected error occurred on a send.
I have tried running Invoke-RestMethod without using the module but get the same error
Invoke-RestMethod -WebSession $myNSSession.WebSession -Method GET -Uri https://<NetScaler IP/nitro/v1/config/service/<Service Name>
When googling this error everything seems to point to certificate issues. I had this initially even on POST method until i added the below to my script
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
So since this works for doing POST i cant see why it wouldn't for a GET!!
another weird thing is, if I put the URL directly into the browser then enter my credentials i get a response in raw text! so it looks like this is an issue with the way i am calling it in PowerShell rather than the NetScaler or the NITRO API!
Someone please help as this is driving me crazy!!
Admitedly i am new to Invoke-RestMethod commands, but try this:
$creds = Get-Credential
$service = Invoke-RestMethod -Uri https://<NetScaler IP/nitro/v1/config/service/<Service Name> -Credential $creds
What you will get is something similar to this:
*errorcode* *message* *serverity* *service*
* 0 Done NONE {#{name=<service name; n..
then type $service.service and you will see more information. whatever attributes are availible will be listed. then just follow the pattern:
$service.service.
I had the same problem with Nitro API (specifically v10.5), and found that setting certificate policies, TLS versions and trust settings had no effect. POST works, GET fails.
The solution for me was to not use the cmdlets and instead drop back to a native .Net method. Below I am still using HTTPS with an internal certificate, hence still setting the callback.
$NSProtocol = "https://"
$NSHostname = "netscaler"
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
$WebRequest = [System.Net.WebRequest]::Create("$NsProtocol$NsHostname/nitro/v1/config/hanode")
$WebRequest.Method = "GET"
$WebRequest.ContentType = "application/json; charset=utf-8";
$WebRequest.Headers.Add("AUTHORIZATION","Basic $([System.Convert]::ToBase64String([System.Text.Encoding]::GetEncoding("ISO-8859-1").GetBytes($nsuser+":"+$nspass)))")
$Response = $WebRequest.GetResponse()
$ReadStream = New-Object System.IO.StreamReader $Response.GetResponseStream()
$HaState = ConvertFrom-Json $ReadStream.ReadToEnd()
Hope that helps.
I've got the following Powershell script:
$url = "http://somerandomapikeyhere#prestashop.dev/my-store/api"
$request = [System.Net.WebRequest]::Create($url)
$response = $request.GetResponse()
echo $response
which attempts to access my Prestashop API REST service.
I can access the URL from Chrome without a problem, but in PowerShell I get a 401 Unauthorized response, which is understandeable as the somerandomapikeyhere bit in the URL needs to get dealt with differently in PowerShell.
The question is, how? Any ideas / advice?
It's not powershel, but the .net class that is not able to take an url in that format. You need to provide the credentials seperate. See here or here.
But if you have powershell v3 you are better off using the build in commands for web requests or even REST requests.
Invoke-restmethod -uri 'prestashop.dev/my-store/api' -credential 'someapikey'
I am looking to make http requests to web pages with powershell, is this possible and if so, how may I achieve this?
Can I make requests to https pages? I am able to make http requests with a bat file but not https, was hoping I could https page requests with powershell.
You can use the usual WebRequest and HttpWebRequest classes provided by the .NET framework.
$request = [System.Net.WebRequest]::Create('http://example.com')
# do something with $request
It's no different from using the same classes and APIs from C#, except for the syntactic differences to PowerShell.
PowerShell v3 also brings Invoke-WebRequest and a few others.
Try this:
(New-Object System.Net.WebClient).DownloadString("http://stackoverflow.com")
WebClient.DownloadString Method (String)
or in PowerShell 3.0,
(Invoke-WebRequest http://stackoverflow.com).content
Invoke-WebRequest
Depending on what you are doing, you can also use System.Net.WebClient, which is a simplified abstraction of HttpWebRequest
$client = new-object system.net.webclient
Look here for difference: What difference is there between WebClient and HTTPWebRequest classes in .NET?
PS: With Powershell v3.0, you have Invoke-WebRequest and Invoke-RestMethod cmdlets which can be used for similar purposes
If all else fails, use Curl from http://curl.haxx.se . You can set everything, including certificate handling, POSTs, etc. Not subtle, but it works and handles all of the odder cases; e.g. you can set the --insecure flag to ignore certificate name issues, expiration, or test status.
You can create HTTP, HTTPS, FTP and FILE requests using Invoke-WebRequest cmdlet. This is pretty easy and gives many options to play around.
Example: To make simple http/https requests to google.com
Invoke-WebRequest -Uri "http://google.com"
More references can be found MSDN
This code works with both ASCII & binary files over https in powershell:
# Add the necessary .NET assembly
Add-Type -AssemblyName System.Net.Http
# Create the HttpClient object
$client = New-Object -TypeName System.Net.Http.Httpclient
# Get the web content.
$task = $client.GetByteArrayAsync("https://stackoverflow.com/questions/7715695/http-requests-with-powershell")
# Wait for the async call to finish
$task.wait();
# Write to file
[io.file]::WriteAllBytes('test.html',$task.result)
Tested on Powershell 5.1.17134.1, Win 10
Try this PowerShell module: https://github.com/toolkitx/simple-request
Install by Install-Module -Name SimpleRequest
Then you can send requests like
$Data = #{
"TokenUrl" = 111
"ClientSecret" = "222"
"ClientId" = "333"
"AuthResource" = "444"
"Username" = "User1"
"Password" = "Password"
"Id" = 99
"Price" = 0.99
"Value" = "Content"
}
$Sample = '
POST https://httpbin.org/post?id={{Id}}
Content-Type: application/json
Authorization: Bearer {{QIBToken}}
{
"id": {{Id}},
"value": "{{Value}}"
}'
$Response = Invoke-SimpleRequest -Syntax $Sample -Context $Data
Please refer to GitHub for detail introductions
This method downloads the content:
# PowerShell 2 version
$WebRequest=New-Object System.Net.WebClient
$WebRequest.UseDefaultCredentials=$true
#$WebRequest.Credentials=(Get-Credential)
$Data=$WebRequest.DownloadData("http://<url>")
[System.IO.File]::WriteAllBytes("<full path of file>",$Data)
# PowerShell 5 version
Invoke-WebRequest -Uri "http://<url>" -OutFile "<full path of file>" -UseDefaultCredentials -ContentType