I'm trying to download and save the text file http://www.gutenberg.org/cache/epub/164/pg164.txt using Powershell. I tried using the code:
$curl http://www.gutenberg.org/cache/epub/164/pg164.txt -OutFile verne.txt
But instead of saving the text file it saved the text file for the page source of http://www.gutenberg.org/ebooks/164?msg=welcome_stranger. I'm wondering if there's something wrong with my code or if I need to use another code.
It's a redirect. If you put the url in the browser you will get the same welcome stranger page. My guess is that they don't want you to access this content in this way. They may require log in, or at the very least, a valid session cookie.
Your link is a redirection, try this :
$uri = 'www.gutenberg.org/ebooks/164.txt.utf-8'
$request = Invoke-WebRequest -Uri $uri -MaximumRedirection 0 -ErrorAction Ignore
if($request.StatusDescription -eq 'found')
{
#redownload the new url (redirection)
$request=Invoke-WebRequest -Uri $request.Headers.Location
$request.ParsedHtml.body.outerText
}
Related
I need some help to get my code to create a Document Set in SharePoint Online using Graph API directly from a PowerShell script using Invoke-RestMethod.
I tested the request under the Graph Explorer portal and it works fine and I get a nice HTTP 201 (OK) as seen on the picture below:
Trying the very same request from my PowerShell script fails and returns HTTP 400 (Bad Request), I can't get the folder created and that is the first step to get the document set created, according to my research and an example found here:
Is it possible to create a project documentset using graph API?
As the first step mentioned in the example above, I need to first create the folder and then proceed to the following steps to achieve the creation of the document set but I can't get this first step done.
My application has the necessary permissions as I tested in the Graph Explorer:
Files.ReadAndWrite.All
Sites.ReadAndWrite.All
Sites.FullControl.All (not required but I had to try this one to make sure!)
I'm on the second step (folder creation) and I can't get past this point, according to the link above, once I get this working I will need to get the new folder ID, and then send a new PATCH to alter its content type to match the desire document set, I hope I can get some help, all the examples are vague and pretty much describe only on what to do but no actual functional code to sample from.
Thanks in advance!
$uri = "https://graph.microsoft.com/v1.0/drives/b!yVnguUBzyUC1PxgTM0JP-_ERFp1PTZFCjycaWZK6yKulBi9Ce_J8RIfF-OkWKE4B/root/children"
$headers = #{
"Authorization" = "$($token.token_type) $($token.access_token)"
"Content-Type" = "application/json"
}
$body = #{
"name" = "Test"
"folder" = {}
"#microsoft.graph.conflictBehavior" = "rename"
}
$request = Invoke-RestMethod -Headers $headers -Body $body -Method Post -Uri $uri
It should work when you modify $body like this
$body = #{
"name" = "Test"
"folder" = #{}
"#microsoft.graph.conflictBehavior" = "rename"
} | ConvertTo-Json
$body is a JSON object and you need to convert it to JSON. For initialing an empty folder object you have to use #{} instead of {}.
Thanks #user2250152, you gave me a great idea by solving part of the problem!
I did add the conversion to JSON as you recommended and decided to add the content-type back to my original header and it did the trick!
Adding the content-type solved the issue but your collaboration was essential, so thank you very much!
Now wish me luck to get the other steps done and achieve the conclusion of this thing!
$headers = #{
"Authorization" = "$($token.token_type) $($token.access_token)"
"Content-Type" = "application/json"
}
I am running this code and its not returning the data to my $response variable as I expected.
$targetUrl = "https://www.tigerrunresort.com/vacation-rentals-homes-search.asp?txtStartDate=8%2F1%2F2022&txtEndDate=8%2F7%2F2022&categoryid=11317&destinationID=0&communityID=0&amenBedrooms=-1&maxPersons=0&advs=&sort=0¤tPage=1&flexibleDates=0&flexibleDatesThreshold=&searchAmenities=&showOnlySpecials=0"
$Response = Invoke-WebRequest -Uri $targetUrl
In rawcontent I should expect to see the string "Sorry" which does not show. What do I need to do to get it to return the values as I would see interactively?
Thanks!
You are not seeing it because it is not actually in the response. It is probably created and appended to the DOM by Javascript. Due to the Invoke-WebRequest not executing Javascript, it is never part of the response. If you are looking to orchestrate browsers (which do execute Javascript and offer access to the DOM) maybe you can try taking a look at Selenium (https://www.selenium.dev/)
For lauching a web request you can use for example start-process like this : start-process microsoft-edge:http://google.com/
Website I am attempting to do this on: https://www.netvendor.net/login
I am trying to login to the website automatically and kick off a script to create users in bulk for us.
If you run an Invoke-WebRequest on this page, it returns information, but nothing about the forms. They are simply not displayed. However, if you view the page source or inspect element, there are clearly forms on the page and they are not composed of JS or anything else that would mess it up. How can I get PowerShell to recognize these fields? I am using the following command:
Invoke-WebRequest -Uri "www.netvendor.net/login" -Method GET -UseBasicParsing
Because of the issue above, I decided I would just POST the information I needed by examining the request. The request requires three things:
email
password
_token
Unfortunately, the token is randomly generated each time a browser session is initiated. If you view source on the page and search for "_token", you will get the parameter that is needed. It doesn't seem like there is any way to retrieve this from the page? I am a bit lost as to what I can do at this point, especially since there is no API or anything else for me to work with.
For all interested, here is the final working script:
$nvlogin = Invoke-WebRequest "https://www.netvendor.net/login" -SessionVariable "netvendor"
$nvtoken = $nvlogin.InputFields.Where({ $_.Name -eq "_token" })[0].Value
$nvbody = #{
"_token" = $nvtoken
"email" = "your.name#website.com"
"password" = 'credentials'
}
Invoke-WebRequest -Uri "https://www.netvendor.net/login" -WebSession $netvendor -Method 'POST' -Body $nvbody
I'm working on a powershell script that will create a printer on my Company Print Server, which is all done and working.
But we have however a location where we list all used IP's, which is a webpage.
So I'm trying to use Invoke-WebRequest to do this. Login to the page works but then when it tries to use the form to add the IP it doesn't.
Login only has one submit button in the form and to add the IP there is three buttons. I will list them in the order they are on the page
Clear the form
Apply
Cancel
So I'm thinking when I Invoke-Webrequest it is using the "clear form" function.
Is there any way I could specify which function to use when sending a webrequest?
I cant remove the "Clear form" button since it is being used frequently, just to get that clear.
$R = Invoke-WebRequest -URI http://localhost -SessionVariable IP
$Form = $R.Forms[0]
$Form.fields["ctlUsername"]=User
$Form.fields["ctlPassword"]=Pass
$R=Invoke-WebRequest -Uri ("http://localhost/" + $Form.Action) -WebSession
$IP -Method POST -Body $Form.Fields
$D = Invoke-WebRequest -URI http://localhost/addIP.php -WebSession $Test
$Form2 = $D.Forms[0]
$Form2.fields["IP"]="172.10.0.1"
$Form2.fields["name"]="Printer300"
$Form2.fields["location"]="Room407"
$Form2.fields["MAC"]="000"
$Form2.fields["ID"]="000"
Invoke-WebRequest -Uri http://localhost/addIP.php -WebSession $IP -Method POST -Body $Form2.Fields
Don't think in terms of "filling in web pages" when it comes to sending those requests. Web pages for a human user. Filling in a form and clicking a button is the manual way of creating an HTTP message with data.
When you post this form:
IP: [172.10.0.1 ]
Name: [Printer300 ]
Location: [Room407 ]
[Apply] [Cancel]
the browser creates a text string like this:
IP=172.10.0.1&Name=Printer300&Location=Room407
and sends it to the target URL.
Invoke-WebRequest can create this format directly from your data. This means, you don't need to "fill in" anything. Just pass a hashtable with the proper keys and values:
Invoke-WebRequest "http://localhost/addIP.php" -WebSession $IP -Method POST -Body #{
IP = "172.10.0.1"
Name = "Printer300"
Location = "Room407"
MAC = "000"
ID = "000"
}
Of course the keys must correspond to the form field names.
That being said, I don't see much of a need for authentication with requests coming from the localhost. You can save yourself some busywork by disabling authentication for all local requests, and requiring it only for remote requests.
Anyway, making a website request a password and then storing that password in clear text in a script is not exactly secure to begin with.
Invoke-WebRequest supports Windows integrated authentication, so that you don't have to bother with usernames and passwords at all. This would be my preferred authentication solution for such tasks.
Found the issue, it needed a hidden value that was not being sent via the form but was added in php function later.
Thanks for all your help.
Considering following code, which is part of a larger powershell script. When the scripts is triggered, a phone call is made by Twilio to 'To' with a text (in the XML $MessagURL) that is read with a TextToSpeech-Engine.
$params = #{ To = $XmlDocument.Response.Dial; From = $number; Url =
$MessageURL}
# Make API request, selecting JSON properties from response
Invoke-WebRequest -URI $url -Method Post -Credential $credential -Body
$params -UseBasicParsing
$MessageURL is an URL to an XML. The XML contains the data for the text that needs to be said when calling the numbeer 'To' (which comes from another XML).
When $MessageURL is to a private public webspace, everything works fine.
When i change the $MessageURL to a azure blob-location, then the Twilio-logic can't read/access the XML-file anymore. The XML on the blob-storage is also free accessable through the internet.
Anyone any ideas why the script has problems with calling an xml on an azure blob-storage?