Ok so I asked a question earlier about a using
webclient.downloadfile($url,$path)
I couldn't open the PDF file because it was coded improperly.
What I now know is that it's because the website ONLY allows Internet Explorer to be used. therefore, I never actually downloaded the PDF, but rather an HTML page..as a PDF. Thus the error. For the website, when any other site is used, a page pops up letting you know so. I found out when I used:
$webclient.downloadstring($url)
..And read through a few lines in the Powershell ISE, coming across the same sentence:
"Detected Incompatible browser. Must use IE 7.0 and up"
Ok. So I did some research and learned about this .AddHeader() Function. I used the following.
$wc.Headers.Add("user-agent", "Windows-RSS-Platform/2.0 (MSIE 9.0; Windows NT 6.1)")
On this website actually. I figured this will trick the site into thinking I am using Internet Explorer 9.0. I typed in $webclient to view all properties of my Net.webclient object and read
Headers: {user-agent}
Sweet I said. Now it will work. But then I tried my .downloadfile function and the same shiz happened.
After running my downloadfile, I checked out the properties for $webclient again and Headers= {} What happened to my user-agent? Can anyone let me know why this is happening or offer any tips to get this working?
This is normal - headers are being reset after the first call made by WebClient. So any subsequent call will have them empty. Here's a proof link from msdn.microsoft.com:
Repeated calls with the same WebClient casues 404
We've found that the second (and subsequent) calls made using a WebClient fail.
This is because any headers are lost. So you need to ensure the custom headers
you use are reset before each call on the same WebClient instance.
Related
I'm struggling with very similar issue to the one described in this topic: .
I use PowerShell Remote session to start Internet Explorer application on Remote Computer (I use 'new-object -com InternetExplorer.Application' command.
Everything works great (the session, object creation...) until I call 'ExecWB(6,2)' method. I want to use this method to automatically print page from IE.
After calling that method nothing happens and, what is more, the com object loses information about itself. I cannot use any of its other methods anymore (which worked correctly earlier). The 'gm' command shows only basic COM object methods.
I have spend a lot of time searching for solution - checking if WMI, RPC services work, checking firewall rules, Internet Explorer security zones settings, printer sharing settings...). But I cannot find solution.
What is more, when I do the same from PowerShell window on target computer - everything works correctly. Problem occurs only when I am using Powershell remote sessions.
Does ExecWB method needs some special settings to use it? I'm trying to print a htm file located on the disk of the target computer (which I am connecting to using Powershell remote session). The file opens without problems but I cannot use ExecWB method.
Every helpful screens are already under the provided link.
Thank you in advance for help.
I maintain a vb.net website. A button on a page shows an AjaxControlToolkit.dll ModalPopupExtender. When I click the button, the page reloads, but no popup appears.
My tests below cover the scenarios I can think of. Please help me diagnose and fix.
This ModalPopupExtender succeeded in the past. The error might have started when we moved to a new server, or when we implemented https, or a browser update, or at some other time since then.
Tried Chrome (latest) and Firefox.
I RDPed into the server and opened the page in Chrome there. ModalPopupExtender succeeded. Same Chrome version as my PC. So, unlikely to be a code issue.
A different page on the same site uses ModalPopupExtender successfully. So, unlikely to be local PC settings.
I put lines of test code immediately before and after ModalPopupExtender.Show(). Both succeed.
Aha - found it!
Solution
The ASPX/HTML referred to http://ajax.googleapis.com. Changing the references to https made ModalPopupExtender.Show() work correctly for me.
Explanation/Diagnosis (if you can clarify further, please comment)
When I checked the html served to my browser, I noticed it defined a javascript function called fn(). The definition for fn() didn't appear in the html served to the server's browser. After the https change, fn() no longer appears in the html I receive. The other page, where ModalPopupExtender worked, didn't have any reference to googleapis.com.
I assume that using http instead of https caused ajax.googleapis.com to provide fn() and that fn() in some way interfered with the normal operation of my ModalPopupExtender.
Here's the fn() definition: (function {var fn = function() {Sys.Extended.UI.ModalPopupBehavior.invokeViaServer('ctl00_cphContent_ModalPopupExtenderConfirm', true); Sys.Application.remove_load(fn);};Sys.Application.add_load(fn);})();
Note: the http also included another javascript function related to the ModalPopupExtender. But there was a similar one on the working page, and in the working version served to the server's browser, and in my fixed version. So, I assume that function is correct.
I am iterating through a CSV file of URLs and using Invoke-WebRequest to get back the innerHTML and href values for links that match a specified criteria however this only works for some URLs and not for others unless I add the parameter -UseBasicParsing which doesn't provide the property access and filtering capabilities I need.
A common denominator is that the ones that don't work all use a www subdomain but a couple of them are still accessible without this but still don't work and I am not sure this should be an issue anyway as other www URLs do work
As mentioned above, I have tried adding UseBasicParsing which does allow a connection but this restricts the data that I have access to. I have also looked at the http headers for the URLS to try and understand what the differences are but am unsure what the issue is.
This functions correctly and returns the innerHTML text and href for each link on the page
$currentRequest = Invoke-WebRequest -Uri https://moz.com/learn/seo/what-
is-seo
$currentRequest | Get-Member
$currentRequest = $currentRequest.Links |
Select innerHTML, href |
WHERE innerHTML -like *SEO*
$currentRequest
Using exactly the same code with the following URL, the console just freezes until the script is exited
https://www.redevolution.com/what-is-seo
When I run the script with the working URL I get a pair of values for each link as shown below
innerHTML : Recommended SEO Companies
href : https://moz.com/community/recommended
With the non working URL as mentioned above the command line just stays at a blinking cursor.
This is just one example and I need to query other data as well so it would be great to understand how I can consistently run Invoke-WebRequest without issues.
Many thanks!!
Mike
Firstly, in the code that "works", i.e your first sample code, you are missing -UseBasicParsing. Now, why this is the case. Documentation here explains why: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-webrequest?view=powershell-5.1
To quote: "By default, script code in the web page may be run when the page is being parsed to populate the ParsedHtml property. Use the -UseBasicParsing switch to suppress this."
If you look at PowerShell v6, the default parsing has been switched to "-UseBasicParsing" (See here: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-webrequest?view=powershell-6)
It is not great, b/c of the reasons you mentioned and unfortunately there is no relief coming (see the comment from PowerShell dev here: https://twitter.com/Steve_MSFT/status/1153456742719639552?s=20)
Not so much an answer, as a long comment...
In PowerShell 5.1, Invoke-WebRequest uses the Internet Explorer engine to parse the html into a DOM, which can also cause execution of any scripts on the page, so it's possible something is going wrong in the script, or the headless Internet Explorer instance doesn't like the page content for whatever reason.
There are other reports elsewhere the same problem - for example Invoke-WebRequest hangs in some cases, unless -UseBasicParsing is used
Adding the -UseBasicParsing switch bypasses Internet Explorer and uses a much simpler internal HTML parser - if you need to extract additional information you can use a HTML parser library like the HtmlAgilityPack or AngleSharp to parse and query the $currentRequest.Content property.
Note that PowerShell Core 6.0 and up have made the -UseBasicParsing switch the default behaviour and there's effectively no way to turn it off, so if you want to write future-proof scripts now it's probably best to find a way to solve your problem using -UseBasicParsing so you dont have to rewrite it if / when you want to move up to PowerShell Core. (See Breaking Changes for PowerShell 6.0 -> Changes to Web Cmdlets)
See How to parse html in Powershell Core for a related question.
I don't understand this.
I have a webapi project setup locally on my own server, that works. I can go to http://mydomain.com/Api/Method and get my expected result no problem, works across 3g and everything too.
Then I have a website hosted somewhere else, in which tried to use this webapi method and this works fine when I test it locally, but as soon as I publish to the web and try it from there it fails with a 404 error?
I've tried both post and get methods and made sure the api method accepted both verbs.
I've tried calling the method both with javascript ajax and from within an MVC controller using a WebRequest.
I just don't understand why it works fine from any webbrowser, but I can't get it to work even with a programmatic WebRequest, shouldn't it be the same?
Nevermind.
Apparently, it was as simple as just setting an accept header with a value of "text/html" (in my case).
I spent way too much time figuring that out. Thanks #I4V for leading me in the right direction.
I am hosting my pages on my doamain. I put the following info. When I go to view the app it shows 405 Method Not Allowed..Not sure what I am doing wrong.
*
The first page load a APP makes is a POST request.
Seems your server could be rejecting them.
Its worth checking the server side error log for further information.
Indeed, it will most probably be a setting on your server. I had the same problem, my server didn't allow POST method on .html pages, changed my canvas page source to .php and all works fine now.
If you can look at your apache logs, it will provide a more clear error that we can help you debug with. Though a 405 error typically signifies a problem with the server and usually stems for a POST request.
I have made the mistake 2 times of creating my site in html instead of using a scripting extension. For example, if you are running on a Windows server and coding in Visual Studio, it is tempting to just create .htm or .html files. However, most web servers will not allow a POST operation to these pages. Instead, create .aspx pages, which will still allow you to use the same exact static HTML and JavaScript. This solved my problem in both cases.