I have been given a task that involves downloading a single file every day from a website. Let's call it "https://test.example.com". I have credentials that allow me to login to the site, where a Flash interface then presents the files that are available for download. After the file is downloaded, it is then processed in a variety of ways. I have already put together the Powershell that handles all that, I am just having a hard time with automating the actual download of the file.
I used the Flash interface to download a few files while watching the network activity, and found that it is actually pulling the file from this URL:
https://test.example.com/link/EBDB7F67EF3B28XX99NCAD9920160423/file.zip
Therefore, I was able to put this together in order to automatically get the file via my PS script:
$url = 'https://test.example.com/link/EBDB7F67EF3B28XX99NCAD9920160423/file.zip'
$output = "C:\Downloads\file.zip"
Invoke-WebRequest -Uri $url -OutFile $output
However, the long string of numbers in the URL changes every day. The only discernible pattern I can find is that the last eight digits are always the date on which that particular file is posted.
Is there a good way to approach this? I've been experimenting with wildcards and patterns, as well as checking the HTML for elements that I can filter, but I am having a hard time finding the correct solution.
This is very hard to automate. You can't drive Flash from the script unless it is specifically designed for that. As I see it now your only options are:
Contact site devs if possible, maybe they can give you a details on function that generates link. This gives me an idea - perhaps you can reverse engineer Flash code to find that function details yourself. Use flash decompiler for this.
Simulate the user browsing the flash site. This can be done in one of the following ways:
Autohotkey - you can record mouse clicking relative to the browser window and execute the script again. Unless flash interface is too dynamic and unpredictive it will work.
Sikuli - another automation language which relies on picture segment recognition.
All above 2.* methods produce fragile automation code as they depend on browser settings (zoom, theme) and even OS settings. For this reason you need to dedicate one machine for that in all probability (virtual machine ofc). Decompiling flash code and re-implementing the url generting code in powershell will make it a reliable 100%.
As somebody said in comments this is not a powershell queestion but browser automation question.
Related
I need to upload a file to OneDrive, via the command line. This will be done through a batch file which is distributed to end users.
From searching on Stack Overflow, I find questions like this one which say that you need to register an app and create an app password, using Azure. I don't have the necessary permissions to do this in the organization where I work, nor can I do anything that requires an admin account. So I can't any install software - I have to use what comes with Windows 10. I can't use VBA either as that's blocked.
I've managed to download files from OneDrive without anything like that, using the process described here:
Open the URL in either of the browser.
Open Developer options using Ctrl+Shift+I.
Go to Network tab.
Now click on download. Saving file isn’t required. We only need the network activity while browser requests the file from the server.
A new entry will appear which would look like “download.aspx?…”.
Right click on that and Copy → Copy as cURL.
Paste the copied content directly in the terminal and append ‘--output file.extension’ to save the content in file.extension since
terminal isn’t capable of showing binary data.
Example:
curl https://xyz.sharepoint.com/personal/someting/_layouts/15/download.aspx?UniqueId=cefb6082%2D696e%2D4f23%2D8c7a%2
…. some long text ….
cCtHR3NuTy82bWFtN1JBRXNlV2ZmekZOdWp3cFRsNTdJdjE2c2syZmxQamhGWnMwdkFBeXZlNWx2UkxDTkJic2hycGNGazVSTnJGUnY1Y1d0WjF5SDJMWHBqTjRmcUNUUWJxVnZYb1JjRG1WbEtjK0VIVWx2clBDQWNyZldid1R3PT08L1NQPg==;
cucg=1’ --compressed --output file.extension
I tried to do something similar after clicking 'upload' on the browser, but didn't find anything useful when trying to filter the requests.
I found these two questions but there is no keyboard shortcut to upload, AFAICT. Also the end user will be uploading a file to a folder I've shared with them from my OneDrive. Opening Chrome or Edge as a minimised window is fine, but I can't just shove a window in their face which automatically clicks on things - they won't like that.
It's just occurred to me that I might be able to use an office application to Save As the file to the necessary onedrive folder, where the keyboard shortcuts are pretty stable, but have no idea how to achieve that via the command line.
The best and more secure way to accomplish this goal I think is going to be with the Rest API for OneDrive.
(Small Files <4MB)
https://learn.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_put_content?view=odsp-graph-online
(Large files)
https://learn.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_createuploadsession?view=odsp-graph-online
You still need a Azure AD App Registration (which your admin should be able to configure for you), to provide API access to services in Azure. Coding with the API is going to be far easier and less complicated, not to mention more versatile.
I am pretty new to IT Administration and I think I have a pretty complicated request but I want to know if it is even possible through Intune. I have an application that I want to deploy and I think I would either need to package the application as a Win32 app or via a Powershell Script. I basically would need to download the zip from this website, extract the contents, open the WCX, silently pass through the prompts, and pin the app to the taskbar. Again I am not sure if this is even possible but this would save me a lot of trouble if there was an easy way to do this. Here is the link.
https://kb.tempworks.com/help/how-to-add-an-enterprise-shortcut-to-the-desktop
To download a file, use Invoke-WebRequest with the -OutFile parameter.
To Install a .wcx you use rundll like this:
rundll32.exe 'tsworkspace,WorkspaceSilentSetup' C:\Path\To\The.wcx
(There's a whole wrapper script for that on the old technet script gallery)
It's impossible by design to pin things to users' task bar or start menu programmatically (well, you could script a bunch of hotkey presses and clicks, but don't).
I want to know if this is doable and get some hints about how to achieve this.
I guess at least it would need a confirmation dialog to run the (possibly evil) macro.
I want to produce an OpenOffice document that will upload itself on save to a hardcoded URL.
Is it possible?
What are the rough steps to achieve this?
My guess is:
bind a macro to the save event
have that macro get the current file binary data
have it post this data to a URL
but before researching about how to do this I need to know if this can be done in the first place.
I don't believe you need a macro to do this, instead depending on your OS you can map a FTP or other type of protocol remotely.
For example in windows you can "Map" a FTP as a drive and this would do exactly the same thing as your describing, you open the file from the FTP and upon saving it will then be written to the remote server. FTP is just an example here, there are other platforms you can use.
If you are at all interested in this method then following the instructions below, otherwise disregard.
Mapping a Network Drive in Windows Vista and 7.
Click on Windows start orb and then click on "computer".
Click on map a network drive.
Map a network drive will then open a new dialog box where you can click "connect to a website"
Simply follow that easy to use wizard and click add a new network location (Choose a custom network location)
Type in your FTP address, including username and password.
Finally name your network location to whatever you want.
Just a suggestion, I really don't think you need any macro unless you plan on distributing these files to other people then yes, but they would need to install that macro/plugin on their open office since there is no way to encode the document itself with such features as far as I know.
So the website constantly changes the data that it displays, and I want to get that data every several seconds and log it in a spreadsheet. The problem is in order to get to the page, I have to have a cookie which I get when I log in. Unfortunately I only know how to program in MATLAB. MATLAB has a function for this, urlread, but it doesn't deal with cookies. What can I do to get to that page? Can anyone help me with this? Point me into a direction where a programing noob like me can succeed please.
You could use wget to download content while using HTTP cookies. I will be using StackOverflow.com as example target. Here are the steps to follow:
1) Obtain the wget command tool. For Mac or Linux, I think it is already available. On Windows, you can get it from the GnuWin32 project or from one of the many other ports (Cygwin, MinGW/MSYS, etc..).
2) Next we need to obtain an authenticated cookie by logging into the website in question. You can use your preferred browser for this.
In Internet Explorer, you can produce it using "File menu > Import and Export > Export Cookies". In Firefox, I used the Cookie Exporter extension to export cookies to text file. For Chrome, there should be similar extensions
Obviously you only need to do this step once, as long as the cookies have not yet expired!
3) Once you locate the cookie file exported, we can use wget to fetch the web page and provide it with this cookie. This of course can be performed from inside MATLAB using the SYSTEM function:
%# fetch page and save it to disk
url = 'http://stackoverflow.com/';
cmd = ['wget --cookies=on --load-cookies=./cookies.txt ' url];
system(cmd, '-echo');
%# process page: I am simply viewing it using embedded browser
web( ['file:///' strrep(fullfile(pwd,'index.html'),'\','/')] )
Parsing the web page is a whole other topic that I will not go into. Once you get the data you seek, you can interact with Excel spreadsheets using the XLSREAD and XLSWRITE functions.
4) Finally you can write this in a function, and make it execute on regular intervals using the TIMER function
Try using the java.net.* classes.
You should be able to use them directly in the MATLAB workspace, as described here: http://www.mathworks.co.uk/help/techdoc/matlab_external/f4863.html
Matlab has built-in functions for web downloading. For http sites, there is webread.m and websave.m. For FTPs, there is mget.m
I have a perl script which when run from the command line generates a text file of data with a specific format for use by another application. The script also prints informational warning messages on stderr. I'm writing a web front end for this. In an ideal world when the user clicks 'submit' on the associated form, a page would be displayed in the browser containing the informational messages, and simultaneously a pop-up would appear allowing the user to save the text file of data to disk. I would like this to work on browsers without javascript enabled, so I think exactly what I want is probably not possible.
Some sites I have seen deal with this kind of thing by displaying the page with the informational messages, and a link to the file to be downloaded. This would seem to mean having to store the files and sorting out some sort of security so that another user cannot download your file (not that this is a big deal for the application in question).
I'm wondering if there is a more elegant way of dealing with this? e.g Is it possible to use multipart messages to somehow achieve returning both pieces of information in one go? Is it possible to pop-up a second window with the informational messages without using javascript? Apologies if these seem like basic questions - my programming knowledge is in the domain of DNA sequence manipulation algorithms rather than web page generation..
If (and only if) the data is quick and easy to generate, do it once for error messages and a second time for download. The link or button of the error-message page would regenerate the results and prompt for download.
This is a bit of a hack since you need to consider what to do if the underlying data changes before the user hits the download link. Be careful to set the header correctly for file download vs normal webpage, eg,
if($submit) {
print header(-type=>'application/octet-stream',
-Content_disposition=>'attachment; filename=foobar.dat');
Gen_Results();
}
To be honest, I'd just use a little javascript anyway since it's a pretty safe assumption now a days. Otherwise, use a "noscript" tag for some alternative.