I'm completely new to Powershell and I know that a number of people use it to automate tasks much in the way bash and c-shell programming is done in *NIX. I've successfully recompiled some ancient analysis software written in FORTRAN that takes individual input files. I now need to somehow run just under 1000 cases with only slightly varied input files. The analysis software writes intermediate files, so for concurrent runs, every run has to be within a different directory. Each case can take up to 40 minutes to solve, so individually running these will take a lot of time and be prone to error.
So now for the question, can Powershell automate this and is there some similar script out there that I can modify to do it?
The automation would need to do the following (as I see it):
Take in an input file with the various runs that have to be run
Create a subdirectory relative to the run name/number
Save a version of the input files with the variables switched in the subdirectory
Run the analysis software in the subdirectory
Look at standard/error output of analysis software to confirm it was successful
Append to a file success or failure of a run
Ideally would be able to run up to some number of analyses concurrently (4-6 for my machine)
If IT reboots the machine (as they do whenever they choose), I'd like to be able to restart where it left off, though I expect the loss of anything that the analysis software was running during the forced reboot.
I've tried recompiling the software with vectorization and automated parallelization and on the tested cases, the convergence time was only minimally reduced, so it is safe to assume that this is effectively single threaded.
Powershell has lots of familiar aliases for Unix users. ls, cat, cp etc are implemented as aliases to native Powershell commands. The commands are not case sensitive. What's more, you can search help even with alias' name. That is,
man ls <=> get-help get-childitem
apropos <=> get-help <keyword>
get-help loop
about_Break
about_Continue
about_do
about_For
about_Foreach
about_Language_Keywords
...
This should help converting an existing script. For the rest, I'll give some hints as the description is somewhat vague.
Get-Content is used to read file contents into a variable: $myVar = cat c:\some\file.txt.
Directory creation is just md.
Capturing exe output is done by assigning to a variable: $exeOutput = c:\myApp.exe
Adding stuff to a file is Add-Content.
Background jobs are started with Start-Job.
Related
I'm not a native English speaker as such pardon some discrepancy in my question. I'm looking at a way to Automate option selection in Programs/Scripts running via PowerShell.
For example:
Start-Process -FilePath "velociraptor.exe" -ArgumentList "config generate -i"
In the above snipper PowerShell will run Velociraptor and initiate the configuration wizard from a Ps1 file. The Wizard has few options. After running it will generate some Yaml files.
As such what would be the way to have PowerShell Script automate the option selection process? I know what the option should be. I looked around but I don't know proper terms to find what I need. Nor am I sure this can be done with PowerShell.
The end goal is to have the Ps1 download Exe, run the config command and continue with choosing the selection based on predefined choices. So far I gotten download and lunching of the velociraptor.exe working. But not sure how to skip the window in screenshot and have PowerShell script do it instead.
I couldn't find a CLI reference for velociraptor at https://www.velocidex.com/, but, generally speaking, your best bet is to find a non-interactive way to provide the information of interest, via dedicated _parameters_ (possibly pointing to an input file).
Absent that, you can use the following technique to provide successive responses to an external program's interactive prompts, assuming that the program reads the responses from stdin (the standard input stream):
$responses = 'windows', 'foo', 'bar' # List all responses here
$responses | velociraptor.exe config generate -i
I got a problem when executing batch file commands through matlab. This batch file includes commands to run simulations in Adams. When I execute the batch file directly from DOS window, it works well. But if I use matlab to execute it (using command dos()), it gives error saying 'cannot check out the license for Adams'.
This confuses me: if the license is incorrect, it should not work no matter I execute the batch file directly in DOS or ask MATLAB to execute it. I also tried to execute other DOS commands through matlab using dos() and it worked well.
Does anyone know what the problem may be?
Such issues are commonly caused by some environment variables being changed or cleared by MATLAB. I have very similar experience on Linux and Mac OS X, where this causes havoc when using system or unix.
In Unix-like systems, MATLAB is started from a shell script where all of this happens. So you can either incorporate missing variables there or in the .matlab7rc.sh in your home directory (the latter is preserved when you upgrade MATLAB and it is much easier to use). I won't go into all the Unix details here.
An alternative workaround is to explicitly set those variables when you issue a system command (e.g. system('export variable=value ; ...')). It is quite a bit of work, but you can then use that MATLAB code on different computers with ease.
On Windows, I'm not completely sure about the exact location of the corresponding files (and whether MATLAB starts in quite a similar way as on Unix). But if they exist, you can probably find it in the MATLAB documentation.
Anyhow, the alternative fix should work here as well.
First you need to diagnose which system variables you need (likely a PATH or anything that has a name related to Adams).
To do so in Windows, run set from the Windows command prompt (cmd.exe) and from within MATLAB. Whatever differs in the output is a possible suspect for your problem.
To inspect just a single variable, you can use the command echo %variablename%.
I will assume that you have found that the suspect environment variable is missing and should be set to value.
The workaround fix is then to run your command in MATLAB as system('set suspect=value & ...') where you replace ... with your original command.
Quite recently, I wrote a few scripts in Perl for a cPanel plugin in which, though most of the code was in Perl, there was quite a lot of system() commands as well which I used to execute shell commands directly.
I am pretty sure that there are Perl modules that I could have used instead. Keeping in mind the time crunch, I thought using the system command was easier (to complete the project in time). In retrospective, I think that was a bad programming practice.
My question is, is there any tradeoff, memory-wise or otherwise when using Perl's modules and using system() commands. For example, what would be the difference in using:
my $directory = "temp";
mkdir $directory;
and
system ("mkdir temp");
Also, if I am to use Perl modules, wouldn't that involve installing a whole lot of modules in the beginning?
The most obvious economy is that, in the first case, your Perl process is creating the directory, while in the second, Perl is starting a new process that runs a command shell which parses the command line and runs the shell mkdir command to create the directory, and then the child process is deleted. You would be creating and deleting a process and running the shell for every call to system: there is no caching of processes or similar economy.
The second thing that comes to mind is that, if your original mkdir fails, it is simple to handle the error in Perl, whereas shelling out to run a mkdir command puts your program at a distance from the error, and it is far more awkward to handle the many different problems that may arise.
There is also the question of maintainability and portability, which will affect you even if you aren't expecting to run your program on more than one machine. Once you abandon control to a system command you have no control over what happens. I could have written a mkdir that will delete your home directory or, less disastrously, your program may find itself on a system where mkdir doesn't exist, or does something slightly different.
In the particular case of mkdir, this is a built-in Perl operator and is part of every Perl installation. There are also many core libraries that require you to put use Module in your program, but are already installed and need no further action.
I am sure others will come up with more reasons to prefer a Perl operator or module over a shell command. In general you should prefer to keep everything you can within the language. There are only a few cases where you have to run a third-party program, and they usually involve custom software that allows you act on proprietary data formats.
I've recently discovered the joy of going through JConsole.exe instead of J.exe to run various scripts. There's generally a noticeable performance gain.
However, sometimes I need to use wd winexec (calling ad-hoc programs for example) and in the console, 11!:0 (wd) support is not available.
Is there a way to send a command from JConsole.exe to the regular Windows command line interpreter? Or maybe a workaround?
You might try the task script. See the script itself for documentation.
J6: ~system/packages/misc/task.ijs',
J7: ~system/main/task.ijs
It contains utilities such as fork_jtask_, spawn_jtask_, shell_jtask_
You can load the script in both versions using: require 'task'
I have a Perl script that contains this code snippet, which calls the system shell to get some files by SFTP and unzip them with WinZip:
# Run script to get files from remote server
system "exec_SFTP.vbs";
# Unzip any files that were retrieved
foreach $zipFile (<*.zip>) {
system "wzunzip $zipFile";
}
Even if some files are retrieved, they are never unzipped, because by the time the files are retrieved and the SFTP connection is closed, the Perl script has already completed the unzip step, with the result that it doesn't find anything to unzip.
My short-term fix is to insert
sleep(60);
before the unzip step, but that assumes that the SFTP connection will finish within 60 seconds, which may sometimes be a gross over-estimate, and other times an under-estimate.
Is there a more sound way to cause Perl to pause until the SFTP connection is closed before proceeding with the unzip step?
Edit: Responders have questioned (and reasonably so) the use of a VB script rather than having Perl do the file transfer. It has to do with security -- the VB script is maintained by others and is authorized to do the SFTP.
Check the code in your *.vbs file. The system function waits for the child process to finish before execution continues. It appears that your *.vbs file is forking a background task to do the FTP and returning immediately.
In a perfect world your script would be rewritten to use Net::SFTP::Foreign and Archive::Extract..
An ugly quick-hackish kind of way might be to create a touch-file before your first system call, alter your sftp-fetching script to delete the file once it is done and have a while like so
while(-e 'touch.file') {
sleep 5;
}
# foreach [...]
Of course, you would need to take care if your .vbs fails and leaves the touchfile undeleted and many other bad side effects. This would be for a quick solution (if none of the other suggestions work) until you get the time to rewrite without system() calls.
You need a way for Perl to wait until the SFTP transfer is done, but as your script is currently written, Perl has no way of knowing this. (It looks like you're combining at least two scripting languages and a (GUI?) SFTP client; this can work, but it's not exactly reliable or robust. Why use VBscript to start the SFTP transfer?)
I can think of four options:
Your Perl script could do the SFTP transfer itself, using something like CPAN's Net::SFTP module, rather than spawning an external job whose status it cannot track.
Your Perl script could spawn a command-line SFTP utility (like PSFTP) that doesn't return until the transfer is done.
Or change exec_SFTP.vbs script to not return until the transfer is done.
If you're currently using a graphical SFTP client and can't switch for whatever reason, I'd recommend using a scripting language like AutoIt instead of Perl. AutoIt has features to wait for windows to change state and so on, so it could more easily monitor for an activity's completion.
Options 1 or 2 would be the most robust and reliable.
The best I can suggest is modifying exec_SFTP.vbs to exit only after the file transfer is complete. system waits for the program it called to complete, so that should solve your problem:
system LIST
system PROGRAM LIST
Does exactly the same thing as "exec LIST", except
that a fork is done first, and the parent process
waits for the child process to complete.
If you can't modify the vbs script to stay alive until it terminates, you may be able to track subprocess creation. If you get subprocess ids, you can monitor them thereby know when the vbs' various offspring terminate.
Win32::Process::Info lets you get a subprocess ids from a running process.
Maybe this is a dumb question, but why not just use the Net::SFTP and Archive::Extract Perl modules to download and unzip the files?
system will not return until the shell it's running the command in has returned; this may be wrong for launching graphical programs and file associations.
See if any of the following help?
system('cscript exec_SFTP.vbs');
use Win32::Process;
use Win32;
Win32::Process::Create(my $proc, 'wscript.exe',
'wscript exec_SFTP.vbs', 0, NORMAL_PRIORITY_CLASS, '.');
$proc->Wait(INFINITE);
Have a look at IPC::Open3
IPC::Open3 - open a process for reading, writing, and error handling using open3()