I would like to come up with a mechanism by which I can share 'data' between different Powershell processes. This would be in order to implement a kind of job system, whereby a function can be run in one Powershell process, complete and then someone communicate its status to a function run from another (distinct) Powershell process...
I guess what I'd ideally like psjob results to be shareable between sessions, but this does not seem to be possible.
I can think of a few dirty ways of achieving this (like O/S environment variables), but am I missing an semi-elegant way?
For example:
Function giveMeNumber
{
$return_vlaue = Get-Random -Minimum -100 -Maximum 100
Return $return_vlaue
}
What are some ways i could get this function to store it's return somewhere and then grab it from another Powershell session (without using a database).
Cheers.
The QA mentioned by Keith refers to using MSMQ, a message queueing feature optionally available on desktop, mobile & server OS's from Microsoft.
It doesn't run by default on desktop OS's so you would have to ensure that the appropriate service was started. Seems like serious overkill to me unless you wanted something pretty beefy.
Of course, the most common choice for this type of task would be a simple shared file.
Alternatively, you could create a TCP listener in each of the jobs that you want to have accept external info. Not done this myself in PowerShell though I know it is possible. Node.JS would be a more familiar environment or Python. Seems like overkill if a shared file would do the job!
Another way would be to use the registry. Though you might consider that cheating since it is actually a database (of a very broken and simplistic sort).
I'm actually not sure that environment variables would work since I know that they can be picky about the parent environment scope (for example setting an env variable in a cmd doesn't make it available outside of the cmd scope by default.
UPDATE: Doh, missed a few! Some of them very obvious. Microsoft have a list:
Clipboard
COM
Data Copy
DDE
File Mapping
Mailslots
Pipes
RPC
Windows Sockets
Pipes was the one I was trying to remember. Windows sockets would be similar to a TCP listener.
Related
This question I asked has resulted in me exploring directly interfacing my FastCGI script to NGINX, rather than using a reverse proxy to Apache. I successfully modified my FastCGI script to run as a daemon using some code I found online:
my $s = FCGI::OpenSocket(':9000',20);
my $request = FCGI::Request( \*STDIN, \*STDOUT, \*STDERR, \%ENV, $s);
# Remaining code stays just as it does when using with Apache's mod_fcgid
while($request->Accept() >= 0) {
# Call core app subroutines.
}
It works, but near as I can tell this has a distinct disadvantage over mod_fcgid: I have one process running which will handle one request at a time and if that process dies, there's nothing to start it back up. There are references on Stack Overflow to code that properly spun off workers, but the sites referenced inevitably seem to have gone offline, much like FastCGI's own site.
So, I'm trying to figure out what I need to add and also -- pardon the pun -- figure out if I need to take a fork in this road. Here are the options that I am trying to consider, if I understand my issues correctly:
Directly implement some sort of forking mechanism, ideally it seems like it should (1) toss off the request to a process/thread/worker -- perhaps one that can stay alive for multiple requests -- and move on to being ready for the next request and (2) be independent enough from the workers that if something goes wrong with a worker, it doesn't bring down the whole system until I catch it and restart the main process (e.g. autorestart processes). If this can be done simply and reliably, this seems to have a huge appeal since the code already works with FastCGI.
Give up on direct FastCGI and convert to PSGI and use an application server to handle these things. Given that I'm using Perl, I'd guess Starman is the logical option, although I've been reading on uwsgi's PSGI support and it sounds almost ideal in "tyrant Emperor" mode, where it could run processes with different privileges, auto restart missing processes, etc.
Option 1 seems intriguing since it requires the least modification to my existing code and a FastCGI script started up without FastCGI still works like a normal CGI script. (I'm not running this code under FastCGI when it is used by sites that are very low traffic).
Option 2, though, feels like it might be more "modern." At least PSGI documentation seems to still be online, for example, and using Starman or uwsgi seem like they take care of the background stuff I need probably better than I would cooking up my own system. Downside: I'd need two startup scripts for my code: one to be used by the PSGI enabled sites and one for sites still running in CGI.
Update: Continuing to explore option 1, I read through this tutorial on Perl fork() which seems somewhat relevant. Would using fork to break off each FastCGI request be a good approach if I go with option 1? I assume I'd be at risk of fork bombing, although if I kept track of the number of forks and issued wait() if ($forks > 10); perhaps that would be a safe approach? (Or perhaps using Parallel::ForkManager to do that process watching.) Or would it be safer and/or more efficient using something like Thread::Queue and passing FastCGI request objects to a set a threads that are reliably already established? There seem to be plenty of pitfalls I might overlook, which then returns me to whether I should opt for Option 2.
I am working on a Perl script which does some periodic processing based on file-system contents.
The overall structure is like this:
# ... initialization...
while(1) {
# ... scan filesystem, perform actions depending on changes detected ...
sleep 5;
}
I would like to add the ability to input some data into this process by means of exposing an interface through HTTP. E.g. I would like to add an endpoint to skip the sleep, but also some means to input data that is processed in the next iteration. Additionally, I would like to be able to query some of the program's status through HTTP (i.e. a simple fork() to run the webserver-part in a separate process is insufficient?)
So far I have already used the Dancer2 framework once but it has a start; call that blocks and thus does not allow any other tasks (like my loop) to run. Additionally, I could of course move the code which is currently inside the loop to an endpoint exposed through Dancer2 but then I would need to call that periodically (though an external program?) which seems to be quite an obscure indirection compared to just having the webserver-part running in background.
Is it possible to unobtrusively (i.e. without blocking the program) add a REST-server capability to a Perl script? If yes: Which modules would be used for the purpose? If no: Should I really implement an external process to periodically invoke a certain endpoint or pursue a different solution altogether?
(I have tried to add a dancer2 tag, but could not do so due to insufficient reputation. Do not be mislead by this: I have so far only tried with Dancer2 not the Dancer (v.1))
You could try to launch your processing loop in a background thread, before you run start;.
See man perlthrtut
You probably want use threads::shared; to declare some variables shared between the REST part and the background thread. Or use dedicated queues/event mechanisms.
I want to copy a load module from one pds to another using REXX.
You could invoke IEBCOPY from within a Rexx, allocating the appropriate datasets to the appropriate ddnames before invoking IEBCOPY.
I'm unable to provide an example as I don't have the facilities/access.
Note that doing so may tie up your terminal/session.
You could also go into a more elaborate solution to build and submit a batch job, perhaps even having a panel front end, driving file tailoring/skeletons.
As #cshneid said you can use IEBCOPY Using IEBCOPY in rexx is basically the same as in JCL but:
use TSO Alloc to allocate the files
Call/invoke the program
if running under ISPF you can use LMCOPY. Roughly the following should work, you may need to issue a LMOPEN / LMClose on the data-ids as well ???
Address ISPEXEC
'LMINIT DATAID(DIDFrom) Dataset(in.data.set)'
'LMINIT DATAID(DIDTo) Dataset(to.data.set)'
'LMCOPY FromId('DIDFrom') FROMMEM(mymem) toId('DIDTo') toMem(newMemberName)'
'LMFREE DATAID(DIDFrom)'
'LMFREE DATAID(DIDto)'
If running foreground, the ISPF services used to have the advantage as they "co-ordinated" there actions with all other ISPF users- Less likely to corrupt the PDS directory. Not sure if this is an advantage any more.
Using just REXX what you want to do is not possible, however, you can invoke IEBCOPY (or your site equivalent) to perform the task for you.
You may want to investigate calling programs like IEBCOPY and passing it the appropriate control cards to perform your task.
I have a moderatly complex script made up of a PS1 file that does Import-Module on a number of PSM1 files, and includes a small amount of global variables that define state.
I have it working as a regular script, and I am now trying to implement it for Remoting, and running into some very odd issues. I am seeing a ton of .NET runtime errors with eventID of 0, and the script seems to work intermittently, with time between attempts seeming to affect results. I know that isn't a very good description of the problem, but I haven't had a chance to test more deeply, and I am just wondering if I am perhaps pushing PowerShell v2 further than it can really handle, trying to do remoting with a complex and large script like this? Or does this look more like something I have wrong in code and once I get that sorted I will get consistent script processing? I am relatively new to PowerShell and completely new to remoting.
The event data is
.NET Runtime version : 2.0.50727.5485 - Application ErrorApplication
has generated an exception that could not be handled. Process ID=0x52c
(1324), Thread ID=0x760 (1888). Click OK to terminate the application.
Click CANCEL to debug the application.
Which doesn't exactly provide anything meaningful. Also, rather oddly, if I clear the event log, it seems like I have a better chance of not having an issue. Once there are errors in the event log the chance of the script failing is much higher.
Any thoughts before I dig into troubleshooting are much appreciated. And suggestions on best practices when troubleshooting Remote scripts also much appreciated.
One thing about v2 remoting is that the shell memory limit is set pretty small - 150 MB. You might try bumping that up to say 1gb like so:
Set-Item WSMan:\localhost\shell\MaxMemoryPerShellMB 1024 -force
I attempting to add some fairly limited PowerShell support in my application: I want the ability to periodically run a user-defined PowerShell script and show any output and (eventually) be able to handle progress notification and user-prompt requests. I don't need command-line-style interactive support, or (I think) remote access or the ability to run multiple simultaneous scripts, unless the user script does that itself from within the shell I host. I'll eventually want to run the script asynchronously or on a background thread, and probably seed the shell with some initial variables and maybe a cmdlet but that's as "fancy" as this feature is likely to get.
I've been reading the MSDN documentation about writing host application code, but while it happily explains how to create a PowerShell object, or Runspace, or RunspacePool, or Pipeline, there's no indication about why one would choose any of these approaches over another.
I think I'm down to one of these two, but I've like some feedback about which approach is a better one to take:
PowerShell shell = PowerShell.Create();
shell.AddCommand(/* set initial state here? */);
shell.AddStatement();
shell.AddScript(myScript);
shell.Invoke(/* can set host! */);
or:
Runspace runspace = RunspaceFactory.CreateRunspace(/* can set host and initial state! */);
PowerShell shell = PowerShell.Create();
shell.Runspace = runspace;
shell.AddScript(myScript);
shell.Invoke(/* can set host here, too! */);
(One of the required PSHost-derived class methods is EnterNestedPrompt(), and I don't know whether the user-defined script I run could cause that to get called or not. If it can, then I'll be responsible for "starting a new nested input loop" (as per here)... if that impacts which path to take above, that would also be good to know.)
Thanks!
What are they?
Pipeline
A Pipeline is a way to concatenate commands inside a powershell script. Example: You "pipe" the output from Get-ChildeItem to Where-Object with | to filter them:
Get-ChildItem | Where-Object {$_}
PowerShell Object
The PowerShell object referes to a powershell session, like the one you would get when you start powershell.exe.
Runspace
Every powershell session has its own runspace (You'll always get output from Get-Runspace). It defines the state of the powershell session. Hence the InitialSessionState object/property of a runspace. You may decide to create a new powershell session, with its own runspace from within a powershell, to enable a kind of multithreading.
RunspacePool
Last but not least the RunspacePool. Like the name says, it's a pool of runspaces (or powershell sessions) that can be used to process a lot of complecated tasks. As soon as one of the runspaces in the pool has finished its task it may take the next task till everything is done. (100 things to do with 10 runspaces: on avarage they process 10 each but one may process 8 while two others process 11...)
When to use what?
Pipeline
The pipeline is used insed of scripts. It makes it easier to build complex scripts and should be used as often as possible.
PowerShell Object
The powershell object is used when ever you need a new powershell session. You can create one inside of an existing script, be it C# or Powershell. It's usefull for easy multithreading. On its own it will create a default session.
Runspace
If you want to create a non standard session of powershell, you can manipulate the runspace object before you create a powershell session with it. It's usefull when you want to share synchronized variables, functions or classes in the extra runspaces. Slightly more complex multithreading.
RunspacePool
Like mentioned before it's a heavy tool for heavy work. When one execution of a script takes hours and you need to do it very often.E.g. In combination with remoting you could simultanly install something on every node of a big cluster and the like.
You are overthinking it. The code you show in samples is a good start. Now you just need to read the result of Invoke() and check the error and warning streams.
PowerShell host provides some hooks that RunSpace can use to communicate with user, like stream and format outputs, show progress, report errors, etc. For what you want to do you do not need PowerShell Host. You can read results back from script execution using PowerShell class, check for errors, warnings, read output streams and show notification to the user using facilities of your application. This will be much more straightforward and effective than write entire PowerShell host to show a message box if errors detected.
Also, PowerShell object HAS a Runspace when it is created, you do not need to give it one. If you need to retain the runspace to preserve the environment just keep entire PowerShell object and clear Commands and all Streams each time after you call Invoke.
The next question you should ask is how to process result of PowerShell::Invoke() and read PowerShell::Streams.