subprocess: stdout and stderror seem interchanged or mixed - subprocess

i'm trying to automate interactive ssh calls (see note 1) as follows:
SSHBINARY = '/usr/bin/ssh'
ASKPASS = '/home/mz0/checkHost/askpass.py'
def sshcmd(host,port,user,password,cmd):
env0 = {'SSH_ASKPASS': ASKPASS, 'DISPLAY':':9999'}
ssh = subprocess.Popen([SSHBINARY,"-T","-p %d" % port,
"-oStrictHostKeyChecking=no", "-oUserKnownHostsFile=/dev/null",
"%s#%s" % (user,host), cmd],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
env=env0,
preexec_fn=os.setsid
)
error = ssh.stderr.readlines()
result = ssh.stdout.readlines()
return error,result
host = 'localhost'
port = 22
user = 'try1'
password = '1try' # unused, hardcoded in ASKPASS
cmd = 'ls'
result1, error1 = sshcmd(host,port,user,password,cmd)
if result1 : print "OUT: %s" % result1
if error1 : print "ERR: %s" % error1
It turns i'm doing something stupid since i get this:
OUT: ["Warning: Permanently added 'localhost' (RSA) to the list of known hosts.\r\n"]
ERR: ['['Desktop\n', ..., 'Videos\n']']
Obviously stdout and stderr are swapped (note 2). Can you kindly point me my error?
note 1: i'm well aware of password-less ssh, dangers of ignoring host key etc. and hate requests on automating interactive ssh as much as you.
note 2: running the same command in shell confirms that stdout and stderr are swapped in my code
ssh -o 'UserKnownHostsFile /dev/null' -o 'StrictHostKeyChecking no' \
try1#localhost ls > ssh-out 2> ssh-error

return error,result
and
result1, error1 = sshcmd(...)
Just swap either of those around.

Related

Save job output from SDSF into a PDS and using ISPF functions in REXX

We periodically runs jobs and we need to save the output into a PDS and then parse the output to extract parts of it to save into another member. It needs to be done by issuing a REXX command using the percent sign and the REXX member name as an SDSF command line. I've attempted to code a REXX to do this, but it is getting an error when trying to invoke an ISPF service, saying the ISPF environment has not been established. But, this is SDSF running under ISPF.
My code has this in it (copied from several sources and modified):
parse arg PSDSFPARMS "(" PUSERPARMS
parse var PSDSFPARMS PCURRPNL PPRIMPNL PROWTOKEN PPRIMCMD .
PRIMCMD=x2c(PPRIMCMD)
RC = isfquery()
if RC <> 0 then
do
Say "** SDSF environment does not exist, exec ending."
exit 20
end
RC = isfcalls("ON")
Address SDSF "ISFGET" PPRIMPNL "TOKEN('"PROWTOKEN"')" ,
" (" VERBOSE ")"
LRC = RC
if LRC > 0 then
call msgrtn "ISFGET"
if LRC <> 0 then
Exit 20
JOBNAME = value(JNAME.1)
JOBNBR = value(JOBID.1)
SMPDSN = "SMPE.*.OUTPUT.LISTINGS"
LISTC. = ''
SMPODSNS. = ''
SMPODSNS.0 = 0
$ = outtrap('LISTC.')
MSGVAL = msg('ON')
address TSO "LISTC LVL('"SMPDSN"') ALL"
MSGVAL = msg(MSGVAL)
$ = outtrap('OFF')
do LISTCi = 1 to LISTC.0
if word(LISTC.LISTCi,1) = 'NONVSAM' then
do
parse var LISTC.LISTCi . . DSN
SMPODSNS.0 = SMPODSNS.0 + 1
i = SMPODSNS.0
SMPODSNS.i = DSN
end
IX = pos('ENTRY',LISTC.LISTCi)
if IX <> 0 then
do
IX = pos('NOT FOUND',LISTC.LISTCi,IX + 8)
if IX <> 0 then
do
address ISPEXEC "SETMSG MSG(IPLL403E)"
EXITRC = 16
leave
end
end
end
LISTC. = ''
if EXITRC = 16 then
exit 0
address ISPEXEC "TBCREATE SMPDSNS NOWRITE" ,
"NAMES(TSEL TSMPDSN)"
I execute this code by typing %SMPSAVE next to the spool output line on the "H" SDSF panel and it runs fine until it gets to this point in the REXX:
114 *-* address ISPEXEC "TBCREATE SMPDSNS NOWRITE" ,
"NAMES(TSEL TSMPDSN)"
>>> "TBCREATE SMPDSNS NOWRITE NAMES(TSEL TSMPDSN)"
ISPS118S SERVICE NOT INVOKED. A VALID ISPF ENVIRONMENT DOES NOT EXIST.
+++ RC(20) +++
Does anyone know why it says I don't have a valid ISPF environment and how I can get around this?
I've done quite a bit in the past with REXX, including writing REXX code to handle line commands, but this is the first time I've tried to use ISPEXEC commands within this code.
Thank you,
Alan

How to exit a REPL from the command line

I'm currently learning Lua and also learning how to work with CMD.
I know how to change a directory path and run my codes and files from that path
but what I don't know and I'm here to ask for is how to get out of a programming language REPL when you start it in CMD
For example to jump into the Lua REPL you should type:
lua53 (--like python3 for the Python language)
then you changed the CMD environment to a Lua compiler and can't access CMD commands such as
dir, cd, cls etc. and everytime when I need to access these commands I have to close the CMD window and open a new one.
Now can you guys tell me am I able to access CMD commands while in the Lua REPL? Or do I have to exit Lua first, and is there any command to exit a REPL?
I'd recommend EOF (Ctrl + D on Unix) rather than SIGKILL (Ctrl + C) because some REPLs (node and python) choose to ignore the latter (perhaps because it's often used for copying text?); python will just print KeyboardInterrupt whereas node will only exit if you press Ctrl + C twice (it will send a message telling you this the first time).
EOF (Ctrl + D) on the other hand immediately exists all REPLs I have used so far.
The Lua REPL stops immediately when it receives either EOF or SIGKILL, so you can use both here.
Edit: As Sowban points out, EOF apparently is entered as Ctrl + Z then Enter in powershell.
You could type ctrl c to exit the process that's running generally.
I suggest to write a REPL by yourself.
But be warned :-)
The main loop with a prompt and the interpreting and executing function/method is mostly the easiest part.
99.99999% is the errorhandling thing.
One of my earliest interpreter language is (A)REXX.
A REPL without any errorhandling is done with...
/* REXX have to start with a comment */
do forever
parse pull input
interpret input
end
Now Lua sandboxed to an _ENV with io and os library and a little bit of errorhandling...
#!/usr/bin/env -S readline-editor /usr/local/bin/lua
-- ^--SHEBANG for Linux --------------------------------------------------------------------------------------
-- interpreter.lua
-- Sandboxed to io and os
-- Lua 5.4 >>-because-> goto label <const> load()
--------------------------------------------------------------------------------------------------------------
local Lua = function(...)
-- Info about OS
io.write(("%s\n"):format(io.popen('uname -a'):read())):flush()
-- Global Setting
debug.setmetatable((1), {__index = math}) -- Add math library to number as methods (like string for strings)
os.setlocale('de_DE.UTF8')
os.setlocale('en_US.UTF8', 'time')
io.write(os.date("[%c]\n" .. ("%s\n"):format(os.setlocale():gsub("%;", "\n")))):flush()
-- io.write(("%s\n"):format(_VERSION)):flush()
-- Label for goto
::lua::
-- Local Setting
local args <const> = args or {...}
local Lua = Lua or true
local cmd = cmd or ""
--------------------------------------------------------------------------------------------------------------
local env <const> = env or setmetatable({os = os, io = io}, -- The _ENV for load()
{__call = function(self, ...)
local self, t = ({...})[1] or self, "" -- First argument becomes self if one
if ({...})[1] == "help" then self = getmetatable(({...})[2]).__index end -- Showing metamethod __index (table)
for k, v in pairs(self) do
t = t .. ("%s => %s\n"):format(k, v)
end
return t
end,
__index = {cg = collectgarbage,
gt = getmetatable,
pairs = pairs,
tn = tonumber,
ts = tostring,
_V = ("%s \27[1;" .. (31):random(36) .. "m(sandboxed)\27[0m"):format(_VERSION)},
__tostring = function(self) return self._V end}) -- end env
--------------------------------------------------------------------------------------------------------------
local prompt = prompt or setmetatable({}, {__tostring = function() return getmetatable(env).__index._V .. "> " end})
local name <const> = name or _VERSION
local result = result or true
--------------------------------------------------------------------------------------------------------------
while Lua do
io.write(tostring(prompt))
cmd = io.read() or 'quit'
if cmd == "quit" then Lua, result = true, true break end
Lua, result = pcall(load("return " .. cmd or false, name, "t", env))
if Lua and result then
io.write(("%s"):format(tostring(result)))
else
goto exception
end
end
--------------------------------------------------------------------------------------------------------------
-- Errorhandler
::exception::
io.write(("%s\n"):format("Exception"))
if not Lua or not result then
io.output(io.stderr)
io.write(("[%s][%s][%s]\n\27[1;31m>>-Exception->\27[0m %s\n"):format(os.date(), _VERSION, cmd, result)):flush()
io.output(io.stdout)
collectgarbage()
goto lua
end
goto lua
end
--------------------------------------------------------------------------------------------------------------
-- EXAMPLE ---------------------------------------------------------------------------------------------------
-- Lua = require("interpreter")
Lua() -- UNCOMMENT-> For direct execution like: /bin/lua interpreter.lua
return Lua -- UNCOMMENT-> For: Lua = require("interpreter")
Ctrl&C quits hardly
Ctrl&D is only an exception
...and os.exit(0) do an clean exit with returncode 0.
Impression

Most simple network protocol for a remote function call?

I am looking for the most simple protocol to program a remote function call, e.g. from Matlab to Julia.
[out1, out2, ...] = jlcall(socket, fname, arg1, arg2, ...);
fname is a string, all other input and output variables are numerical arrays (or other structures known to both sides) on Linux, Windows as option.
Client: open connection, block, pack and transmit
Server: receive, unpack, process, pack and transmit back
Client: receive, unpack, close connection and continue
The solutions I've seen (tcp, zmq) were built with old versions and do no longer work.
Protocol could (should?) be limited to do the pack/transmit - receive/unpack work.
UPDATE
Here is what I have come up with using pipes:
function result = jlcall(varargin)
% result = jlcall('fname', arg1, arg2, ...)
% call a Julia function with arguments from Matlab
if nargin == 0 % demo
task = {'foo', 2, 3}; % demo fun, defined in jsoncall.jl
else
task = varargin;
end
% create pipe and write function and parameter(s) to pipe
pipename = tempname;
pipe = java.io.FileOutputStream(pipename);
pipe.write(uint8(jsonencode(task)));
pipe.close;
% run Julia and read result back
system(sprintf('julia jsoncall.jl %s', unixpath(pipename)))
fid = fopen(pipename, 'r');
c = fread(fid);
result = jsondecode(char(c'));
fclose(fid);
function path_unix = unixpath(path_pc)
%convert path to unix version
path_unix = path_pc;
path_unix(strfind(path_unix,'\'))='/';
# jsoncall.jl
using JSON3 # get JSON3.jl from repository first
function foo(a,b) # demo function
a+b, a*b
end
jsonfile = ARGS[1] # called as > julia jsoncall.jl <json_cmdfile>
io = open(jsonfile, "r") # open IOStream for read
data = read(io) # read UTF8 data from stream
close(io) # close stream
cmd = JSON3.read(String(data)) # unpack stream into [fun, farg] array
fun = Symbol(cmd[1]) # first element is Julia function name,
result = #eval $fun(cmd[2:end]...) # others are function arguments
io = open(jsonfile, "w") # open IOStream for write
write(io, JSON3.write(result)) # (over-)write result back to stream
close(io) # close stream
Open points:
my first use of pipes/streams
output formatting: where Julia outputs a tuple of two, Matlab creates an an nx2 array.
replace json by msgpack for performance, might help with type formatting as well.
Your comments are welcome!
Here is a stripped down way of doing it. If you are going to vary your functions and arguments, a REST as in the comments server is going to be more flexible and less likely to pose a security risk (as you are eval()ing arbitrary code in some cases).
#server code
using Sockets
const port = 6001
const addr = ip"127.0.0.1"
const server = listen(addr, port)
while true
try
#info "Server on $port awaiting request..."
sock = accept(server)
#info "Server connected."
msg = strip(readline(sock))
#info "got message $msg"
fstr, argstr = split(msg, limit=2)
x = parse(Float64, argstr) # or other taint checks here...
ans = eval(Meta.parse(fstr * "($x)"))
#info "server answer: $ans"
write(sock, "$ans\n")
catch y
#info "exiting on condition: $y"
end
end
# client code
using Sockets
port = 6001
server = ip"127.0.0.1"
sock = connect(server, port)
#info "Client connected to $server"
func = "sinpi"
x = 0.5
#info "starting send"
write(sock, "$func $x\n")
flush(sock)
#info "flushed send"
msg = strip(readline(sock)) # read one line of input and \n, remove \n
ans = parse(Float64, msg)
println("answer is $ans")
close(sock)

Preview app doesn't open since I installed MACOS Catalina

import PIL
img = PIL.Image.new("RGB", (100,100))
img.show()
The error message:
FSPathMakeRef(/Applications/Preview.app) failed with error -43.
Following from Sean True's answer, an even quicker but temporary fix is to simply make a symbolic link to Preview.app in the old location. In the terminal run
ln -s /System/Applications/Preview.app /Applications/Preview.app
This fixed the problem for me.
There's an official fix in github for Pillow 7, but I'm still on 6.
This appears to be a PIL ImageShow issue, with the PIL MacViewer using /Applications/Preview.app as an absolute path to the OSX Preview app.
It's not there in Catalina. I did a quick hack to ImageShow.py changing /Applications/Preview.app to just Preview.app and the issue went away. That might or might not still work on pre-Catalina OSX, but I don't have an easy way to test.
It has apparently moved to /System/Applications/Preview.app so a quick check at run time would probably cover both cases.
elif sys.platform == "darwin":
class MacViewer(Viewer):
format = "PNG"
options = {'compress_level': 1}
preview_locations = ["/System/Applications/Preview.app","/Applications/Preview.app"]
preview_location = None
def get_preview_application(self):
if self.preview_location is None:
for pl in self.preview_locations:
if os.path.exists(pl):
self.preview_location = pl
break
if self.preview_location is None:
raise RuntimeError("Can't find Preview.app in %s" % self.preview_locations)
return self.preview_location
def get_command(self, file, **options):
# on darwin open returns immediately resulting in the temp
# file removal while app is opening
pa = self.get_preview_application()
command = "open -a %s" % pa
command = "(%s %s; sleep 20; rm -f %s)&" % (command, quote(file),
quote(file))
return command
def show_file(self, file, **options):
"""Display given file"""
pa = self.get_preview_application()
fd, path = tempfile.mkstemp()
with os.fdopen(fd, 'w') as f:
f.write(file)
with open(path, "r") as f:
subprocess.Popen([
'im=$(cat);'
'open %s $im;'
'sleep 20;'
'rm -f $im' % pa
], shell=True, stdin=f)
os.remove(path)
return 1

Is there any way to solve the argv unpacking problem

Not enough values to unpack argv variable
from sys import argv
script, user_name = argv
prompt = '>'
print("Hi %s I am the %s script." % (user_name, script))
print("I would like to ask you a few questions.")
print("Do you like me %s?" % user_name)
likes = input(prompt)
print("Where do you live %s?" % user_name)
lives = input(prompt)
print("What kind of computer do you have?")
computer = input(prompt)
print("""
Alright, so you said %r about liking me.
You live in %r. Not sure where that is.
And you have a %r computer. Nice.
""" % (likes, lives, computer))
ERROR:
line 3, in
script, user_name = argv
ValueError: not enough values to unpack (expected 2, got 1)