IBM i: ssh remotehost /QOpenSys/usr/bin/db2 \""select 123 from sysibm.sysdummy1"\" returns blank output - db2

Question

I have found a nice hint from Kevin Adler:
$ ln -s /QOpenSys/usr/bin/qsh /QOpenSys/usr/bin/db2
which runs great in QSH both locally and also over ssh
https://kadler.io/2018/05/29/calling-qsh-utilities-from-pase.html#
/QOpenSys/usr/bin/db2 "select 123 from sysibm.sysdummy1"
and also
ssh remotehost /QOpenSys/usr/bin/db2 ""select 123 from sysibm.sysdummy1""
returns the expected 123
123
1 RECORD(S) SELECTED.
This is a handy shortcut, otherwise the command would have to be expressed as:
ssh remotehost /QOpenSys/usr/bin/system -eiqs ""CALL QSYS/QP0ZCALL ('/usr/bin/db2' 'select 123 from sysibm.sysdummy1')""
Yet on some hosts, even if /QOpenSys/usr/bin/db2 works fine locally, it returns blank output when being called over ssh. (no problem with other commands over ssh so far, all systems mentioned here are IBM i on 7.3 or 7.4)
Although there is a blank output from some of the hosts of ours for the /QOpenSys/usr/bin/db2 command the instructions get executed correctly.
create table, insert, delete... all of this gets done
ssh remotehost system ""QSYS/DSPOBJD OBJ(SCO/ROB123) OBJTYPE(*FILE)""
CPF2105: Object ROB123 in SCO type *FILE not found.
$
ssh remotehost /QOpenSys/usr/bin/db2 ""create table sco.rob123 (Number int, Name varchar(10), City varchar(10))""
$
ssh remotehost system ""QSYS/DSPOBJD OBJ(SCO/ROB123) OBJTYPE(*FILE)""
Display Object Description - Basic Page
1
5770SS1 V7R4M0 190621 REMOTEHOST 22-08-25 11:22:42 EEDS
T
Library . . . . . . . : SCO Library ASP device . : *SYSBAS
Library ASP group . . : *SYSBAS
Object Type Attribute Size Text
ROB123 *FILE PF 40960
* * * * * E N D O F L I S T I N G * * * * *
$
ssh remotehost system ""DSPFD FILE(SCO/ROB123) TYPE(*ALL)"" | grep "Total records"
Total records . . . . . . . . . . . . . . : 0
$
ssh remotehost /QOpenSys/usr/bin/db2 ""insert into sco.rob123 values (1, 'jenda', 'crhof')""
$
ssh remotehost system ""DSPFD FILE(SCO/ROB123) TYPE(*ALL)"" | grep "Total records"
Total records . . . . . . . . . . . . . . : 1
$
ssh remotehost /QOpenSys/usr/bin/db2 ""delete from sco.rob123""
$
ssh remotehost system ""DSPFD FILE(SCO/ROB123) TYPE(*ALL)"" | grep "Total records"
Total records . . . . . . . . . . . . . . : 0
$
ssh remotehost /QOpenSys/usr/bin/db2 ""insert into sco.rob123 values (1, 'jenda', 'crhof')""
$
ssh remotehost system ""DSPFD FILE(SCO/ROB123) TYPE(*ALL)"" | grep "Total records"
Total records . . . . . . . . . . . . . . : 1
$
ssh remotehost /QOpenSys/usr/bin/db2 ""select * from sco.rob123""
$
ssh -v remotehost /QOpenSys/usr/bin/db2 ""select * from sco.rob123""
OpenSSH_8.0p1, OpenSSL 1.1.1l 24 Aug 2021
debug1: Reading configuration data /QOpenSys/QIBM/ProdData/SC1/OpenSSH/etc/ssh_config
debug1: Connecting to remotehost [10.1.1.10] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_rsa type 0
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: identity file /home/user/.ssh/id_dsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519 type -1
debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
debug1: identity file /home/user/.ssh/id_xmss type -1
debug1: identity file /home/user/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug1: Authenticating to remotehost:22 as 'user'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:WOjO7zT1MIS3NeMhPq786/0GVGbZYlGXhDnFOlRpL7k
debug1: Host 'remotehost' is known and matches the ECDSA host key.
debug1: Found key in /home/user/.ssh/known_hosts:50
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/user/.ssh/id_rsa RSA SHA256:mP1UusmDO6fvA+67200rZHR+XC2hMZTITEejMuWZRZw
debug1: Will attempt key: /home/user/.ssh/id_dsa
debug1: Will attempt key: /home/user/.ssh/id_ecdsa
debug1: Will attempt key: /home/user/.ssh/id_ed25519
debug1: Will attempt key: /home/user/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2
-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key: /home/user/.ssh/id_rsa RSA SHA256:mP1UusmDO6fvA+67200rZHR+XC2hMZTITEejMuWZRZw
debug1: Server accepts key: /home/user/.ssh/id_rsa RSA SHA256:mP1UusmDO6fvA+67200rZHR+XC2hMZTITEejMuWZRZw
debug1: Authentication succeeded (publickey).
Authenticated to remotehost ([10.1.1.10]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00#openssh.com want_reply 0
debug1: Remote: /home/user/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /home/user/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending command: /QOpenSys/usr/bin/db2 "select * from sco.rob123"
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow#openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 3132, received 2860 bytes, in 0.4 seconds
Bytes per second: sent 7799.9, received 7122.5
debug1: Exit status 0
$
And I keep failing to find out why the output for this particular command /QOpenSys/usr/bin/db2 on a handful of our systems is blank.
Does any one, please, have any clue?
Thank you

Answer 1

it came out it was not specifically about db2 but more in general the use of
/QOpenSys/usr/bin/system (/QOpenSys/QIBM/ProdData/OS400/PASE/bin/system)
did not provide any output at all when called over ssh
Resolution:
The IBM /QOpenSys/usr/bin/qsh
contains at the end of the script
...
exec /QOpenSys/usr/bin/system -eiqs "CALL QSYS/QP0ZCALL ('/usr/bin/$basename' $args)"
and after omitting the "s" parameter:
exec /QOpenSys/usr/bin/system -eiq "CALL QSYS/QP0ZCALL ('/usr/bin/$basename' $args)"
all /usr/bin/... commands wrapped into /QOpenSys/usr/bin/system provide the output even when called over ssh.
So now it is time to inspect CCSIDs
( https://www.ibm.com/docs/en/i/7.3?topic=utilities-pase-i-system-utility )