Record Level Data truncation on Mainframe server while doing SFTP from spark server - scala

Question

Please read this fully.
I am working on sending a csv file via SFTP from spark application developed in scala to the mainframe server. I am using jsch (java secure channel) package version 0.1.53 version to accomplish the SFTP connection from spark server to mainframe server. I am facing issue that on the mainframe server, the csv file gets truncated to 1024 bytes per record line.
After research, I found that on the mainframe, we have options like using "lrecl" and "recfm" to control the length of each record in the file and the format of that record. But I am unable to integrate these options on scala. I found this answer on stackoverflow which was meant for implementation in Java. When I use the same logic on scala, I am getting the below error:
EDC5129I No such file or directory., file: /+recfm=fb,lrecl=3000 at
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846)
at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2198)
at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2215)
at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:1565)
at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:1526)
Scala code block using the jsch library for establishing the SFTP connection and transferring file is as below:
session = jsch.getSession(username, host, port)
session.setConfig("PreferredAuthentication","publickey")
session.setConfig("MaxAuthTries",2)
System.out.println("Created SFTP Session")
val sftpSessionConfig: Properties = new Properties()
sftpSessionConfig.put("StrictHostKeyChecking","no")
session.setConfig(sftpSessionConfig)
session.connect() //Connect to session
System.out.println("Connected to SFTP Session")
val channel = session.openChannel("sftp")
channel.connect()
val sftpChannel = channel.asInstanceOf[ChannelSftp]
sftpChannel.ls("/+recfm=fb,lrecl=3000") //set lrecl and recfm ---> THROWING ERROR HERE
sftpChannel.put(sourceFile, destinationPath,ChannelSftp.APPEND) //Push file from local to mainframe
Is there any way where we can set these options as the configuration in my scala code using the jsch library? I also tried using the spring-ml's spark-sftp package. But this package also has the problem of data truncation on the mainframe server.
Please help as this issue has become very critical blocker to my project.
EDIT: Updated question with scala code block

Answer 1

From this presentation Dovetail SFTP Webinar on slide 21:
ls /+recfm=fb,lrecl=80
it seems to me there is one '/' too many in your code.
From the error message, I think the SFTP server has the current path in the UNIX file system. You do not set the data set high level qualifier (HLQ) for the data set, do you? I can't see it in the code. Again from the above presentation, do a cd before the ls:
cd //your-hlq-of-choice
This will do two things:
Change the current working directory to the MVS data set side.
Set the HLQ to be used.
Sorry I cannot test myself; I do not know scala.

Answer 2

First, what SFTP server is running on z/OS? If it is the one provided with z/OS (not Dovetail) the command you are executing isn't supported and you will receive a message like Can't ls: "/+recfm=fb,lrecl=80" not found. Which would be valid because that is not valid file. Everything to the right of the / would be considered part of the filename.
I converted your code to Java as I'm not familiar with Scala and didn't have time to learn it. Here was my code sample I used.
import com.jcraft.jsch.JSch;
import java.util.Properties;
import java.util.Vector;
class sftptest {
static public void main(String[] args) {
String username = "ibmuser";
String host = "localhost";
int port = 10022; // Note, my z/OS is running in a docker container so I map 10022 to 22
JSch jsch = new JSch();
String sourceFile = "/";
String destinationPath ="/";
String privateKey = "myPrivateKey";
try {
jsch.addIdentity(privateKey); //add private key path and file
com.jcraft.jsch.Session session = jsch.getSession(username, host, port);
session.setConfig("PreferredAuthentication","password");
session.setConfig("MaxAuthTries", "2");
System.out.println("Created SFTP Session");
Properties sftpSessionConfig = new Properties();
sftpSessionConfig.put("StrictHostKeyChecking","no");
session.setConfig(sftpSessionConfig);
session.connect(); //Connect to session
System.out.println("Connected to SFTP Session");
com.jcraft.jsch.ChannelSftp channel = (com.jcraft.jsch.ChannelSftp) session.openChannel("sftp");
channel.connect();
// com.jcraft.jsch.Channel sftpChannel = (ChannelSftp) channel;
// channel.ls("/+recfm=fb,lrecl=3000"); //set lrecl and recfm ---> THROWING ERROR HERE
// channel.ls("/"); //set lrecl and recfm ---> THROWING ERROR HERE
Vector filelist = channel.ls("/");
for(int i=0; i<filelist.size();i++){
System.out.println(filelist.get(i).toString());
}
// channel.put(sourceFile, destinationPath, com.jcraft.jsch.ChannelSftp.APPEND); //Push file from local to mainframe
} catch (Exception e) {
System.out.println("Exception "+e.getMessage());
}
}
}
For my case I did use an ssh key and not a password. The output with your ls method is:
Created SFTP Session
Connected to SFTP Session
Exception No such file
dropping the + and everything to the right you get:
Created SFTP Session
Connected to SFTP Session
drwxr-xr-x 2 OMVSKERN SYS1 8192 May 13 01:18 .
drwxr-xr-x 7 OMVSKERN SYS1 8192 May 13 01:18 ..
-rw-r--r-- 1 OMVSKERN SYS1 0 May 13 01:18 file 1
-rw-r--r-- 1 OMVSKERN SYS1 0 May 13 01:18 file 2
The main issue is that the z/OS appears to not support the syntax you are using which is provided by a specific SFTP implementation by Dovetail.
If you do not have Dovetail I recommend that since you are sending CSV files that are generally variable in length that you send them as a USS file so that the lines will be properly translated and will be of variable length. Transfer them to USS (regular Unix on z/OS) and then copy them to an MVS file that has a RECFM of VB. Assuming the file is already allocated you could do a cp myuploadedFile.csv "//'MY.MVS.FILE'"