Is there a way to copy quickly files from remote location to local in pyspark - pyspark

Question

I'm copying files from remote location using lftp using mget parameter. The task takes approximately 2 min to copy 50 xml files from a sftp machine to my local Unix machine. I'd like to be able to copy 20k files. An XML file is approx ~15kb. The dataframe df_files contains the list of all the XML files that I'd like to copy.
I've tried the code below with 20 thousand files, it seems to take few hours in order to create a dataframe with those files.
for row in df_files.tolist():
print row
cmd_p1 = """lftp sftp://username:password!#remotelocation-e "lcd /var/projects/u_admin/folder/;mget /var/projects/storage/folder/"""+row
cmd_p2 = """;bye " """
cmd_get_xml = cmd_p1+cmd_p2
s=subprocess.call(cmd_get_xml,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
j=0
for row in df_file.itertuples(index=True, name='Pandas'):
print getattr(row,'filename')
if j==0:
acq = sqlContext.read.format("com.databricks.spark.xml").option("rowTag","Message").load("file:///var/projects/u_admin/folder/"+df_file['filename'].iloc[j])
schema = acq.schema
else :
acq2 = sqlContext.read.format("com.databricks.spark.xml").option("rowTag","Message").load("file:///var/projects/u_admin/folder/"+df_file['filename'].iloc[j], schema = schema)
acq = acq.union(acq2)
I'd like to be able to copy those files for the least amount of time.

Answer 1

First, get all your .xml files into one directory using SCP module for Paramiko. Assuming that you .xml files have the same schema since you are able to do a union on the same, once you have all these xml files in one directory, you can directly read the entire directory rather than reading files individually.
This solution will save a lot of time that you spend in the for loop.
import paramiko
from scp import SCPClient
def createSSHClient(server, port, user, password):
client = paramiko.SSHClient()
client.load_system_host_keys()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(server, port, user, password)
return client
ssh = createSSHClient(server, port, user, password)
scp = SCPClient(ssh.get_transport())
Then call scp.get() or scp.put() to do SCP operations.
acq_all = sqlContext.read.format("com.databricks.spark.xml").option("rowTag","Message").load("file:///var/projects/u_admin/folder/", schema = schema)
I understand your use-case might be a little different, since you're also having an if-else block, but the schema is same, so it can be done once the files are read.
You can read one file, to get the right schema or you can define it by yourself before the read.