How to upload text file to FTP from Databricks notebook - pyspark

Question

I tried to find a solution but nothing. I am new in this, so please help me if you know the solution.
Thanks!

Answer 1

Ok, I found a solution.
#copy file from ADLS to SFTP
from ftplib import FTP_TLS
from azure.datalake.store import core, lib, multithread
import pandas as pd
keyVaultName = "yourkeyvault"
#then you need to configure keyvault with ADLS
#set up authentification for ADLS
tenant_id = dbutils.secrets.get(scope = keyVaultName, key = "tenantId")
username = dbutils.secrets.get(scope = keyVaultName, key = "appRegID")
password = dbutils.secrets.get(scope = keyVaultName, key = "appRegSecret")
store_name = 'ADLSStoridge'
token = lib.auth(tenant_id = tenant_id, client_id = username, client_secret = password)
adl = core.AzureDLFileSystem(token, store_name=store_name)
#create secure connection with SFTP
ftp = FTP_TLS('ftp.xyz.com')
#add credentials
ftp.login(user='',passwd='')
ftp.prot_p()
#set sftp directory path
ftp.cwd('folder path on FTP')
#load file
f = adl.open('adls path of your file')
#write to SFTP
ftp.storbinary('STOR myfile.csv', f)

Answer 2

In Databricks, you can access files stored in ADLS using any one of the method described below.
There are three ways of accessing Azure Data Lake Storage Gen2:
Mount an Azure Data Lake Storage Gen2 filesystem to DBFS using a service principal and OAuth 2.0.
Use a service principal directly.
Use the Azure Data Lake Storage Gen2 storage account access key directly.
Steps to mount and access the files in your filesystem as if they were local files:
To mount a Azure Data Lake Storage Gen2 or a folder inside a container, use the following command:
Syntax:
configs = {"fs.azure.account.auth.type": "OAuth",
"fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
"fs.azure.account.oauth2.client.id": "<appId>",
"fs.azure.account.oauth2.client.secret": "<password>",
"fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/<tenant>/oauth2/token",
"fs.azure.createRemoteFileSystemDuringInitialization": "true"}
dbutils.fs.mount(
source = "abfss://<container-name>#<storage-account-name>.dfs.core.windows.net/folder1",
mount_point = "/mnt/flightdata",
extra_configs = configs)
Example:
After mounting the ADLS, you can access in your filesystem as if they were local files, for example:
df = spark.read.csv("/mnt/flightdata/flightdata.csv", header="true")
display(df)
Example:
Reference: Databricks - Azure Data Lake Storage Gen2.
Hope this helps.

Answer 3

I have found a workaround for accessing the files outside Databricks (using sftp software like WinSCP/FileZilla).
These are the steps I followed -
I downloaded the awscli using this documentation https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html in Databricks's cluster's terminal.
I ran this command in my Databricks notebook -
!aws s3api put-object-acl --bucket S3-bucket-name --key
path/to-s3/file.txt --acl bucket-owner-full-control
By doing these 2 steps I was able to access the file outside Databricks.