Get s3 folder partitions with AWS Glue create_dynamic_frame.from_options - pyspark

Question

I'm creating a dynamic frame with create_dynamic_frame.from_options that pulls data directly from s3.
However, I have multiple partitions on my raw data that don't show up in the schema this way, because they aren't actually part of the data, they're part of the s3 folder structure.
If the partitions were part of the data I could use partitionKeys=["date"] , but date is not a column, it is a folder.
How can I detect these partitions using from_options or some other mechanism?
I can't use a glue crawler to detect the full schema including partitions, because my data is too nested for glue crawlers to handle.

Answer 1

Another way to deal with partitions is to register them in the Glue Data catalog either manually or programmatically.
You can use the glue client in a lambda function.
The example below is from a lambda function triggered upon file landing in a datalake, that parses the file name to create partitions in a target table in the glue catalog. (You can find a use case with Adobe Analytics in aws archives from where I got the code amazon-archives/athena-adobe-datafeed-splitter.)
import boto3
def create_glue_client():
"""Create a return a Glue client for the region this AWS lambda job is running in"""
current_region = os.environ['AWS_REGION'] # With Glue, we only support writing to the region where this code runs
return boto3.client('glue', region_name=current_region)
def does_partition_exist(glue_client, database, table, part_values):
"""Test if a specific partition exists in a database.table"""
try:
glue_client.get_partition(DatabaseName=database, TableName=table, PartitionValues=part_values)
return True
except glue_client.exceptions.EntityNotFoundException:
return False
def add_partition(glue_client, database_name, s3_report_base, key):
"""Add a partition to the target table for a specific date"""
# key_example = 01-xxxxxxxx_2019-11-07.tsv.gz
partition_date = key.split("_")[1].split(".")[0]
# partition_date = "2019-11-07"
year, month, day = partition_date.split('-')
# year, month, day = [2019,11,07]
part = key[:2]
# part = "01"
if does_partition_exist(glue_client, database_name, "target_table", [year, month, day, part]):
return
# get headers python list from csv file in S3
headers = get_headers(s3_headers_path)
# create new partition
glue_client.create_partition(
DatabaseName=database_name,
TableName="target_table",
PartitionInput={
"Values": [year, month, day, part],
"StorageDescriptor": storage_descriptor(
[{"Type": "string", "Name": name} for name in headers], #columns
'%s/year=%s/month=%s/day=%s/part=%s' % (s3_report_base, year, month, day, part) #location
),
"Parameters": {}
}
)
def storage_descriptor(columns, location):
"""Data Catalog storage descriptor with the desired columns and S3 location"""
return {
"Columns": columns,
"Location": location,
"InputFormat": "org.apache.hadoop.mapred.TextInputFormat",
"OutputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
"SerdeInfo": {
"SerializationLibrary": "org.apache.hadoop.hive.serde2.OpenCSVSerde",
"Parameters": {
"separatorChar": "\t"
}
},
"BucketColumns": [], # Required or SHOW CREATE TABLE fails
"Parameters": {} # Required or create_dynamic_frame.from_catalog fails
}
In your case with an existing file structure you can use the function below to iterate on the file keys in your raw bucket and parse them to create partitions :
s3 = boto3.resource('s3')
def get_s3_keys(bucket):
"""Get a list of keys in an S3 bucket."""
keys = []
resp = s3.list_objects_v2(Bucket=bucket)
for obj in resp['Contents']:
keys.append(obj['Key'])
return keys
Then you can build your dynamic frame as follows :
datasource = glueContext.create_dynamic_frame.from_catalog(
database = "database_name",
table_name = "target_table",
transformation_ctx = "datasource1",
push_down_predicate = "(year='2020' and month='5' and day='4')"
)

Answer 2

Another workaround to this is to show the file paths of the data using attachFilename option (similar to input_file_name in spark) as a column and parse the partitions manually from the string paths.
df = glueContext.create_dynamic_frame.from_options(
connection_type='s3',
connection_options = {"paths": paths, "groupFiles": "none"},
format="csv",
format_options = {"withHeader": True,
"multiLine": True,
"quoteChar": '"',
"attachFilename": "source_file"
},
transformation_ctx="table_source_s3"
)