Preprocessing weird data in pyspark - pyspark

Question

I am working with a set of climate data that come in a very strange distribution and is difficult to work with. I decided to work with pyspark because it is a large volume of data, you know, with the idea of saving time.
The data format is .ascii/.text/.dat, whatever you want to call it, And the distribution is as follows:
Date 1
Value 1
Value 2
Value 3
Value 4
Value 5
Value 6
Value 7
Value 8
Value 9
Value 10
Value 11
Value 12
.
.
.
.
.
Value 101178
Date 2
Value 1
Value 2
Value 3
Value 4
Value 5
Value 6
Value 7
Value 8
Value 9
Value 10
Value 11
Value 12
.
.
.
.
.
Value 101178
That is, it is a table composed of tables of 101178 data distributed in 6 columns (16863 rows).
In case the explanation is not very clear, I attach a link to a small fragment of the file. (the original file is >50GB)
https://drive.google.com/file/d/1-aJRTWzpQ5lHyZgt-h7DuEY5GpYZRcUh/view?usp=sharing
My idea is to generate a matrix with the following structure:
Date 1
Date 2
Date n
Value 1
Value1.2
Value1.n
Value 2
Value2.2
Value2.n
Value n
Valuen.2
Valuen.n
I have tried to make the question as clear as possible. As I said I am working with pyspark so if anyone has any solution to do this data processing using this tool I would be very grateful.
Thank you all very much!

Answer 1

I managed to get it very close, but not exactly, to your expected dataframe structure.
To test code's output I made this dummy dataset to play with, because it's super hard to follow lots of number in your original dataset
+--------------------------------------------+
|value |
+--------------------------------------------+
| 1990010100 0 0 24|
| 001 002 003 004 005 006 |
| 007 008 009 010 011 012 |
| 013 014 015 016 017 018 |
| 019 020 021 022 023 024 |
| 1990010101 0 0 24|
| 101 102 103 104 105 106 |
| 107 108 109 110 111 112 |
| 113 114 115 116 117 118 |
| 119 120 121 122 123 124 |
| 1990010102 0 0 24|
| 201 202 203 204 205 206 |
| 207 208 209 210 211 212 |
| 213 214 215 216 217 218 |
| 219 220 221 222 223 224 |
+--------------------------------------------+
And here is the complete code that I tested out. The main idea is to mark the date block and its records somehow so they can join with each other.
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql import types as T
from pyspark.sql import Window as W
# I'm not sure what cluster do you have, or you're going to run on your local machine, so I added this piece here for your reference. The most important thing is you would want to provide enough resources to handle entire dataset all at once.
spark = (SparkSession
.builder
.master('local[*]')
.appName('SO')
.config('spark.driver.cores', '4')
.config('spark.driver.memory', '4g')
.config('spark.executor.cores', '4')
.config('spark.executor.memory', '4g')
.getOrCreate()
)
# load raw weird data and apply some transformation to make it "readable" or "processable"
df = (spark
.read.text('t2.dat')
.withColumn('id', F.row_number().over(W.orderBy(F.lit(1)))) # making ID per row, very important to mark the dates
.withColumn('value', F.trim(F.col('value'))) # trim spaces before and after
.withColumn('value', F.split(F.col('value'), '\s+')) # turn single line to individual values
)
# +------------------------------+---+
# |value |id |
# +------------------------------+---+
# |[1990010100, 0, 0, 24] |1 |
# |[001, 002, 003, 004, 005, 006]|2 |
# |[007, 008, 009, 010, 011, 012]|3 |
# |[013, 014, 015, 016, 017, 018]|4 |
# |[019, 020, 021, 022, 023, 024]|5 |
# |[1990010101, 0, 0, 24] |6 |
# |[101, 102, 103, 104, 105, 106]|7 |
# |[107, 108, 109, 110, 111, 112]|8 |
# |[113, 114, 115, 116, 117, 118]|9 |
# |[119, 120, 121, 122, 123, 124]|10 |
# |[1990010102, 0, 0, 24] |11 |
# |[201, 202, 203, 204, 205, 206]|12 |
# |[207, 208, 209, 210, 211, 212]|13 |
# |[213, 214, 215, 216, 217, 218]|14 |
# |[219, 220, 221, 222, 223, 224]|15 |
# +------------------------------+---+
# Extracting available date blocks
date_df = (df
.where(F.size('value') == 4)
.withColumn('grp', ((F.col('id') - 1) / 5).cast('int')) # replace 5 with 16864 when run with your actual dataset
.select('grp', F.col('value')[0].alias('date'))
)
date_df.show(10, False)
# +---+----------+
# |grp|date |
# +---+----------+
# |0 |1990010100|
# |0 |1990010101|
# |0 |1990010102|
# +---+----------+
# Extracting available value blocks
value_df = (df
.where(F.size('value') == 6)
.withColumn('grp', ((F.col('id') - 1) / 5).cast('int')) # replace 5 with 16864 when run with your actual dataset
.groupBy('grp')
.agg(F.collect_list('value').alias('value'))
.withColumn('value', F.flatten('value'))
)
# +---+------------------------------------------------------------------------------------------------------------------------+
# |grp|value |
# +---+------------------------------------------------------------------------------------------------------------------------+
# |0 |[001, 002, 003, 004, 005, 006, 007, 008, 009, 010, 011, 012, 013, 014, 015, 016, 017, 018, 019, 020, 021, 022, 023, 024]|
# |1 |[101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124]|
# |2 |[201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224]|
# +---+------------------------------------------------------------------------------------------------------------------------+
# join them together and "explode" array to different rows
joined_df = (date_df
.join(value_df, on=['grp'])
.withColumn('value', F.explode('value'))
)
# +---+----------+-----+
# |grp|date |value|
# +---+----------+-----+
# |0 |1990010100|001 |
# |0 |1990010100|002 |
# |0 |... |... |
# |0 |1990010100|023 |
# |0 |1990010100|024 |
# |1 |1990010101|101 |
# |1 |1990010101|102 |
# |1 |... |... |
# |1 |1990010101|123 |
# |1 |1990010101|124 |
# |2 |1990010102|201 |
# |2 |1990010102|202 |
# |2 |... |... |
# |2 |1990010102|223 |
# |2 |1990010102|224 |
# +---+----------+-----+
# now, joined_df is basically holding your entire dataset, it's totally up to you how do you want to handle it.
# One option is you can save each date as a partition of one Hive table.
# Another option is saving each date as a file.
# It's just for the sake of simplicity when you'll ever need to read that painful dataset again.
for date in [row['date'] for row in date_df.collect()]:
(joined_df
.where(F.col('date') == date)
.write
.mode('overwrite')
.csv(date)
)

Answer 2

After your comment I have revised my answer, not that I have used 'pandas' and their dataframe, this should directly translate to spark if necessary.
Also I think your data is corrupt as the last array is not the correct length MY CODE DOES NOT HANDLE THIS so you will need to using the captured value from the regex expected_values = m.group(4) etc.
WARNING: The := operator requires Python 3.8... but you can fix that up if you need to
NOTES:
The header of each 'section' is captured by the regex and is used to form the column name
Split the file on the date row:
import pandas as pd
import numpy as np
import re
from pathlib import Path
header = re.compile(r"^\s+(\d{10})\s+(\d*)\s+(\d*)\s+(\d*)$")
df = pd.DataFrame()
with open("t2.dat", "r") as ifp:
rows = []
date = None
count = 0
while line := ifp.readline():
# Get the header and start a new file
if m := header.match(line):
# We have a header so convert to array then flatten to a vector
# before appending to the dataframe.
if rows and date:
df[date] = np.array(rows, dtype=float).flatten(order="C")
rows = []
# Get the header
date = m.group(1)
else:
rows.append(line.strip().split())
print(f"Appending the last {len(rows)*len(rows[0])} values")
df[date] = np.array(rows, dtype=float).flatten(order="C")
Output an abbreviated form (there is 1 column per date with 101178 rows:
1990010100 1990010101 1990010102 1990010103 1990010104
0 10.4310 10.0490 9.7269 9.3801 9.0038
1 10.3110 9.9225 9.5431 9.1758 8.7899
2 10.2290 9.8144 9.4156 9.0304 8.6171
3 10.1500 9.7154 9.2999 8.8890 8.4713
4 9.8586 9.3968 8.9156 8.4328 7.9764
... ... ... ... ... ...
101173 -1.5511 -1.5472 -1.5433 -1.5251 -1.5399
101174 -1.8659 -1.8719 -1.8485 -1.8481 -1.8325
101175 -1.9044 -1.8597 -1.7963 -1.8094 -1.7653
101176 -2.0564 -2.0404 -1.9779 -1.9893 -1.9521
101177 -2.1842 -2.2840 -2.3216 -2.2794 -2.2655