Converting Postgres Function to Impala UDF or a function in Spark - postgresql

Question

I have a postgres function that is called in a query. Its similar to this sample:
CREATE OR REPLACE FUNCTION test_function(id integer, dt date, days int[], accts text[], flag boolean) RETURNS float[] AS $$
DECLARE
pt_dates date[];
pt_amt integer[];
amt float[];
BEGIN
if cleared then
pt_dates := array(select dt from tabl);
pt_amt := array(select amt from tab1);
if array_upper(days, 1) is not null then
for j in 1 .. array_upper(days, 1)
loop
amt+=amt;
end loop;
end if;
return amt;
END;
$$ LANGUAGE plpgsql;
If I wish to convert this to in to the Data Lake Environment, which is the best way to do it? Impala UDF? or Spark UDF ? or Hive UDF? In Impala UDF, how do I access the impala database? if I write Spark UDF can I use it in the impala-shell?
Please advise.

Answer 1

There's a lot lot lot of questions in your 1 post. So i'll choose just the Spark related question.
You have this SQL query that represents the data processing you wish to perform.
Here is a general formula to do this with Spark:
Take some amount of data, move it to S3
Go into AWS EMR and create a new cluster
SSH into the master node, and run pyspark console
once it has started, you can read in your S3 data via rdd = sc.readText("s3://path/to/your/s3/buckets/")
apply a schema to it with a map function rdd2 = rdd.map(..add schema..)
convert that rdd2 into a dataframe and store that as a new var. rdd2DF = rdd2.toDF()
perform a rdd2DF.registerTempTable('newTableName') on that
write a SQL query and store the result: output = sqlContext.sql("SELECT a,b,c FROM newTableName")
show the output: output.show()
Now i know this is literally too high level to be a specific answer to your question, but everything i just said is very google'able.
And this is an example of a separated Compute and Storage scenario leveraging EMR with Spark and SparkSQL to process a lot of data with SQL queries.