spark OOM on simple read and write - postgresql

Question

I'm using spark to read from a postgres table and dump it to Google cloud storage as json. The table is quite big, many 100's of GBs. The code is relatively straightforward (plz see below) but it fails with OOM. It seems like spark is trying to read the entire table in memory before starting to write it. Is this true? How can I change the behavior such that it reads and writes in a streaming fashion?
Thanks.
SparkSession sparkSession = SparkSession
.builder()
.appName("01-Getting-Started")
.getOrCreate();
Dataset<Row> dataset = sparkSession.read().jdbc("jdbc:postgresql://<ip>:<port>/<db>", "<table>", properties);
dataset.write().mode(SaveMode.Append).json("gs://some/path");

Answer 1

There are a couple of overloaded DataFrameReader.jdbc() methods that are useful for splitting up JDBC data on input.
jdbc(String url, String table, String[] predicates, java.util.Properties connectionProperties) - the resulting DataFrame will have one partition for each predicate given, e.g.
String[] preds = {“state=‘Alabama’”, “state=‘Alaska’”, “state=‘Arkansas’”, …};
Dataset<Row> dataset = sparkSession.read().jdbc("jdbc:postgresql://<ip>:<port>/<db>", "<table>", preds, properties);
jdbc(String url, String table, String columnName, long lowerBound, long upperBound, int numPartitions, java.util.Properties connectionProperties) - Spark will divide the data based on a numeric column columnName into numPartitions partitions between lowerBound and upperBound inclusive, e.g.:
Dataset<Row> dataset = sparkSession.read().jdbc("jdbc:postgresql://<ip>:<port>/<db>", "<table>", “<idColumn>”, 1, 1000, 100, properties);