Why is linear regression wrong for pyspark? - pyspark

Question

I kept getting wrong answers so I tried it on something very, very basic, and it was still wrong.
input file:
1 1:1
2 1:2
3 1:3
4 1:4
from pyspark.ml.regression import LinearRegression
# Load training data
training = spark.read.format("libsvm").load("stupid.txt")
lr = LinearRegression(maxIter=100, regParam=0.3, loss='squaredError')
# Fit the model
lrModel = lr.fit(training)
# Print the coefficients and intercept for linear regression
print("Coefficients: %s" % str(lrModel.coefficients))
print("Intercept: %s" % str(lrModel.intercept))
# Summarize the model over the training set and print out some metrics
trainingSummary = lrModel.summary
print("numIterations: %d" % trainingSummary.totalIterations)
print("objectiveHistory: %s" % str(trainingSummary.objectiveHistory))
trainingSummary.residuals.show()
print("RMSE: %f" % trainingSummary.rootMeanSquaredError)
print("r2: %f" % trainingSummary.r2)
Should have gotten coefficients [1] and intercept 0.
Instead got
Coefficients: [0.7884394856681294]
Intercept: 0.52890128583

Answer 1

It looks like the issue is the regParam parameter that you’re using. If I run it with that set to 0, which causes normal OLS to take place, we get the expected output:
Code:
from pyspark.ml.regression import LinearRegression
from pyspark.ml.linalg import Vectors
training = spark.createDataFrame([
(1.0, Vectors.dense(1.0)),
(2.0, Vectors.dense(2.0)),
(3.0, Vectors.dense(3.0)),
(4.0, Vectors.dense(4.0))], ["label", "features"])
lr = LinearRegression(maxIter=100, regParam=0, loss='squaredError')
# Fit the model
lrModel = lr.fit(training)
# Print the coefficients and intercept for linear regression
print("Coefficients: %s" % str(lrModel.coefficients))
print("Intercept: %s" % str(lrModel.intercept))
# Summarize the model over the training set and print out some metrics
trainingSummary = lrModel.summary
print("numIterations: %d" % trainingSummary.totalIterations)
print("objectiveHistory: %s" % str(trainingSummary.objectiveHistory))
trainingSummary.residuals.show()
print("RMSE: %f" % trainingSummary.rootMeanSquaredError)
print("r2: %f" % trainingSummary.r2)
Output:
Coefficients: [1.0]
Intercept: 0.0
numIterations: 1
objectiveHistory: [0.0]
+---------+
|residuals|
+---------+
| 0.0|
| 0.0|
| 0.0|
| 0.0|
+---------+
RMSE: 0.000000
r2: 1.000000
It seems like the regParam > 0 is being used as an L2 regularization term, and preventing the model from performing a normal OLS process.