PCA() got an unexpected keyword argument 'k' - pyspark

Question

I am trying t perform pca from a spark application using PySpark API on a python script. I doing This way:
pca = PCA(k=3, inputCol="features", outputCol="pcaFeatures")
PCAmodel = pca.fit(data)
when I run those two code line in the pyspark shell it work and return good results, but in an application script, I am getting the type of error:
PCA() got an unexpected keyword argument 'k'
PS: In both case I am using Spark 2.2.0
where is the problem? why it does work in the PySpark shell and not for the application?

Answer 1

You probably imported from ml in one case:
from pyspark.ml.feature import PCA
and mllib in the other:
from pyspark.mllib.feature import PCA

Answer 2

Are you sure you have not also imported PCA from scikit-learn, after you imported it from PySpark in your application script?
spark.version
# u'2.2.0'
from pyspark.ml.feature import PCA
from sklearn.decomposition import PCA
# PySpark syntax with scikit-learn PCA function
pca = PCA(k=3, inputCol="features", outputCol="pcaFeatures")
# Error:
TypeError: __init__() got an unexpected keyword argument 'k'
Reversing the order of imports will not produce the error (not shown).

Answer 3

Try renaming your classes:
from pyspark.ml.feature import PCA as PCAML
from sklearn.decomposition import PCA as PCASK
pca_ml = PCAML(k=3, inputCol="features", outputCol="pcaFeatures")
There should be no confusion, then, which one you call.