Unable to connect to redshift from spark - scala

Question

I am trying to read data from redshift to spark 1.5 using scala 2.10
I have built the spark-redshift package and added the amazon JDBC connector to the project, but I keep getting this error:
Exception in thread "main" java.lang.NoClassDefFoundError: com/amazonaws/auth/AWSCredentials
I have authenticated in the following way:
val hadoopConf = sc.hadoopConfiguration
hadoopConf.set("fs.s3n.impl","org.apache.hadoop.fs.s3native.NativeS3FileSystem")
hadoopConf.set("fs.s3n.awsAccessKeyId", "ACCESSKEY")
hadoopConf.set("fs.s3n.awsSecretAccessKey","SECRETACCESSKEY")
val df: DataFrame = sqlContext.read.format("com.databricks.spark.redshift")
.option("url","jdbc:redshift://AWS_SERVER:5439/warehouseuser=USER&password=PWD")
.option("dbtable", "fact_time")
.option("tempdir", "s3n://bucket/path")
.load()
df.show()

Answer 1

Concerning your first error java.lang.NoClassDefFoundError: com/amazonaws/auth/AWSCredentials I repeat what I said in the comment :
You have forgot to ship your AWS dependency jar in your spark app jar
And about the second error, I'm not sure of the package but it's more likely to be the org.apache.httpcomponents library you need. (I don't know for what you are using it thought!)
You can add the following to your maven dependencies :
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>4.4.3</version>
</dependency>
and you'll need to assembly the whole.
PS: You'll always need to provide the libraries when they are not installed. You must also becareful with the size of the jar you are submitting because it can harm performance.