Write to s3 bucket directly using Spark/Scala - scala

Question

I am trying to write parquet file in the s3 bucket in Spark/scala program but not sure what's the best way to do it. After a bit of research, I found below option :
Add below dependency in the code :
"com.amazonaws" % "aws-java-sdk" % "1.11.690",
"org.apache.hadoop" % "hadoop-aws" % "2.6.0"
then, set the required config in the code :
spark.sparkContext.hadoopConfiguration.set("fs.s3a.access.key", "access key")
// Replace Key with your AWS secret key (You can find this on IAM
spark.sparkContext.hadoopConfiguration.set("fs.s3a.secret.key", "secrete key")
spark.sparkContext.hadoopConfiguration.set("fs.s3a.endpoint", "end point", "us-east-1")
But, I am getting below error :
Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID...
I am inclined to go with above option as this would give me flexibility to directly write the file to S3:
jdbcWriteDF.write.format("parquet")
.mode(SaveMode.Overwrite)
.save("s3a://cs-dev/cmc_site_qa.parquet")
With the AWSClient option, I need to generate it first on my local and then upload it on the S3 bucket.
I know the keys are correct as when I was using in one of my jave process, it was working fine..
AmazonS3 s3client = AmazonS3ClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(s3BucketAccessKey, s3BucketSecretKey)))
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(s3BucketEndpoint, "us-east-1"))
.build();
return s3client;