I'm trying to setup an Alpakka S3 for files upload purpose. Here is my configs:
alpakka s3 dependency:
...
"com.lightbend.akka" %% "akka-stream-alpakka-s3" % "0.20"
...
Here is application.conf:
akka.stream.alpakka.s3 {
buffer = "memory"
proxy {
host = ""
port = 8000
secure = true
}
aws {
credentials {
provider = default
}
}
path-style-access = false
list-bucket-api-version = 2
}
File upload code example:
private val awsCredentials = new BasicAWSCredentials("my_key", "my_secret_key")
private val awsCredentialsProvider = new AWSStaticCredentialsProvider(awsCredentials)
private val regionProvider = new AwsRegionProvider { def getRegion: String = "us-east-1" }
private val settings = new S3Settings(MemoryBufferType, None, awsCredentialsProvider, regionProvider, false, None, ListBucketVersion2)
private val s3Client = new S3Client(settings)(system, materializer)
val fileSource = Source.fromFuture(ByteString("ololo blabla bla"))
val fileName = UUID.randomUUID().toString
val s3Sink: Sink[ByteString, Future[MultipartUploadResult]] = s3Client.multipartUpload("my_basket", fileName)
fileSource.runWith(s3Sink)
.map {
result => println(s"${result.location}")
} recover {
case ex: Exception => println(s"$ex")
}
When I run this code I get:
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
What can be a reason?
The certificate problem arises for bucket names containing dots.
You may switch to
akka.stream.alpakka.s3.path-style-access = true to get rid of this.
We're considering making it the default: https://github.com/akka/alpakka/issues/1152
Related
I am trying to push records to AWS S3 with Aplakka S3 library. The issue is due to security issues, I have to "assume role", and my IAM user doesn't have access to PUT. with aws cli, I could have successfully pushed to s3 with --profile parameter in aws s3 --profile <profile>. I want to know HOW TO ASSUME ROLE IN ALPAKKA S3 LIBRARY?
my application.conf file has the credentials as in https://github.com/akka/alpakka/blob/v3.0.4/s3/src/main/resources/reference.conf
:
alpakka.s3 {
# default values for AWS configuration
aws {
# to use the same configuration as if credentials.provider = default
credentials {
# static credentials
provider = default //static
access-key-id = <> //valid access key exists in original code
secret-access-key = <>
}
and my PUTing code is:
object Experiments extends App{
//AWS S3 configs
val s3BucketName = config.getString("alpakka.s3.bucket_name")
val s3BucketRegion = config.getString("alpakka.s3.bucket_region")
val bucket_path = config.getString("alpakka.s3.path_inside_bucket")
Source.single(record)
.map { Record => println("Record value: " + Record)
val K_record = //some parsing for the record into JSON
//push to S3
val bucketKey = s"$bucket_path/${K_record.session_id}.json"
try{
Source.single(ByteString(K_record.featureSet))
.runWith(S3.multipartUpload(bucket = s3BucketName, key = bucketKey))
}
catch{
case e2:Exception => println(s"S3 pushing error: ${e2.getMessage}")
}
}
.runWith(Sink.ignore)
I followed this document: https://docs.aws.amazon.com/dynamodb-encryption-client/latest/devguide/java-examples.html and setup encryption client and mapper to encrypt an items and batchsave into Table but it is not working and throwing errors as given below
stack Trace details:
ERROR Client: Application diagnostics message: User class threw exception: java.lang.IllegalAccessError: tried to access class com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMappingsRegistry from class com.amazonaws.services.dynamodbv2.datamodeling.AttributeEncryptor at com.amazonaws.services.dynamodbv2.datamodeling.AttributeEncryptor.getModelClassMetadata(AttributeEncryptor.java:156) at com.amazonaws.services.dynamodbv2.datamodeling.AttributeEncryptor.transform(AttributeEncryptor.java:65) at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.transformAttributes(DynamoDBMapper.java:2180) at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.batchWrite(DynamoDBMapper.java:1229) at com.amazonaws.services.dynamodbv2.datamodeling.AbstractDynamoDBMapper.batchSave(AbstractDynamoDBMapper.java:193) at com.amazon.payrolldatalakeemr.awsoperations.DDBOperations$.batchSaveInDDB(DDBOperations.scala:40)
Config details:
AWSJavaSDKExternalRelease = 1.11.x;
# Spark dependencies
Spark-core = 2.2.1;
Spark-sql = 2.2.1;
DaxJavaClient = 1.0;
ANTLR-Runtime = 3.5.x;
DynamoDbGrammar = 1.0;
Lombok = 1.16.x;
LombokUtils = 1.1;
Maven-com-amazonaws_aws-dynamodb-encryption-java = 1.x;
Mapper code:
def getDynamoDBMapper(region: String): DynamoDBMapper = {
val cmkArn = "*****************************"
val kms: AWSKMS = AWSKMSClientBuilder.standard().withRegion(region).build()
val cmp: DirectKmsMaterialProvider = new DirectKmsMaterialProvider(kms, cmkArn)
val encryptor: DynamoDBEncryptor = DynamoDBEncryptor.getInstance(cmp)
val mapperConfig: DynamoDBMapperConfig = DynamoDBMapperConfig.builder.withSaveBehavior(DynamoDBMapperConfig.SaveBehavior.CLOBBER).build
new DynamoDBMapper(ddclient, mapperConfig, new AttributeEncryptor(encryptor))
}
Resolved after adding spark property:
--conf spark.driver.userClassPathFirst=true
Using these parameters:
canada {
hosts = ["dd.weather.gc.ca"]
username = "anonymous"
password = "anonymous"
port = 5671
exchange = "xpublic"
queue = "q_anonymous_gsk"
routingKey = "v02.post.observations.swob-ml.#"
requestedHeartbeat = 300
ssl = true
}
I can connect to a weather service in Canada using NewMotion/Akka, but when I try op-rabbit, I get:
ACCESS_REFUSED - access to exchange 'xpublic' in vhost '/' refused for user 'anonymous'
[INFO] [foo-akka.actor.default-dispatcher-7] [akka://foo/user/$a/connection] akka://foo/user/$a/connection connected to amqp://anonymous#{dd.weather.gc.ca:5671}:5671//
[INFO] [foo-op-rabbit.default-channel-dispatcher-6] [akka://foo/user/$a/connection/$a] akka://foo/user/$a/connection/$a connected
[INFO] [foo-akka.actor.default-dispatcher-4] [akka://foo/user/$a/connection/confirmed-publisher-channel] akka://foo/user/$a/connection/confirmed-publisher-channel connected
[INFO] [foo-akka.actor.default-dispatcher-4] [akka://foo/user/$a/connection/$b] akka://foo/user/$a/connection/$b connected
[ERROR] [foo-akka.actor.default-dispatcher-3] [akka://foo/user/$a/subscription-q_anonymous_gsk-1] Connection related error while trying to re-bind a consumer to q_anonymous_gsk. Waiting in anticipating of a new channel.
...
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - access to exchange 'xpublic' in vhost '/' refused for user 'anonymous', class-id=40, method-id=10)
The following works in NewMotion/Akka:
val inQueue = "q_anonymous_gsk"
val inExchange = "xpublic"
val canadaQueue = canadaChannel.queueDeclare(inQueue, false, true, false, null).getQueue
canadaChannel.queueBind(canadaQueue, inExchange, inQueue)
val consumer = new DefaultConsumer(canadaChannel) {
override def handleDelivery(consumerTag: String, envelope: Envelope, properties: BasicProperties, body: Array[Byte]) {
val s = fromBytes(body)
if (republishElsewhere) {
// ...
}
}
}
canadaChannel.basicConsume(canadaQueue, true, consumer)
but using op-rabbit like this:
val inQueue = "q_anonymous_gsk"
val inExchange = "xpublic"
val inRoutingKey = "v02.post.observations.swob-ml.#""
val rabbitCanada: ActorRef = actorSystem.actorOf(Props(classOf[RabbitControl], connParamsCanada))
def runSubscription(): SubscriptionRef = Subscription.run(rabbitCanada) {
channel(qos = 3) {
consume(topic(queue(inQueue), List(inRoutingKey))) {
(body(as[String]) & routingKey) { (msg, key) =>
ack
}
}
}
}
I get the ACCESS_REFUSED error near the top of this post. Why? How do I fix this if I want to use op-rabbit?
Have you tried to use the correct vhost with permission to anonymous user
My question is how to use a Buffered stream when using Scalaj-Http.
I have written the following code which is a complete working example that will download a file from Hadoop HDFS using HttpFS. My goal is to handle very large files and this will require using a buffered approach with multiple I/O writes to a local file.
I have not been able to find documentation on how to use a stream with the ScalaJ-Http interface. I am interested in an example for both download and upload that can handle large multi GB files. My code below uses in memory buffering which is appropriate for only prototyping.
import scalaj.http._
import ujson.Js
import java.text.SimpleDateFormat
import java.net.SocketTimeoutException
import java.io.InputStream
import java.io.BufferedOutputStream
import java.io.FileOutputStream
import java.io.FileNotFoundException
object CopyFileFromHdfs {
def main(args: Array[String]) {
val host = "hadoop.example.com"
val user = "root"
var dstFile = ""
var srcFile = ""
val operation = "OPEN"
val port = 14000
System.setProperty("sun.net.http.allowRestrictedHeaders", "true")
if (args.length != 2)
{
println("Error: Missing or too many arguments")
println("Usage: CopyFileFromHdfs <srcfile> <dstfile>")
System.exit(1)
}
srcFile = args(0)
dstFile = args(1)
// ********************************************************************************
// Create the URL string that we will use to connect to Hadoop HttpFS
//
// The string will look like this:
// http://root#123.456.789.012:14000/webhdfs/v1/?user.name=root&op=OPEN
// ********************************************************************************
val url = makeHttpfsUrl(host, user, srcFile, operation, port)
// ********************************************************************************
// Using HTTP, call the HttpFS server
//
// Exceptions:
// java.net.SocketTimeoutException
// java.net.UnknownHostException
// java.lang.IllegalArgumentException
// Remote Exceptions:
// java.io.FileNotFoundException
// com.sun.jersey.api.NotFoundException
// ********************************************************************************
try {
var response = Http(url)
.timeout(connTimeoutMs = 1000, readTimeoutMs = 5000)
.asBytes
// ********************************************************************************
// Check for an error. We are expecting an HTTP 200 response
// ********************************************************************************
if (response.code < 200 || response.code > 299)
{
val data = ujson.read(response.body)
printf("Error: Cannot download file: %s\n", dstFile)
println(removeQuotes(data("RemoteException")("message").str))
println(removeQuotes(data("RemoteException")("exception").str))
System.exit(1)
}
val is = new FileOutputStream(dstFile)
val bs = new BufferedOutputStream(is)
bs.write(response.body, 0, response.body.length)
bs.close()
is.close()
} catch {
case e: SocketTimeoutException => {
printf("Error: Cannot connect to host %s on port %d\n", host, port)
println(e)
System.exit(1);
}
case e: Exception => {
printf("Error (other): Cannot download file %s\n", srcFile)
println(e)
System.exit(1);
}
}
printf("Success: File downloaded. %s -> %s\n", srcFile, dstFile)
System.exit(0)
}
// ********************************************************************************
// The Json strings are surrounded by quotes.
// This function will remove them (only at the start and the end).
// ********************************************************************************
def removeQuotes(str: String): String = {
// This expression will delete quotes at the beginning and end of a string
return str.replaceAll("^\"|\"$", "");
}
// ********************************************************************************
// Create the URL string that we will use to connect to Hadoop HttpFS
//
// The string will look like this:
// http://root#123.456.789.012:14000/webhdfs/v1/?user.name=root&op=LISTSTATUS
// ********************************************************************************
def makeHttpfsUrl(
host: String,
user: String,
hdfsPath: String,
operation: String,
port: Integer) : String = {
var url = "http://" + user + "#" + host + ":" + port.toString + "/webhdfs/v1"
if (hdfsPath(0) == '/')
url += hdfsPath
else
url += "/" + hdfsPath
url += "?user.name=" + user + "&op=" + operation
return url
}
}
I need to be able to send a certificate file (.pem, I think), with a get request using scala and dispatch.
How do you do that?
Based on the Java code in #sbridges sample, I came up with the following Scala code using dispatch. It creates a custom SSL context containing the certificates you provide (and only those; the default store of trusted root certificates is not used by this code when verifying the remote host).
class SslAuthenticatingHttp(certData: SslCertificateData) extends Http {
override val client = new AsyncHttpClient(
(new AsyncHttpClientConfig.Builder).setSSLContext(buildSslContext(certData)).build
)
private def buildSslContext(certData: SslCertificateData): SSLContext = {
import certData._
val clientCertStore = loadKeyStore(clientCertificateData, clientCertificatePassword)
val rootCertStore = loadKeyStore(rootCertificateData, rootCertificatePassword)
val keyManagerFactory = KeyManagerFactory.getInstance("SunX509")
keyManagerFactory.init(clientCertStore, clientCertificatePassword.toCharArray)
val keyManagers = keyManagerFactory.getKeyManagers()
val trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm())
trustManagerFactory.init(rootCertStore)
val trustManagers = trustManagerFactory.getTrustManagers()
val context = SSLContext.getInstance("TLS")
context.init(keyManagers, trustManagers, null)
context
}
private def loadKeyStore(keyStoreData: Array[Byte], password: String): KeyStore = {
val store = KeyStore.getInstance(KeyStore.getDefaultType)
store.load(new ByteArrayInputStream(keyStoreData), password.toCharArray)
store
}
}
case class SslCertificateData (
clientCertificateData: Array[Byte],
clientCertificatePassword: String,
rootCertificateData: Array[Byte],
rootCertificatePassword: String)
which would be used as in:
val certificateData = SslCertificateData(/* bytes from .jks file for client cert here */, "secret!",
/* bytes from .jks file for root cert here */, "also secret!")
val http = new SslAuthenticatingHttp(certificateData)
val page = http(req OK as.String)
println(page())
Note that this keeps the certificate data in memory, which is not the most secure way to do it and consumes memory unnecessarily. It may in many cases be more suitable to store an InputStream or a filename in the SslCertificateData case class.
I am assuming you want to do https with client certificates. I think this needs to be set up at the jvm level, there is a good explanation here how to do it.
There seems to be a way to do this with ning directly, as explained here,
the code is copied below,
// read in PEM file and parse with commons-ssl PKCS8Key
// (ca.juliusdavies:not-yet-commons-ssl:0.3.11)
RandomAccessFile in = null;
byte[] b = new byte[(int) certFile.length()];
in = new RandomAccessFile( certFile, "r" );
in.readFully( b );
char[] password = hints.get( "password" ).toString().toCharArray();
PKCS8Key key = new PKCS8Key( b, password );
// create empty key store
store = KeyStore.getInstance( KeyStore.getDefaultType() );
store.load( null, password );
// cert chain is not important if you override the default KeyManager and/or
// TrustManager implementation, IIRC
store.setKeyEntry( alias, key.getPrivateKey(), password, new DefaultCertificate[0] );
// initialize key and trust managers -> default behavior
KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance( "SunX509" );
// password for key and store have to be the same IIRC
keyManagerFactory.init( store, password );
KeyManager[] keyManagers = keyManagerFactory.getKeyManagers();
TrustManagerFactory tmf = TrustManagerFactory.getInstance( TrustManagerFactory.getDefaultAlgorithm() );
tmf.init( store );
TrustManager[] trustManagers = tmf.getTrustManagers();
// override key and trust managers with desired behavior - for example
// * 'trust everything the server gives us' -> TrustManager#checkServerTrusted
// * 'always return a preset alias to use for auth' -> X509ExtendedKeyManager#chooseClientAlias, X509ExtendedKeyManager#chooseEngineClientAlias
for ( int i = 0; i < keyManagers.length; i++ )
{
if ( keyManagers[i] instanceof X509ExtendedKeyManager )
{
AHCKeyManager ahcKeyManager = new AHCKeyManager( (X509ExtendedKeyManager) keyManagers[i] );
keyManagers[i] = ahcKeyManager;
}
}
for ( int i = 0; i < trustManagers.length; i++ )
{
if ( tm instanceof X509TrustManager )
{
AHCTrustManager ahcTrustManager = new AHCTrustManager( manager, (X509TrustManager) trustManagers[i] );
trustManagers[i] = ahcTrustManager;
}
}
// construct SSLContext and feed to AHC config
SSLContext context = SSLContext.getInstance( "TLS" );
context.init( keyManagers, trustManagers, null );
ahcCfgBuilder.setSSLContext(context);