I followed this document: https://docs.aws.amazon.com/dynamodb-encryption-client/latest/devguide/java-examples.html and setup encryption client and mapper to encrypt an items and batchsave into Table but it is not working and throwing errors as given below
stack Trace details:
ERROR Client: Application diagnostics message: User class threw exception: java.lang.IllegalAccessError: tried to access class com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMappingsRegistry from class com.amazonaws.services.dynamodbv2.datamodeling.AttributeEncryptor at com.amazonaws.services.dynamodbv2.datamodeling.AttributeEncryptor.getModelClassMetadata(AttributeEncryptor.java:156) at com.amazonaws.services.dynamodbv2.datamodeling.AttributeEncryptor.transform(AttributeEncryptor.java:65) at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.transformAttributes(DynamoDBMapper.java:2180) at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.batchWrite(DynamoDBMapper.java:1229) at com.amazonaws.services.dynamodbv2.datamodeling.AbstractDynamoDBMapper.batchSave(AbstractDynamoDBMapper.java:193) at com.amazon.payrolldatalakeemr.awsoperations.DDBOperations$.batchSaveInDDB(DDBOperations.scala:40)
Config details:
AWSJavaSDKExternalRelease = 1.11.x;
# Spark dependencies
Spark-core = 2.2.1;
Spark-sql = 2.2.1;
DaxJavaClient = 1.0;
ANTLR-Runtime = 3.5.x;
DynamoDbGrammar = 1.0;
Lombok = 1.16.x;
LombokUtils = 1.1;
Maven-com-amazonaws_aws-dynamodb-encryption-java = 1.x;
Mapper code:
def getDynamoDBMapper(region: String): DynamoDBMapper = {
val cmkArn = "*****************************"
val kms: AWSKMS = AWSKMSClientBuilder.standard().withRegion(region).build()
val cmp: DirectKmsMaterialProvider = new DirectKmsMaterialProvider(kms, cmkArn)
val encryptor: DynamoDBEncryptor = DynamoDBEncryptor.getInstance(cmp)
val mapperConfig: DynamoDBMapperConfig = DynamoDBMapperConfig.builder.withSaveBehavior(DynamoDBMapperConfig.SaveBehavior.CLOBBER).build
new DynamoDBMapper(ddclient, mapperConfig, new AttributeEncryptor(encryptor))
}
Resolved after adding spark property:
--conf spark.driver.userClassPathFirst=true
I'm trying to create a simple akka system in Scala Worksheet but each time I try, I get this error.
com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'akka'
at com.typesafe.config.impl.SimpleConfig.findKeyOrNull(tsets.sc:148)
at com.typesafe.config.impl.SimpleConfig.findKey(tsets.sc:141)
at com.typesafe.config.impl.SimpleConfig.findOrNull(tsets.sc:168)
at com.typesafe.config.impl.SimpleConfig.find(tsets.sc:180)
at com.typesafe.config.impl.SimpleConfig.find(tsets.sc:185)
at com.typesafe.config.impl.SimpleConfig.getString(tsets.sc:242)
at akka.actor.ActorSystem$Settings.<init>(tsets.sc:166)
at akka.actor.ActorSystemImpl.<init>(tsets.sc:533)
at akka.actor.ActorSystem$.apply(tsets.sc:139)
at akka.actor.ActorSystem$.apply(tsets.sc:116)
at #worksheet#.system$lzycompute(tsets.sc:9)
at #worksheet#.system(tsets.sc:9)
at #worksheet#.get$$instance$$system(tsets.sc:9)
at A$A9$.main(tsets.sc:35)
at #worksheet#.#worksheet#(tsets.sc)
Even though the Scala worksheet is in the exact same directory as the application.conf file.
The akka entry for the file looks something like this.
akka {
loglevel = ??
actor {
provider = ??
}
remote {
log-remote-lifecycle-events = ??
enabled-transports =??
netty.tcp {
hostname = ??
port = ??
maximum-frame-size = ??
maximum-frame-size = ??
send-buffer-size = ??
send-buffer-size = ??
receive-buffer-size = ??
receive-buffer-size = ??
}
}
cluster {
roles = ["frontend"]
seed-nodes = ??
use-dispatcher = c??
failure-detector {
threshold = ??
acceptable-heartbeat-pause = ??
heartbeat-interval = ??
heartbeat-request {
expected-response-after = ??
}
}
}
}
I have even tried adding this to the build.sbt
assemblyMergeStrategy in assembly := {
case PathList("application.conf") => MergeStrategy.concat
}
One of the possible variants is calling ConfigFactory.parseFile with resolve (if you have some substitutions like receive-buffer-size = ${send-buffer-size} in your config file):
val confPath = getClass.getResource("/application.conf")
val config = ConfigFactory.parseFile(new File(confPath.getPath)).resolve()
config.getString("akka.loglevel")
Your application.conf file should be in resources folder and your worksheet can be placed in scala folder.
Update
If you use Intellij IDEA. There is a workaround for this
Go to File > Settings > Languages & Frameworks > Scala > Worksheet
Untick "Run config in the compiler process" and press "OK"
So you can use ConfigFactory.load() freely
I used IDEA with version 2018.1.5
I'm trying to connect to IBM Cloud Object Storage from IBM Data Science Experience:
access_key = 'XXX'
secret_key = 'XXX'
bucket = 'mybucket'
host = 'lon.ibmselect.objstor.com'
service = 'mycos'
sqlCxt = SQLContext(sc)
hconf = sc._jsc.hadoopConfiguration()
hconf.set('fs.cos.myCos.access.key', access_key)
hconf.set('fs.cos.myCos.endpoint', 'http://' + host)
hconf.set('fs.cose.myCos.secret.key', secret_key)
hconf.set('fs.cos.service.v2.signer.type', 'false')
obj = 'mydata.tsv.gz'
rdd = sc.textFile('cos://{0}.{1}/{2}'.format(bucket, service, obj))
print(rdd.count())
This returns:
Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: java.io.IOException: No FileSystem for scheme: cos
I'm guessing I need to use the 'cos' scheme based on the stocator docs. However, the error suggests stocator isn't available or is an old version?
Any ideas?
Update 1:
I have also tried the following:
sqlCxt = SQLContext(sc)
hconf = sc._jsc.hadoopConfiguration()
hconf.set('fs.cos.impl', 'com.ibm.stocator.fs.ObjectStoreFileSystem')
hconf.set('fs.stocator.scheme.list', 'cos')
hconf.set('fs.stocator.cos.impl', 'com.ibm.stocator.fs.cos.COSAPIClient')
hconf.set('fs.stocator.cos.scheme', 'cos')
hconf.set('fs.cos.mycos.access.key', access_key)
hconf.set('fs.cos.mycos.endpoint', 'http://' + host)
hconf.set('fs.cos.mycos.secret.key', secret_key)
hconf.set('fs.cos.service.v2.signer.type', 'false')
service = 'mycos'
obj = 'mydata.tsv.gz'
rdd = sc.textFile('cos://{0}.{1}/{2}'.format(bucket, service, obj))
print(rdd.count())
However, this time the response was:
Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: java.io.IOException: No object store for: cos
at com.ibm.stocator.fs.ObjectStoreVisitor.getStoreClient(ObjectStoreVisitor.java:121)
...
Caused by: java.lang.ClassNotFoundException: com.ibm.stocator.fs.cos.COSAPIClient
The latest version of Stocator (v1.0.9) that supports fs.cos scheme is not yet deployed on Spark aaService (It will be soon). Please use the stocator scheme "fs.s3d" to connect to your COS.
Example:
endpoint = 'endpointXXX'
access_key = 'XXX'
secret_key = 'XXX'
prefix = "fs.s3d.service"
hconf = sc._jsc.hadoopConfiguration()
hconf.set(prefix + ".endpoint", endpoint)
hconf.set(prefix + ".access.key", access_key)
hconf.set(prefix + ".secret.key", secret_key)
bucket = 'mybucket'
obj = 'mydata.tsv.gz'
rdd = sc.textFile('s3d://{0}.service/{1}'.format(bucket, obj))
rdd.count()
Alternatively, you can use ibmos2spark. The lib is already installed on our service. Example:
import ibmos2spark
credentials = {
'endpoint': 'endpointXXXX',
'access_key': 'XXXX',
'secret_key': 'XXXX'
}
configuration_name = 'os_configs' # any string you want
cos = ibmos2spark.CloudObjectStorage(sc, credentials, configuration_name)
bucket = 'mybucket'
obj = 'mydata.tsv.gz'
rdd = sc.textFile(cos.url(obj, bucket))
rdd.count()
Stocator is on the classpath for Spark 2.0 and 2.1 kernels, but the cos scheme is not configured. You can access the config by executing the following in a Python notebook:
!cat $SPARK_CONF_DIR/core-site.xml
Look for the property fs.stocator.scheme.list. What I currently see is:
<property>
<name>fs.stocator.scheme.list</name>
<value>swift2d,swift,s3d</value>
</property>
I recommend that you raise a feature request against DSX to support the cos scheme.
It looks like cos driver is not properly initialized. Try this configuration:
hconf.set('fs.cos.impl', 'com.ibm.stocator.fs.ObjectStoreFileSystem')
hconf.set('fs.stocator.scheme.list', 'cos')
hconf.set('fs.stocator.cos.impl', 'com.ibm.stocator.fs.cos.COSAPIClient')
hconf.set('fs.stocator.cos.scheme', 'cos')
hconf.set('fs.cos.mycos.access.key', access_key)
hconf.set('fs.cos.mycos.endpoint', 'http://' + host)
hconf.set('fs.cos.mycos.secret.key', secret_key)
hconf.set('fs.cos.service.v2.signer.type', 'false')
UPDATE 1:
You also need to ensure stocator classes are on the classpath. You can use packages system by exceuting pyspark in the following way:
./bin/pyspark --packages com.ibm.stocator:stocator:1.0.24
This works with swift2d and cos scheme.
UPDATE 2:
Just follow Stocator documentation (https://github.com/CODAIT/stocator). It contains all details how to install it, what branch to use, etc.
I found the same issue, and to solve it I just changed environment:
Within IBM Watson Studio, if you start a a Jupyter notebook in an environment without a pre-configured spark cluster, than you get that error. Installing PySpark is not enough.
Instead, if you start a notebook with the Spark cluster available, you will be just fine.
You have to set .config("spark.hadoop.fs.stocator.scheme.list", "cos") along with some others fs.cos... configurations.
Here's an end-to-end snippet code example that works (tested with pyspark==2.3.2 and Python 3.7.3):
from pyspark.sql import SparkSession
stocator_jar = '/path/to/stocator-1.1.2-SNAPSHOT-IBM-SDK.jar'
cos_instance_name = '<myCosIntanceName>'
bucket_name = '<bucketName>'
s3_region = '<region>'
cos_iam_api_key = '*******'
iam_servicce_id = 'crn:v1:bluemix:public:iam-identity::<****************>'
spark_builder = (
SparkSession
.builder
.appName('test_app'))
spark_builder.config('spark.driver.extraClassPath', stocator_jar)
spark_builder.config('spark.executor.extraClassPath', stocator_jar)
spark_builder.config(f"fs.cos.{cos_instance_name}.iam.api.key", cos_iam_api_key)
spark_builder.config(f"fs.cos.{cos_instance_name}.endpoint", f"s3.{s3_region}.cloud-object-storage.appdomain.cloud")
spark_builder.config(f"fs.cos.{cos_instance_name}.iam.service.id", iam_servicce_id)
spark_builder.config("spark.hadoop.fs.stocator.scheme.list", "cos")
spark_builder.config("spark.hadoop.fs.cos.impl", "com.ibm.stocator.fs.ObjectStoreFileSystem")
spark_builder.config("fs.stocator.cos.impl", "com.ibm.stocator.fs.cos.COSAPIClient")
spark_builder.config("fs.stocator.cos.scheme", "cos")
spark_sess = spark_builder.getOrCreate()
dataset = spark_sess.range(1, 10)
dataset = dataset.withColumnRenamed('id', 'user_idx')
dataset.repartition(1).write.csv(
f'cos://{bucket_name}.{cos_instance_name}/test.csv',
mode='overwrite',
header=True)
spark_sess.stop()
print('done!')
I could extract value for
val ranking_score_path = cf.getString(stg + ".input.path.ranking_score")
.replaceAll("_replace_date_", this_date)
and
val output_path = cf.getString(stg + ".output.path.hdfs") + tomz_date + "/"
but not
val AS_HOST = cf.getString(stg + ".output.path.aerospike.host")
println("AS_HOST = " + AS_HOST)
I have tried
replacing . with _,
adding commas
but didnt work.
error log
Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'production.output.path.aerospike'
at com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:152)
at ...
application.conf
production {
input {
path {
local = "/home/aduser/tmp/"
hdfs = "/user/aduser/tmp_vincent/CPA/_replace_date_/intermediate/l1/"
ranking_score = "/home/aduser/plt/item_performance/pipeline/cpa/output/_replace_date_/predict_output/ranking_score.csv"
}
}
output {
path {
local = "/home/aduser/tmp/"
hdfs = "/user/aduser/dyson/display/"
aerospike {
host = "0.0.0.0"
port = 3000
namespace = "test"
set = "spark-test2"
}
}
}
}
Reply # Comment #1
the cf is very long but the important part is as follows
... ore.csv"}},"output":{"path":{"hdfs":"/user/aduser/dyson/display/","local":"/home/aduser/tmp/"}}},"sun":{"arch": ...
Effort #1: Replaced part of the application.conf
path {
local = "/home/aduser/tmp/"
hdfs = "/user/aduser/dyson/display/"
ae_host = "0.0.0.0"
ae_port = 3000
ae_namespace = "test"
ae_set = "spark-test2"
}
and changed the calling method
val AS_HOST = cf.getString(stg + ".output.path.ae_host")
println("AS_HOST = " + AS_HOST)
but still getting errors
Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'production.output.path.ae_host'
at com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:152)
I have found the problem after several hours, my application.conf was at the same level with src.main.scala and it worked partially for reasons which I don't know. It worked perfectly after creating src.main.resources and putting application.conf inside.
I've just updated Liquidsoap to 1.3.0 and now get_process_lines does not return anything.
def get_request() =
# Get the URI
lines = get_process_lines("curl http://localhost:3000/api/v1/liquidsoap/next/my-radio")
log("liquidsoap curl returns #{lines}")
uri = list.hd(default="",lines)
log("liquidsoap will try and play #{uri}")
# Create a request
request.create(uri)
end
I read on the CHANGELOG
- Moved get_process_lines and get_process_output to utils.liq, added optional env parameter
Does it mean I have to do something to use utils.liq in my script now ?
The full script is as follows
set("log.file",false)
set("log.stdout",true)
set("log.level",3)
def apply_metadata(m) =
title = m["title"]
artist = m["artist"]
log("Now playing: #{title} by #{artist}")
end
# Our custom request function
def get_request() =
# Get the URI
lines = get_process_lines("curl http://localhost:3000/api/v1/liquidsoap/next/my-radio")
log("liquidsoap curl returns #{lines}")
uri = list.hd(default="",lines)
log("liquidsoap will try and play #{uri}")
# Create a request
request.create(uri)
end
def my_safe(s) =
security = sine()
fallback(track_sensitive=false,[s,security])
end
s = request.dynamic(id="s",get_request)
s = on_metadata(apply_metadata,s)
s = crossfade(s)
s = my_safe(s)
# We output the stream to an icecast
# server, in ogg/vorbis format.
log("liquidsoap starting")
output.icecast(
%mp3(id3v2=true,bitrate=128,samplerate=44100),
host = "localhost",
port = 8000,
password = "PASSWORD",
mount = "myradio",
genre="various",
url="http://www.myradio.fr",
description="My Radio",
s
)
Of course the API is working
$ curl http://localhost:3000/api/v1/liquidsoap/next/my-radio
annotate:title="Chamakay",artist="Blood Orange",album="Cupid Deluxe":http://localhost/stream/3.mp3
A more simple example :
lines = get_process_lines("echo hi")
log("lines = #{lines}")
line = list.hd(default="",lines)
log("line = #{line}")
returns the following logs
2017/05/05 15:24:42 [lang:3] lines = []
2017/05/05 15:24:42 [lang:3] line =
Many thanks in advance for your help !
geoffroy
The issue was fixed in liquidsoap 1.3.1
Fixed:
Fixed run_process, get_process_lines, get_process_output when compiling with OCaml <= 4.03 (#437, #439)
https://github.com/savonet/liquidsoap/blob/1.3.1/CHANGES#L12