Getting Object tag value in AWS S3 - scala

I am using scala to get information about my objects that are in S3 I am intersted in getting each object I have inside S3 Bucket his tag value so far I have accomplished this code to get me information about my objects but did not succeeded in getting its tagging value: I used this code in scala:
def retrieveObjectTags(keyName: String): Unit ={
try {
println("Listing objects")
val req: ListObjectsV2Request =
new ListObjectsV2Request().withBucketName(bucketName).withMaxKeys(2)
var result: ListObjectsV2Result = null
do {
result = client.listObjectsV2(req)
for (objectSummary <- result.getObjectSummaries) {
println(
" - " + objectSummary.getKey + " " + "(size = " + objectSummary.getSize +
")")
println(objectSummary.getETag)
}
println("Next Continuation Token : " + result.getNextContinuationToken)
req.setContinuationToken(result.getNextContinuationToken)
} while (result.isTruncated == true);
}catch {
case ase: AmazonServiceException => {
println(
"Caught an AmazonServiceException, " + "which means your request made it " +
"to Amazon S3, but was rejected with an error response " +
"for some reason.")
println("Error Message: " + ase.getMessage)
println("HTTP Status Code: " + ase.getStatusCode)
println("AWS Error Code: " + ase.getErrorCode)
println("Error Type: " + ase.getErrorType)
println("Request ID: " + ase.getRequestId)
}
case ace: AmazonClientException => {
println(
"Caught an AmazonClientException, " + "which means the client encountered " +
"an internal error while trying to communicate" +
" with S3, " +
"such as not being able to access the network.")
println("Error Message: " + ace.getMessage)
}
}
// val getTaggingRequest = new GetObjectTaggingRequest(bucketName,keyName)
// var getTagResult = client.getObjectTagging(getTaggingRequest)
//println(getTaggingRequest)
var tag: Tag = new Tag()
println("tag name:" + tag.getValue)
}
as for the remarked lines I have encounter a problem with it, what other way I can use to solve this problem?

Related

cannot capture NumberFormatException in a unit test

i have a unit test which have to fail at purpose, but I cannot capture it, so it is weird.
This is how it looks the csv file:
curva;clase;divisa;rw
AED_FXDEP;OIS;AED;240,1000
ARS :Std;6m;ARS;240
AUD_CALMNY_DISC;OIS;AUD;169.7056275
AUD_DEPO_BBSW;6m;AUD;169.7056275
AUD_DEPO_BBSW;6m;AUD;
And this is the content of the json schema file:
{"type" : "struct","fields" : [ {"name" : "curve","type" : "string","nullable" : false}, {"name":"class", "type":"string", "nullable":false}, {"name":"currency", "type":"string", "nullable":false}, {"name":"rw", "type":"string","nullable":false} ]
I think it is self explainable, the last line of the csv has an empty field and that is not permitted, the exception is clear, NumberFormatException because you can create a number with an empty value. I want to catch the exception in the unit test, why I can't reach it?
This is the code that provokes the exception:
try{
val validateGenericFile : Boolean = CSVtoParquet.validateGenericCSV(pathCSVWithHeaderWithErrors,
pathCurvasJsonSchemaWithDecimal,
_nullValue,
_delimiter,
sc,
sqlContext)
//never reach!
Assert.assertTrue(validateGenericFile)
} catch {
case e:NumberFormatException => Assert.assertTrue("ERROR! " + e.getLocalizedMessage,false)
case ex:Exception => Assert.assertTrue("ERROR! " + ex.getLocalizedMessage,false)
} finally {
println("Done testValidateInputFilesFRTBSTDES436_WithErrors!")
}
the method validateGenericCSV looks like:
val myDfWithCustomSchema = _sqlContext.read.format("com.databricks.spark.csv").
option("header", "true").
option("delimiter", _delimiter).
option("nullValue", _nullValue).
option("mode","FAILFAST").
schema(mySchemaStructType).
load(fileToReview)
var finallyCorrect : Boolean = true
var contLinesProcessed = 1
try{
//this line provokes the exception!
val myArray = myDfWithCustomSchema.collect
var contElementosJson = 0
var isTheLineCorrect: Boolean = true
myArray.foreach { elem =>
println("Processing line with content: " + elem)
for (myElem <- myList) {
val actualType = myElem.`type`
val actualName = myElem.name
val actualNullable = myElem.nullable
if (contElementosJson == myList.size) {
contElementosJson = 0
}
if (actualType == "string") {
val theField = elem.getString(contElementosJson)
val validatingField: Boolean = theField.isInstanceOf[String]
isTheLineCorrect = validatingField && !((theField == "" || theField == null) && !actualNullable)
contElementosJson += 1
if (!isTheLineCorrect){
finallyCorrect=false
println("ATTENTION! an empty string chain. " + "Check this field " + actualName + " in the csv file, which should be a " + actualType + " according with the json schema file, can be nullable? " + actualNullable + " isTheLineCorrect? " + isTheLineCorrect)
}
} else if (actualType == "integer") {
val theField = elem.get(contElementosJson)
val validatingField: Boolean = theField.isInstanceOf[Integer]
isTheLineCorrect = validatingField && !((theField == "" || theField == null) && !actualNullable)
contElementosJson += 1
if (!isTheLineCorrect){
finallyCorrect=false
println("ATTENTION! an empty string chain. " + "Check this field " + actualName + " in the csv file, which should be a " + actualType + " according with the json schema file, can be nullable? " + actualNullable + " isTheLineCorrect? " + isTheLineCorrect)
}
} else if (actualType.startsWith("decimal")) {
val theField = elem.get(contElementosJson)
val validatingField: Boolean = theField.isInstanceOf[java.math.BigDecimal]
isTheLineCorrect = validatingField && !((theField == "" || theField == null) && !actualNullable)
contElementosJson += 1
if (!isTheLineCorrect){
finallyCorrect=false
println("ATTENTION! an empty string chain. " + "Check this field " + actualName + " in the csv file, which should be a " + actualType + " according with the json schema file, can be nullable? " + actualNullable + " isTheLineCorrect? " + isTheLineCorrect)
}
} else {
println("Attention! se está intentando procesar una columna del tipo " + actualType + " que no está prevista procesar. Comprobar.")
}
} //for
contLinesProcessed += 1
} //foreach))
} catch {
//NEVER REACHED! why????
case e:NumberFormatException => throw e
case ex:Exception => throw ex
}
Why the NumberFormatException is never reached within in validateGenericCSV method?
UPDATE
i have modified these lines:
case e:NumberFormatException => Assert.assertTrue("ERROR! " + e.getLocalizedMessage,true)
case ex:Exception => Assert.assertTrue("ERROR! " + ex.getLocalizedMessage,true)
for these lines:
case e:NumberFormatException => Assert.assertTrue("ERROR! " + e.getLocalizedMessage,false)
case ex:Exception => Assert.assertTrue("ERROR! " + ex.getLocalizedMessage,false)
The same error, my problem is that I cannot reach to the catch sentences when the exception happens!
Thank you
When inspecting the stack trace, we can see the following:
ERROR! Job aborted due to stage failure: Task 1 in stage 1.0 failed 1 times, most recent failure: Lost task 1.0 in stage 1.0 (TID 2, localhost): java.lang.NumberFormatException
Spark is a distributed computing framework. The NumberFormatException is taking place remotely at one of the executors while processing a task. Spark gets a TaskFailure from that executor and propagates the exception wrapped in a org.apache.spark.SparkException to the action that triggered the materialization of the computation: the .collect() method in this specific case.
If we would like to get the reason behind the failure, we can use ex.getCause.
In practical terms we will have something like this snippet:
catch {
case ex:Exception if ex.getCause.getClass == classOf[NumberFormatException] => Assert.fail("Number parsing failed" + e.getLocalizedMessage)
case ex:Exception => Assert.fail(...)
}
The test won't fail because Assert.assertTrue(...,true) does not fail. assertTrue fails if the second parameter is false but not when it's true.

Gatling: Web Socket Open and Initialization Index page is giving an Error

When I'm initializing(POST INDEX PAGE) my index page, it is giving me the following error.
KO bodyString.find.transform.exists failed, could not extract: transform crashed: Unexpected character ('2' (code 50)): was expecting comma to separate OBJECT entries
Code:
.exec(http("POST INDEX PAGE")
.post("/?v-1471231389581")
// .headers(Map("Content-Type" -> "application/json; charset=UTF-8"))
.formParam("v-browserDetails","1")
.formParam("theme", "mytheme")
.formParam("v-appId", appver)
.formParam("v-sh", "1200")
.formParam("v-sw", "1920")
.formParam("v-cw", "147")
.formParam("v-ch", "1047")
.formParam("v-curdate", "1470999686031")
.formParam("v-tzo", "-330")
.formParam("v-dstd", "0")
.formParam("v-rtzo", "-330")
.formParam("v-dston", "false")
.formParam("v-vw", "147")
.formParam("v-vh", "0")
.formParam("v-loc", baseurl + "/")
.formParam("v-wn", appver + "-0.7179318188297512")
.check(Checker.httpChecker)
).pause(1 seconds)
Checker:
val httpChecker = bodyString.transform {
(resp, session) =>
val state = new VaadinState;
println("\n resp :"+resp+"\n")
println("\n session :"+session+"\n")
println("Started user " + session.get("userName").as[String] + " " + session.get("password").as[String]);
state.userName = session.get("userName").as[String];
HttpRequestCreator.userStates += (session.get("userName").as[String] -> state);
//Find and store jsessionId
val url = new URL( new AppConfig().getBaseURL() );
val jsessionCookie = session("gatling.http.cookies").as[CookieJar].get(Uri.create( url.getProtocol+"://" + url.getHost+url.getPath+"/")).find(_.getName == "JSESSIONID");
state.jsessionid = jsessionCookie.getOrElse(null).getValue;
println("\n jsession id"+state.jsessionid+"\n")
println("\n resp :"+resp+ "\n")
state.readJsonState(state.httpResponseToValidJsonString(resp));
}
After that I've tried without posting those parameters. Then the sync id of the response is giving -1 even though it says web socket is open.
10:41:22.143 [DEBUG] i.g.h.a.w.WsActor - Received text message on websocket 'gatling.http.webSocket':237|for(;;);[{"changes":{},"resources":{},"locales":{},"meta":{"appError":{"caption":"Communication problem","url":null,"message":"Take note of any unsaved data, and <u>click here</u> or press ESC to continue.","details":null}},"syncId":-1}

Yet another "Failed to validate oauth signature and token"

I've this problem that many others have been through. I'm doing everything correct but still i get this annoying "Failed to validate oauth signature and token" error :)
Well, something got to be wrong I guess..
I'm trying to obtain a request_token by making a post to "https://api.twitter.com/oauth/request_token" with headers:
Authorization:
OAuth oauth_consumer_key="MYVq....................ywj2g",
oauth_nonce="m8NG0s4oc87AOIpuILafAeI1YoMv5Mu9",
oauth_signature="Bxb%252FFIfOG9KLVj%252FUNdV%252FycVlGPs%253D",
oauth_signature_method="HMAC-SHA1",
oauth_timestamp="1378976842",
oauth_version="1.0"
But it complains about signature and token.
Is my signature invalid somehow?
And for this request I dont need a token right??
I can't figure out whats wrong.
Here's some of my getRequestToken code:
val oauth_consumer_key: String = CONSUMER_KEY
val oauth_nonce: String = generateNonce()
val oauth_timestamp: String = (System.currentTimeMillis / 1000).toString
var oauth_signature: String = ""
val oauth_signature_method: String = "HMAC-SHA1"
val oauth_version: String = "1.0"
val PARAMETER_STRING: String =
"oauth_consumer_key=" + oauth_consumer_key + "&" +
"oauth_nonce=" + oauth_nonce + "&" +
"oauth_signature_method=" + oauth_signature_method + "&" +
"oauth_timestamp=" + oauth_timestamp + "&" +
"oauth_version=" + oauth_version
val BASE_STRING: String =
"POST&" + URLEncoder.encode("https://api.twitter.com/oauth/request_token", "UTF-8") + "&" + URLEncoder.encode(PARAMETER_STRING, "UTF-8")
oauth_signature = getSignature(CONSUMER_SECRET, BASE_STRING, "HmacSHA1")
val AUTHORIZATION = "OAuth " +
"oauth_consumer_key=\"" + URLEncoder.encode(oauth_consumer_key, "UTF-8") +
"\", oauth_nonce=\"" + URLEncoder.encode(oauth_nonce, "UTF-8") +
"\", oauth_signature=\"" + URLEncoder.encode(oauth_signature, "UTF-8") +
"\", oauth_signature_method=\"" + URLEncoder.encode(oauth_signature_method, "UTF-8") +
"\", oauth_timestamp=\"" + URLEncoder.encode(oauth_timestamp, "UTF-8") +
"\", oauth_version=\"" + URLEncoder.encode(oauth_version, "UTF-8") + "\""
WS.url("https://api.twitter.com/oauth/request_token").withHeaders("Authorization" -> AUTHORIZATION).post(Results.EmptyContent()).map(response => {
if(response.status != 200) Logger.error(response.body) //THIS IS WHERE I GET THE ERROR
else {
if((response.json \ "oauth_callback_confirmed").as[String] == "true") {
REQUEST_TOKEN = (response.json \ "oauth_token").as[String]
REQUEST_SECRET = (response.json \ "oauth_token_secret").as[String]
requestDone.success(true)
}
}
})
Ok so I've got everyting to work (without the oauth_callback parameter, because if I add this I get the error again).
I get the Request_token, which is valid because when I manually paste the authenticate url in the browser together with the generated request token I get redirected to twitter authenticate page and then a correct callback is made and the result is correct also. (token, token_secret, user_id and screen_name)
But my code seem to ignore my redirect to this authorize page.
requestToken_future.map { result =>
Redirect("https://api.twitter.com/oauth/authenticate?oauth_token="+REQUEST_TOKEN)
}
If I put a Logger inside the brackets it shows the log in my terminal window. But that Redirect seems to just be ignored. Never goes off.
You haven't included the oauth_callback parameter which is required. See the documentation here.

FileMaker 10 Pro Insert data by Asp.Net

I am trying to insert the data in FM but get the parse error serached number of fourms with no luck.
ERROR [HY000] [DataDirect][ODBC SequeLink driver][ODBC Socket][DataDirect][ODBC FileMaker driver][FileMaker]Parse Error in SQL
enter code here
StringBuilder sbAddBarcode = new StringBuilder();
sbAddBarcode.Append("insert into BarCode (PONumber,Description,Model,[Serial Number])");
sbAddBarcode.Append("values");
sbAddBarcode.Append(" ("+ barcode.PONumber + ",");
sbAddBarcode.Append(" '" + barcode.Description +"',");
sbAddBarcode.Append(" '" + barcode.ModelNumber +"')");
//sbAddBarcode.Append(" '" + barcode.SerialNumber +"')");
fmCommand = new OdbcCommand(sbAddBarcode.ToString(), fmcon);
fmCommand.CommandType = CommandType.Text;
fmCommand.Connection = fmcon;
try
{
fmcon.Open();
fmCommand.ExecuteNonQuery();
}
catch (OdbcException oe)
{
throw new Exception(oe.Message);
}
catch (Exception ex)
{
throw new Exception(ex.Message);
}
finally
{
fmcon.Close();
}
Echo your sbAddBarcode value. The SQL you're building does look to be invalid.

Error C#.NET 3.0: The best overloaded method match for 'CreatePageSection(ref string, ref string, ref object)' has some invalid arguments

I have got the following problem. I encountered an error regarding the default parameters. I added a simple piece of code for an overload. Now I get in the new code (line 3: CreatePageSection(sItemID, "", null);) gives the error mentioned in the title.
I looked for the answer in the other topics, but I can't find the problem. Can someone help me?
The code is found here:
public void CreatePageSection(ref string sItemID)
{
CreatePageSection(sItemID, "", null);
}
public void CreatePageSection(ref string sItemID, ref string sFrameUrl, ref object vOnReadyState)
{
if (Strings.InStr(msPresentPageSections, "|" + sItemID + "|", 0) > 0) {
return;
}
msPresentPageSections = msPresentPageSections + sItemID + "|";
string writeHtml = "<div class=" + MConstants.QUOTE + "PageSection" + MConstants.QUOTE + " id=" + MConstants.QUOTE + "Section" + sItemID + "Div" + MConstants.QUOTE + " style=" + MConstants.QUOTE + "display: none;" + MConstants.QUOTE + ">";
this.WriteLine_Renamed(ref writeHtml);
//UPGRADE_WARNING: Couldn't resolve default property of object vOnReadyState. Click for more: 'ms-help://MS.VSCC.v90/dv_commoner/local/redirect.htm?keyword="6A50421D-15FE-4896-8A1B-2EC21E9037B2"'
//UPGRADE_NOTE: IsMissing() was changed to IsNothing_Renamed(). Click for more: 'ms-help://MS.VSCC.v90/dv_commoner/local/redirect.htm?keyword="8AE1CB93-37AB-439A-A4FF-BE3B6760BB23"'
writeHtml = " <iframe id=" + MConstants.QUOTE + sItemID + "Frame" + MConstants.QUOTE + " name=" + MConstants.QUOTE + sItemID + "Frame" + MConstants.QUOTE + " frameborder=" + MConstants.QUOTE + "0" + MConstants.QUOTE + (!string.IsNullOrEmpty(sFrameUrl) ? " src=" + MConstants.QUOTE + sFrameUrl + MConstants.QUOTE : "") + ((vOnReadyState == null) ? "" : " onreadystatechange=" + MConstants.QUOTE + Convert.ToString(vOnReadyState) + MConstants.QUOTE) + ">";
this.WriteLine_Renamed(ref writeHtml);
writeHtml = " </iframe>";
this.WriteLine_Renamed(ref writeHtml);
writeHtml = "</div>";
this.WriteLine_Renamed(ref writeHtml);
}
you must pass params by reference
public void CreatePageSection(ref string sItemID)
{
var missingString = String.Empty;
object missingObject = null;
CreatePageSection(ref sItemID, ref missingString, ref missingObject);
}
Since your are not manipulating sFrameUrl and vOnReadyState, remove the ref keyword from those parameters.
See: http://msdn.microsoft.com/en-us/library/14akc2c7(v=vs.71).aspx
hth
Mario