I'm attempting to return the source of a page using the wget command to String using command :
val url: String = "https://morningconsult.com/alert/house-passes-employee-stock-options-bill-aimed-startups/"
import sys.process._
val result: String = ("wget -qO- " + url !).toString
println("result : " + result);
but return value is 0. This is output of code :
result : 0
How to access the return value of wget in a variable , in this case the source value of the url ?
! returns the exit code of the process (0 in your case). If you need the output of the process you should use !! instead:
val url = "https://morningconsult.com/alert/house-passes-employee-stock-options-bill-aimed-startups/"
import sys.process._
val result = ("wget -qO- " + url !!).toString
println("result : " + result)
(relevant documentation)
Related
I am trying to integrate pytest and locust and facing the issue PytestCollectionWarning: cannot collect test class because it has a init constructor
import json
import pytest
import requests
from locust import SequentialTaskSet, task
request_url = ""
module_name = "shifts"
class TestShifts(SequentialTaskSet):
def get_shifts_data(self, login, request_data):
global request_url
name = "Get Shifts Data"
request_url = login.get_base_url() + request_data['path']
with self.client.get(request_url, catch_response=True, name=name, headers=login.get_header_get_request()) \
as get_shift_response:
if get_shift_response.status_code == 200:
get_shift_response.success()
return json.loads(get_shift_response.content)
else:
get_shift_response.failure("Get Shifts Failure")
#pytest.mark.usefixtures("login", "request_data")
def test_delete_shift(self, login, request_data):
global request_url
name = "Delete Shift with name :" + request_data['requestBody']['name']
shifts_data_dict = self.get_shifts_data(login, request_data)
for shift in shifts_data_dict['shifts']:
if shift['name'] == request_data['requestBody']['name']:
with self.client.delete(request_url + "/" + shift['id'], catch_response=True, name=name,
headers=login.get_header_get_request()) as shift_delete_response:
if shift_delete_response.status_code == 204:
shift_delete_response.success()
else:
shift_delete_response.failure("Shift Delete Failure")
I am using MongoDB Driver Java API to convert BSON to JSON.
I have test code like this.
String input = "{ \"timestamp\" : 1486064586641 }";
org.bson.Document doc = org.bson.Document.parse(input);
System.out.println("input = " + input);
System.out.println("output = " + doc.toJson());
The output is:
input = { "timestamp" : 1486064586641 }
output = { "timestamp" : { "$numberLong" : "1486064586641" } }
Is there an easy way to make the output look like the input?
BSON Documnet's toJson method supports only output to MongoDB Extended JSON (STRICT or SHELL format). If you want to have regular JSON, you can use com.mongodb.util.JSON class:
String input = "{ \"timestamp\" : 1486064586641 }";
org.bson.Document doc = org.bson.Document.parse(input);
System.out.println("input = " + input);
System.out.println("output (SHELL) = " + doc.toJson(new JsonWriterSettings(JsonMode.SHELL)));
System.out.println("output (STRICT) = " + doc.toJson(new JsonWriterSettings(JsonMode.STRICT)));
System.out.println("output (JSON) = " + com.mongodb.util.JSON.serialize(doc));
This will generate following output:
input = { "timestamp" : 1486064586641 }
output (SHELL) = { "timestamp" : NumberLong("1486064586641") }
output (STRICT) = { "timestamp" : { "$numberLong" : "1486064586641" } }
output (JSON) = { "timestamp" : 1486064586641}
Natalja's answer is excellent, but if you are using the Mongo Java driver 3.8.2 upwards you will notice some deprecation warnings. If you want the output to look like the input you can use RELAXED JsonWriterSettings mode.
Below you can see an example with the possible modes and how the JSON will looks like. There are also some deprecation warnings and alternatives to the deprecated code:
String input = "{ \"timestamp\" : 1486064586641 }";
org.bson.Document doc = org.bson.Document.parse(input);
System.out.println("input = " + input);
JsonWriterSettings shellMode = JsonWriterSettings.builder().outputMode(JsonMode.SHELL).build();
System.out.println("output (SHELL) = " + doc.toJson(shellMode));
JsonWriterSettings strictMode = JsonWriterSettings.builder().outputMode(JsonMode.STRICT).build();
System.out.println("output (STRICT) = " + doc.toJson(strictMode)); // deprecated - use extended like below
JsonWriterSettings extendedMode = JsonWriterSettings.builder().outputMode(JsonMode.EXTENDED).build();
System.out.println("output (EXTENDED) = " + doc.toJson(extendedMode));
JsonWriterSettings relaxed = JsonWriterSettings.builder().outputMode(JsonMode.RELAXED).build();
System.out.println("output (RELAXED) = " + doc.toJson(relaxed));
System.out.println("output (JSON) = " + com.mongodb.util.JSON.serialize(doc)); // deprecated - use relaxed like above
Also note that the JsonWriterSettings constructor is deprecated and you can use as an alternative the builder method like e.g:
JsonWriterSettings.builder().outputMode(JsonMode.SHELL).build()
My question is how to use a Buffered stream when using Scalaj-Http.
I have written the following code which is a complete working example that will download a file from Hadoop HDFS using HttpFS. My goal is to handle very large files and this will require using a buffered approach with multiple I/O writes to a local file.
I have not been able to find documentation on how to use a stream with the ScalaJ-Http interface. I am interested in an example for both download and upload that can handle large multi GB files. My code below uses in memory buffering which is appropriate for only prototyping.
import scalaj.http._
import ujson.Js
import java.text.SimpleDateFormat
import java.net.SocketTimeoutException
import java.io.InputStream
import java.io.BufferedOutputStream
import java.io.FileOutputStream
import java.io.FileNotFoundException
object CopyFileFromHdfs {
def main(args: Array[String]) {
val host = "hadoop.example.com"
val user = "root"
var dstFile = ""
var srcFile = ""
val operation = "OPEN"
val port = 14000
System.setProperty("sun.net.http.allowRestrictedHeaders", "true")
if (args.length != 2)
{
println("Error: Missing or too many arguments")
println("Usage: CopyFileFromHdfs <srcfile> <dstfile>")
System.exit(1)
}
srcFile = args(0)
dstFile = args(1)
// ********************************************************************************
// Create the URL string that we will use to connect to Hadoop HttpFS
//
// The string will look like this:
// http://root#123.456.789.012:14000/webhdfs/v1/?user.name=root&op=OPEN
// ********************************************************************************
val url = makeHttpfsUrl(host, user, srcFile, operation, port)
// ********************************************************************************
// Using HTTP, call the HttpFS server
//
// Exceptions:
// java.net.SocketTimeoutException
// java.net.UnknownHostException
// java.lang.IllegalArgumentException
// Remote Exceptions:
// java.io.FileNotFoundException
// com.sun.jersey.api.NotFoundException
// ********************************************************************************
try {
var response = Http(url)
.timeout(connTimeoutMs = 1000, readTimeoutMs = 5000)
.asBytes
// ********************************************************************************
// Check for an error. We are expecting an HTTP 200 response
// ********************************************************************************
if (response.code < 200 || response.code > 299)
{
val data = ujson.read(response.body)
printf("Error: Cannot download file: %s\n", dstFile)
println(removeQuotes(data("RemoteException")("message").str))
println(removeQuotes(data("RemoteException")("exception").str))
System.exit(1)
}
val is = new FileOutputStream(dstFile)
val bs = new BufferedOutputStream(is)
bs.write(response.body, 0, response.body.length)
bs.close()
is.close()
} catch {
case e: SocketTimeoutException => {
printf("Error: Cannot connect to host %s on port %d\n", host, port)
println(e)
System.exit(1);
}
case e: Exception => {
printf("Error (other): Cannot download file %s\n", srcFile)
println(e)
System.exit(1);
}
}
printf("Success: File downloaded. %s -> %s\n", srcFile, dstFile)
System.exit(0)
}
// ********************************************************************************
// The Json strings are surrounded by quotes.
// This function will remove them (only at the start and the end).
// ********************************************************************************
def removeQuotes(str: String): String = {
// This expression will delete quotes at the beginning and end of a string
return str.replaceAll("^\"|\"$", "");
}
// ********************************************************************************
// Create the URL string that we will use to connect to Hadoop HttpFS
//
// The string will look like this:
// http://root#123.456.789.012:14000/webhdfs/v1/?user.name=root&op=LISTSTATUS
// ********************************************************************************
def makeHttpfsUrl(
host: String,
user: String,
hdfsPath: String,
operation: String,
port: Integer) : String = {
var url = "http://" + user + "#" + host + ":" + port.toString + "/webhdfs/v1"
if (hdfsPath(0) == '/')
url += hdfsPath
else
url += "/" + hdfsPath
url += "?user.name=" + user + "&op=" + operation
return url
}
}
i have a unit test which have to fail at purpose, but I cannot capture it, so it is weird.
This is how it looks the csv file:
curva;clase;divisa;rw
AED_FXDEP;OIS;AED;240,1000
ARS :Std;6m;ARS;240
AUD_CALMNY_DISC;OIS;AUD;169.7056275
AUD_DEPO_BBSW;6m;AUD;169.7056275
AUD_DEPO_BBSW;6m;AUD;
And this is the content of the json schema file:
{"type" : "struct","fields" : [ {"name" : "curve","type" : "string","nullable" : false}, {"name":"class", "type":"string", "nullable":false}, {"name":"currency", "type":"string", "nullable":false}, {"name":"rw", "type":"string","nullable":false} ]
I think it is self explainable, the last line of the csv has an empty field and that is not permitted, the exception is clear, NumberFormatException because you can create a number with an empty value. I want to catch the exception in the unit test, why I can't reach it?
This is the code that provokes the exception:
try{
val validateGenericFile : Boolean = CSVtoParquet.validateGenericCSV(pathCSVWithHeaderWithErrors,
pathCurvasJsonSchemaWithDecimal,
_nullValue,
_delimiter,
sc,
sqlContext)
//never reach!
Assert.assertTrue(validateGenericFile)
} catch {
case e:NumberFormatException => Assert.assertTrue("ERROR! " + e.getLocalizedMessage,false)
case ex:Exception => Assert.assertTrue("ERROR! " + ex.getLocalizedMessage,false)
} finally {
println("Done testValidateInputFilesFRTBSTDES436_WithErrors!")
}
the method validateGenericCSV looks like:
val myDfWithCustomSchema = _sqlContext.read.format("com.databricks.spark.csv").
option("header", "true").
option("delimiter", _delimiter).
option("nullValue", _nullValue).
option("mode","FAILFAST").
schema(mySchemaStructType).
load(fileToReview)
var finallyCorrect : Boolean = true
var contLinesProcessed = 1
try{
//this line provokes the exception!
val myArray = myDfWithCustomSchema.collect
var contElementosJson = 0
var isTheLineCorrect: Boolean = true
myArray.foreach { elem =>
println("Processing line with content: " + elem)
for (myElem <- myList) {
val actualType = myElem.`type`
val actualName = myElem.name
val actualNullable = myElem.nullable
if (contElementosJson == myList.size) {
contElementosJson = 0
}
if (actualType == "string") {
val theField = elem.getString(contElementosJson)
val validatingField: Boolean = theField.isInstanceOf[String]
isTheLineCorrect = validatingField && !((theField == "" || theField == null) && !actualNullable)
contElementosJson += 1
if (!isTheLineCorrect){
finallyCorrect=false
println("ATTENTION! an empty string chain. " + "Check this field " + actualName + " in the csv file, which should be a " + actualType + " according with the json schema file, can be nullable? " + actualNullable + " isTheLineCorrect? " + isTheLineCorrect)
}
} else if (actualType == "integer") {
val theField = elem.get(contElementosJson)
val validatingField: Boolean = theField.isInstanceOf[Integer]
isTheLineCorrect = validatingField && !((theField == "" || theField == null) && !actualNullable)
contElementosJson += 1
if (!isTheLineCorrect){
finallyCorrect=false
println("ATTENTION! an empty string chain. " + "Check this field " + actualName + " in the csv file, which should be a " + actualType + " according with the json schema file, can be nullable? " + actualNullable + " isTheLineCorrect? " + isTheLineCorrect)
}
} else if (actualType.startsWith("decimal")) {
val theField = elem.get(contElementosJson)
val validatingField: Boolean = theField.isInstanceOf[java.math.BigDecimal]
isTheLineCorrect = validatingField && !((theField == "" || theField == null) && !actualNullable)
contElementosJson += 1
if (!isTheLineCorrect){
finallyCorrect=false
println("ATTENTION! an empty string chain. " + "Check this field " + actualName + " in the csv file, which should be a " + actualType + " according with the json schema file, can be nullable? " + actualNullable + " isTheLineCorrect? " + isTheLineCorrect)
}
} else {
println("Attention! se está intentando procesar una columna del tipo " + actualType + " que no está prevista procesar. Comprobar.")
}
} //for
contLinesProcessed += 1
} //foreach))
} catch {
//NEVER REACHED! why????
case e:NumberFormatException => throw e
case ex:Exception => throw ex
}
Why the NumberFormatException is never reached within in validateGenericCSV method?
UPDATE
i have modified these lines:
case e:NumberFormatException => Assert.assertTrue("ERROR! " + e.getLocalizedMessage,true)
case ex:Exception => Assert.assertTrue("ERROR! " + ex.getLocalizedMessage,true)
for these lines:
case e:NumberFormatException => Assert.assertTrue("ERROR! " + e.getLocalizedMessage,false)
case ex:Exception => Assert.assertTrue("ERROR! " + ex.getLocalizedMessage,false)
The same error, my problem is that I cannot reach to the catch sentences when the exception happens!
Thank you
When inspecting the stack trace, we can see the following:
ERROR! Job aborted due to stage failure: Task 1 in stage 1.0 failed 1 times, most recent failure: Lost task 1.0 in stage 1.0 (TID 2, localhost): java.lang.NumberFormatException
Spark is a distributed computing framework. The NumberFormatException is taking place remotely at one of the executors while processing a task. Spark gets a TaskFailure from that executor and propagates the exception wrapped in a org.apache.spark.SparkException to the action that triggered the materialization of the computation: the .collect() method in this specific case.
If we would like to get the reason behind the failure, we can use ex.getCause.
In practical terms we will have something like this snippet:
catch {
case ex:Exception if ex.getCause.getClass == classOf[NumberFormatException] => Assert.fail("Number parsing failed" + e.getLocalizedMessage)
case ex:Exception => Assert.fail(...)
}
The test won't fail because Assert.assertTrue(...,true) does not fail. assertTrue fails if the second parameter is false but not when it's true.
When I'm initializing(POST INDEX PAGE) my index page, it is giving me the following error.
KO bodyString.find.transform.exists failed, could not extract: transform crashed: Unexpected character ('2' (code 50)): was expecting comma to separate OBJECT entries
Code:
.exec(http("POST INDEX PAGE")
.post("/?v-1471231389581")
// .headers(Map("Content-Type" -> "application/json; charset=UTF-8"))
.formParam("v-browserDetails","1")
.formParam("theme", "mytheme")
.formParam("v-appId", appver)
.formParam("v-sh", "1200")
.formParam("v-sw", "1920")
.formParam("v-cw", "147")
.formParam("v-ch", "1047")
.formParam("v-curdate", "1470999686031")
.formParam("v-tzo", "-330")
.formParam("v-dstd", "0")
.formParam("v-rtzo", "-330")
.formParam("v-dston", "false")
.formParam("v-vw", "147")
.formParam("v-vh", "0")
.formParam("v-loc", baseurl + "/")
.formParam("v-wn", appver + "-0.7179318188297512")
.check(Checker.httpChecker)
).pause(1 seconds)
Checker:
val httpChecker = bodyString.transform {
(resp, session) =>
val state = new VaadinState;
println("\n resp :"+resp+"\n")
println("\n session :"+session+"\n")
println("Started user " + session.get("userName").as[String] + " " + session.get("password").as[String]);
state.userName = session.get("userName").as[String];
HttpRequestCreator.userStates += (session.get("userName").as[String] -> state);
//Find and store jsessionId
val url = new URL( new AppConfig().getBaseURL() );
val jsessionCookie = session("gatling.http.cookies").as[CookieJar].get(Uri.create( url.getProtocol+"://" + url.getHost+url.getPath+"/")).find(_.getName == "JSESSIONID");
state.jsessionid = jsessionCookie.getOrElse(null).getValue;
println("\n jsession id"+state.jsessionid+"\n")
println("\n resp :"+resp+ "\n")
state.readJsonState(state.httpResponseToValidJsonString(resp));
}
After that I've tried without posting those parameters. Then the sync id of the response is giving -1 even though it says web socket is open.
10:41:22.143 [DEBUG] i.g.h.a.w.WsActor - Received text message on websocket 'gatling.http.webSocket':237|for(;;);[{"changes":{},"resources":{},"locales":{},"meta":{"appError":{"caption":"Communication problem","url":null,"message":"Take note of any unsaved data, and <u>click here</u> or press ESC to continue.","details":null}},"syncId":-1}