set shell mode for mongodb [duplicate] - mongodb

I am using MongoDB Driver Java API to convert BSON to JSON.
I have test code like this.
String input = "{ \"timestamp\" : 1486064586641 }";
org.bson.Document doc = org.bson.Document.parse(input);
System.out.println("input = " + input);
System.out.println("output = " + doc.toJson());
The output is:
input = { "timestamp" : 1486064586641 }
output = { "timestamp" : { "$numberLong" : "1486064586641" } }
Is there an easy way to make the output look like the input?

BSON Documnet's toJson method supports only output to MongoDB Extended JSON (STRICT or SHELL format). If you want to have regular JSON, you can use com.mongodb.util.JSON class:
String input = "{ \"timestamp\" : 1486064586641 }";
org.bson.Document doc = org.bson.Document.parse(input);
System.out.println("input = " + input);
System.out.println("output (SHELL) = " + doc.toJson(new JsonWriterSettings(JsonMode.SHELL)));
System.out.println("output (STRICT) = " + doc.toJson(new JsonWriterSettings(JsonMode.STRICT)));
System.out.println("output (JSON) = " + com.mongodb.util.JSON.serialize(doc));
This will generate following output:
input = { "timestamp" : 1486064586641 }
output (SHELL) = { "timestamp" : NumberLong("1486064586641") }
output (STRICT) = { "timestamp" : { "$numberLong" : "1486064586641" } }
output (JSON) = { "timestamp" : 1486064586641}

Natalja's answer is excellent, but if you are using the Mongo Java driver 3.8.2 upwards you will notice some deprecation warnings. If you want the output to look like the input you can use RELAXED JsonWriterSettings mode.
Below you can see an example with the possible modes and how the JSON will looks like. There are also some deprecation warnings and alternatives to the deprecated code:
String input = "{ \"timestamp\" : 1486064586641 }";
org.bson.Document doc = org.bson.Document.parse(input);
System.out.println("input = " + input);
JsonWriterSettings shellMode = JsonWriterSettings.builder().outputMode(JsonMode.SHELL).build();
System.out.println("output (SHELL) = " + doc.toJson(shellMode));
JsonWriterSettings strictMode = JsonWriterSettings.builder().outputMode(JsonMode.STRICT).build();
System.out.println("output (STRICT) = " + doc.toJson(strictMode)); // deprecated - use extended like below
JsonWriterSettings extendedMode = JsonWriterSettings.builder().outputMode(JsonMode.EXTENDED).build();
System.out.println("output (EXTENDED) = " + doc.toJson(extendedMode));
JsonWriterSettings relaxed = JsonWriterSettings.builder().outputMode(JsonMode.RELAXED).build();
System.out.println("output (RELAXED) = " + doc.toJson(relaxed));
System.out.println("output (JSON) = " + com.mongodb.util.JSON.serialize(doc)); // deprecated - use relaxed like above
Also note that the JsonWriterSettings constructor is deprecated and you can use as an alternative the builder method like e.g:
JsonWriterSettings.builder().outputMode(JsonMode.SHELL).build()

Related

REST API: pass parameters to preparedQuery in Vertx pgclient

This is the response from my API:
"The number of parameters to execute should be consistent with the expected number of parameters = [2] but the actual number is [0]."
I use pgClient, Vert.x 3.9.3 and consume various REST API's without problems, but....in this query (probably it's wrong)
private static final String SELECT_CBA = "select art.leyenda, $1::numeric(10,3) as cantidad, uni.abreviatura, \r"
+ "round(((art.precio_costo * (art.utilidad_fraccionado/100)) + art.precio_costo) * $2::numeric(10,3),2) as totpagar \r"
+ "FROM public.articulos art join public.unidades uni on uni.idunidad = art.idunidad \r"
+ "WHERE (substring(art.codigobarra,1,2) = \'$3\' and substring(art.codigobarra,3,6) = \'$4\')";
Some explanation
$1 and $2 are the same parameters; $2 and $3 must be quoted parameters.
This is my rest verticle:
poolClient = PgPool.pool(vertx, options, poolOptions);
bizArticulo = new BizArticulo(poolClient);
Router router = Router.router(vertx);
router.route("/api/articulos*").handler(BodyHandler.create());
router.get("/api/articulos").handler(bizArticulo::getAll);
router.get("/api/articulos/:cantcomp1/:cantcomp2/:tipoprod/:prodpadre").handler(bizArticulo::getOneReadingBarcode);
Where :cantcomp1 -> $1, :cantcomp2 -> $2, :tipoprod -> $3 and $4 -> :prodpadre
and finally, this my "business"
public void getOneReadingBarcode(RoutingContext routingContext) {
HttpServerResponse response = routingContext.response();
pgClient
.preparedQuery(SELECT_CBA)
.execute(ar -> {
if (ar.succeeded()) {
RowSet<Row> rows = ar.result();
List<Articulo> articulos = new ArrayList<>();
rows.forEach(row -> {
articulos.add(fromBarCode(row));
});
response.putHeader("content-type", "application/json; charset=utf-8")
.setStatusCode(200)
.end(Json.encodePrettily(articulos));
} else {
System.out.println("Failure: " + ar.cause().getMessage());
response.putHeader("content-type", "application/json; charset=utf-8")
.end(Json.encodePrettily(ar.cause().getMessage()));
}
});
}
In Postman, I wrote:
192.168.0.15:8092/api/articulos/0.750/0.750/20/021162; where I assume the parameters match, but it returns the error mentioned above.
¿What's wrong? Any help would be appreciated
Ernesto
Ok, I answer to myself...
First: little fix to my query (SELECT_CB), I put the parameters $1, $2... in parentheses..
private static final String SELECT_CBA = "select art.leyenda, ($1::numeric(10,3)) as cantidad, uni.abreviatura, \r"
+ "round(((art.precio_costo * (art.utilidad_fraccionado/100)) + art.precio_costo) * ($2::numeric(10,3)),2) as totpagar \r"
+ "FROM public.articulos art join public.unidades uni on uni.idunidad = art.idunidad \r"
+ "WHERE (substring(art.codigobarra,1,2) = ($3) and substring(art.codigobarra,3,6) = ($4))";
Second: got the params from context (see router.context in previous question)
Double cantComprada1 = Double.parseDouble(routingContext.request().getParam("cantcomp1"));
Double cantComprada2 = Double.parseDouble(routingContext.request().getParam("cantcomp2"));
String tipoProducto = routingContext.request().getParam("tipoprod");
String productoPadre = routingContext.request().getParam("prodpadre");
and Third: put the params as arguments
pgClient
.preparedQuery(SELECT_CBA)
.execute(Tuple.of(cantComprada1, cantComprada2, tipoProducto, productoPadre), ar -> {
......
......
}
All work's fine

cannot capture NumberFormatException in a unit test

i have a unit test which have to fail at purpose, but I cannot capture it, so it is weird.
This is how it looks the csv file:
curva;clase;divisa;rw
AED_FXDEP;OIS;AED;240,1000
ARS :Std;6m;ARS;240
AUD_CALMNY_DISC;OIS;AUD;169.7056275
AUD_DEPO_BBSW;6m;AUD;169.7056275
AUD_DEPO_BBSW;6m;AUD;
And this is the content of the json schema file:
{"type" : "struct","fields" : [ {"name" : "curve","type" : "string","nullable" : false}, {"name":"class", "type":"string", "nullable":false}, {"name":"currency", "type":"string", "nullable":false}, {"name":"rw", "type":"string","nullable":false} ]
I think it is self explainable, the last line of the csv has an empty field and that is not permitted, the exception is clear, NumberFormatException because you can create a number with an empty value. I want to catch the exception in the unit test, why I can't reach it?
This is the code that provokes the exception:
try{
val validateGenericFile : Boolean = CSVtoParquet.validateGenericCSV(pathCSVWithHeaderWithErrors,
pathCurvasJsonSchemaWithDecimal,
_nullValue,
_delimiter,
sc,
sqlContext)
//never reach!
Assert.assertTrue(validateGenericFile)
} catch {
case e:NumberFormatException => Assert.assertTrue("ERROR! " + e.getLocalizedMessage,false)
case ex:Exception => Assert.assertTrue("ERROR! " + ex.getLocalizedMessage,false)
} finally {
println("Done testValidateInputFilesFRTBSTDES436_WithErrors!")
}
the method validateGenericCSV looks like:
val myDfWithCustomSchema = _sqlContext.read.format("com.databricks.spark.csv").
option("header", "true").
option("delimiter", _delimiter).
option("nullValue", _nullValue).
option("mode","FAILFAST").
schema(mySchemaStructType).
load(fileToReview)
var finallyCorrect : Boolean = true
var contLinesProcessed = 1
try{
//this line provokes the exception!
val myArray = myDfWithCustomSchema.collect
var contElementosJson = 0
var isTheLineCorrect: Boolean = true
myArray.foreach { elem =>
println("Processing line with content: " + elem)
for (myElem <- myList) {
val actualType = myElem.`type`
val actualName = myElem.name
val actualNullable = myElem.nullable
if (contElementosJson == myList.size) {
contElementosJson = 0
}
if (actualType == "string") {
val theField = elem.getString(contElementosJson)
val validatingField: Boolean = theField.isInstanceOf[String]
isTheLineCorrect = validatingField && !((theField == "" || theField == null) && !actualNullable)
contElementosJson += 1
if (!isTheLineCorrect){
finallyCorrect=false
println("ATTENTION! an empty string chain. " + "Check this field " + actualName + " in the csv file, which should be a " + actualType + " according with the json schema file, can be nullable? " + actualNullable + " isTheLineCorrect? " + isTheLineCorrect)
}
} else if (actualType == "integer") {
val theField = elem.get(contElementosJson)
val validatingField: Boolean = theField.isInstanceOf[Integer]
isTheLineCorrect = validatingField && !((theField == "" || theField == null) && !actualNullable)
contElementosJson += 1
if (!isTheLineCorrect){
finallyCorrect=false
println("ATTENTION! an empty string chain. " + "Check this field " + actualName + " in the csv file, which should be a " + actualType + " according with the json schema file, can be nullable? " + actualNullable + " isTheLineCorrect? " + isTheLineCorrect)
}
} else if (actualType.startsWith("decimal")) {
val theField = elem.get(contElementosJson)
val validatingField: Boolean = theField.isInstanceOf[java.math.BigDecimal]
isTheLineCorrect = validatingField && !((theField == "" || theField == null) && !actualNullable)
contElementosJson += 1
if (!isTheLineCorrect){
finallyCorrect=false
println("ATTENTION! an empty string chain. " + "Check this field " + actualName + " in the csv file, which should be a " + actualType + " according with the json schema file, can be nullable? " + actualNullable + " isTheLineCorrect? " + isTheLineCorrect)
}
} else {
println("Attention! se está intentando procesar una columna del tipo " + actualType + " que no está prevista procesar. Comprobar.")
}
} //for
contLinesProcessed += 1
} //foreach))
} catch {
//NEVER REACHED! why????
case e:NumberFormatException => throw e
case ex:Exception => throw ex
}
Why the NumberFormatException is never reached within in validateGenericCSV method?
UPDATE
i have modified these lines:
case e:NumberFormatException => Assert.assertTrue("ERROR! " + e.getLocalizedMessage,true)
case ex:Exception => Assert.assertTrue("ERROR! " + ex.getLocalizedMessage,true)
for these lines:
case e:NumberFormatException => Assert.assertTrue("ERROR! " + e.getLocalizedMessage,false)
case ex:Exception => Assert.assertTrue("ERROR! " + ex.getLocalizedMessage,false)
The same error, my problem is that I cannot reach to the catch sentences when the exception happens!
Thank you
When inspecting the stack trace, we can see the following:
ERROR! Job aborted due to stage failure: Task 1 in stage 1.0 failed 1 times, most recent failure: Lost task 1.0 in stage 1.0 (TID 2, localhost): java.lang.NumberFormatException
Spark is a distributed computing framework. The NumberFormatException is taking place remotely at one of the executors while processing a task. Spark gets a TaskFailure from that executor and propagates the exception wrapped in a org.apache.spark.SparkException to the action that triggered the materialization of the computation: the .collect() method in this specific case.
If we would like to get the reason behind the failure, we can use ex.getCause.
In practical terms we will have something like this snippet:
catch {
case ex:Exception if ex.getCause.getClass == classOf[NumberFormatException] => Assert.fail("Number parsing failed" + e.getLocalizedMessage)
case ex:Exception => Assert.fail(...)
}
The test won't fail because Assert.assertTrue(...,true) does not fail. assertTrue fails if the second parameter is false but not when it's true.

Return value of process to String

I'm attempting to return the source of a page using the wget command to String using command :
val url: String = "https://morningconsult.com/alert/house-passes-employee-stock-options-bill-aimed-startups/"
import sys.process._
val result: String = ("wget -qO- " + url !).toString
println("result : " + result);
but return value is 0. This is output of code :
result : 0
How to access the return value of wget in a variable , in this case the source value of the url ?
! returns the exit code of the process (0 in your case). If you need the output of the process you should use !! instead:
val url = "https://morningconsult.com/alert/house-passes-employee-stock-options-bill-aimed-startups/"
import sys.process._
val result = ("wget -qO- " + url !!).toString
println("result : " + result)
(relevant documentation)

How do you list out all output / generate files from a Gradle Build?

Currently I am using the following to generate list of output files which I want for another plugin which does Github releases.
[
"${buildDir}/libs/${project.name}-${project.version}.jar",
"${buildDir}/libs/${project.name}-${project.version}-javadoc.jar",
"${buildDir}/libs/${project.name}-${project.version}-sources.jar"
]
I am not very satisfied with this as I am manually specifying each type of file (doc, source, class). I am looking to make this more generic where I can reuse without any modification for any project which might have more or less outputs and also may not be in the default location and some outputs may not even be Java outputs. I.e. changing output location, the tasks to run or plugging elsewhere in the script should not break this part of the script and should be reusable without editing. The question is how do you get a list of all the outputs as a List of String like above.
More detailed code snippet:
...
plugins {
id 'co.riiid.gradle' version '0.4.2'
}
...
ext {
org = 'MicroTrader'
repo = org + '/' + project.name
releaseNotesFile = 'RELEASE_NOTES.md'
host = 'github.com'
hostUrl = 'https://' + host + '/'
orgUrl = hostUrl + org
websiteUrl = hostUrl + repo
vcsUrl = websiteUrl + '.git'
scmUrl = 'scm:git#' + host + ':' + repo + '.git'
issueTrackerUrl = websiteUrl + '/issues'
ossrhUsername = ((project.hasProperty('ossrhUsername') ? ossrhUsername : System.getenv('OSSRH_USER')) ?: System.getProperty('ossrh.user')) ?: ' '
ossrhPassword = ((project.hasProperty('ossrhPassword') ? ossrhPassword : System.getenv('OSSRH_PASSWORD')) ?: System.getProperty('ossrh.password')) ?: ' '
bintray_user = ((project.hasProperty('bintray_user') ? bintray_user : System.getenv('BINTRAY_USER')) ?: System.getProperty('bintray.user')) ?: ' '
bintray_api_key = ((project.hasProperty('bintray_api_key') ? bintray_api_key : System.getenv('BINTRAY_KEY')) ?: System.getProperty('bintray.apikey')) ?: ' '
artifactory_user = ((project.hasProperty('artifactory_user') ? artifactory_user : System.getenv('ARTIFACTORY_USER')) ?: System.getProperty('artifactory.user')) ?: ' '
artifactory_password = ((project.hasProperty('artifactory_password') ? artifactory_password : System.getenv('ARTIFACTORY_PASSWORD')) ?: System.getProperty('artifactory.password')) ?: ' '
github_api_key = ((project.hasProperty('github_api_key') ? github_api_key : System.getenv('GITHUB_KEY')) ?: System.getProperty('github.apikey')) ?: ' '
}
...
github {
owner = project.org
repo = project.name
token = project.github_api_key
tagName = project.version
targetCommitish = 'master'
name = 'v' + project.version
body = Files.exists(Paths.get('' + projectDir + '/' + releaseNotesFile)) ?
new File('' + projectDir + '/' + releaseNotesFile).text :
(Files.exists(Paths.get('' + baseDir + '/' + releaseNotesFile)) ?
new File('' + baseDir + '/' + releaseNotesFile).text : '')
assets = [
"${buildDir}/libs/${project.name}-${project.version}.jar",
"${buildDir}/libs/${project.name}-${project.version}-javadoc.jar",
"${buildDir}/libs/${project.name}-${project.version}-sources.jar"
]
}
...
Try reading the outputs property on all of the tasks in your build used to generate the artifacts:
project(':my-sub-project').jar.outputs
Another option is to use the archives configuration from all of your sub-projects:
project(':my-sub-project').configurations.archives.allArtifacts

Logging mongodb query in java

I am using mongo-java-driver-2.9.1 for interacting with mongodb, I want to log the query that are fired on to the mongodb server. e.g. In java for inserting the document this is the code that I write
DBCollection coll = db.getCollection("mycollection");
BasicDBObject doc = new BasicDBObject("name", "MongoDB")
.append("type", "database")
.append("count", 1);
coll.insert(doc);
for this, equivalent code in "mongo" client for inserting document in mongodb is
db.mycollection.insert({
"name" : "MongoDB",
"type" : "database",
"count" : 1
})
I want to log this second code, is there any way to do it?
I think the MongoDB Java driver has not logging support so you have to write your logging Message by your own. Here an Example:
DBCollection coll = db.getCollection("mycollection");
BasicDBObject doc = new BasicDBObject("name", "MongoDB")
.append("type", "database")
.append("count", 1);
WriteResult insert = coll.insert(doc);
String msg = "";
if(insert.getError() == null){
msg = "insert into: " + collection.toString() +" ; Object " + q.toString());
//log the message
} else {
msg = "ERROR by insert into: " + collection.toString() +" ; Object " + q.toString());
msg = msg + " Error message: " + insert.getError();
}
//log the message